Adolescents who are diagnosed with attention-deficit/hyperactivity disorder (ADHD) exhibit numerous problem behaviours such as ‘sustaining attention to academic tasks, resisting short-term interests for the sake of longer-term goals, generating intrinsic motivation, understanding consequences, planning, and inhibiting excessive motor activity’ (Levine & Anshel, Reference Levine and Anshel2011, p. 297). Sibley et al. (Reference Sibley, Pelham, Molina, Gnagy, Waschbusch, Garefino and Karch2012) documented that problems associated with youths with ADHD include delinquency, school dropout, and early initiation of sexual behaviour. In schools, the absenteeism rates of these at-risk youths are high, and lessons are often disrupted as they are frequently inattentive and off-task (Barbaresi, Katusic, Colligan, Weaver, & Jacobsen, Reference Barbaresi, Katusic, Colligan, Weaver and Jacobsen2007; Robb et al., Reference Robb, Sibley, Pelham, Foster, Molina, Gnagy and Kuriyan2011). In addition, these youths are also often in a state of emotional angst due to problems with family, friends, and partners (Sibley et al., Reference Sibley, Pelham, Molina, Gnagy, Waschbusch, Garefino and Karch2012). If teachers are not in control of the classroom and have no effective way to manage such students, little or no learning can take place and the problem behaviour can escalate (Stronge, Ward, & Grant, Reference Stronge, Ward and Grant2011).
Behavioural interventions have been found to improve on-task behaviour and reduce disruptive behaviour in students with ADHD (Fabiano & Pelham, Reference Fabiano and Pelham2003). Some of these school behavioural interventions include positive reinforcement, ‘pull out’ services for individualised training, and daily behaviour report cards (DBRCs), which provide immediate, frequent, and salient feedback on student performance (Antshel et al., Reference Antshel, Hargrave, Simonescu, Kaul, Hendricks and Faraone2011). The use of a DBRC helps improve behaviour by documenting or giving daily feedback on student behaviour to teachers, students, and parents in a nontechnical form. A DBRC is an intervention tool to facilitate academic skill attainment and remediate behavioural problems (Pyle & Fabiano, Reference Pyle and Fabiano2017; Vannest, Davis, Davis, Mason, & Burke, Reference Vannest, Davis, Davis, Mason and Burke2010). According to Vannest et al. (Reference Vannest, Davis, Davis, Mason and Burke2010) and Chafouleas, Riley-Tillman, and McDougal (Reference Chafouleas, Riley-Tillman and McDougal2002), DBRCs should have the following characteristics: (a) specification of behaviour; (b) daily rating of the occurrences of target behaviour; (c) sharing of information with parents, teachers, and students; (d) using as a monitoring tool on the effectiveness of an intervention; and (e) delivery of rewards or consequences based on student behaviour. A DBRC intervention involves the collaboration of home and school to improve student behaviour and achieve more positive school outcomes, leading to a reduced need for costly special education services (Fabiano et al., Reference Fabiano, Vujnovic, Pelham, Waschbusch, Massetti, Pariseau and Volker2010; Iznardo, Rogers, Volpe, Labelle, & Robaey, Reference Iznardo, Rogers, Volpe, Labelle and Robaey2017; Owens et al., Reference Owens, Holdaway, Zoromski, Evans, Himawan, Girio-Herrera and Murphy2012). A number of studies have shown the effectiveness of DBRCs in increasing work completion (Jurbergs, Palcic, & Kelley, Reference Jurbergs, Palcic and Kelley2007), reducing talking out behaviour (Burkwist, Mabee, & McLaughlin, Reference Burkwist, Mabee and McLaughlin1987), increasing on-task behaviour and reducing disruptions (Kelley & McCain, Reference Kelley and McCain1995), as well as addressing other problematic student behaviour (Owens et al., Reference Owens, Holdaway, Zoromski, Evans, Himawan, Girio-Herrera and Murphy2012; Williams, Noell, Jones, & Gansle, Reference Williams, Noell, Jones and Gansle2012).
With the advent of the internet in recent years, the use of online tools for teaching and home–school communication has become more prevalent with teachers (Phillips & Olson, Reference Phillips and Olson2013). There are, however, limited published studies that have explored the effectiveness of the electronic versions of DBRC. The DBRC intervention in the research by Williams et al. (Reference Williams, Noell, Jones and Gansle2012) was implemented using email. They utilised a group experimental research design with 46 elementary students to investigate the effectiveness of emailed DBRCs over 3 weeks. They found that emailed DBRCs between teachers and parents were effective in decreasing preschool students’ disruptive behaviour. Vannest, Burke, Payne, Davis, and Soares (Reference Vannest, Burke, Payne, Davis and Soares2011) presented descriptive case studies on the use of electronic DBRCs (e-DBRC©) with three students with challenging behaviour and learning difficulties. They reported that students who were monitored closely via the e-DBRC intervention by teachers and parents showed improved behaviour. Vannest and colleagues therefore recommended the use of e-DBRC as a behavioural interventional strategy as well as a progress monitoring tool, especially when combined with frequent use and contingent reinforcement. With only two studies published, there is insufficient evidence documenting the effectiveness of online DBRC.
One of the main challenges of home–school collaboration is garnering parental support, particularly from families with lower socioeconomic status (SES), as parents from lower SES families typically report less participation in school activities (Zhang, Hsu, Kwok, Benz, & Bowman-Perrott, Reference Zhang, Hsu, Kwok, Benz and Bowman-Perrott2011). Although Vannest et al. (Reference Vannest, Davis, Davis, Mason and Burke2010) found that parent involvement had a positive impact on the effectiveness of a DBRC intervention, Pyle and Fabiano (Reference Pyle and Fabiano2017) did not find similar effects of parental involvement on the outcomes of DBRC interventions in their meta-analysis on DBRC. Pyle and Fabiano found that level of home–school communication, as defined by (a) provision of reinforcement at home, (b) availability of home training on DBRC for parents, and (c) provision of student's behaviour feedback at home, was not a moderating variable. Therefore, instead of ruling out the use of DBRC with parents with low school participation, perhaps additional school support is needed to ensure that the student and parent can benefit from a DBRC. In the current study, the online DBRC was adapted such that a school mentor was involved to support the parents in the intervention to monitor and guide the students. This mentor worked together with the parents in this modified online DBRC approach. The purpose of the study was to examine the (a) efficacy of implementing an online DBRC, with the involvement of school mentors, to decrease off-task behaviour in students with ADHD; and (b) perceptions of the students, parents, and teachers regarding the acceptability and effectiveness of the online DBRC intervention.
Method
Participants and Setting
Setting
Ethics approval from the teacher education institute's ethics review committee was obtained before the commencement of this study. This study took place in a secondary school in Singapore that catered to approximately 800 youths, between 13 and 16 years of age. These students were at risk for learning and behavioural difficulties, as they had failed their national school examination, known as the Primary School Leaving Examination. As such, as compared to other public schools, this school had more supporting school personnel involved in discipline and counselling. There were four discipline masters and school counsellors, an education psychologist, and a trained special education teacher. The majority of students in this school came from disadvantaged families with low SES. Class size averaged from 15 to 20 students per class.
Students
Three secondary school students participated in the study. Students were included based on the following criteria: (a) a formal diagnosis of ADHD obtained from school records, (b) high levels of off-task behaviour compared to peers during initial observations, and (c) teacher referral for problematic off-task behaviour via office discipline referrals. Written parent consent and student assent were obtained before the participants were included in the study.
Andy was 15 years 4 months and was enrolled in eighth grade at the secondary school. The form and subject teachers supported his inclusion in the study due to his high level of off-task behaviour during lessons. Andy was diagnosed with ADHD and had a high number of office discipline referrals. During classroom observations, Andy was observed to exhibit many off-tasks behaviours. Andy's intact family of four was of low SES. His mother supported and participated in this study.
Fred was 13 years old, enrolled in seventh grade, and had a formal ADHD diagnosis. Fred's teachers supported his inclusion in the study, as he was hyperactive and prone to rash behaviour (e.g., running out of the classroom without permission, swearing at classmates and some teachers). During classroom observations, Fred was observed to be frequently off task. He also had a high number of office discipline referrals. Fred was from a single-parent family of low SES. His mother consented to be a parent participant.
Lisa was 13 years 1 month and was enrolled in seventh grade. Lisa was strongly recommended by her teachers for this study, as she had been diagnosed with ADHD and was susceptible to mood swings and anger outbursts. Lisa was sometimes withdrawn and refused to engage with friends and teachers at times; at other times, she exhibited over-friendly behaviour. She also invaded others’ personal space due to her overeagerness to appear helpful, and yet on occasions threatened to beat up her friends who were moments ago her best buddies. Lisa generally stayed on task except for those occasions of temper tantrums when she had high levels of off-task behaviour. Lisa's parents were separated, both parents were working, and the family was of low SES. Lisa had to alternate between both parents’ homes every week. Her father agreed to participate as a parent participant.
Teachers
Teachers selected to participate in this study were the homeroom teachers and subject teachers of the identified students. Only teachers who were teaching the participants at least two or more periods per day were included in the study. In total, 17 teachers provided written consent and participated in this study. Seven, six, and five teachers were involved in rating the daily behaviour of Andy, Fred, and Lisa, respectively.
Mentors
There were three mentors in this study, and all the mentors provided written consent. Andy and Lisa were both assigned to the education psychologist, whereas Fred was assigned to the discipline master and the special education teacher. Fred had two mentors, as the discipline master had other responsibilities and could not meet with Fred daily, and thus the special education teacher was enlisted to co-mentor Fred. The mentors had an important role in the implementation of the online DBRC intervention. These mentors provided the emotional support and guidance for these students who lacked home support.
Independent Variable
Online DBRC
An online DBRC system was designed and set up to facilitate ease of use and improve promptness of feedback from the teacher to other stakeholders. Teachers rated students’ behaviour directly to an online system instead of using a pen and paper checklist. Teachers accessed the online DBRC via an internet browser, either on their computer or smartphone. Only the researcher, teachers, and mentors had access to the online DBRC.
After each lesson, teachers evaluated the off-task behaviour of the participating students and input their ratings in the online DBRC immediately. The teacher also provided the student with an overview of his/her behaviour for that lesson, reminded him/her to meet the mentor at the end of the day, and show good classroom behaviour for the next lesson.
With the online DBRC, the teacher selected the student and subject of the lesson from a dropdown list before entering the date. Next, the teacher rated the students against these three behaviour descriptors: (a) keep quiet (verbal) and not make noise (with objects); (b) stay seated as well as be still and attentive, not fidget; and (c) keep a distance from others, not touch others’ property (respect personal space). For each behaviour descriptor, the teacher rated the extent to which the student exhibited the behaviour: ‘No’ (0 score), ‘Sometimes’ (1), ‘Most of the time’ (2), and ‘All the time’ (3). An algorithm would then generate an overall behaviour rating of the student (i.e., Needs Improvement, Fair, or Well Done). The student would be rated as (a) Needs Improvement if the student received a ‘No’ in any of the three behaviour descriptors; (b) Fair if the student received a rating of ‘Sometimes’ or ‘Most of the time’ in any of the three behaviour descriptors; and (c) Well Done if the student did not receive any ‘No’ and ‘Sometimes’ ratings, and received at least one rating of ‘All the time’. Once the overall behaviour rating was generated, the teacher verbally reviewed the rating with the student privately immediately after the lesson using a provided script. The script read, ‘You got a (rating) for your behaviour today as you did/did not perform the/these (on-task behaviours). Please meet your mentor later today and try to do better/keep up your good work in your next lesson’. The teacher told the student his/her rating for the lesson and explained the reason for the rating.
In addition, the online DBRC system would also convert each teacher's rating of the student's on-task behaviour to a performance scale of between 0% to 100% (i.e., the total rating score of a teacher divided by the total possible score of 9, then multiplied by 100). Next, an overall mean teacher rating of the student's on-task behaviour was then calculated (i.e., performance scale of all the teachers of the student divided by the number of teachers) to measure the average teacher rating of the student's on-task behaviour. At the end of the school day, the mentor would use the overall mean teacher rating and a given script to review the student's on-task behaviour with the respective student. The script read, ‘You got an overall (%) for your on-task behaviour today as you did/did not perform the/these (on-task behaviours). For this performance, you will receive a (reward based on performance). Please show this (DBRC printout) to your parent and try to do better/keep up your good work during your lessons tomorrow’. The mentor then printed the DBRC and placed the hard copy in a folder for the student to bring home for the parent's signature. The parents in this study chose to have the printed DBRC delivered to them by their child as they did not have easy access to the internet. The student returned the folder with the signed DBRC to the mentor the next day when they met up after school for the review session.
The criteria to determine the rewards or consequences was individualised for each participating student depending on the severity of their behavioural problem, as the online DBRC system was customised to the needs of the individual student to better manage their unique symptoms of ADHD. For instance, to be awarded a stamp reward, Andy needed to obtain at least a 70% overall mean teacher rating, whereas Lisa and Fred required a 65% and 60% overall mean teacher rating respectively. Students chose rewards based on a reward menu; for example, students could exchange three stamps for a free drink, or 12 stamps for a fast-food meal.
Teacher and mentor training
The researcher conducted a 1-hour training session to brief teachers on the research, and train teachers on the usage of the online DBRC system. During the training session, the researcher (a) described the purpose of the study, (b) explained the operational definitions of on-task and off-task behaviours, and (c) demonstrated the use of the online DBRC system and provided teachers with opportunities for guided and independent practice. The three mentors selected for this study underwent a 30-minute training session to monitor and review the students’ behaviour daily using the DBRC and provide advice, guidance, and encouragement, as well as how to mete out the rewards or consequences.
Dependent Measures
Off-task behaviour
The dependent measure for this study was the students’ off-task behaviour in the classroom. Off-task behaviours were operationally defined as (a) talking out and making noise (i.e., any audible vocalisation and/or noises made with objects, without the teacher's permission or that is/are not part of instructional activity); (b) out of seat, fidgety, and inattentive (i.e., student's body breaks contact with their seat for more than three seconds, fidgeting with own things or parts of his/her body, being distracted from task); and (c) touching others or their things (i.e., the student touches other students or their things when it is not part of the instructional activity, whether or not with aggressive intent). Observation sessions were 20 minutes. Using a 15-second interval with partial interval sampling, off-task behaviours were monitored and observed throughout each interval and recorded as occurring or not occurring. To obtain the percentage of intervals in which the off-task behaviour occurred, the number of intervals an off-task behaviour was coded was divided by the total number of intervals observed and multiplied by 100.
For Andy, observations of off-task behaviour took place four times a week during his English lesson. Observations of off-task behaviour for Fred took place during his mathematics lesson, which was held four times a week. Observations of off-task behaviour for Lisa also took place four times a week during her mathematics lesson. The days and periods for observations were consistent across the weeks. English and mathematics lessons in this school consisted of two to three consecutive 30-minute periods. However, the 20-minute data collection by the observers was not set at a specific time within the lesson. For example, observations of Andy took place on Mondays at the start of his three-period English lesson, whereas on Tuesdays, observations were conducted during the second half of his two-period English lesson.
Observers and observer training
Two female school administrative staff, who were blind to the study hypotheses, were employed as observers in this study. The researcher trained the two observers. Observer training included having the observers memorise the definitions of off-task behaviours, role-plays, and practice observation sessions in which selected students were observed and coded by one or two observers at a time. Observers were required to obtain interrater agreement above 80% for six out of seven practice sessions prior to data collection.
Interobserver Agreement
Interobserver agreement checks were assessed on at least 50% of the total observations and were collected by the two trained independent observers. Interobserver agreement for off-task behaviour was measured by comparing the two records for agreement for each session, and percentage agreement was calculated by dividing the number of agreements by the total number of opportunities to agree (the number of agreements plus disagreements for each session), and then multiplied by 100. The average interobserver agreement for the measure of off-task behaviour was 87% (range: 80–93%), 88% (range: 81–96%), and 91% (range: 81–100%) for Andy, Fred, and Lisa, respectively.
Treatment Fidelity
The observers were also trained to conduct treatment fidelity. Fidelity checks for teachers were conducted in all of the intervention sessions during the scheduled observations of the dependent measures. The observers checked if teachers had followed the two-step procedures of (a) reviewing the rating for that lesson with the student and (b) entering the rating into the online DBRC system. The observers then recorded in the observation sheet whether the teacher had or had not adhered to the procedures. Fidelity was calculated by dividing the total number of observed procedural steps by the total number of expected procedural steps and multiplying by 100. Average treatment fidelity of teachers for this study was 66% (range: 50–100%). The treatment fidelity data for teachers were occasionally at 50%. During those sessions, the procedural error was that the teachers did not input feedback into the online DBRC system immediately after the lessons as the teachers had to attend to their next lessons in other classrooms. In these cases, they provided only verbal feedback to the students before rushing off to their next lessons. Teachers then inputted their feedback into the online system at the end of the day.
Fidelity checks for mentors and parents were conducted for all the intervention sessions. The mentors’ adherence was based on the printed DBRC with mentors’ signatures. Parents’ adherence to the intervention protocol to determine if they had reviewed their child's performance was assessed by examining the signatures on the printed DBRCs that the student returned to the mentor teacher the next day. The mentors’ and parents’ fidelity was 100% across intervention phase as determined by the signatures of both mentors and parents on the printed DBRC.
Procedures and Research Design
A single-case research using a multiple baseline across participants design (Kazdin, Reference Kazdin2011) was employed to test the effects of online DBRC on students’ off-task behaviour. In single-case design research, the participant serves as his or her own control when baseline data are collected to compare with the intervention data. Unlike group experimental designs, where a large sample size is expected, single-case design research studies typically demonstrate the functional relationship between the intervention and target behaviour with a smaller sample size of at least three different participants (Horner et al., Reference Horner, Carr, Halle, McGee, Odom and Wolery2005).
Baseline
During the baseline phase, the teachers responded to students’ off-task behaviour as they normally would, which typically comprised ignoring, verbal reprimand, or sending the student to the discipline master.
Intervention
Following the baseline period, each student was assigned to a mentor who explained what the DBRC was and how it would be used to rate his/her classroom behaviour. The definition of the off-task behaviours was described to the students. The teachers of the participating student followed the steps outlined in the DBRC intervention protocol and indicated the behaviour rating of each student after every lesson. After school, the student went to the mentor for a daily review. During the 5- to 10-minute review, the mentor accessed the online DBRC system and viewed the ratings given by the various subject teachers as well as the overall behaviour rating and determined the reward or consequence. As the parents did not have regular access to the internet, the mentor printed the DBRC and reminded the student to show it to the parent. The parent acknowledged the report by signing the printout.
Maintenance
During maintenance, the online DBRC intervention was discontinued. Due to the upcoming examination period, a maintenance phase was implemented only with Andy. Andy did not see the mentor after school daily and teachers did not rate his behaviour during lessons. Maintenance data were collected once a week during English lessons, for two weeks, after the intervention phase.
Data Analysis
Data collected for each student were graphed to allow for visual examination of trends across phases. In the current study, based on the data recorded by observers, there was a decrease in percentage of off-task behaviour exhibited by students measured across the days in which DBRC intervention was implemented with mentor and parental involvement. The data were then visually analysed between and within phases to scrutinise the variability of the data and identify any perceptible trends in the behaviour.
Treatment Acceptability
DBRC intervention with parental involvement is a collaborative home and school intervention program, including parents as partners with the school in teaching at-risk youths. Treatment acceptability of DBRC was assessed at the conclusion of the study using an adapted Intervention Rating Profile 15 (IRP-15; Martens, Witt, Elliott, & Darveaux, Reference Martens, Witt, Elliott and Darveaux1985). The adapted IRP-15 is a 15-item scale that uses Likert-style rating, ranging from 1 to 6 for strongly disagree to strongly agree, to assess teacher's acceptability of the online DBRC intervention. Some sample items from the adapted IRP-15 were The DBRC intervention is effective in changing the student's problem behaviour, I would be willing to use this DBRC intervention in the classroom setting, and This DBRC intervention would be appropriate for a variety of students. An adapted IRP-15 was used to gather parents’ perception of the online DBRC intervention. Some items in the parents’ adapted IRP-15 were The DBRC is an acceptable intervention for my child, My child's problem behaviour warrants the use of this DBRC intervention, and This DBRC intervention is reasonable for my child's behaviour problem. Scores ranged from 15 to 90, with high scores indicating high acceptability. To evaluate treatment acceptability from the students’ perspective, an adapted Children's Intervention Rating Profile (CIRP; Witt & Elliott, Reference Witt and Elliott1985) was used in which students rated the seven items of the CIRP using a 6-point Likert scale ranging from 1 (disagree very strongly) to 6 (agree very strongly). Sample items in the adapted CIRP were The method used to help me was good and I think the method used would help me do better in school. Scores ranged from 7 to 42, with high scores indicating high acceptability.
Results
Summary of Findings
Figure 1 presents the percentage of off-task behaviour for Andy, Fred, and Lisa.
The Percentage of Off-Task Behaviour During Baseline and DBRC Intervention for Andy, Fred, and Lisa.
Andy
During baseline, Andy was off task on average 38% of the time across four sessions. Although the baseline data were variable, at the fourth data point, off-task behaviour increased to a high of 62%. As a result of this high off-task data point and the upward baseline trend, the decision to administer the intervention was made. When the DBRC intervention was introduced, Andy's off-task behaviour fell to an average of 21% across six sessions. The intervention data showed a decreasing trend. During the maintenance phase, off-task behaviour was at 7% one week after the intervention ended and 0% two weeks after. Andy's intervention data were more stable compared to baseline, and there was an immediate decrease in his off-task behaviour during the intervention phase.
Fred
Fred demonstrated an average occurrence of off-task behaviour of 69% across seven baseline sessions. His data pattern for off-task behaviour was fairly stable, with an upward trend, during the baseline phase. His highest percentage of observed off-task behaviour in a session during baseline was 89% and the lowest was a high of 55% of off-task behaviour. During the DBRC intervention, although there was a high of 91% of off-task behaviour in one session (he was rude to a teacher and that became a school discipline case), Fred's off-task behaviour showed a downward trend with an average of 50% across the seven intervention sessions. Although Fred was a highly impulsive student and prone to react rashly and aggressive to any threats or the smallest triggers, a visual analysis suggested that the impact of the DBRC intervention was immediate for Fred. The DBRC intervention had impacted him positively with improved behaviour reflected in four data points below the lowest baseline data point, with the lowest off-task behaviour at 22%.
Lisa
Lisa's average off-task behaviour was 22% during baseline with a gradual decreasing trend. Her peak off-task behaviour was observed at 49% as compared to her lowest percentage of off-task behaviour of 6% across this phase. Due to the impending school examination, Lisa was able to undergo intervention for only five sessions. Lisa registered the highest occurrence of off-task behaviour at 28% during intervention, which was a marked improvement from her peak of 49% during baseline. In addition, upon the intervention phase, there was an immediate decrease in off-task behaviour and the intervention trend line showed a clear decrease in the level of off-task behaviour.
Treatment Acceptability
Teachers’ and parents’ acceptability of the online DBRC intervention was measured using the adapted IRP-15. The mean IRP score for 17 teachers was 71.8, which represented high levels of acceptability given a maximum total score of 90, and the IRP scores from these teachers ranged from 53 to 90 in the context of a possible range of scores from 15 to 90. The mean rating for all the items was 4.77 and the mean ratings ranged from 4.53 to 5.06 for the 15 items, which indicated high acceptability of the online DBRC intervention by teachers.
The IRP scoring was slightly lower for the three participating parents, with a mean score of 67.7. The scores from individual parents were 43, 74, and 86, indicating that parents viewed the DBRC as a moderately acceptable intervention. The mean rating was 4.51 for the overall items and the mean item ratings ranged from 4.0 to 4.67. These scores indicated moderate acceptability of the DBRC intervention by parents.
The three participating students rated the online DBRC intervention through the adapted CIRP. The mean CIRP score from the students was high at 34.3, and the scores from individual students were 33, 35, and 35. The mean item score was 4.52 and the mean item ratings ranged from 3.7 to 5.7. The CIRP scores indicated high acceptability of the intervention by the students, with the minimum CIRP score at 33 and the highest at 35 (maximum CIRP score was 42). The students viewed the DBRC intervention favourably. They were of the opinion that it would help them do better in school and considered that DBRC would also be beneficial to other students.
Discussions
The first aim of this study was to investigate the efficacy of using online DBRC intervention to decrease off-task behaviour in students with ADHD in a Singapore public school for at-risk youths, and three main outcomes were found. First, students exhibited decreased off-task behaviour during the intervention phase of the research as compared to baseline. Specifically, off-task behaviour decreased when the students were monitored closely by their teachers and they had to acquire parents’ acknowledgement of the DBRC daily. In this study, Andy was the only participant who was subjected to the longest period of online DBRC intervention and underwent a maintenance phase. He showed sustained reduction in off-task behaviour during maintenance. The results therefore suggest the need for home–school communication through DBRC to improve student behaviour, and preferably over a sustained period of time as suggested by Owens et al. (Reference Owens, Holdaway, Zoromski, Evans, Himawan, Girio-Herrera and Murphy2012). Owens et al. emphasised the importance of persisting with a behavioural classroom intervention, as an intervention needs to be followed through over a period of time in order to reap positive results.
Second, a unique characteristic of this study's implementation of DBRC was the use of mentors to provide feedback and administer rewards or consequences. Bailey, Wolf, and Phillips (Reference Bailey, Wolf and Phillips1970) reported that the home-based reinforcement implemented by external agents was effective even if parents or school teachers were not involved in the implementation as the reinforcer. The results in the current study also showed that the review of student behaviour and reinforcement of rewards to modify behaviour need not be administered by parents in order to be effective, as reflected in the study by Bailey et al. (Reference Bailey, Wolf and Phillips1970). The mentors in this study were responsible for administering the rewards to the participating students upon reviewing their daily behaviour ratings by their teachers. Findings indicated that the mentors had been effective as off-task behaviour had reduced for the three participants, and informal feedback from the students indicated that they liked the online DBRC procedures and preferred to have the mentors review with them instead of their parents.
Third, the online DBRC system allowed for the coordinated implementation of DBRC across several teachers, facilitated by a mentor, for each student. Traditional paper DBRC interventions were usually conducted with one teacher rating the performance of one student and sending the report home to the parent (e.g., LeBel, Chafouleas, Britner, & Simonsen, Reference LeBel, Chafouleas, Britner and Simonsen2013). In this study, each student had at least five teachers rating the student's performance. The mentor would then provide feedback and reward the students based on the teachers’ ratings. The teachers easily accessed the online DBRC system to enter their rating of the student's performance, and the mentors were also able to conveniently keep track of the ratings and provided feedback to the student. Due to ease of use and speed of communication, and the lower probability of the student losing the DBRC slips, the online DBRC may be a more efficient system within a school setting. A traditional paper DBRC system would require more concerted effort by a coordinator to ensure that teachers’ DBRC ratings slips are passed on to the mentor.
The second aim of this current study relates to the acceptability of the online DBRC intervention in the eyes of the students, parents, and teachers. According to the treatment acceptability responses, the mean IRP score for teachers was 71.8 out of a total score of 90, which indicates high acceptability of the online DBRC intervention by teachers. This willingness and acceptance in teachers to adopt intervention strategies for students with ADHD would eventually influence the positive outcomes of students with behavioural problems (Zentall & Javorsky, Reference Zentall and Javorsky2007). Parents’ average IRP score of 67.7 suggested that parents perceived the DBRC as a moderately acceptable intervention. The IRP scores from Andy's and Fred's parents were 86 and 74 respectively, but the mean score was lowered due to the low scoring of 43 by Lisa's parent. As Lisa had to alternate between both parents’ homes every week, her father felt that the acknowledgement of the DBRC slips was an additional responsibility that he had to keep track of. The participating students perceived the DBRC favourably with their high mean CIRP score of 34.3 over a maximum of 42. Overall, students were perceived as responding positively to the intervention by parents and teachers, and the intervention itself was acceptable to all concerned (i.e., students, parents, and teachers).
Research Issues and Limitations
Despite the overall effectiveness of the online DBRC intervention and the general acceptability of the intervention, some limitations warrant noting. In this study, aside from the DBRC, there were components of teacher feedback and token reinforcement where students were rewarded with stamps for meeting the performance target, and the stamps were then exchanged for tangible items. Therefore the decrease in off-task behaviour could not be attributed to the online DBRC alone. In addition, although off-task behaviour decreased, students’ engagement in the lesson was not monitored in this study. Anecdotal reports from teachers, however, suggested higher engagement in the lessons.
In this study, we utilised a multiple baseline design, with a minimum of six phases and at least three data points per phase. Although baseline data for the second and third participant were collected over seven and 11 baseline sessions respectively, only four baseline data points were collected with the first participant, Andy, and this fell short of the minimum five data points per phase recommended by the What Works Clearinghouse (WWC) Pilot Single-Case Design Standards (WWC, 2017). Therefore, this study met the WWC Pilot Single-Case Design Standards with Reservations. In addition, Andy's baseline data showed large variability in off-task behaviour, with a low of around 20% to a high of over 60%. More baseline data should have been collected to show a clearer trend of his off-task behaviour before the online DBRC was implemented.
One of the major limitations of this study is the timing of the research. First, observation sessions were not fixed at a specific time of the lesson (e.g., beginning of the lesson) for all the observations. Data collection in this study was constrained by the availability of the observers, who were school administrative staff. This may have contributed to the variability in the students’ off-task behaviour across the observation sessions. Second, this study was conducted relatively close to the year end examination period of the Singapore school term; consequently, the observed student behaviour during baseline and intervention phase might not have been representative of the rest of the school year. In addition, due to time constraints, the intervention phase was short and the long-term effects of the DBRC were not studied for Fred and Lisa. If the study had been conducted for a longer period of time, the effectiveness of the DBRC intervention may have produced stronger and sustained results. Furthermore, due to the school examination period, a maintenance phase was conducted only with Andy, but the result suggested that the decreased off-task behaviour could be sustained even after the intervention was removed. Maintenance phase conducted with all three participants would have provided the generalisation effects of the DBRC across time and further validate intervention results.
Another limitation is that although many teachers completed the online DBRC rating for each participant, behavioural data were only collected in limited settings (i.e., one class setting for each student). As behavioural data were not collected in other settings, it is not known whether the students’ behaviour improved in the other classroom settings. Conversely, would the effect of the online DBRC be similar if only the teacher in the observed classroom setting had completed the online DBRC rating? Further research is thus warranted to evaluate the effects of online DBRC with or without the involvement of multiple teachers.
In this study, there were occasions where some teachers were not able to complete the online DBRC at the end of class, hence reducing the treatment fidelity of the online DBRC intervention. Although teachers generally agreed that the online DBRC intervention was effective to address the off-task behaviour of the participating students, some teachers questioned whether it was feasible to implement the online DBRC intervention for multiple students in a class effectively. This concern was raised as the general student population in the school was at-risk students and had similar misbehaviours that were exhibited by all three participants. Consequently, although the intervention received high acceptance by teachers based on the teacher-rated IRP, implementing this DBRC intervention in a class for several students with a high level of off-task behaviour might require further adjustments to the approach. There will be a need to simplify the intervention, as it may not be feasible in its current form. Further studies to look into DBRC interventions for multiple students in a class might be necessary to determine the efficacy in addressing off-task behaviour.
Implications for Future Research
The results of this study support the strong research base that a DBRC is an effective intervention to decrease off-task behaviour in students (Bailey et al., Reference Bailey, Wolf and Phillips1970; Jurbergs et al., Reference Jurbergs, Palcic and Kelley2007; Jurbergs, Palcic, & Kelley, Reference Jurbergs, Palcic and Kelley2010; Owens et al., Reference Owens, Holdaway, Zoromski, Evans, Himawan, Girio-Herrera and Murphy2012; Pelham & Fabiano, Reference Pelham and Fabiano2008). However, to broaden this knowledge base and support DBRC as an evidence-based intervention, this study should be replicated with more students and in various schools. Additionally, the time committed to carrying out the DBRC should be extended to allow greater benefits due to improved behaviour outcomes. Further studies should explore the efficiency and acceptability of online DBRC as compared to traditional paper-based DBRC. Future research should also involve a maintenance phase for all participants in which implementation of the intervention is discontinued but data are collected after a significant period of time to determine if the effectiveness of using DBRC to decrease off-task behaviour can be maintained.
The results of this study found that mentors, who provided feedback and rewards to the students, had been effective in guiding students using the DBRC intervention, which was demonstrated by the reduction of off-task behaviour for all three students. Parental involvement was low in this online DBRC intervention as the parents signed the DBRC sheet only to acknowledge they had reviewed the teachers’ ratings of their child's behaviour for that day. Published research studies show mixed results in terms of the optimal level of parent participation in DBRCs. Although Vannest et al. (Reference Vannest, Davis, Davis, Mason and Burke2010) found that DBRC interventions with the highest home–school communication showed significantly stronger effect sizes compared to studies with low home–school communication, Pyle and Fabiano (Reference Pyle and Fabiano2017) established that for students with ADHD the level of home–school communication was not a moderating variable in the effectiveness of a DBRC. Future research could therefore explore DBRCs with different levels of parental involvement.
Although DBRC is more popular among elementary school and special education teachers, DBRCs are also commonly used across populations of other school-age students, such as preschool or high school students (Chafouleas, Riley-Tillman, & Sassu, Reference Chafouleas, Riley-Tillman and Sassu2006). This could be a possible future study, to determine the necessary conditions (e.g., particular groups of students, different settings, and specific problem behaviour) for effective use of DBRC. Future research should also explore using DBRC intervention with other types of interventions; for instance, implementing DBRC intervention along with social skills training, or incorporating functional behavioural assessment, or integrating DBRC as part of a school-wide positive behaviour support system. The combination of interventions from these studies would establish if more effective results can be obtained; that is, greater reduction of off-task behaviour and sustained on-task behaviour.
Conclusion
Teachers may lack the necessary skillset and knowledge to provide the guidance and support to effectively help students who have ADHD (Reid, Vasa, Maag, & Wright, Reference Reid, Vasa, Maag and Wright1994; Zentall & Jarvosky, Reference Zentall and Javorsky2007). Having an array of evidence-based behavioural intervention tools to assist teachers would offer more effective support and guidance for the students concerned. The purpose of this research was to examine the efficacy of implementing online DBRC to decrease off-task behaviour in students with ADHD. Despite the noted limitations, this study provides additional support of the effectiveness of DBRC and therefore extends the knowledge base of the effectiveness of DBRC intervention for at-risk youths.
