Introduction
Attention-deficit/hyperactivity disorder (AD/HD) is a prevalent neurodevelopmental disorder (Polanczyk, Willcutt, Salum, Kieling, & Rohde, Reference Polanczyk, Willcutt, Salum, Kieling and Rohde2014) and afflicted individuals suffer long-term deficits (Karam et al., Reference Karam, Breda, Picon, Rovaris, Victor, Salgado and Bau2015). While medication is the most effective and widely used treatment in children with AD/HD (Wolraich et al., Reference Wolraich, Brown, Brown, DuPaul, Earls, Feldman and Davidson2011), concerns are expressed regarding side effects (Efron, Jarman, & Barker, Reference Efron, Jarman and Barker1997; Graham et al., Reference Graham, Banaschewski, Buitelaar, Coghill, Danckaerts, Dittmann and Grp2011; Vitiello, Reference Vitiello2008), a lack of evidence for long-term effects (Wang et al., Reference Wang, Volkow, Wigal, Kollins, Newcorn, Telang and Swanson2013), and poor adherence (Berger, Dor, Nevo, & Goldzweig, Reference Berger, Dor, Nevo and Goldzweig2008) — these findings underpin the urgent need for alternative treatments. Moreover, with promising results of home-based digital therapeutic in AD/HD (e.g. Kollins et al., Reference Kollins, DeLoss, Cañadas, Lutz, Findling, Keefe and Faraone2020) and compromised service for mental health disorders during the COVID-19 pandemic (Moreno et al., Reference Moreno, Wykes, Galderisi, Nordentoft, Crossley, Jones and Arango2020), considerable attention has been paid to the possibility of alternative treatments with remote access (Chew et al., Reference Chew, Ong, Lei, Rajendram, Grisan, Verma and Gunasekerans2020; Lim, Lim-Ashworth, & Fung, Reference Lim, Lim-Ashworth and Fung2020).
The increasing understanding of AD/HD lays the foundations for developing new treatments. Models have attributed AD/HD symptoms to atypical functions such as abnormal central nervous system arousal and deficient executive functions (EFs) (Barkley, Reference Barkley1997; Sergeant, Reference Sergeant2005; Sonuga-Barke, Reference Sonuga-Barke2005; Zentall & Zentall, Reference Zentall and Zentall1983). These models provide a base for developing new interventions, as AD/HD symptoms may be alleviated by training to improve atypical functions. Two such interventions are cognitive training (CT) and neurofeedback training (NFT). CT approaches for children with AD/HD usually target the EF subcomponents of working memory (WM) and/or response inhibition (RI) based on research showing deficits in these processes (Lijffijt, Kenemans, Verbaten, & van Engeland, Reference Lijffijt, Kenemans, Verbaten and van Engeland2005; Willcutt, Doyle, Nigg, Faraone, & Pennington, Reference Willcutt, Doyle, Nigg, Faraone and Pennington2005). NFT aims to promote awareness and control of brain states (e.g. resting state, attentional state, relaxed state) to allow more effective state modulation in response to situational demands (Arns, Heinrich, & Strehl, Reference Arns, Heinrich and Strehl2014; Lofthouse, Arnold, Hersch, Hurt, & DeBeus, Reference Lofthouse, Arnold, Hersch, Hurt and DeBeus2012). As the two types of training may address different aspects of atypical functions in children with AD/HD, the efficacy of a combined approach integrating CT with NFT has also been examined, conceptualised as neurocognitive training (Johnstone et al., Reference Johnstone, Roodenrys, Blackman, Johnston, Loveday, Mantz and Barratt2012, Johnstone, Roodenrys, Johnson, Bonfield, & Bennett, Reference Johnstone, Roodenrys, Johnson, Bonfield and Bennett2017).
While empirical evidence supports the efficacy of CT, NFT, and neurocognitive approaches (e.g. Arns et al., Reference Arns, Heinrich and Strehl2014; Jiang & Johnstone, Reference Jiang and Johnstone2015; Jiang, Johnstone, Sun, & Zhang, Reference Jiang, Johnstone, Sun and Zhang2018; Johnstone, Roodenrys, Phillips, Watt, & Mantz, Reference Johnstone, Roodenrys, Phillips, Watt and Mantz2010, Reference Johnstone, Roodenrys, Blackman, Johnston, Loveday, Mantz and Barratt2012, Reference Johnstone, Roodenrys, Johnson, Bonfield and Bennett2017), three issues should be considered. Firstly, comparisons of the efficacy of different protocols are needed. Theoretically, a combined training approach with multiple training targets should be more efficacious than single-target training (i.e. CT or NFT alone). Secondly, as the usefulness of a training approach depends on the extent to which the outcomes transfer to broader situations (Klingberg, Reference Klingberg2010), transferability should be compared between different approaches (Cortese et al., Reference Cortese, Ferrin, Brandeis, Buitelaar, Daley and Dittmann2015, Reference Cortese, Ferrin, Brandeis, Holtmann, Aggensteiner, Daley and Zuddas2016; Sonuga-Barke et al., Reference Sonuga-Barke, Brandeis, Cortese, Daley, Ferrin and Holtmann2013; Sonuga-Barke, Brandeis, Holtmann, & Cortese, Reference Sonuga-Barke, Brandeis, Holtmann and Cortese2014). Transfer effects are usually classified as ‘near’ or ‘far’ in psychological research (Barnett & Ceci, Reference Barnett and Ceci2002). Near transfer refers to training gains in situations that are similar in context to the training, while far transfer refers to training gains in situations that have little overlap with the training context (Barnett & Ceci, Reference Barnett and Ceci2002). Thirdly, previous studies have been criticised for the use of non-optimal control groups/conditions, which may lead to biased conclusions (Arns et al., Reference Arns, Heinrich and Strehl2014; Shipstead, Redick, & Engle, Reference Shipstead, Redick and Engle2012; Sonuga-Barke et al., Reference Sonuga-Barke, Brandeis, Cortese, Daley, Ferrin and Holtmann2013). For example, as CT and NFT typically involves several sessions, improved performance in experimental groups compared to waiting-list control groups may stem from expectation or placebo effects and/or more prolonged contact with clinicians/researchers (Arns et al., Reference Arns, Heinrich and Strehl2014; Shipstead et al., Reference Shipstead, Redick and Engle2012). Thus, there is a need to control for non-specific factors such as the amount of client–therapist interaction or training sessions, time commitment, and expectation (Arns et al., Reference Arns, Heinrich and Strehl2014).
In the current study, the effects of three home-based training protocols (i.e. CT alone, NFT alone, and CT combined with NFT) were compared regarding near and far transfer effects with non-specific factors controlled. Along with AD/HD symptoms, measures that play an important role in AD/HD theories are considered as outcome measures, including EFs and resting-state electroencephalograph (RS-EEG) activity. The outcome measures were classified as measures of either near or far transfer based on the similarity of the functional context between the training content and the measure (Barnett & Ceci, Reference Barnett and Ceci2002).
A single-case experimental design (SCED) was adopted for the current study. SCED includes frequent pre- and post-intervention measurement of dependent variables, with each participant serving as their own control. The internal validity of the experiment is addressed by the requirement for the dependent variables to change only after introducing the independent variables. External validity is addressed by replication within or between participants (Dallery, Cassidy, & Raiff, Reference Dallery, Cassidy and Raiff2013). Compared with group designs, SCED requires fewer participants but more frequent observations. A well-controlled SCED is ranked as Level 1 evidence by the Oxford Centre for Evidence-based Medicine alongside randomised controlled trials. There are several variants of SCED, such as the ABAB design, multiple-baseline design, and alternating treatments design (Dallery et al., Reference Dallery, Cassidy and Raiff2013). As the effects of the interventions to be used in this study are likely to be maintained over a period of time, a multiple-baseline SCED is appropriate (Dallery et al., Reference Dallery, Cassidy and Raiff2013). In the multiple-baseline design, each participant should undergo two phases: a baseline phase and an intervention phase. The only difference between the two phases is the involvement of the intervention. Experimental effects are reflected in the differences between two phases (Tate, Perdices, McDonald, Togher, & Rosenkoetter, Reference Tate, Perdices, McDonald, Togher and Rosenkoetter2014).
As near transfer effects have been reported previously in CT and NFT studies (e.g. Johnstone et al., Reference Johnstone, Roodenrys, Blackman, Johnston, Loveday, Mantz and Barratt2012; Karbach & Kray, Reference Karbach and Kray2009), it is predicted that there will be near transfer effects in both the CT condition and the NFT condition. Furthermore, as the combined training condition involves both CT and NFT training, similar near transfer effects are predicted. Also, as the combined condition has been shown to alleviate AD/HD symptoms (Jiang & Johnstone, Reference Jiang and Johnstone2015; Johnstone et al., Reference Johnstone, Roodenrys, Johnson, Bonfield and Bennett2017), a far transfer effect on AD/HD symptoms is predicted.
Methods
Participants
Twelve children were initially recruited. One participant dropped out due to health issues, and another as a result of not adhering to the intervention plan. Subsequently, two more participants were recruited. Demographic information is displayed in Table 1.
Demographic Information for Cases in Each Group
Note: CT = cognitive training condition; NFT = neurofeedback training condition; Combined = cognitive plus neurofeedback training condition; F = female; M = male; C = combined subtype; I = inattention subtype.
The participants were recruited at Peking University Sixth Hospital in Beijing, China and were diagnosed by highly experienced paediatric psychiatrists. All participants met the following inclusion criteria: (1) screened by the Clinical Diagnostic Interviewing Scales (Barkley, Reference Barkley1997), a structured clinical interview based on the DSM-IV, (2) no history of head trauma with loss of consciousness, (3) no history of neurological illness or other severe diseases, (4) no history of other psychiatric disorders described in the DSM-IV, (5) naïve to any pharmacological treatment for AD/HD, and (6) an IQ higher than 80 on the Wechsler Intelligence Scale III for children.
Design
Each participant was randomised to one of the three training conditions (CT, NFT, or Combined); therefore, each condition consisted of four cases. Each case completed a baseline phase, followed by an intervention phase. The baseline durations within a training condition were varied, as outlined below. The duration of the intervention phases was equal for each condition. Participants and their parents were not informed that there were two phases across sessions.
Training Phases
During the intervention phase, participants were required to complete 20 training sessions at home over a 7-week period. They completed three sessions (Monday, Wednesday, and Friday) per week for the first 6 weeks and 2 sessions (Wednesday and Friday) in the last week. Each training session took approximately 25 min to complete. The intervention was completed on a 7-inch tablet device and delivered using a combination of software (i.e. Focus Pocus software application) and EEG hardware (i.e. NeuroSky Mindwave Mobile EEG device). The training software has been fully introduced in other papers (e.g. Jiang & Johnstone, Reference Jiang and Johnstone2015; Johnstone et al., Reference Johnstone, Roodenrys, Johnson, Bonfield and Bennett2017). Briefly, the software application presents different types of training as games, with the difficulty level of each game being dependent on performance. Each training session consisted of 14 games. The CT condition consisted of seven WM games and seven RI games. The NFT condition consisted of seven games linked to attention-related EEG activity (i.e. mainly frontal beta power) and seven linked to relaxation-related EEG activity (i.e. mainly frontal alpha power). The Combined condition consisted of eight NFT games (four attention-related and four relaxation-related), three WM, and three RI games.
Active control was adopted to maximise internal experimental validity (e.g. expectation). In the baseline phase, participants were required to wear the portable EEG device and to complete strategy games on the tablet. The ‘training’ frequency and length of the strategy games were the same as in the intervention phase; 25 min per session and 3 sessions per week. The baseline duration was predetermined. Within each training condition, the baseline duration for the first three cases was randomised from 2-, 3-, and 4-weeks without replacement, and the duration for the last case was randomised from 2-, 3-, and 4-weeks.
Measures
Three types of measures were used to assess transfer effects — EFs, RS-EEG, and AD/HD symptoms. We adopted two approaches to measuring EFs: computerised tasks for measuring lab-based EF performance and a questionnaire for measuring everyday EF performance. The computerised EF tasks were not gamified and involved the 2-back task for measuring WM (Jaeggi, Buschkuehl, Perrig, & Meier, Reference Jaeggi, Buschkuehl, Perrig and Meier2010) and the Stop-signal task for measuring RI (Logan, Van Zandt, Verbruggen, & Wagenmakers, Reference Logan, Van Zandt, Verbruggen and Wagenmakers2014). Performance accuracy from the 2-back task and stop-signal reaction time (SSRT) from the stop-signal task were selected as dependent variables. The Behavior Rating Inventory of Executive Function (BRIEF) parent version was used to measure everyday EFs (Gioia, Isquith, Guy, & Kenworthy, Reference Gioia, Isquith, Guy and Kenworthy2000), with a focus on two subscales for measuring everyday WM and RI. Although both approaches claim to measure EFs, previous research has reported low or no correlations between them (Barkley & Fischer, Reference Barkley and Fischer2011; McAuley, Chen, Goos, Schachar, & Crosbie, Reference McAuley, Chen, Goos, Schachar and Crosbie2010; Toplak, Bucciarelli, Jain, & Tannock, Reference Toplak, Bucciarelli, Jain and Tannock2009), suggesting that they are tapping different aspects of EFs.
RS-EEG was measured by a 14-channel wireless EEG headset device (Emotiv EPOC). The device records EEG from 14 scalp locations (AF3, F7, F3, FC5, T7, P7, P8, T8, FC6, F4, F8, AF4, O1, and O2) at 128 Hz. Four minutes of RS-EEG was recorded while the participant was resting with eyes closed, followed by 4 min during an eyes-open resting condition. EEG pre-processing followed previous studies in this area (e.g. Barry, Clarke, Johnstone, McCarthy, & Selikowitz, Reference Barry, Clarke, Johnstone, McCarthy and Selikowitz2009; Clarke et al., Reference Clarke, Barry, Dupuy, Heckel, McCarthy, Selikowitz and Johnstone2011; Zhang et al., Reference Zhang, Johnstone, Roodenrys, Luo, Li, Wang and Sun2018). All channels were band-pass filtered from 1 to 70 Hz with a 50 Hz notch filter. Visual inspection was used to identify and exclude sections of EEG trace containing gross artefacts. The Independent Component Analysis function in EEGLAB (Delorme & Makeig, Reference Delorme and Makeig2004) identified and excluded components related to eye and muscle movements; this is a semi-automatic process using the ADJUST toolbox (Mognon, Jovicich, Bruzzone, & Buiatti, Reference Mognon, Jovicich, Bruzzone and Buiatti2011). The EEG traces were then segmented into 4 s epochs. These epochs were Fourier transformed using a Hamming window, with EEG spectral power summed within four frequency bands: delta (1.5−3.5 Hz), theta (3.5−7.5 Hz), alpha (7.5−12.5 Hz), and beta (12.5−25 Hz). Baseline arousal (absolute alpha power averaged across channels in the eyes-closed condition, Barry, Clarke, Johnstone, & Rushby, Reference Barry, Clarke, Johnstone and Rushby2008), frontocentral theta/beta ratio (TBR, obtained by averaging the ratio of FC5 and FC6; Zhang, Li, et al., Reference Zhang, Li, Wu, Zhao, Song, Liu and Sun2017; Zhang, Roodenrys, et al., Reference Zhang, Roodenrys, Li, Barry, Clarke, Wu and Sun2017), frontal alpha (absolute alpha power average at AF3, F7, F3, F4, F8, and AF4) and frontal beta (absolute beta power average at AF3, F7, F3, F4, F8, and AF4) were selected as the dependent variables.
AD/HD symptoms were rated by the participant's parents. The AD/HD Rating Scale was used to rate the severity of Inattention (IA) and Hyperactivity–Impulsivity (HI) symptoms.
Measures were classified as near or far transfer outcomes according to their similarities to three training conditions (Table 2). Caution should be taken to the classification of EF tasks for the CT condition. The training content of the CT condition approximated computerised EF tasks. As previously mentioned, EFs measured by computerised tasks differ from those measured by the BRIEF (Barkley & Fischer, Reference Barkley and Fischer2011; McAuley et al., Reference McAuley, Chen, Goos, Schachar and Crosbie2010; Toplak et al., Reference Toplak, Bucciarelli, Jain and Tannock2009). Thus, the performance on the computerised 2-back and Stop-signal tasks was regarded as near transfer for CT, while the performance on BRIEF was regarded as far transfer.
Near and Far Transfer Tasks for Each Condition
Note: CT = cognitive training; NFT = neurofeedback training; Combined = cognitive plus neurofeedback training; BRIEF = the parent version of Behavior Rating Inventory of Executive Function; RS-EEG = resting-state EEG, including global alpha, fronto-cental theta/beta ratio, frontal alpha, and frontal beta; AD/HD symptoms = measured by the AD/HD Rating Scale.
Procedure
Ethics approval was obtained from the Ethics Committee of Peking University Health Science Centre and the University of Wollongong Human Research Ethics Committee (HE 16/032). Informed consent was obtained from the parent or guardian of each participant prior to accessing any record or testing.
The first assessment session was conducted before the commencement of the baseline phase at Peking University Sixth Hospital. Children and their parents were given a demonstration and instructions in the use of the EEG headset and strategy games for the baseline phase. After this phase, children and their parents were guided to use the Focus Pocus software. Children completed both phases at home. The participants and their parent(s) attended the hospital for an assessment session on the weekend during both phases.
Data Analysis
Visual analysis is frequently used in SCED research, with a focus on specific features to examine intervention effects (WWC, 2014). Elements of the analysis include (1) Level — the mean score for the data within a phase, (2) Trend — the slope for the data within a phase, and (3) Variability — the range or standard deviation of data within a phase. Next, features that show the difference between phases are examined: (4) Immediacy — the change between the level of the last three observation points in one phase and that of the first three observation points in the next phase, (5) Overlap — the proportion of data between phases overlapped, and (6) Consistency — the consistency of data in similar phases.
Although often used in SCED, visual analysis has been criticised for lacking decision guidelines, being potentially biased by trends in the baseline phases, and insensitivity to subtle changes (Harrington & Velicer, Reference Harrington and Velicer2015). These shortcomings have motivated the development of statistical analysis techniques for use in SCED research. Tau-U is one such statistical method, based on non-parametric inference, combining non-overlap and trend analysis (Parker, Vannest, Davis, & Sauber, Reference Parker, Vannest, Davis and Sauber2011). The rationale of this technique is that there should be little or no overlap between data in the baseline and intervention phases if the intervention is showing an effect. Also, the technique examines any trends in the baseline phase and can correct for this in the phase comparison. Compared to visual analysis, Tau-U provides a more objective and sensitive way to detect intervention effects (Parker et al., Reference Parker, Vannest, Davis and Sauber2011). However, it does not provide information about the immediacy of the effect. Hence, Tau-U analysis was firstly conducted to detect the intervention effects, with the visual analysis used as an adjunct analysis to examine the immediacy of the effect. Also, the level and consistency were examined to recheck any effects detected by Tau-U analysis.
Results
This study adopted the ‘three demonstrations’ criterion (i.e. the conclusion that the intervention showed an effect on a variable was drawn only when at least three cases showed the phase difference) to assess if the intervention had an effect (Kratochwill et al., Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2013). Due to internet connection issues, not all training sessions were completed online with adaptive difficulty for some participants. Instead, offline-mode training with non-adaptive difficulties was completed when not able to train online. The exact sessions in which the participants competed training offline are shown in Figures 1–3.
The CT Condition Outcomes
Completion
Figure 1 displays the dependent variables across sessions for participants (P4, P5, P9, and P12) in the CT condition. P4, P9, and P10 completed all training sessions online (i.e. with the training software adjusting the game's difficulty level adaptively according to their performance on each game). P5 completed 15 sessions online and 5 sessions offline (i.e. with self-directed difficulty level) due to technical challenges.
The performance of the participants in the CT condition across sessions. The tiny red bars indexed the sessions in which the participant completed training offline. SSRT = stop-signal reaction time; WM and inhibition subscales = two subscales in the parent version of Behavior Rating Inventory of Executive Function.
The baseline phase for P9 was one week longer than the assigned baseline duration due to a minor technical challenge with the tablet. External artefacts largely contaminated P9's EEG data from the fourth session. As a result, RS-EEG measures are missing.
Intervention effects on EFs
Table 3 displays the Tau-U results for the EFs. Three demonstrations of the intervention effect were shown for inhibition rated by BRIEF. The Tau-U analyses indicated that P4, P5, and P9 showed significantly reduced inhibition scores in the intervention phase. Visual analysis indicated that the BRIEF inhibition subscale in the baseline phase (meanBP) was larger than in intervention phase (meanIP) for P4 (meanBP: 24 vs. meanIP: 20), P5 (meanBP: 15 vs. meanIP: 11), and P9 (meanBP: 18 vs. meanIP: 13). As the changes were not immediate in relation to the introduction of CT, it appears that the effects were delayed.
Tau Analysis on EFs for Each Group
Note: CT = cognitive training condition; NFT = neurofeedback training condition; Combined = cognitive plus neurofeedback training condition; SSRT = stop-signal reaction time; BRIEF = the parent version of Behavior Rating Inventory of Executive Function. * marks p less than or close to 0.05.
Cases P5 and P9 showed improvement in the intervention phase for 2-back. Cases P4 and P5 showed improvement in WM rated by BRIEF. Case P12 showed improvement in SSRT. However, the number of effect demonstrations did not meet the criterion of three demonstrations for these measures.
Intervention effects on RS-EEG
Table 4 displays the Tau-U results for the RS-EEG measures. In the intervention phase, P4 and P5 showed an increase in frontal alpha, and P5 showed an increase in global alpha — thus, these variables did not show the effect criterion. No effect was found for the CT condition on frontal beta.
Tau Analysis on the RS-EEG Measures for Each Group
Note: CT = cognitive training condition; NFT = neurofeedback training condition; Combined = cognitive plus neurofeedback training condition; TBR = fronto-cental theta/beta ratio. * marks p less than or close to 0.05.
Intervention effects on AD/HD symptoms
Table 5 displays Tau-U results for the ADHD-RS scores. P9 and P12 showed a decrease in HI, and P5 showed a decrease in IA, indicating that there was no effect for the CT condition on AD/HD symptoms.
Tau Analysis on the AD/HD Symptoms for Each Group
Note: CT = cognitive training condition; NFT = neurofeedback training condition; Combined = cognitive plus neurofeedback training condition; IA = Inattention; HI = Hyperactivity–Impulsivity. * marks p less than or close to 0.05.
The NFT Condition Outcomes
Completion
Figure 2 displays the dependent variables across sessions for participants (P1, P3, P6, and P10) in the NFT condition. P10 completed all sessions online. P1 completed 18 sessions online and 2 sessions offline. P3 completed 13 sessions online and 7 sessions offline. P6 completed 14 sessions online and 6 sessions offline.
The performance of the participants in the NFT condition across sessions. SSRT = stop-signal reaction time; WM and inhibition subscales = two subscales in the parent version of Behavior Rating Inventory of Executive Function.
External artefacts largely contaminated P10's EEG raw data from their third session so RS-EEG measures from this session are missing.
Intervention effects on EFs
Table 3 displays the Tau-U results for the EFs. Tau-U analysis indicated that P1, P3, and P6 showed significantly higher 2-back response accuracy in the intervention phase. Visual analysis of 2-back response accuracy indicated that the meanBP was smaller than the meanIP for P1 (meanBP: 59% vs. meanIP: 72%), P3 (meanBP: 73% vs. meanIP: 84%), and P6 (meanBP: 52% vs. meanIP: 78%). As NFT was introduced in the intervention phase, 2-back accuracy was improved without delay.
Case P5 showed a decrease in SSRT. Case P9 showed an increased WM score rated by BRIEF. No cases showed a change in inhibition rated by BRIEF, indicating that there was no effect for the NFT condition on these variables.
Intervention effects on RS-EEG
Table 4 displays the Tau-U results for the RS-EEG measures. Tau-U analysis demonstrated that cases P1, P6, and P10 showed an increase in the frontal alpha power in the intervention phase. Visual analysis of frontal alpha power indicated that meanBP was smaller than meanIP for P1 (meanBP: 17.0 μv2 vs. meanIP: 30.8 μv2), P6 (meanBP: 10.8 μv2 vs. meanIP: 20.3 μv2), and P10 (meanBP: 7.4 μv2 vs. meanIP: 19.6 μv2). As NFT was introduced in the intervention phase, the frontal alpha power change was rapid in P1 and P6 but delayed in P10.
P1 and P10 showed an increase in frontal beta and global alpha, and P10 showed a decrease in TBR in the intervention phase; however, the number of demonstrations did not meet the effect criterion.
Intervention effects on AD/HD symptoms
Table 5 displays the Tau-U results for the AD/HD symptoms. Case P5 showed a decrease in IA. No changes were found for HI. These results indicate no effect for the NFT condition on AD/HD symptoms.
Combined Condition Outcomes
Completion
Figure 3 displays the dependent variables across sessions for participants (P2, P7, P13, and P14) in the Combined condition. P2 and P7 completed all sessions online. P13 and P14 completed 10 sessions online and 10 sessions offline. P14 was unable to train for 1 week of the intervention phase due to travel. The data for that week (the ninth) were not recorded, but one more week was added to the intervention phase.
The performance of the participants in the combined condition across sessions. SSRT = stop-signal reaction time; WM and inhibition subscales = two subscales in the parent version of Behavior Rating Inventory of Executive Function.
Intervention effects on EFs
Table 3 displays the Tau-U results for the EFs. Cases P2 and P13 showed an increase in the 2-back accuracy. Case P2 showed a decrease in SSRT and inhibition score rated by BRIEF. No significant changes were found for WM rated by BRIEF. Together, these results indicate that there was no effect for the Combined condition on the EF measures.
Intervention effects on RS-EEG
Table 4 displays the Tau-U results for the RS-EEG measures. P2 showed an increase in global alpha. No changes were found in the other RS-EEG measures, indicating that there was no effect for the Combined condition on the RS-EEG measures.
Intervention effects on AD/HD symptoms
Table 5 displays the Tau-U results for the AD/HD symptoms. Case P13 showed a decrease in IA. Case P2 showed a decrease in HI. These results indicate that the Combined condition had no impact on AD/HD symptoms.
Discussion
This study aimed to compare the transfer effects of three promising at-home interventions in children with AD/HD by using a multiple-baseline SCED. Twelve participants completed 20 sessions of training at home. The current study found that the CT condition reduced the BRIEF inhibition score, the NFT condition increased frontal alpha power and 2-back accuracy, and no changes were present in EFs, RS-EEG, or AD/HD symptoms for the Combined condition.
A near transfer effect was shown in the NFT condition. NFT in this study aimed to improve state-regulation ability via enhancing modulation of EEG frontal alpha and beta activity. This study showed that global alpha activity was increased after the NFT, which suggests that the focal alpha NFT has a near transfer effect to modify broader alpha activity. This is important as the reduced global alpha activity has been regarded as a biomarker for a certain proportion of children with AD/HD (Clarke et al., Reference Clarke, Barry, Dupuy, Heckel, McCarthy, Selikowitz and Johnstone2011).
In terms of far transfer, both CT and NFT showed their effects on measures dissimilar to the training. BRIEF was considered as a far transfer measure for CT as it indexes aspects of EFs unrelated to the EFs trained in CT. A reduced inhibition score was observed after CT. Following the outcome explanations of BRIEF (Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000), the reduced score indicated that the CT condition showed an improved ability to resist impulses and to stop their inappropriate behaviours in daily situations. This suggests that the CT condition had a far transfer effect in improving day-to-day inhibitory performance in children with AD/HD. Similar findings have been reported in children with AD/HD (Johnstone et al., Reference Johnstone, Roodenrys, Phillips, Watt and Mantz2010; Reference Johnstone, Roodenrys, Blackman, Johnston, Loveday, Mantz and Barratt2012) and also other domains; for example, after inhibition training, adults show an increased ability to inhibit chocolate consumption (Houben & Jansen, Reference Houben and Jansen2011) and alcohol use (Houben, Wiers, & Jansen, Reference Houben, Wiers and Jansen2011) in daily life.
The far transfer for NFT was also shown on EF. The performance on the 2-back WM task, which was not part of NFT, was also improved. This far transfer may be due to the relationship between brain resting activity and task-related performance. Recent cognitive neuroscience research attaches more importance to brain resting activity (Northoff, Duncan, & Hayes, Reference Northoff, Duncan and Hayes2010; Raichle, Reference Raichle2009) and shows that brain resting activity can predict N-back WM performance (e.g. Zou et al., Reference Zou, Ross, Gu, Geng, Zuo, Hong and Yang2013). Hence, modifying brain resting activity may, in turn, change WM performance.
In contrast, the Combined condition did not show any transfer effects. As the Combined condition involved both CT and NFT, evidence of improvements in the CT and NFT conditions was expected to be seen in the Combined condition. A closer inspection of the training parameters and completion may explain these results. An implicit rationale behind the condition comparison was to explore the transfer effects of the three conditions over the same training duration. Each condition required 20 sessions of training of about 25 min per session. As a consequence of controlling for total training duration, the time spent on CT in the Combined condition was half as much as in the CT condition; similarly, the time spent on NFT was half as much as in the NFT condition. Thus, the training time may not have been long enough to achieve the transfer effects seen in the CT and NFT conditions. Furthermore, it should be noted that two participants in the Combined condition completed only half of the training with the adaptive difficulty due to technical issues. Although in offline mode, participants were guided to select challengeable difficulty each time, the self-selected training difficulty may not have been sufficiently challenging. Groups who train at a single- or fixed-difficulty level show limited training effects (Johnstone et al., Reference Johnstone, Roodenrys, Phillips, Watt and Mantz2010; Motter, Devanand, Doraiswamy, & Sneed, Reference Motter, Devanand, Doraiswamy and Sneed2016). Together, the reduced training time on each training component and the involvement of the non-adaptive training may have impeded the Combined condition to produce the transfer effects shown in the CT and NFT conditions.
Some other improvements were shown in the current study but with only one or two demonstrations. Following the ‘minimum 3-demonstration criterion’ (Kratochwill et al., Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2013), these improvements are not regarded as intervention effects, and they indeed may be caused by factors other than the experimental manipulations (Tate et al., Reference Tate, Perdices, McDonald, Togher and Rosenkoetter2014). However, it is also possible that there are individual differences in obtained training benefits, a notion supported by studies in other populations. In healthy populations, researchers point out that motivation and implicit attitude to intelligence may influence training effects, and individuals will not benefit from training in the same way (Jaeggi, Buschkuehl, Shah, & Jonides, Reference Jaeggi, Buschkuehl, Shah and Jonides2014). A similar proposition was put in clinical research, with patients’ character affecting training outcomes (Vinogradov, Fisher, & de Villers-Sidani, Reference Vinogradov, Fisher and de Villers-Sidani2012). Hence, the improvements that did not reach the effect level may be caused by the different reactivity to the training in children with AD/HD, which encourages future studies to consider individual differences in obtaining training effects.
Transfer to untrained situations is important for assessing training efficacy (Klingberg, Reference Klingberg2010). Transfer as a result of CT and NFT has been less examined (Cortese et al., Reference Cortese, Ferrin, Brandeis, Buitelaar, Daley and Dittmann2015, Reference Cortese, Ferrin, Brandeis, Holtmann, Aggensteiner, Daley and Zuddas2016; Sonuga-Barke et al., Reference Sonuga-Barke, Brandeis, Cortese, Daley, Ferrin and Holtmann2013, Reference Sonuga-Barke, Brandeis, Holtmann and Cortese2014). The current study suggests that CT and NFT can benefit untrained abilities, such as executive functions and resting brain states, which contribute to behavioural difficulties in children with AD/HD (Barkley, Reference Barkley1997; Clarke et al., Reference Clarke, Barry, Dupuy, Heckel, McCarthy, Selikowitz and Johnstone2011; Nigg, Willcutt, Doyle, & Sonuga-Barke, Reference Nigg, Willcutt, Doyle and Sonuga-Barke2005). It should also be noted that no effects on AD/HD symptoms were found for the conditions, which questions the utilisation of these protocols on their own. Future studies may examine if the CT and NFT are better than medication treatment in terms of the improved measures in this study, which may better help children with AD/HD.
To our knowledge, this is the first study that has evaluated and compared the elements of neurocognitive training in children with AD/HD with SCED. In the course of this study, it became apparent that this design has some practical advantages as well as limitations for evaluating neurocognitive training. SCED has an advantage over traditional group designs in requiring a smaller sample size, which is helpful when large groups of participants with a specific type/severity of a disorder are difficult to recruit. In this study, a substantial number of participants would need to be recruited if adopting a group design (three experimental groups plus one control group). However, it should also be noted that SCED requiring repetitive testing sessions is more demanding for each participant. As a result, children may struggle with the volume of training and observation sessions, which may affect both the training benefits and evaluation results.
There were also other limitations to this study. The minimum number of observations points in the baseline phase was 3, which meets the WWC research standards but with reservations (WWC, 2014). The WWC recommends collecting at least 5 points to describe a phase. Moreover, the study determined whether there was an effect by the ‘three demonstrations’ criterion (Kratochwill et al., Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2013). However, the criterion is just based on the convention in SCED research in which three replications represent a higher likelihood of causal effects (Horner, Swaminathan, Sugai, & Smolkowski, Reference Horner, Swaminathan, Sugai and Smolkowski2012), and there is no formal expression of the criterion (Kratochwill et al., Reference Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf and Shadish2013). For example, although the minimum number of attempts to demonstrate effects are mentioned, the maximum is seldom discussed. Moreover, there was no female participant in the CT and NFT groups, which may limit the generalisation of the current findings. Also, although the questionnaires used in this study have good psychometric properties, the behaviour was assessed weekly, and this differs from conventional usage. For example, BRIEF is often assessed based on the past 6-months of behaviour. The reliability and validity of weekly measures are uncertain. Furthermore, there was no follow-up test for the training effects, so it is uncertain how long the obtained effects may last.
Conclusion
The current study provides preliminary results of comparing three non-pharmacological and home-based training protocols in terms of transfer effects in children with AD/HD. Based on Tau-U analysis and visual inspection, the results indicated that the CT and NFT showed limited transfer effects on EFs. The Combined condition did not demonstrate any transfer effect. More importantly, no effects on AD/HD symptoms were found for those interventions. Together, these preliminary results do not support the application of CT and NFT on their own as the treatment for children with AD/HD. Practical and theoretical suggestions were made for further examining the training effects and optimising training protocols.
Funding
This work was supported by the Capital Characteristic Clinic Project (Z171100001017089); the Natural Science Foundation of Jiangsu Province (BK20210816); the National Natural Sciences Foundation of China (81971284, 81771479); and the Key scientific research projects of capital health development (2020-1-4111).
Conflict of interest
The authors declare that they have no conflict of interest.
