INTRODUCTION
Executive functions are a set of inter-related abilities that facilitate purposeful, goal-oriented behavior (Lezak, Reference Lezak1995). These abilities emerge early in life and continue to develop until mid to late adolescence or early adulthood (Romine & Reynolds, Reference Romine and Reynolds2005). Executive functions play an important role in the development of other abilities during this period, including learning and memory skills (Bjorklund & Douglas, Reference Bjorklund, Douglas and Cowan1997; Schlagmüller & Schneider, Reference Schlagmüller and Schneider2002), reading and math proficiency (McClelland, Cameron, Connor, Farris, Jewkes, & Morrison, Reference McClelland, Cameron, Connor, Farris, Jewkes and Morrison2007; St. Clair-Thompson & Gathercole, Reference St. Clair-Thompson and Gathercole2006), social-emotional competence (Riggs, Jahromi, Razza, Dillworth-Bart, & Mueller, Reference Riggs, Jahromi, Razza, Dillworth-Bart and Mueller2006), and level of adaptive functioning (Blair & Peters, Reference Blair and Peters2003). Impairments in executive functions are also considered to be a core feature of several developmental disorders, including autism (Russo, Flanagan, Iarocci, Berringer, Zelazo, & Burack, Reference Russo, Flanagan, Iarocci, Berringer, Zelazo and Burack2007) and Attention-Deficit/Hyperactivity Disorder (ADHD: Barkley, Reference Barkley1997; Willcutt, Doyle, Nigg, Faraone, & Pennington, Reference Willcutt, Doyle, Nigg, Faraone and Pennington2005). Given the involvement of executive functions in both typical and atypical development, they have become the focus of considerable clinical interest and empirical study.
The Behavior Rating Inventory of Executive Function (BRIEF) is a questionnaire that was developed to provide clinicians with a means of assessing the executive functions of youth in an ecologically valid manner (Gioia, Isquith, Guy, & Kenworthy, Reference Gioia, Isquith, Guy and Kenworthy2000a, Reference Gioia, Isquith, Guy and Kenworthy2000b). The BRIEF is based on the premise that parents and teachers can provide useful information about the executive functions of youth by reporting on their behavior outside of the testing environment. An overall index of executive dysfunction is provided by the Global Executive Composite, which is comprised of two subordinate indices called the Behavioral Regulation Index and the Metacognition Index. The Behavioral Regulation Index is comprised of 3 scales, including Inhibit (e.g., delay or stop impulsive behaviors), Shift (e.g., change tasks and adapt to new situations) and Emotional Control (e.g., modulate mood appropriately). The Metacognition Index is comprised of 5 scales, including Initiate (e.g., generate ideas, start new tasks), Working Memory (e.g., sustain one’s focus, keep information in mind), Plan/Organize (e.g., think prospectively, follow a plan), Organization of Materials (e.g., clean-up after oneself), and Monitor (e.g., check one’s work for errors, monitor the effect of one’s behavior on other people).
In clinical settings, the BRIEF has been used to evaluate the executive functions of children and adolescents presenting with a wide range of concerns. Studies have shown that children diagnosed with ADHD have higher scores on many of the BRIEF scales compared with children who do not have this disorder (Mahone, Cirino, et al., Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002; Pratt, Campbell-LaVoie, Isquith, Gioia, & Suy, 2000, as cited in Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000b; Toplack, Bucciarelli, Jain, & Tannock, Reference Toplack, Bucciarelli, Jain and Tannock2009). Similar findings have been obtained from children born with extremely low birth weight (Taylor, 2000, as cited in Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000b), children with myelomeningocele and hydrocephalus (Mahone, Zabel, Levey, Verda, & Kinsman, Reference Mahone, Zabel, Levey, Verda and Kinsman2002), children diagnosed with autism spectrum disorders (Gilotty, Kenworthy, Sirian, Black, & Wagner, Reference Gilotty, Kenworthy, Sirian, Black and Wagner2002), and children who have experienced a focal brain lesion or severe traumatic brain injury (Anderson, Anderson, Northam, Jacobs, & Mikiewicz, Reference Anderson, Anderson, Northam, Jacobs and Mikiewicz2002; Conklin, Salorio, & Slomine, Reference Conklin, Salorio and Slomine2008; Mangeot, Armstrong, Colvin, Yeates, & Taylor, Reference Mangeot, Armstrong, Colvin, Yeates and Taylor2002).
The extent to which the BRIEF assesses executive dysfunction has been empirically examined using participants drawn from diverse clinical groups and spanning a relatively broad age range (Table 1). A general trend to have emerged from this literature is that the BRIEF is not typically associated with complex measures of executive function (e.g., Anderson et al., Reference Anderson, Anderson, Northam, Jacobs and Mikiewicz2002; Mahone, Cirino et al., Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002; Vriezen & Pigott, Reference Vriezen and Pigott2002), although there has been the occasional exception (e.g., Mangeot et al., Reference Mangeot, Armstrong, Colvin, Yeates and Taylor2002). In contrast, measures of executive function that are thought to tap more circumscribed abilities have yielded inconsistent findings, with some studies reporting a lack of association (e.g., Conklin et al., Reference Conklin, Salorio and Slomine2008; Niendam, Horwitz, Bearden, & Cannon, Reference Niendam, Horwitz, Bearden and Cannon2007) and other studies reporting associations that are small to moderate in magnitude (e.g., Toplack et al., Reference Toplack, Bucciarelli, Jain and Tannock2009). Although the BRIEF is sensitive to behavioral disruption and impairment, it is unclear whether the questionnaire is a measure of executive dysfunction per se.
Summary of studies examining associations between parent ratings on the BRIEF and performance-based tasks of executive function
Note
INH = Inhibition, SFT = Shift, EC = Emotional Control, INT - Initiate, WM = Working Memory, P/O = Plan/Organize, OM = Organization of Materials, MON = Monitor, BRI = Behavioral Regulation Index, MI = Metacognition Index, GEC = Global Executive Composite
a Examined associations with WM.
b Examined associations with INH, SFT, WM, P/O.
c Examined associations with INH, SFT, EC, INT, WM, P/O, OM, MON.
d Examined associations with BRI and MI.
e Examined associations with BRI, MI, GEC.
f Examined associations with INH, WM, BRI, MI, GEC.
In an effort to further elucidate the nature of the BRIEF, we examined associations between the Behavioral Regulation Index and Metacognition Index and a variety of cognitive, behavioral, and academic measures that were collected from a diverse sample of youth between 6 and 15 years of age. These measures included parent and teacher ratings of inattentive and hyperactive-impulsive symptoms in youth, parent and teacher ratings of behavioral and social-emotional problems experienced by youth, youth’s scores on measures of reading and math proficiency, and youth’s scores on measures of inhibition, performance monitoring, and working memory. Although these latter tasks provide a rather narrow conceptualization of the executive function construct, they were included in our study because they are widely recognized as core executive functions (Huizinga, Dolan, & van der Molen, Reference Huizinga, Dolan and van der Molen2006; Miyake, Friedman, Emerson, Witzki, & Howerter, Reference Miyake, Friedman, Emerson, Witzki and Howerter2000; Ridderinkhof, van den Wildenberg, Segalowitz, & Carter, Reference Ridderinkhof, van den Wildenberg, Segalowitz and Carter2004) and have been studied extensively in typically and atypically developing youth (for reviews see Pennington & Ozonoff, Reference Pennington and Ozonoff1996; Welsh, Reference Welsh, Molfese and Molfese2002).
Our predictions were informed by differing views of what the BRIEF measures. If the questionnaire is a general measure of behavioral disruption and impairment, then we would expect the Behavioral Regulation Index and Metacognition Index to be strongly associated with ratings of inattentive and hyperactive-impulsive symptoms in youth and with ratings of how well youth functioned in their everyday environments. We did not expect to see similar associations with the other measures that were administered. In contrast, if the questionnaire is a more specific measure of executive dysfunction, then we would expect the Behavioral Regulation Index and Metacognition Index to be strongly associated with youth’s scores on measures of inhibition, performance monitoring, and working memory. Because cognitive aspects of executive function are known to play an important role in the acquisition of academic skills (McClelland et al., Reference McClelland, Cameron, Connor, Farris, Jewkes and Morrison2007; St. Clair-Thompson & Gathercole, Reference St. Clair-Thompson and Gathercole2006), we would further expect the Metacognition Index to be strongly associated with youth’s proficiency in reading and math.
METHOD
Participants
Data from 97 participants 6 to 15 years of age were included in this study. Participants were drawn from an outpatient clinic in an urban pediatric hospital and included youth who were referred for attention, learning and/or behavioral problems (i.e., clinical group) and youth who were recruited to serve as healthy controls (i.e., control group). Information regarding mental health concerns was obtained from parents and teachers in semi-structured clinical interviews including the Parent Interview for Child Symptoms (Ickowicz, Schachar, Sugarman, Chen, Millette, & Cook, Reference Ickowicz, Schachar, Sugarman, Chen, Millette and Cook2006) and the Teacher Telephone Interview (Tannock, Hum, Masellis, Humphries, & Schachar, Reference Tannock, Hum, Masellis, Humphries and Schachar2002). Interviews were conducted by an individual with a Master’s degree who was trained in developmental psychopathology and who worked under the supervision of a registered clinician. Diagnoses were made by the clinician based on criteria in the Diagnostic and Statistics Manual of Mental Disorders, 4th Edition (American Psychiatric Association, 1994). Intellectual and academic testing was conducted by a trained psychometrist who also worked under the supervision of a registered clinician. Participants were excluded if they had a full scale IQ (Wechsler, Reference Wechsler1991, Reference Wechsler2003) of less than 70 or greater than 130 or invalid data on any of the measures that were used in the study. Demographic information is presented in Table 2.
Demographic variables
**p < .001, *p < .05
Procedure
A detailed account of our clinical research protocol has appeared elsewhere and will be reviewed only briefly (Schachar et al., Reference Schachar, Chen, Logan, Ornstein, Crosbie and Ickowicz2004). Informed consent was obtained from the participant and his or her parent/guardian by a trained health professional. Before the session, behavioral questionnaires were sent to the child’s home to be completed by the child’s parent/guardian and teacher. Completed questionnaires were returned to our clinic at the time of the appointment. During the session, each child worked individually with a trained psychology assistant. Academic and cognitive tasks were administered in randomized order. As the child worked, the parent/guardian completed a clinical interview in a separate room of the clinic. A clinical interview also was completed with the child’s teacher at a separate time over the phone. All youth participating in the study were free of medication 48 hr before the appointment. This protocol was approved by the Research Ethics Board at The Hospital for Sick Children and is consistent with the Helsinki Declaration.
Measures
Behavior Rating Inventory of Executive Function
The parent form of the BRIEF is an 86-item questionnaire that was developed to assess the executive functions of youth 5 to 18 years of age (Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000a). Each item loads onto one of eight scales. Three scales yield a summary measure called the Behavioral Regulation Index and five scales yield a summary measure called the Metacognition Index. These indices reflect the extent to which youth engage in behaviors that may be indicative of impairment in one or more aspects of executive function. Chronbach’s α for the Behavioral Regulation Index is .96 for clinic-referred youth and .94 for youth drawn from a normative sample. Chronbach’s α for the Metacognition Index is .96 for clinic-referred youth and .96 for youth drawn from a normative sample (Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000b).
Conners’ Rating Scales – Revised:Long
The CRS-R:L assesses symptoms of ADHD in youth 3 to 17 years of age (Conners, Reference Conners2001a, Reference Conners2001b). The parent form consists of 80 items and the teacher form consists of 59 items. Both forms of the questionnaire include an “L” scale reflecting symptoms of inattention (e.g., difficulty sustaining attention, being highly distractible) and an “M” scale reflecting symptoms of hyperactivity-impulsivity (e.g., restless, always on the go). Chronbach’s α for the L and M scales is .93 and .90 for males and .91 and .88 for females, respectively (Conners, Reference Conners2001c). There is significant overlap between items comprising the L and M scales of the CRS-R:L and items comprising the Inhibit and Working Memory scales of the BRIEF.
Ontario Child Health Study
The OCHS is a comprehensive questionnaire that assesses multiple aspects of child and adolescent health (Boyle et al., Reference Boyle, Offord, Hofman, Catlin, Byles and Cadman1987). Items from the parent and teacher forms of the questionnaire were used to evaluate impairment associated with behavioral and social-emotional problems experienced by youth (e.g., quality of relationships, engagement in activities, school truancy). The items we selected to examine impairment have been used in other studies for the same purpose (Lindsay, Offord, Boyle, & Racine, Reference Lindsay, Offord, Boyle and Racine1995; Sanford, Offord, Boyle, Peace, & Racine, Reference Sanford, Offord, Boyle, Peace and Racine1992). These items were summed to create two composite scores reflecting overall level of impairment as reported by parents and teachers, respectively. There was no overlap between the items we selected from the OCHS and items comprising the eight scales of the BRIEF or items comprising the L and M scales of the CRS-R:L.
Wide Range Achievement Test 3
The WRAT3 is a screen of academic proficiency that may be used across the life span (Wilkinson, Reference Wilkinson1993a). The Reading and Arithmetic subtests were administered in this study. Reading required participants to identify letters (if younger than 8 years) and/or read words aloud (if 8 years or older). Arithmetic required participants to answer orally presented words problems (if younger than 8 years) and/or solve written math problems (if 8 years or older). In people 6 to 15 years of age, Chronbach’s α is between .94 and .96 for Reading and between .88 and .94 for Arithmetic (Wilkinson, Reference Wilkinson1993b).
Stop-signal task
The stop-signal task is a well-established measure of inhibitory ability and performance monitoring (Lijffijt, Kenemans, Verbaten, & van Engeland, Reference Lijffijt, Kenemans, Verbaten and van Engeland2005; Oosterlaan, Logan, & Sergeant, Reference Oosterlaan, Logan and Sergeant1998; Schachar et al., Reference Schachar, Chen, Logan, Ornstein, Crosbie and Ickowicz2004). Our version of the stop signal task was presented in 4 blocks of 24 trials. At the beginning of each trial, a central fixation appeared for 500 ms. After the fixation disappeared, an X or an O appeared at the center of the computer screen for 1000 ms. Participants identified the letter by making a speeded key press response. After a blank intertrial interval of 2000 ms the next trial was presented. On 25% of trials, the appearance of the letter was followed by an auditory tone that signaled participants to inhibit their response. Timing of the signal was determined using a dynamic tracking algorithm (Logan & Cowan, Reference Logan and Cowan1994) such that participants were able to inhibit their response on approximately 50% of trials. Stop signal reaction time (SSRT) served as a measure of inhibition and was obtained by subtracting the mean delay of the stop signal from the mean time taken to respond to the letters. Post-error slowing (PES) served as a measure of performance monitoring and was obtained by subtracting the mean RT of correct trials following failed inhibit trials from the overall correct mean RT.
N-back task
The n-back task is a well-established measure of working memory (Owen, McMillan, Laird, & Bullmore, Reference Owen, McMillan, Laird and Bullmore2005). In our study, a spatial version of the 1-back task was presented in 4 blocks of 40 trials. At the beginning of each trial, a central fixation appeared for 500 ms. After the fixation disappeared, a white box appeared at a location on the computer screen for 1000 ms. Participants determined whether the location of the box on the present trial was the same as the location of the box 1 trial previous by making a speeded key press response. After a blank intertrial interval of 2000 ms the next trial was presented. Target accuracy (NBACC) served as a measure of working memory. Accuracy was inflected so that positive scores denoted worse performance, consistent with the other cognitive measures used in this study.
RESULTS
Bivariate Correlations
Raw scores were converted into T scores using appropriate age and gender norms (BRIEF, CRS:R-L, WRAT3) or were transformed into residual scores that were corrected for age (PES, NBACC), gender (OCHS teacher form), or age and gender (SSRT). Bivariate correlations were initially inspected to assess the overall pattern of results (Table 3). The Behavioral Regulation Index and Metacognition Index were significantly inter-correlated (r = .75), as were parent and teacher ratings of youth’s inattentiveness and hyperactivity-impulsivity (rs = .35 to .66), parent and teacher ratings of youth’s ability to function (r = .49), youth’s scores on measures of reading and math (r = .51), and youth’s scores on measures of inhibition, performance monitoring, and working memory (rs = .25 to .59). The two indices of the BRIEF were significantly correlated with parent and teacher ratings of ADHD symptoms (rs = .34 to .81), with parent and teacher ratings of impairment (rs = .32 to .63), and with youth’s level of academic proficiency (rs = −.23 to −.45). Although correlations between the two indices of the BRIEF and youth’s performance on measures of executive function were smaller in magnitude, a somewhat stronger relationship was observed with working memory (rs = .19 to .26) than with inhibition or performance monitoring (rs = <.01 to .12).
Correlations between the BRIEF and Behavioral, Cognitive, and Academic measures
**p < .01, *p < .05
Principal Components Analysis
To reduce the amount of data and facilitate interpretation of results, all measures (except the BRIEF) were entered into a principal components analysis with varimax rotation and Kaiser normalization. Principal components analysis produced a 3-factor solution that accounted for 64% of the variance in the data. Factor loadings and the proportion of variance explained by each factor are presented in Table 4. The first factor included parent and teacher ratings of inattention and hyperactivity-impulsivity in youth as well as parent and teacher perceptions of behavioral and social-emotional problems that were experienced by youth. This factor was interpreted as an index of behavioral disruption and impairment. The second factor included SSRT and PES from the stop-signal task and NBACC from the 1-back task. Because these variables reflect inhibition, performance monitoring, and working memory, this factor was interpreted as an index of executive function. The third factor included performance on tests of reading and arithmetic and was interpreted as an index of academic ability.
Factor loadings and explained variance from the principal components analysis
Regression Analyses
Hierarchical regression analyses were conducted to determine the proportion of variance in each BRIEF index that was explained by the three factors identified in the principal components analysis after possible confounds were statistically controlled. In each regression, either the Behavioral Regulation Index or Metacognition Index served as the dependent measure. Sex, group membership, and IQ were entered as independent measures in the first step and factors representing problematic behavior, executive function, and academic ability were entered as independent measures in the second step. The final step included all possible 2-way interactions between variables that were entered in the first and second steps. None of these interactions were significant in either analysis (ps > .10), indicating that the relationship between the 2 indices of the BRIEF and the 3 factors identified in the principal components analysis did not vary according to sex, group membership, or IQ.
As shown in Table 5, 27% of the variance in the Behavioral Regulation Index and 26% of the variance in the Metacognition Index was uniquely explained by the three factors identified in the principal components analysis after sex, group membership, and IQ were statistically controlled. Inspection of individual B weights revealed that the Behavioral Regulation Index and Metacognition Index were positively associated with parent and teacher ratings of behavioral disruption and impairment, indicating that youth who had higher ratings on the BRIEF were more likely to have higher ratings of inattentive and/or hyperactive-impulsive symptoms and greater difficulty functioning in their everyday environments. Because these findings may reflect a common reporting source (i.e., the parent), we re-ran the analyses using only teacher ratings of behavioural disruption and impairment. In so doing, the same pattern of associations was observed. Inspection of individual B weights further revealed that the Metacognition Index was negatively associated with youth’s scores on measures of academic proficiency, indicating that that youth who had higher ratings on this particular index were more likely to have difficulties in reading and math. Neither index of the BRIEF was significantly associated with youth’s scores on performance-based tasks of executive function (ps > .10). Of note, when each of the eight BRIEF scales were separately treated as a dependent measure in the analyses, Inhibit, Shift, and Emotional Control showed the same pattern of results as the Behavioral Regulation Index and Initiate, Working Memory, Plan/Organize, Organization of Materials, and Monitor showed the same pattern of results as the Metacognition Index.
Regression analyses examining the relationship between the BRIEF and factors representing problem behavior, executive function, and academic ability
**p < .001, *p < .05
DISCUSSION
This study was undertaken to further elucidate the nature of the BRIEF. To address this aim, we examined associations between the BRIEF and a variety of cognitive, behavioral, and academic measures in a sample of youth who were drawn from an outpatient mental health clinic or who were recruited to serve as healthy controls. Measures were selected to represent a broad range of domains and included parent and teacher ratings of inattentive and hyperactive-impulsive symptoms in youth, parent and teacher ratings of behavioral and social-emotional problems experienced by youth, youth’s scores on measures of reading and math proficiency, and youth’s performance on the Stop Signal and 1-back tasks. These latter tasks tap circumscribed aspects of the executive function construct and, to our knowledge, have not previously been used in conjunction with the BRIEF.
A robust finding was that the Behavioral Regulation Index and Metacognition Index were strongly associated with parent and teacher ratings of attention problems in youth and with behavioral and social-emotional problems experienced by youth in their everyday lives. Although this finding is consistent with research suggesting that the BRIEF is sensitive to behavioral disruption and impairment, limitations of our design precluded us from determining whether the strength of these associations reflected variance attributable to a common underlying trait (i.e., trait variance) or variance that was attributable to the use of a common method (i.e., method variance). One approach that has been used to resolve this issue is the multitrait-multimethod design, in which two or more traits are assessed using two or more unique methods (for a more detailed discussion see Marsh & Grayson, Reference Marsh, Grayson and Hoyle1995). Although this approach requires intensive resources, it may provide a fruitful avenue for future explorations of the BRIEF.
The aforementioned finding may cause one to speculate that the BRIEF is a measure of ADHD—a common disorder of childhood that is characterized by six or more symptoms of inattention and/or hyperactivity-impulsivity and associated impairment across multiple settings. Correlations between the BRIEF and ADHD questionnaires are typically moderate to large in magnitude, as are correlations between the BRIEF and ADHD scales on more general questionnaires of pathology and adaptive function (Burmeister, Hannay, Copeland, Fletcher, Boudousquie, & Dennis, Reference Burmeister, Hannay, Copeland, Fletcher, Boudousquie and Dennis2005; Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000b; Mahone, Cirino, et al., Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002; Sullivan & Riccio, Reference Sullivan and Riccio2006; Toplack et al., Reference Toplack, Bucciarelli, Jain and Tannock2009). Previous work has shown that children with ADHD have higher scores on the BRIEF compared with their unaffected peers (Mahone, Cirino, et al., Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002; Pratt et al. 2000, cited in Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000b; Toplack et al., Reference Toplack, Bucciarelli, Jain and Tannock2009), that the Working Memory scale is particularly sensitive to the diagnosis of ADHD (Isquith & Gioia, Reference Isquith and Gioia2000; McCandless & O’Laughlin, Reference McCandless and O’Laughlin2007), and that the Inhibit scale reliably differentiates between Inattentive and Combined subtypes of the disorder (Isquith & Gioia, Reference Isquith and Gioia2000; McCandless & O’Laughlin, Reference McCandless and O’Laughlin2007; Riccio, Homack, Pizzitola Jarratt, & Wolfe, Reference Riccio, Homack, Pizzitola Jarratt and Wolfe2006). More recent work also has shown that ratings on the Working Memory, Inhibit, Shift, and Plan/Organize scales are good predictors of ADHD status (Toplack et al., Reference Toplack, Bucciarelli, Jain and Tannock2009). This research may suggest that the BRIEF is primarily useful as a diagnostic tool for ADHD. However, other work has demonstrated that the BRIEF is elevated in youth presenting with a variety of issues (Anderson et al., Reference Anderson, Anderson, Northam, Jacobs and Mikiewicz2002; Gilotty et al., Reference Gilotty, Kenworthy, Sirian, Black and Wagner2002; Mahone, Zabel et al., Reference Mahone, Zabel, Levey, Verda and Kinsman2002; Mangeot et al., Reference Mangeot, Armstrong, Colvin, Yeates and Taylor2002; Taylor, 2000, as cited in Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000b) and that it may show robust associations with behavioral disruption and impairment in unaffected youth who do not have ADHD or any other disorder (as demonstrated in our study). Although the BRIEF is sensitive to the symptoms and impairment that characterize ADHD, the scope of the BRIEF encompasses a broader range of concerns.
Another finding to emerge from our study was a strong association between the Metacognition Index and youth’s proficiency on measures of reading and math (see Waber, Gerber, Turcios, Forbes, & Wagner, Reference Waber, Gerber, Turcios, Forbes and Wagner2006, for similar results involving the teacher version of the BRIEF). These results are partly consistent with those of Mahone, Cirino, et al. (Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002), who found that the Behavioral Regulation Index and Metacognition Index were associated with youth’s scores on a composite of math, although not to youth’s scores on a composite of reading. In comparison to the relatively basic measures of academic performance that were used in our study, Mahone, Cirino, et al. (Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002) administered measures of single-word reading, reading comprehension, numerical calculations, and math reasoning to youth who were diagnosed with ADHD and/or Tourette Syndrome or who served as healthy controls. Although it is unclear why we failed to find an association between the Behavioral Regulation Index and math proficiency or why Mahone, Cirino, et al. (Reference Mahone, Cirino, Cutting, Cerrone, Hagelthorn and Hiemenz2002) failed to find an association between the Metacognition Index and reading proficiency, these discrepancies may reflect differences in the academic measures that were used in our studies and/or differences in the characteristics of our respective samples. It is likely that all scales of the BRIEF include items that are necessary for success at school; however, items reflecting self-regulatory skills (e.g., the Behavioral Regulation Index) and items reflecting metacognitive skills (e.g., the Metacognition Index) may be differentially sensitive to the kinds of school-related demands that are encountered by students of different ages (i.e., preschool vs. elementary vs. high school students). This is an issue that will be interesting to explore in future research.
Although the BRIEF scales have names that correspond to specific aspects of the executive function construct, we found no significant associations between the Behavioral Regulation Index or the Metacognition Index and youth’s scores on measures of inhibition, performance monitoring, and working memory. Similar null findings have been reported in other studies using different performance-based tasks of executive function (see Table 1). An illustrative example is provided by Vriezen and Pigott (Reference Vriezen and Pigott2002), who compared parent ratings on the BRIEF to children’s performance on the Wisconsin Card Sorting Test, Trail Making Test, and Verbal Fluency Test. All children had sustained moderate to severe traumatic brain injuries approximately 3 years before their enrollment in the study. Of these children, nearly a third had scores on the Behavioral Regulation Index, Metacognition Index, and Global Executive Composite that were in the clinically significant range (defined as a T score above 65). However, scores on these indices were not significantly correlated with children’s performance on any of the executive function measures (rs ranged from .03 to .26).
At present, reasons for the apparent dissociation between parent and teacher ratings on the BRIEF and youth’s scores on performance-based tasks of executive function are not well-understood. One set of interpretations is based on the premise that these measures assess different aspects of the same underlying construct. For example, it has been suggested that the executive function construct can be fractionated into a behavioral component that is assessed by the BRIEF and a cognitive component that is assessed by performance-based tasks (e.g., Anderson et al., Reference Anderson, Anderson, Northam, Jacobs and Mikiewicz2002). Although this has been offered as one possible explanation for null findings in the literature, it is inconsistent with recent neuroimaging findings suggesting that the two sets of measures share a common neuroanatomical substrate (Mahone, Martin, Kates, Hay, & Horska, Reference Mahone, Martin, Kates, Hay and Horska2009). An alternative, and perhaps more plausible explanation, is that performance-based tasks assess underlying skills whereas the BRIEF assesses the application of those skills at home and at school. It may be the case that environmental variables mediate this relationship, which would explain why youths’ scores on performance-based tasks do not necessarily correspond to parent and teacher ratings on the BRIEF. To our knowledge, potential mediators of this relationship have not yet been empirically examined.
Another interpretation is that performance-based tasks of executive function lack ecological validity due to the manner in which they are typically administered. Testing usually occurs in environments that are designed to minimize distractions, maximize support, and provide individuals with a high degree of structure (e.g., clear instructions, well-specified goals). Because these conditions bear little resemblance to the ones in which we typically function, it has been suggested that performance-based tasks do not engage the same set of skills that are required in naturalistic settings (Burgess, Reference Burgess and Rabbitt1997).
In contrast to this view, recent work in the education literature has demonstrated that youth’s scores on performance-based tasks of executive function are related to their experiences in at least one naturalistic setting—that of the school. Youth’s scores on performance-based tasks have been shown to predict proficiency in specific academic skills (McClelland et al., Reference McClelland, Cameron, Connor, Farris, Jewkes and Morrison2007; St. Clair-Thompson & Gathercole, Reference St. Clair-Thompson and Gathercole2006), achievement on national curriculum assessments (Gathercole & Pickering, Reference Gathercole and Pickering2000; Jarvis & Gathercole, Reference Jarvis and Gathercole2003), risk of grade retention (Biederman et al., Reference Biederman, Monuteaux, Doyle, Seidman, Wilens and Ferrero2004), and teacher perceptions of student function (Diamantopoulou, Rydell, Thorell, & Bohlin, Reference Diamantopoulou, Rydell, Thorell and Bohlin2007). These findings provide preliminary support for the ecological validity of performance-based tasks of executive function. In future studies, it will be important to examine associations between these tasks and a greater variety of real-world outcomes.
A final interpretation is that the BRIEF does not measure executive functions to the extent that is commonly believed. As has already been mentioned, numerous studies have failed to find associations between parent and teacher ratings on the BRIEF and performance-based tasks of executive function (see, for example, Table 1). Many of these tasks were developed to assess specific facets of the executive function construct, were validated with brain lesioned patients (e.g., Drewe, Reference Drewe1974; Shallice, Reference Shallice1982), and have neuroanatomical substrates that have since been well-specified (e.g., Newman, Carpenter, Varma, & Just, Reference Newman, Carpenter, Varma and Just2003; Smith, Taylor, Brammer, & Rubia, Reference Smith, Taylor, Brammer and Rubia2004). In future studies, it will be important to verify the validity of the BRIEF by comparing the questionnaire to naturalistic tasks that require the application of executive functions in more complex contexts.
Although our study raises questions about the relation of the BRIEF to performance-based tasks that are commonly used to assess executive functions, it supports the use of the BRIEF as a clinical tool for assessing a broad range of concerns. In clinical settings, the BRIEF may be used to identify youth who are experiencing behavioral difficulties and who may be at increased risk for the development of social and school-related problems. When used in conjunction with other assessment tools, the BRIEF can help to further delineate the nature of difficulties that are experienced by youth and inform decisions regarding psychological intervention and educational planning.
ACKNOWLEDGMENTS
We thank Catherine Munns for assistance with the literature review, Troy Climans, Jonathan Lipszyc, Shalaine Payne, and Mehereen Wadiwalla for their input regarding this manuscript, and the families whose time and effort made this project possible. This project was supported by the Canadian Institute for Health Research (CIHR) grants 64277/44070. The authors have no conflict of interest to disclose.
