INTRODUCTION
Children recovering from concussion face a range of problems that can negatively impact school learning and performance during recovery (Halstead et al., Reference Halstead, McAvoy, Devore, Carl, Lee and Logan2013; Ransom et al., Reference Ransom, Vaughan, Pratson, Sady, McGill and Gioia2015). Concussion symptoms, such as headaches, fatigue, impaired concentration, and slowed processing speed, may temporarily impair the student’s ability to maintain pace in their academic curriculum (Sady, Vaughan, & Gioia, Reference Sady, Vaughan and Gioia2011). In turn, efforts to engage in cognitively challenging tasks while still symptomatic can exacerbate concussion symptoms and impede academic activities, including problems taking notes, difficulty understanding new material, and requiring more time for assignments (Sady et al., Reference Sady, Vaughan and Gioia2011). A multimodal assessment battery, including post-concussion symptoms and cognitive performance, has been recommended as the most effective means to evaluate concussion (Echemendia et al., Reference Echemendia, Iverson, McCrea, Macciocchi, Gioia, Putukian and Comper2013; McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvořák, Echemendia and Turner2013). It is a logical extension that this approach may have value in identifying factors related to academic impairment for the student. Empirical evidence currently does not exist, however, to guide clinicians as to which tools may best detect and predict who is at greatest risk for adverse academic outcomes during recovery.
An evidence-based assessment (EBA) approach can address this question by examining the various sources of information gathered in the clinical setting (e.g., personal risk factors and validated assessment measures) to derive the best set of predictors of academic outcomes (Youngstrom, Reference Youngstrom2013). The evidence-based practice movement has been a cornerstone of modern healthcare since it was introduced in 1992 (Evidence-Based Medicine Working Group, 1992), although its application to clinical neuropsychology is still in the early stages (Chelune, Reference Chelune2010). By integrating empirical research with clinical expertise, evidence-based standards aim to maximize prediction of outcomes and improve quality of life (Chelune, Reference Chelune2010; Straus, Glasziou, Richardson, & Haynes, Reference Straus, Glasziou, Richardson and Haynes2011). EBA methodology is advantageous in clinical settings as it can streamline assessments to answer specific clinical questions applied to a given patient, while reducing cost and burden of extensive evaluations. Examining a set of clinical factors that are predictive of current status or current treatment needs allows for a rapid, evidence-based, decision to occur at the time of the appointment. This has a methodological advantage over prediction of a delayed outcome, in that outcome status can change over time or become influenced by other factors not yet measured.
In the evaluation of a child with a concussion, specific tools can be administered to answer the question, “Is this student at a high risk for problems with academics?” The EBA can inform clinicians’ school-related treatment planning during concussion recovery by refining the predictive capability of known risk factors, clinical measures, or functional impairment to the likelihood of post-injury school problems. Relying on predefined probabilities to determine relative risk for specific outcomes, high versus low probability of academic problems, allows the clinician to target individualized treatments, fostering a personalized approach to intervention (Straus et al., Reference Straus, Glasziou, Richardson and Haynes2011; Youngstrom, Reference Youngstrom2014). Children with concussions who are determined to be at a higher risk of academic problems could receive additional guidance and support to reduce the chance of more persisting adverse outcomes (Gioia, Reference Gioia2014).
In defining the post-concussion risk for perceived academic problems, we applied the following six EBA procedures, outlined by Youngstrom (Reference Youngstrom2014), Chelune (Reference Chelune2010), Straus et al. (Reference Straus, Glasziou, Richardson and Haynes2011), and Hunsley and Mash (Reference Hunsley and Mash2007) with responses relevant to the current study in italics:
-
1. Evaluate presenting problems and complaints and convert these into answerable questions by defining a measurable outcome variable. In this study, we answer the question as to which measures are most useful for determining who is at risk for perceived post-injury academic problems. We first define our outcome variable (high versus low self-reported academic problems), which in the absence of a “gold standard” can be done using latent class analysis modeling. These procedures have been described elsewhere to establish group membership when a true “gold standard” is not defined for the variable of interest ( Pepe, Reference Pepe2004 ; Zhou, Obuchowski, & McClish, Reference Zhou, Obuchowski and McClish2002 ).
-
2. Obtain benchmarks from a sample that is representative of the clinical patients/setting for whom this information is intended to estimate the base rate of the clinical outcome in that particular setting. In this study, we estimate the base rate of high levels of perceived academic problems within a specialty concussion clinic.
-
3. Choose relevant test measures (or “index tests”) that are posited to generate meaningful data to answer the specific question. We examine common domains of functional disruption following concussion—symptom burden, executive dysfunction, cognitive exertion, and neuropsychological test performance—as predictors of perceived academic problems following a concussion.
-
4. Determine the utility of each index test by examining receiver operating characteristic (ROC) analyses and diagnostic efficiency statistics (i.e., the trade-off between sensitivity and specificity). We examine the predictive validity of measures of symptom severity, executive dysfunction, cognitive performance, and exertional response for the dichotomous outcome of high versus low self-reported academic problems.
-
5. Compare index tests to one another to identify the most powerful index test(s) for answering the question in a given setting by examining the relative predictive values of each measure. We use the Venkatraman method for comparing ROC curves ( Venkatraman & Begg, Reference Venkatraman and Begg1996 ) to directly compare the predictive value for measures of symptom severity, executive dysfunction, cognitive performance, and exertional responses.
-
6. Examine the unique contribution of key predictors above and beyond background variables (e.g., demographic variables) in predicting the outcome of interest. In this study, we explore sex, school grade, and pre-injury patient history separately and in combination with index tests to further refine our multimodal battery for predicting perceived academic problems.
This study aimed to demonstrate the utility of an EBA approach to establish a set of predictive measures to identify students at risk for perceived academic problems during concussion recovery. We hypothesized that each component of the battery: (1) would discriminate students reporting high and low levels of academic problems significantly better than chance (procedures 1 through 5), and (2) would provide significant incremental validity unique to each measure (procedure 6).
METHODS
Participants
The sample consisted of 142 children and adolescents (Age: M=14.95 years; SD=1.80; range, 11–18 years; male=59%) evaluated nonconsecutively in an outpatient concussion clinic within 4 weeks of injury (median=16 days; range, 7–28 days). Participants completed a clinical assessment battery to measure perceived academic problems, symptoms, executive functioning, exertional symptom response, and cognitive performance. Criteria for study inclusion were as follows: clinician confirmed diagnosis of concussion within the past 28 days using the Center for Disease Control’s published criteria (Center for Disease Control and Prevention & National Center for Injury Prevention and Control, 2015), 10 years of age or older to allow appropriate completion of self-report questionnaires due to reading/ language limitations, and the child had to have returned to school by the time of the evaluation.
A process of data screening was conducted within the clinical database to exclude ineligible cases (n=22). Specifically, patients who presented to the clinic but who did not meet criteria for a diagnosis of concussion were excluded. In addition, individuals with highly atypical symptom presentations that were not consistent with standard developmental (e.g., attention deficit/hyperactivity disorder ADHD; learning disability, LD) or other neurological or psychiatric diagnoses (e.g., migraine, depression, etc.) were not eligible for the study. These include children with clinical presentations complicated by a high degree of psychosocial stressors (e.g., parental divorce, litigation, or school avoidance). Finally, cases with intellectual disability, diagnosis of severe anxiety/depression, and autism, were excluded due to possible limitations in the ability to accurately respond to self-report measures.
To maximize the generalizability of our sample to other patients seen in specialty clinics, participants were intentionally included with symptoms ranging from none/mild symptoms (i.e., recovered or nearly so) to those reporting high symptom levels (presumably still actively concussed). To further increase clinical representativeness and generalizability, children with premorbid LDs, ADHD, or behavior disorders were also included in the sample. The presence of these premorbid difficulties was coded in the database using information from the clinicians’ interview as well as a parent-report history form that asked parents to indicate the presence of several pre-existing conditions. The majority of participants attended public school (65%), and parents reported average to above average grades before injury (86% reported their child primarily earned As/Bs).
Measures
Outcome measure
The Concussion Learning Assessment and School Survey (CLASS) is a questionnaire completed by children and adolescents (ages, 10 to 1 years) to assess the perception of academic functioning following concussion (Ransom et al., Reference Ransom, Vaughan, Pratson, Sady, McGill and Gioia2015). The CLASS has been found to help identify students who have not yet recovered from concussion by showing higher levels of concern, more self-reported post-injury academic problems, and greater difficulty in classes than their recovered peers (Ransom et al., Reference Ransom, Vaughan, Pratson, Sady, McGill and Gioia2015). Furthermore, the total number of self-reported academic problems is positively correlated with ratings of symptom severity across self- and parent-report (Ransom et al., Reference Ransom, Vaughan, Pratson, Sady, McGill and Gioia2015).
The outcome measure of interest in this study was a specific question on the CLASS, in which participants were asked to indicate which problems were present that were either new or had worsened post-injury (“check all that apply”). This question assesses seven possible problems (listed in parentheses) that were grouped in two broad categories: (1) problems associated with interfering symptoms (i.e., headaches, fatigue, problems concentrating), and (2) problems associated with diminished academic skills (i.e., difficulty taking notes, difficulty understanding class material, spending more time on homework, problems studying).
Index tests
The Post-Concussion Symptom Inventory (PCSI; Sady, Vaughan, & Gioia, Reference Sady, Vaughan and Gioia2014) was used as a measure of self and parent-reported symptom severity. The 17-item child version of the PCSI (3 response choices) was administered to 18 students under the age of 12 and the 21-item adolescent version was administered to 124 students aged 13–18 years (7 response choices). Adjusted post-injury score totals were computed by subtracting retrospective baseline (RBL) ratings of pre-injury symptom report from post-injury report. Mean score ratings were calculated between the child and adolescent versions.
To calculate a comparable metric across all versions of the PCSI given the different number of items and response categories, the child version mean scores were multiplied by two. We chose this multiplier because it resulted in a distribution of scores that was very similar (no significant differences in mean, standard deviation, or range) in those who completed the children’s version of the PCSI compared to those who were over 13 years of age and completed the adolescent version. In addition, we ran all of our ROC and logistic regression analyses without these 18 participants, and there were no substantive differences in the results, other than a loss of power that made one of the interactions non-significant.
A modified version of the Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Kenworthy, & Guy, Reference Gioia, Isquith, Guy and Kenworthy2000) was used as a parent- and self-report measure of executive functions. The modified BRIEF includes 34 and 35 items on the parent- and self-report versions, respectively, using a 5-point response scale. Three symptom validity (low endorsement items) are embedded in the measure to detect possible over-reporting and endorsement of unrealistic symptoms (i.e., forgets where bedroom is located, cannot remember friends’ names, has difficulty chewing food). This modified version was found to produce factors along three dimensions; Emotional (i.e., Emotional Control), Behavioral Regulation (i.e., Inhibition), and Cognitive Regulation (i.e., Working Memory, Planning/Organization, and Task Completion; McGill, Gerst, Isquith, & Gioia, Reference McGill, Gerst, Isquith and Gioia2011). These factors have been shown to reliably measure change in behavioral manifestations of executive functioning following concussion (McGill et al., Reference McGill, Gerst, Isquith and Gioia2011). As with the PCSI, children and their parents provided RBL ratings of pre-injury functioning on each item together with ratings of current post-injury functioning. To calculate the post-injury change in functioning, raw score totals were adjusted by subtracting RBL ratings from post-injury ratings.
The Children’s Exertional Effects Rating Scale (ChEERS) is a rating scale designed to capture dynamic change in symptom severity following cognitive activity. The ChEERS is completed by the patient before and after a session of neuropsychological testing (Sady, McGill, Gerst, & Gioia, Reference Sady, McGill, Gerst and Gioia2013). Four symptoms were measured, including headache, fatigue, concentration problems, and irritability. Each symptom was presented with a scale from 0 (not experiencing symptom) to 10 (severe). Children ages 12 and younger were provided with cartoon faces to provide additional context for response options. The Exertional Effects Index (EEI) was calculated by subtracting the pre-test ratings from the post-testing ratings, and negative EEI scores (higher pre-test than post-test) were set to zero for analysis.
Participants underwent a battery of computerized cognitive tests. Fifteen children ages 11 to 12 completed the Multimodal Assessment of Cognition & Symptoms for Children (MACS; Newman, Reesman, Vaughan, & Gioia, Reference Newman, Reesman, Vaughan and Gioia2013) and 127 participants ages 12 years, 6 months to 18 completed the Immediate Post Concussion Assessment and Testing (ImPACT; Lovell, Collins, Podell, Powell, & Maroon, Reference Lovell, Collins, Podell, Powell and Maroon2000). The MACS generates two composite standard scores, Memory and Speed. On ImPACT, two similar composite scores were calculated from the four ImPACT subcomposite scores by transforming the scores from percentiles to standard scores and then averaging the two memory and two speed measures (Schatz & Maerlender, Reference Schatz and Maerlender2013).
Procedure
Parents provided demographic information, including the child’s age, level of schooling, gender, race, ethnicity, and the presence of specific pre-injury mental health and medical diagnoses (e.g., anxiety, depression, attention deficit/hyperactivity disorder, LD, and headache/migraines; see Table 1). Children and their parents completed rating scales, including the outcome data from the CLASS, and performed cognitive testing described above at the time of their clinical evaluation. Since information was gathered in the clinic setting, the order of administration (e.g., questionnaires or testing first) varied. Exertion ratings were standardly collected at the start and end of the combined testing/questionnaire session. The contemporaneous administration of the predictor and criterion tests is generally considered desirable, in that having a short lag reduces the amount that the target condition is likely to change (in this case, number of self-reported academic problems) relative to current symptom or cognitive status (Bossuyt et al., Reference Bossuyt, Reitsma, Bruns, Gatsonis, Glasziou, Irwig and de Vet2003).
Descriptive statistics for clinical and demographic variables and bivariate tests of association with perceived school problems status at visit 1 (N=142)
While it would be valuable to know how a current assessment may predict future outcomes such as recovery, identifying current status and to predict current treatment needs is similarly useful (Findling et al., Reference Findling, Jo, Frazier, Youngstrom, Demeter, Fristad and Horwitz2013; Kraemer, Reference Kraemer1992). Data were collected with Institutional Review Board approval (Children’s National Health System), with informed consent obtained from parents and assent from children and adolescents. The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Data Analysis
Data analyses were performed with SPSS statistical software version 22 (IBM SPSS Statistics, IBM Corporation), Latent Gold 5.0 (Statistical Innovations, Inc.; Vermunt & Magidson, Reference Vermunt and Magidson2005), and R 3.2 (R Core Team, 2013). We first engaged in detailed data examination and cleaning. We examined distributions on all continuous measures (e.g., PCSI, modified BRIEF monitor) and eliminated individuals whose scores exceeded three standard deviations from the mean, to avoid undue influence of statistical outliers on the results.
We also used several methods to screen for unusual or excessive reporting on rating scales. On the PCSI, we examined previously collected samples, both uninjured and injured (see Sady et al., Reference Sady, Vaughan and Gioia2014), for symptoms that were rarely endorsed at a high level. In these samples, less than 5% of both groups endorsed more than five rare symptoms in the extreme (a rating of 2 on the 8- to 12-year-old PCSI, or a rating of 5–6 on the 13- to 18-year-old PCSI). In our sample, only one individual endorsed five rare symptoms at a high level. Since this individual did not endorse an overall extreme level of symptoms and they were not identified as a multivariate outlier in subsequent analyses, they were retained.
On the modified BRIEF-Monitor, we examined the three symptom validity items that were included to screen for possible over-reporting and endorsement of unrealistic symptoms. Two children and two parents endorsed two of three items, and one child and one parent endorsed three of three; none of these comprised a parent-child pair. Examination of the outcome variables for the elevated cases found that the number of reported school problems was within 1 SD of the mean, so these individuals were also not excluded.
Finally, we examined the influence of time since injury on our variables, given that the range was from 2 to 28 days. The two outcome groups differed significantly, with the high perceived problems group being on average 2.5 days longer post-injury (see Table 1). However, when added to the models, days since injury did not predict group membership after accounting for self-report questionnaires, so we did not include it in the final set of analyses. We also evaluated the predictors with respect to time since injury, as we wanted to determine whether there was any evidence of systematic recall bias on preinjury ratings (which were used to compute adjusted ratings). There were no significant correlations between number of days since injury and mean retrospective ratings on any of the self-report or parent report measures (r<|.05|; p>.56 for all four).
Following data screening, we applied the following set of statistical procedures to achieve our aims, corresponding to the six EBA procedures identified above. First, to establish a distinct grouping outcome variable for self-reported academic problems on the CLASS, Latent Class Analysis (LCA) identified a two-class solution based on the distribution of the responses in the study sample, following standard procedures for a “missing gold standard” scenario (Hagenaars & McCutcheon, Reference Hagenaars and McCutcheon2002; Zhou et al., Reference Zhou, Obuchowski and McClish2002). Individuals within a given latent class category are, therefore, similar to one another with respect to their responses on the CLASS (i.e., high vs. low self-reported academic problems). The LCA is a latent variable model, essentially a factor analysis with a categorical factor, and class membership is assigned probabilistically based on the constellation of observed responses. Independent sample t tests quantified criterion validity, or the association of each predictor to the LCA category. Because LCA is a latent variable approach, there were not set classification thresholds on each observed indicator. For example, moderate scores on several items could result in classification in the “high problems” group, but so could severe scores on a smaller set of items.
For the second and third steps, we examined the frequency of high and low self-reported academic problems to establish the base rate within our specialty concussion clinic, and we evaluated the relationship between each index test and the outcome to assess relevance.
Fourth, ROC analyses were conducted to evaluate the trade-off between diagnostic specificity and sensitivity for each of the index tests in distinguishing students reporting high and low academic problems. The area under the curve (AUC) quantified the accuracy of the measure in predicting group membership. Chance levels of diagnostic performance correspond to an AUC of .50, while an AUC of 1.0 reflects perfect performance. Benchmarks provided by Rice and Harris (Reference Rice and Harris2005) indicate that AUCs in the mid .50s are small and not clinically useful; AUCs in the mid .60s are medium in size and may provide some incremental clinical value, but are not sufficient in isolation; AUCs in the low to mid .70s reflect large, clinically informative values; AUCs in the low .80s are excellent; AUCs in the high .80s are exceptional under clinically rigorous designs and are often highly informative; while AUCs in the .90s are extraordinary. In practice, Youngstrom (Reference Youngstrom2014) highlighted that rating scales producing AUCs greater than .90 are more likely to indicate design flaws rather than a high level of discriminative validity (e.g., Youngstrom, Meyers, Youngstrom, Calabrese, & Findling, 2006).
Fifth, significance tests (Venkatraman & Begg, Reference Venkatraman and Begg1996) were conducted to directly compare index tests in their ability to discriminate individuals reporting high and low academic problems.
Sixth, a hierarchical logistic regression was used to test the incremental validity of the index tests at discriminating outcome groups, after accounting for the effects of demographic and pre-injury history. Additionally, logistic regression models explored whether the utility of particular tests depends upon other factors (i.e., potential demographic/background moderators). Demographic and pre-injury history variables, as well as self- and parent-report of symptoms and executive dysfunction (PCSI and the modified BRIEF), cognitive performance measures, and ratings of exertional response (EEI), and the interactions between demographic and pre-injury history and all other predictors were entered hierarchically into a logistic regression model. Continuous variables were mean-centered to reduce multicollinearity and improve interpretability of interaction terms. The order of entry for variables in the regression model was chosen to examine the incremental predictive validity of assessment measures with increasing examiner and patient burden (i.e., demographics, questionnaires, cognitive testing/EEI).
By using this order of entry, a more burdensome (time- and cost-consuming) measure would only be recommended for clinical use in identifying risk of self-reported academic problems if it showed incremental predictive power above less burdensome measures. Pairwise interactions between demographics and questionnaires, cognitive testing, and the EEI (21 total) were included to assess whether the discriminative power is significantly better amongst a particular subsample (e.g., whether the PCSI was equally beneficial for both genders). All nonsignificant interactions were iteratively eliminated from the final regression model to conserve power and improve precision of estimates. Two significant interactions were retained within the final model.
RESULTS
The following results were obtained within the aforementioned six procedures.
-
1. Evaluate presenting problems and complaints and convert these into answerable questions by defining a measureable outcome. Latent class analysis was conducted on the patient reported number of academic problem on the CLASS to establish two criterion groups, high and low perceived academic problems. Table 1 presents the study demographics, pre-injury characteristics, and clinical measures, comparing latent class groups with t tests. Significant differences were found between groups on self/parent-reported pre-injury history of ADHD/LD, anxiety, depression, and/or headache/migraine, in addition to self- and parent-report of symptoms and executive dysfunction (PCSI and the modified BRIEF) and ratings of exertional response (EEI). No differences in demographics, aspects of pre-injury history, or cognitive performance measures were found between the two latent class criterion groups.
-
2. Obtain benchmarks from a sample that is representative of the clinical patients/setting for whom this information is intended, in order to estimate the base rate of the clinical outcome in that particular setting. In our sample of adolescents who presented to a specialty concussion clinic, 79 (56%) were categorized as reporting a “low” number of academic problems (M=1.61; SD=1.10) and 63 (44%) were categorized as reporting a “high” number of perceived academic problems (M=5.29; SD=1.25) based on patient report on the CLASS.
-
3. Choose relevant test measures (or “index tests”) that are posited to generate meaningful data to answer the specific questions. To examine the pairwise relation between the criterion variable and each of the clinical predictors, we examined chi square analyses, analyses of variance, point biserial, and Pearson correlations (Tables 1 and 2). Report of high academic problems was associated with pre-injury history (i.e., history of ADHD/LD, depression, anxiety, or headache/migraine) in addition to greater overall post-concussion symptom severity, executive problems, and exertional response, but was not associated with cognitive performance measures. We also ran a supplemental confirmatory factor analysis to check that the measures covered distinct constructs, rather than a single method variance factor. A single factor solution, consistent with the shared method hypothesis, fit badly: chi-squared (2 df)=43.68; p<.000005; comparative fit index (CFI)=.905 Tucker Lewis index (TLI)=.716, whereas a two-factor model for the distinct constructs provides more acceptable fit, chi-squared (1 df)=16.28; p<.00005; CFI=.965; TLI=.792. Overall, the findings provided preliminary evidence that the majority of selected index tests are relevant to the prediction of self-reported academic problems.
-
4. Determine the utility of each index test by examining diagnostic efficiency statistics (i.e., the trade-off between sensitivity and specificity) via ROC analyses: Table 3 and Figure 1 present the ROC curves for the eight clinical measures. According to Rice and Harris’ criteria (2005), self-reported ratings on both the PCSI and the modified BRIEF demonstrated excellent accuracy in distinguishing participants reporting high academic problems from those reporting low academic problems, earning AUCs of 0.80 (p<.001) and 0.84 (p<.001), respectively. These findings indicate that a randomly selected student reporting high academic problems scores higher than a student reporting low academic problems 80% of the time on the PCSI and 84% of the time on the modified BRIEF. Parent report on the PCSI and the modified BRIEF demonstrated large, clinically informative accuracy, with AUCs of .79 (p<.001) and .74 (p<.001), respectively. In contrast, performance on cognitive measures did not significantly distinguish students reporting high and low academic problems (Processing Speed AUC=.56; Memory AUC=.57, ns), although the EEI measuring the exertional response resulting from this cognitive effort demonstrated a large, clinically informative AUC of .70 (p<.001).
-
5. Compare index tests to one another to identify the most powerful index test(s) for answering the question in a given setting by examining the relative predictive values of each measure. Venkatraman tests (Venkatraman & Begg, Reference Venkatraman and Begg1996) were used to directly compare the index tests with clinically informative AUCs to one other to examine significant differences in their ability to discriminate among students reporting high and low academic problems. Of all possible comparisons (eight total), the only significant difference was observed between ratings on the modified BRIEF, with self-report identifying students reporting high academic problems significantly better than parent-report (p<.03). However, this finding was not significant after correcting for multiple comparisons.
-
6. Examine the unique contribution of the key predictors as well as background variables (e.g., demographic variables) in predicting perceived academic problems. First, Pearson and point biserial correlations quantified the relation among predictors to check the degree of multicollinearity (Table 2). To examine the second study hypothesis, hierarchical logistic regression assessed the incremental utility of index tests to predict group membership of low versus high levels of self-reported academic problems. Nonsignificant interactions were trimmed from the final model.
ROC curves for the eight clinical measures.
Correlations among variables
a Pre-injury history includes diagnoses of ADHD, learning disability, anxiety, depression, or personal history of headaches/migraines.
b Coded such that low academic problems=0 and high academic problems=1.
c Standard Score; all others are raw scores, adjusted for retrospective ratings of pre-injury functioning.
d Phi Coefficient.
e Point-biserial correlation; all others are Pearson r correlations.
*p<.05, **p<.01, ***p<.001, two-tailed.
AUC from ROC analyses identifying students reporting school problems at visit 1 with index tests and moderators
Note. Benchmarks provided by Rice and Harris (Reference Rice and Harris2005) indicate that AUCs in the mid .50s are small and not clinically useful; AUCs in the mid .60s are medium in size and may provide some incremental clinical value, but are not sufficient in isolation; AUCs in the low to mid .70s reflect large, clinically informative values; AUCs in the low .80s are excellent; AUCs in the high .80s are exceptional under clinically rigorous designs and are often highly informative; while AUCs in the .90s are extraordinary.
Results indicated that demographics and pre-injury history did not provide significant prediction when entered in the first block of the model (p=.07; see Table 4). Comparison of log-likelihood ratios for models with and without self-and parent-report measures showed statistically significant improvement with addition of self-report measures in the second block (χ 2 =54.43; p<.001), but not after addition of parent-report measures in the third block (χ 2 =2.22; p=.33). In the fourth block, there was a significant interaction between self-reported PCSI and level of schooling on the prediction of self-reported academic problems, and between self-reported executive dysfunction and pre-injury history on the prediction of self-reported academic problems (χ 2 =11.742; p=.003). Specifically, the discriminative power of self-reported PCSI scores is stronger for high school students compared with elementary/ middle school students (B=1.87; p=.03).
Logistic regression model identifying students reporting high levels of post-injury school problems
a Presence of pre-injury history of ADHD, LD, anxiety, depression, or history of headache/migraine.
b ImPACT and MACS Processing Speed and Memory composite standard scores. High scores indicate greater difficulty.
However, ROC analysis of elementary/middle school students alone showed that self-reported PCSI significantly predicted perceived academic problems, albeit not as strongly as high school students (AUC=.67; p=.04 and AUC=.88; p<.001, respectively). Furthermore, greater self-reported adjusted scores on the modified BRIEF significantly increased the likelihood of high self-reported academic problems, particularly in individuals with a pre-injury history of ADHD/LD, depression, anxiety, or headache/migraine (B=0.13; p=.02). However, self-reported BRIEF scores demonstrated equally significant discriminative power in individuals with and without pre-injury risk factors (AUC=.85; p<.001 for both groups). In blocks 5 and 6, addition of cognitive performance and exertional response did not explain a unique portion of the variance above and beyond demographics and self- and parent-reported measures (χ 2 =1.13; p=.77).
In sum, self-reported ratings of symptoms predicted perceived academic problems in all students, but did so more strongly for high school students, and self-reported executive dysfunction predicted high perceived academic problems for all individuals. Parent report of symptoms and executive dysfunction, performance on cognitive measures, and exertional response did not account for additional significant variance
DISCUSSION
Children and adolescents who sustain a concussion are at risk for perceived academic problems during and immediately following recovery, including such difficulties as understanding new concepts, headaches or fatigue interfering with productivity, and difficulty completing work on time (Ransom et al., Reference Ransom, Vaughan, Pratson, Sady, McGill and Gioia2015). Defining the most useful predictors of who is most likely to perceive academic problems would better guide the clinician in using an efficient and targeted evaluation to improve clinical decision-making and management. This study applied an EBA approach, using a set of six statistical procedures, to examine a multimodal set of assessment measures—including patient demographics, pre-injury history, parent- and self-report of symptoms and executive functions, cognitive performance measures, and exertional response—to identify those measures that best differentiate children at risk for perceived post-injury academic problems.
We posited that each component of the battery (1) would discriminate students reporting high and low levels of academic problems significantly better than chance, and (2) would provide significant incremental validity unique to each measure. We first defined the clinical outcome using Latent Class Analysis to categorize individuals into high and low self-reported academic problems and examined base rates within our specialty concussion clinic sample. Next, we used ROC analyses to examine the discriminative validity of the assessment battery’s components in identifying students reporting high and low levels of academic problems. We directly compared the utility of each measure to one another through a series of Venkatraman tests. Finally, we conducted a hierarchical logistic regression to explore the incremental validity of measures above and beyond other measures.
Results demonstrated partial support for the first study hypothesis, as we found significant discriminative validity of self- and parent-reported measures of symptom severity and executive functions, as well as the student’s exertional response during testing. In contrast, cognitive performance measures did not significantly differentiate students reporting high and low levels of academic problems. The second study hypothesis was partially supported as well, indicating that self-reported symptoms and executive dysfunction were the best predictors of perceived academic problems, although the strength of prediction for symptom status was dependent on level of schooling. Parent-report of symptoms or executive dysfunction and exertional response, although significant by themselves, did not provide significant incremental validity above and beyond self-report measures in predicting perceived academic outcome. Cognitive performance did not predict perceived academic problems above and beyond other measures.
Several implications can be drawn from this study. First, self-reported measures of symptom severity and executive dysfunction are the strongest predictors of perceived academic problems during concussion recovery. Although parent-reported measures and exertional response are valid indicators for identifying at-risk students, clinicians evaluating children with concussions might consider prioritizing self-report of post-concussion symptoms and executive dysfunction to reduce the cost and burden of an extensive evaluation. Second, this study highlights the methodological advantages of an EBA approach in identifying clinically relevant tools in answering important questions related to school outcomes.
EBA can provide a unique perspective for the field of neuropsychology. Refining research questions in the context of EBA (procedure 1) can yield specific, measurable outcomes that are well suited to clinical assessment. While this study explored relative risk for perceived post-concussion academic problems, such a model could easily be applied to a variety of other clinical questions relevant to children with concussions (e.g., risk for prolonged recovery) or other aspects of neuropsychology (e.g., risk for post-treatment effects in pediatric cancer survivors). To evaluate clinical outcomes without a clearly defined “gold standard” for identification, we demonstrate the use of latent class analysis to provide a meaningful outcome criterion (McCutcheon, Reference McCutcheon1987; Zhou et al., Reference Zhou, Obuchowski and McClish2002).
Evaluating the utility of neuropsychological assessment tools with an EBA approach (procedures 3 and 4) can define their applicability to important diagnostic and functional outcomes. ROC analysis can provide a clinically relevant measure of the strength of the relationship between a clinical predictor and the outcome of interest, perhaps more so than traditional effect sizes such as a bivariate correlation or Cohen’s d (Kraemer et al., Reference Kraemer, Kazdin, Offord, Kessler, Jensen and Kupfer1999), which are helpful for understanding group-level differences but do not address classification accuracy for individual cases.
The AUC statistic of the ROC analysis reflects the predictive validity of each assessment measure and makes a compelling case for their clinical utility in selecting a test battery. Examining clinical factors that predict a patient’s current status allows for an empirically based decision to occur at the time of the appointment which can be more advantageous than predicting a delayed outcome, as the outcome status can change over time or become influenced by other factors. In the evaluation of pediatric concussion, self- and parent-reported measures of symptoms and executive functions demonstrated significant predictive utility in identifying students reporting high academic problems, above and beyond performance on cognitive measures and would, therefore, be considered as primary indicators of this important outcome for a school-aged child.
Examining the incremental validity of the measures within the neuropsychological assessment (procedure 6) defines the “added value” of each component. By exploring the relative amount of variance explained by measures in a hierarchical model, a parsimonious set of clinical measures can be defined with empirical evidence to most efficiently answer the question at hand. This study highlighted the unique contribution of self-reported symptom measures (PCSI and modified BRIEF) to identify students at high risk for perceived adverse academic effects relative to the more limited information added by parent-reported measures.
Furthermore, more fine-grained examination of the interaction effects between level of schooling and self-report measures also showed the strength of symptom report in high school students. An important consideration regarding the EBA approach is that although some measures in the battery may not add a unique contribution to the prediction of an outcome, corroborating data may, at times, provide clinically meaningful information. For example, athletes sometimes underreport post-injury symptoms in an effort to prematurely return to competition; parent report may better elucidate the level of post-injury symptoms and/or executive dysfunction in the event of questionable self-report validity. Although the parent reports in this study did not provide significant incremental validity beyond the self-report, they nevertheless did demonstrate a strong relationship with the outcome variable, and could substitute as a surrogate predictive indicator if needed.
Limitations
Several limitations are evident in the present study. One limitation lies in the use of a self-reported criterion variable (e.g., self-reported post-injury school problems) without an independent, corroborating measure of school problems such as course grades or teacher report of performance. Although participants were asked to report new/worsened post-injury academic problems, recall bias or heightened anxiety/concern may have impacted perception of past or current difficulties. Recall bias is a systematic influence on ratings of behaviors or other phenomena that occurred in the past. In the field of concussion, a “good old days” bias has been reported in adults, such that ratings of preinjury functioning tend to be more positive than reality would suggest (Iverson, Lange, Brooks, & Rennison, Reference Iverson, Lange, Brooks and Rennison2010; Mittenberg, DiGiulio, Perrin, & Bass, Reference Mittenberg, DiGiulio, Perrin and Bass1992), but much of the research on this phenomenon relates to long post-injury intervals (many months to years).
Iverson et al. (Reference Iverson, Lange, Brooks and Rennison2010) found that the bias was most apparent in individuals who also failed multiple validity tests (not simply those involved in litigation, as previous research had suggested). Brooks et al. (Reference Brooks, Kadoura, Turley, Crawford, Mikrogianakis and Barlow2014) found that parents’ preinjury ratings declined over the first month post-injury, suggestive of the good old days bias in children with concussion; however, this longitudinal design introduces the possibility of the decrease being accounted for by the test–retest attenuation that is commonly seen with multiple administrations of rating scales (e.g., Gioia, Reference Gioia2015; Gioia et al., Reference Gioia, Isquith, Guy and Kenworthy2000). In fact, the screening analyses in the current cross-sectional design showed no association between time since injury and level of retrospective preinjury ratings. Similarly, days since injury was not a significant predictor of our outcome once more relevant variables (such as current symptom status) were entered into the models. Taken together, there was no evidence of systematic bias in recall over the first month post-injury in this study.
In addition, validity of self-report could not be confirmed in the absence of objective measures. There are multiple reasons participants could have reported a high level of academic problems, including individuals who were truly experiencing problems and those who rated a high number of problems due to anxiety, a feeling of helplessness or of being overwhelmed with missed work, or even exaggeration or feigning. While we screened our data to avoid including those with obvious symptom exaggeration, it is possible that some participants were not presenting their true status. It is possible that exertional effects may have generally influenced responses on child self-report measures, as highly exertional patients may not have responded as carefully to questions. We would not expect these effects to be specific to any one measure, however, as the order of test administration varied across examiners.
Method variance is a second limitation in that the strongest relationships were found among the self-report measures of symptoms, executive dysfunction, and academic problems. Supplemental analyses confirmed that a single, method variance factor fit these indicators poorly, allaying concern that results were driven by shared source variance and highlighting that results from this study are not driven by self-report measures predicting self-report measures. It also is reassuring that parent-reported measures also discriminated the classes, albeit not showing incremental validity after controlling for self-report, showing validity for a distinct source. Future research should examine academic outcomes across reporters, including self-report, parent-report, teacher report, and other external indicators of academic performance (e.g., test grades, report cards). Similarly, future studies can use this EBA model to examine the utility of a range of predictors, as results from this study are limited to the use of these specific measures (PCSI, modified BRIEF, ChEERS) and may not apply to other measures.
An inability to account for change in cognitive performance from pre-injury to post-injury may explain the limited utility of the cognitive performance measures. For other clinical predictors, a change score (i.e., the difference between reports of RBL and current symptoms/executive dysfunction) allowed for an examination of change in functioning compared to an individual’s own pre-injury level of problems, which may be more sensitive to perceived concussion related academic problems than a single post-injury test score. Further investigation of the link between cognitive performance and problems might be bolstered by the addition of a “hold” test as a control for preinjury performance (i.e., a reading or vocabulary test).
An additional limitation is the sole inclusion of students and their parents who sought evaluation and treatment in a specialty concussion clinic, possibly reflecting a more affected sample with a more prominent self/parent-reported pre-injury history of ADHD/LD, anxiety/mood concerns, and/or headaches/migraine. Finally, our analyses were limited to a clinical sample and did not include an uninjured contrast group. This was by design, however, as these results were meant to generalize best to concussion clinic settings, where the key comparison is not between those with or without a concussion, but rather among those with a concussion to identify who is at risk for perceived academic problems.
CONCLUSION
This study demonstrates a systematic, multi-procedure method highlighting the utility of an EBA model in establishing a multimodal toolset for identifying students at risk for self-reported post-injury academic problems. Students recovering from concussion are at risk for perceived academic problems during recovery, as predicted by measures of symptom severity, executive dysfunction, and exertional response. Application of an EBA model offers an important perspective in the management of pediatric concussion by defining the key tools that best predict this adverse outcome for children and adolescents. These findings inform clinicians about the measures that best predict perceived academic problems following concussion, allowing them to make an informed decision on the selection of research-validated assessment measures.
The relationships between self-reported symptoms/executive functioning problems and perceived school problems elucidated in this study provide a springboard for further investigation of several questions. For example, it will be important to look at when these perceived problems resolve in relation to recovery, how well perceived problems map onto more objective measures of decreased school performance, and how perceived problems relate to other important indicators of injury status. Additional research should also investigate the types of accommodations and strategies that are effective in alleviating, or at least minimizing long-term impact of, these difficulties during recovery.
These findings are a first step in defining the relationship between predictors of school-related outcomes, and an important next step is to provide metrics for clinical use, including what specific scores on the identified measures result in a “high risk” designation for any given individual. Follow-up work by this author team will provide guidance on the clinical application of the EBA model, articulating the at-risk scores on the PCSI, modified BRIEF, and ChEERS that would alert the clinician to take action to meet concurrent treatment needs and to identify those at risk for possible post-injury school problems.
ACKNOWLEDGMENTS
This work was supported by the Eunice Kennedy Shriver Intellectual and Developmental Disabilities Research Center (IDDRC; P30HD040677) and the Clinical and Translational Science Award (UL1TR000075). Conflict of interest. None.
