INTRODUCTION
Community members frequently present to hospital emergency departments (EDs) with mild traumatic brain injury (mTBI; Cassidy et al., Reference Cassidy, Carroll, Peloso, Borg, von Holst and Holm2004; Faul, Xu, Wald, & Coronado, Reference Faul, Xu, Wald and Coronado2010). In the United States, for example, approximately 2.5 million ED visits, hospitalizations, or deaths were associated with TBI in 2010 (over 80% of which are categorized as mTBI based on acute injury characteristics; Centers for Disease Control and Prevention, 2015). Despite the fact that most mTBIs occur in civilian, patient-based populations (Bazarian, McClung, Shah, et al., Reference Bazarian, McClung, Shah, Cheng, Flesher and Kraus2005; Centers for Disease Control and Prevention, 2015), arguably more scientific progress has been made to understand the effects of the injury, and improve its assessment and management, in the subset of mTBIs precipitated by sports participation (i.e., sport-related concussion).
One such advance has been the development of standardized assessment tools that facilitate the injury’s identification and clinical management. Computerized neurocognitive test (CNT) batteries, in particular, have been developed and are now widely used for the assessment of athletes with mTBI. Of the available CNT batteries (all of which emphasize neurocognitive assessment and in some cases include a symptom checklist), ImPACT (Immediate Post-Concussion and Cognitive Testing) is the most widely used and studied in sports settings, although other tools have been developed for athlete (e.g., Axon Sports), military (Automated Neuropsychological Assessment Metrics and Defense Automated Neurobehavioral Assessment, i.e., ANAM and DANA, respectively), and other populations (e.g., CNS Vital Signs).
Given that most mTBIs occur outside the athletic arena, it is important to determine whether the neurocognitive assessment tools developed for athletes provide clinical utility for the broader mTBI population. The ED setting is particularly challenging for diagnosing and treating mTBI—the majority of mTBI patients in this setting do not realize their symptoms are indicative of mTBI (Delaney, Abuzeyad, Correa, & Foxford, Reference Delaney, Abuzeyad, Correa and Foxford2005), and most who appear to meet CDC criteria for mTBI do not receive this diagnosis (Powell, Ferraro, Dikmen, Temkin, & Bell, Reference Powell, Ferraro, Dikmen, Temkin and Bell2008). Of those who are diagnosed, a large minority (37.2%) are not given recommendations for mTBI follow-up by their physicians (Bazarian, McClung, Cheng, Flesher, & Schneider, Reference Bazarian, McClung, Cheng, Flesher and Schneider2005).
A major factor that complicates the identification of mTBI in ED and other civilian mTBI populations is the high prevalence of comorbid personal and clinical features that mimic the effects of mTBI, thereby rendering it difficult to isolate the effects of the injury (Furger, Nelson, Lerner, & McCrea, Reference Furger, Nelson, Lerner and McCrea2016; Luoto et al., Reference Luoto, Tenovuo, Kataja, Brander, Ohman and Iverson2013; Mathias, Harman-Smith, Bowden, Rosenfeld, & Bigler, Reference Mathias, Harman-Smith, Bowden, Rosenfeld and Bigler2014). To the degree that neurocognitive tests are sensitive to mTBI in ED patients, their use could facilitate its identification and the triaging of affected patients into appropriate education and clinical management opportunities.
The automated nature of CNT administration (alongside other features; see Covassin, Elbin, & Stiller-Ostrowski, Reference Covassin, Elbin and Stiller-Ostrowski2009; Meehan, d’Hemecourt, Collins, Taylor, & Comstock, Reference Meehan, d’Hemecourt, Collins, Taylor and Comstock2012) may facilitate their adoption in medical settings over traditional paper-and-pencil neuropsychological tests. However, very few studies have examined the performance of CNTs in patients recruited from EDs. One study of ImPACT found statistically slower reaction time/processing speed (but equivalent verbal and visual memory) in mTBI versus control participants tested in the ED (Peterson, Stull, Collins, & Wang, Reference Peterson, Stull, Collins and Wang2009), whereas another study found a moderate effect of mTBI on immediate memory (as measured by the Rapid Screen of Concussion computerized battery) but nonsignificant effects of six other indices after adjusting for confounding variables (recent alcohol use; Sheedy, Geffen, Donnelly, & Faux, Reference Sheedy, Geffen, Donnelly and Faux2006).
Project Head to Head was a prospective study of the reliability, validity, and clinical utility of four CNTs for the assessment of mTBI that enrolled subjects in both sport and civilian (ED) samples from 2012 to 2014. Here, we present results from the ED arm of the study, which collected data on three CNTs (ANAM, DANA, ImPACT). (Corresponding findings from the sports arm of the study have been published previously; Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016.) We present the test–retest reliability and validity (group-level sensitivity/discrimination) of these neuropsychological measures in patients recruited from an ED with uncomplicated mTBI (as well as matched trauma controls) who were assessed at several time points post-injury.
Findings for the CNTs are compared to those of a brief postconcussive symptom checklist (Sport Concussion Assessment Tool – 3; SCAT3). Based on the small body of prior work on the performance of these measures to detect mTBI in similar patient populations, we expected to observe significant and moderate-to-large differences between mTBI and control groups in SCAT3 symptoms, with modest but statistically significant differences between groups on some CNT measures. Test–retest reliability evaluated in the control group was expected to be modest, given findings of limited stability of these measures in uninjured athlete populations (e.g., Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016), where sample characteristics (as compared to this ED sample) could be expected to maximize measurement stability.
METHOD
Participants
The sample was derived from participants in Project Head to Head, which enrolled a convenience sample of ED patients treated between September 2012 and May 2014 at an academic medical center that serves the emergent needs of the local community as well as being the region’s only level I trauma center. Participants completed informed consent before their first evaluation and were compensated $210 for their time and effort in completing all assessments. Testing procedures were approved by the Institutional Review Board at the Medical College of Wisconsin.
We monitored each patient being treated in the ED and attempted to approach any patient who appeared to meet inclusion and exclusion criteria specified below (e.g., age 18–45 years old, exposed to a common cause of mTBI; Faul et al., Reference Faul, Xu, Wald and Coronado2010). Possible causes of mTBI included motor vehicle-traffic crashes, falls, assaults, and struck by/against an object events. Figure 1 is a flow chart of each screening and enrollment stage. Of the 181 participants who completed informed consent, 98 met criteria for mTBI and 83 were enrolled as trauma controls. One of these subjects did not complete any assessment procedures after consenting to participate, 2 were withdrawn after the first assessment when it was discovered they met exclusionary criteria (neurologic disorder and positive head CT), and 4 were excluded due to evidence of invalid neurocognitive test results (see the Data Analyses section), yielding a final sample for analysis of 94 mTBI participants and 80 controls.
Patient screening and recruitment flow chart. Screening of emergency department (ED) admissions occurred in real-time, and patients who met eligibility criteria were considered for approach. As study procedures were conducted outside the ED, patients who were approached about the study were allowed to either schedule a study appointment or consent to be contacted about the study at a later time.
Inclusion and Exclusion Criteria
Our definition of mTBI was based on that of the study sponsor, the U.S. Department of Defense: “mTBI is defined as an injury to the brain resulting from an external force and/or acceleration/deceleration mechanism from an event such as a blast, fall, direct impact, or motor vehicle accident which causes an alteration in mental status typically resulting in the temporally related onset of symptoms such as headache, nausea, vomiting, dizziness/balance problems, fatigue, insomnia/sleep disturbances, drowsiness, sensitivity to light/noise, blurred vision, difficulty remembering, and/or difficulty concentrating” (Helmick et al., Reference Helmick, Guskiewicz, Barth, Cantu, Kelly, McDonald and Warden2006, p. 2).
Inclusion criteria for participation were age (18–45 years, the age range of interest to the study sponsor), loss of consciousness less than 30 min, posttraumatic amnesia less than 24 hr, no acute intracranial findings on brain imaging (if available), proficiency in English, and presenting for the initial assessment within 72 hr of injury. Subjects were excluded if they had an injury that precluded participation in the study protocol (e.g., hand injury that prevented use of a computer mouse), current diagnosis of a psychotic disorder,Footnote 1 history or clinical suspicion of other conditions (e.g., epilepsy, stroke, dementia) known to cause cognitive dysfunction, and history of moderate or severe TBI. Trauma controls met the inclusion criteria but sustained orthopedic injuries rather than mTBIs. Controls were also excluded if they had an mTBI within the last 6 months.
Assessment Protocol
The study protocol involved completion of post-injury examinations conducted within 72 hr of injury and at 8 (±1), 15 (±2), and 45 (±5) days post-injury. Mean (SD) time from injury to follow-up was 39.03 (21.62) hr, 7.94 (1.16) days, 14.63 (1.54) days, and 43.94 (3.93) days for the 72-hr, 8-, 15-, and 45-day time points, respectively. Tests were individually proctored by a research assistant in a quiet setting, nearly always with only one participant being examined at a time. Participants were either first assessed immediately after being medically discharged from the ED or they made an appointment to return another time. The assessment began with a one-on-one interview of contact information, demographics, and health history information followed by a neuropsychological assessment battery. Follow-up assessments began with an interview about subjects’ recoveries followed by the same neuropsychological assessment battery. To reduce the burden on participants to come to our office for assessment, day 8 follow-up appointments were completed via phone and only obtained recovery and SCAT3 symptom ratings (see below).
The testing protocol consisted of, in order, the: Wechsler Test of Adult Reading (WTAR; Wechsler, Reference Wechsler2001), CNT #1, Standardized Assessment of Concussion (SAC; McCrea et al., Reference McCrea, Kelly, Randolph, Kluge, Bartolic, Finn and Baxter1998), Sport Concussion Assessment Tool – 3rd edition (SCAT3) symptom checklist (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013), CNT #2, Green’s Medical Symptom Validity Test (MSVT; Green, Reference Green2003), Satisfaction With Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, Reference Diener, Emmons, Larsen and Griffin1985), Brief Symptom Inventory-18 (BSI-18; Derogatis, Reference Derogatis2001), Posttraumatic Stress Disorder Checklist – Civilian Version (PCL-C; Weathers, Litz, Huska, & Kean, Reference Weathers, Litz, Huska and Kean1994) and the Balance Error Scoring System (BESS; Guskiewicz, Ross, & Marshall, Reference Guskiewicz, Ross and Marshall2001). The MSVT was not performed at the day 15 assessment.
Each subject took two of three CNTs: Automatic Neuropsychological Assessment Metrics (ANAM v. 4.3; Vista Life Sciences), Defense Automated Neurobehavioral Assessment [DANA; U.S. Navy Bureau of Medicine and Surgery (BUMED)] and Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT, Online version; ImPACT Applications Inc.). These were selected by the study’s principal investigator and advisors to match the most widely used CNTs in sports medicine and military settings at the time of study design. CNT pairs were filled sequentially during the course of the study with order of administration counterbalanced across participants. Once assigned to an order of administration, each subject completed the same order at all follow-up examinations.
Neuropsychological Tests
ANAM and ImPACT have been described in more detail in publications from the sports arm of this study and are summarized more briefly here (Chin, Nelson, Barr, McCrory, & McCrea, Reference Chin, Nelson, Barr, McCrory and McCrea2016; Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016; Nelson, Pfaller, Rein, & McCrea, Reference Nelson, Pfaller, Rein and McCrea2015).
ANAM
The version of ANAM used was comprised of eight subtests: Simple Reaction Time (SRT), Code Substitution-Learning (CDS), Procedural Reaction Time (PRT), Mathematical Processing (MTH), Matching to Sample (M2S), Code Substitution-Delayed (CDD), Simple Reaction Time 2 (SR2), and Go/No-Go (GNG), producing one throughput score for each subtest (except GNG, for which the D-prime score was used) as a well as Composite Score that aggregates the subtests’ throughput scores (Vincent et al., Reference Vincent, Roebuck-Spencer, Lopez, Twillie, Logan, Grate and Gilliland2012).
DANA
The DANA BRIEF examination is comprised of nine cognitive tasks: Simple Reaction Time (SRT), Code Substitution (CDS), Procedural Reaction Time (PRT), Spatial Processing (SP), Choice Reaction/Go/No-Go (GNG), Code Substitution Recall (CDR), Matching to Sample (M2S), Memory Search (MS), and a second trial of Simple Reaction Time (SR2). Each subtest produces a throughput score. Psychological screening measures included in DANA were not evaluated for the purpose of this study.
ImPACT
ImPACT is comprised of six tasks which yield four neurocognitive composite scores: Verbal Memory (VERM), Visual Memory (VISM), Visual Motor Speed (VMS), and Reaction Time (RT).
SCAT3 symptom checklist
The SCAT3 (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013) symptom checklist is a 22-item self-report checklist of post-concussive symptoms (each rated from 0 to 6, with the resulting symptom severity scores ranging from 0 to 132). Higher scores reflect greater symptom burden.
Data Analyses
Sample and measures
Analyses included the aforementioned indices derived from each CNT as well as the SCAT3 symptom severity score. Subjects with evidence of invalid test performance, as determined by failure of one or more indices of the MSVT, were excluded from analyses by time point (at day 15, where the MSVT was not administered, we used the more conservative estimate of any MSVT failure across the other time points). In all, only 4 subjects failed to provide any valid assessment data, resulting in 94 mTBI and 80 control subjects reported on below. Cell sample sizes for the valid data analyzed here are presented in Table 3. Analyses did not exclude subjects based on the performance validity output embedded in ImPACT and ANAM due to: (a) our preference to not bias results in favor of select neurocognitive measures (e.g., due to the fact that DANA does not contain performance validity measures), (b) because some performance validity measures (in particular those of ImPACT) were developed for the purpose of establishing invalid performance in healthy athletes assessed pre-injury (and, therefore, are of unclear validity in this injured trauma sample), and (c) to maximize the sample size available for each analysis.
However, because we were interested in the degree to which the CNT performance validity criteria affected our results, major analyses (e.g., test–retest reliability, effect sizes of mTBI vs. control group differences) were re-run for the three CNTs after excluding subjects whose embedded performance validity measures indicated questionably valid data. For DANA, we applied the major validity criteria used by ANAM given the high level of similarity between these tests (i.e., excluding relevant subtests where accuracy<56%; see Nelson et al., Reference Nelson, Pfaller, Rein and McCrea2015). The findings of these additional analyses (which yielded extremely modest changes in results from those of the full sample) are summarized in the Results section.
Stability
Test–retest reliability for each scale was quantified for the control sample using both Pearson correlations (r) and intraclass correlations (ICC; two-way mixed; absolute agreement). Although the injured status of this control group could be expected to diminish performance stability, this was nevertheless of interest to provide a comparison to prior published data on the stability of these measures in healthy athletes (Chin et al., Reference Chin, Nelson, Barr, McCrory and McCrea2016; Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016). Test–retest intervals were selected from varying combinations of 72-hr, day 15, and day 45 time points to achieve a range of intervals. This yielded mean intervals of 13 (72-hr vs. day 15), 29 (day 15 vs. day 45), and 42 (72-hr vs. day 45) days. To extract an overall stability metric across indices within each CNT, correlation coefficients were converted to z scores using the Fisher transformation, averaged together, and back-transformed into the mean correlation coefficients presented in Table 2.
Sensitivity
Group (mTBI, control)×Time (72-hr, day 15, day 45) repeated-measures analyses of variance were computed for each CNT subscale. Greenhouse-Geisser correction was used where needed for violations of the sphericity assumption. Follow-up t tests were conducted where appropriate to examine between-group differences at each time point. Adjustment for multiple comparisons was performed using the false discovery rate method (Benjamini & Hochberg, Reference Benjamini and Hochberg1995). Cohen’s d effect sizes of mTBI versus control group differences were computed across measures and time points. Next, receiver operating characteristic (ROC) curves were produced for each index to provide another clinically relevant metric regarding the value of each measure for individual decision making about group status.
RESULTS
Sample Characteristics
The demographic makeup and acute injury characteristics of the sample are presented in Table 1. The groups were closely matched on variables including gender, age, race, socioeconomic status, estimated verbal intellectual ability (WTAR score), mTBI history, and litigation related to injury. The mTBI group had somewhat higher representation of subjects with a self-reported history of psychiatric disorder (n=8 vs. 1 in the control group; p=.031). Of these, four reported bipolar depression (two of whom had a comorbid anxiety disorder), two reported only unipolar depression, two reported an unspecified mood disorder, and three reported an anxiety disorder. Eight of these subjects were taking a psychiatric medication at the time of enrollment. Exclusion of subjects with psychiatric histories from analyses, however, had negligible effects on the main results of this study (e.g., mTBI vs. control group effect sizes reported Table 4 declined at a mean of d =.01).
Sample characteristics
Note. ADHD=attention deficit-hyperactivity disorder; WTAR=Wechsler Test of Adult Reading standard score; SES=Hollingshead socioeconomic status. Missingness was n=1 on number of prior mTBIs and ADHD, 9 on litigation status, and 8 on worker’s compensation status.
Among the injured sample, 36.8% endorsed loss of consciousness, 15.8% posttraumatic amnesia, and 6.3% retrograde amnesia. We could not confirm loss of consciousness with witnesses and the report of loss of consciousness could simply represent posttraumatic amnesia. The overall distribution of CNT pairings across the sample evaluated in this manuscript was: 38.2% ANAM-DANA, 12.9% ANAM-ImPACT, and 48.9% DANA-ImPACT.
The percentage of the sample that completed day 15 and 45 follow-up assessments was 89.1% and 83.9%, respectively. Those who did versus did not complete the day 15 assessment were less likely to have attention deficit-hyperactivity disorder (ADHD) (7.8% vs. 26.34%; χ2[1]=6.55; p=.010). Similarly, those who did versus did not complete the day 45 assessment were less likely to have ADHD (5.5% vs. 32.1%; χ2[1]=18.78; p<.001). Otherwise, all other demographic, history, injury, and 72-hr assessment measures were equivalent between groups (p>.05).
Trauma control subjects self-reported injuries to a variety of bodily regions: back (61.3%), neck (50.0%), upper extremity (45.0%), lower extremity (45.0%), trunk (18.8%), and head (16.3%). The 13 controls coded positive for head trauma reported lacerations, bruises, or other injuries to the head but did not otherwise meet criteria for mTBI (i.e., did not endorse altered mental status or mTBI symptoms). Excluding these subjects from analyses did not impact the primary results (e.g., M change in Cohen’s d values reported in Table 4 was d=.01).
Finally, although litigation status was unrelated to injury group and, therefore, had no effect on the major study analyses, given the relatively high prevalence of the sample’s reported involvement in litigation and the relevance of this variable to the neuropsychology audience, data pertaining to the magnitude of litigation-related effects on the neurocognitive performance and mTBI symptoms are presented in the Supplementary Materials. To summarize the results briefly, subjects’ who endorsed any involvement in litigation related to their injuries tended to report more mTBI symptoms and perform more poorly on neurocognitive measures, with effects small-to-medium in magnitude. Given the relatively crude manner in which this variable was assessed,Footnote 2 however, these data should be interpreted cautiously.
Test–Retest Reliability of Symptom and Neurocognitive Indices
Table 2 displays the test–retest reliability of the symptom and neuropsychological measures obtained at varying intervals using data from the trauma control group. The SCAT3 symptom severity score had reliability coefficients over .70 for the briefer test–retest interval (Pearson vs. ICC=.78 vs. .74), with somewhat lower magnitudes for the longer test–retest intervals (e.g., Pearson r for 29-day and 42-day intervals=.60 and .55, respectively). For the CNTs, 44–63% of Pearson reliability coefficients were over .6 across all test–retest intervals (12 of 27 for ANAM, 17 of 27 for DANA, and 5 of 12 for ImPACT, respectively), and 25–33% of coefficients were over .7 (9 of 27, 8 of 27, and 3 of 12 across all test–retest intervals for ANAM, DANA, and ImPACT, respectively). Similar results were found using ICCs (33–63% of coefficients over .6 and 19–30% over .7 across CNTs and test–retest intervals). Excluding subjects who failed embedded effort indices for each CNT had minimal effect on test–retest reliability coefficients (e.g., across all time points, Pearson stability coefficients improved by a mean of .06, .01, and .01 for ANAM, DANA, and ImPACT, respectively).
Test–retest reliability of SCAT3 and CNT measures in trauma controls using Pearson (and intraclass) correlations
Note. 13-day interval=72-hour to day 15 assessment; 29-day interval=day 15 to day 45 assessment; 42-day interval=72-hour to day 45 assessment; SCAT3=Sport Concussion Assessment Tool—3; SRT=Simple reaction time; CDS=code substitution-learning; PRT=procedural reaction time; MTH=mathematical processing; M2S=matching to sample; CSD=code substitution-delayed; SR2=simple reaction time 2; GNG=go no-go; SP=spatial processing; CDR=code substitution-recall; MS=memory search; SR2=simple reaction time trial 2; VERM=verbal memory composite; VISM=visual memory composite; VMS=visual motor speed composite; RT=reaction time composite.
Between-Group Differences and Effect Sizes on Symptom and Neurocognitive Measures
Table 3 provides descriptive statistics for the mTBI and control groups across measures and time points, while Table 4 provides the between-group differences expressed as Cohen’s d. The Supplementary Materials (Tables S1–S2) provides additional statistical output to support the information provided in Table 3. SCAT3 symptom severity score was the only variable that demonstrated statistically significant between-group differences at any of the acute and subacute time points (72-hr, day 8, and day 15 d=.78, .60, and .49, respectively, FDR-adjusted ps<.05; all in the expected direction of higher symptoms for mTBI vs. control subjects). The symptom score was not significantly different between the groups after correction for multiple comparisons at day 45 (d=.35). No CNT measure demonstrated statistically significant group differences at any of the post-injury time points.
Descriptive statistics M (SD)
Note. SCAT3 Symptom variable reflects the symptom severity score. SCAT3 symptom severity day 8 M (SD) for mTBI and control group=25.23 (22.57) and 13.47 (16.15), respectively.
mTBI versus trauma control group effect sizes (Cohen’s d)
aSCAT3 Day 8 d=−.60. Comparisons are all scaled such that negative values reflect worse performance in the mTBI group. Values are bolded where p<.05 after adjustment for multiple comparisons. SCAT3=Sport Concussion Assessment Tool—3; SRT=Simple reaction time; CDS=code substitution-learning; PRT=procedural reaction time; MTH=mathematical processing; M2S=matching to sample; CSD=code substitution-delayed; SR2=simple reaction time 2; GNG=go no-go; SP=spatial processing; CDR=code substitution-recall; MS=memory search; SR2=simple reaction time trial 2; VERM=verbal memory composite; VISM=visual memory composite; VMS=visual motor speed composite; RT=reaction time composite.
Before adjustment for multiple comparisons, only one main effect of Group (of 22 CNT variables) was significant (ANAM simple reaction time 2 Group p=.022) and no Group × Time interactions were significant. In terms of magnitude or clinical meaningfulness of group differences, the neurocognitive measures demonstrated quite small effect sizes even at 72hr (absolute value of 72-hr ANAM ds=.06 to .60 with all but one under .3; DANA ds=.00 to .19; ImPACT ds=.05 to .13; across these measures 15 were in the expected direction and 10 were not). Excluding subjects who failed embedded effort indices for each CNT had minimal effect on group differences (e.g., across all time points, d values became stronger in the expected direction by a mean of .05, .05, and .01 for ANAM, DANA, and ImPACT, respectively).
ROC Curves
Table 5 displays the area under the ROC curve (AUC) as another metric of the ability of these clinical measures to discriminate between the mTBI and orthopedically-injured control groups. The best performing measure (SCAT-3 symptom severity score) provided only fair discrimination between groups (AUC=.72 at 72-hr), whereas discrimination was poor for neurocognitive measures (M AUC for ANAM, DANA, and ImPACT at 72-hr=.56, 49, and .50, respectively).
Area under the ROC curve
aSCAT3 Day 8 AUC=.68. Values are bolded where p<.05 after adjustment for multiple comparisons. SCAT3=Sport Concussion Assessment Tool-3; SRT=Simple reaction time; CDS=code substitution-learning; PRT=procedural reaction time; MTH=mathematical processing; M2S=matching to sample; CSD=code substitution-delayed; SR2=simple reaction time 2; GNG=go no-go; SP=spatial processing; CDR=code substitution-recall; MS=memory search; VERM=verbal memory composite; VISM=visual memory composite; VMS=visual motor speed composite; RT=reaction time composite.
DISCUSSION
In this prospective study of the utility of three computerized neurocognitive test batteries (ANAM, DANA, ImPACT) for the assessment of mTBI in patients recruited from an ED, none provided more than minimal discrimination between mTBI and trauma control groups within 72 hr of injury (M=39 hr) or at later time points (days 15 and 45 post-injury). For example, the mean Cohen’s d (scaled with negative values in the direction of more impairment for mTBI vs. control groups) at 72 hr for ANAM, DANA, and ImPACT were −.16, .02, and .00, respectively. These findings contrast those previously published from the sports arm of this study, which found stronger (generally medium-to-large) between-group differences in performance on ANAM, Axon, and ImPACT within 24 hr (M=19 hr) of injury in a sample of high school and collegiate athletes (Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016).
That effect sizes were much smaller for ED versus sport samples likely reflects one major difference between the two arms of the study—the use of injured controls in the ED sample. As nonspecific injury- and treatment-related factors present in trauma populations broadly can contribute to similar neuropsychological sequelae as mTBI (e.g., Babikian et al., Reference Babikian, Satz, Zaucha, Light, Lewis and Asarnow2011; Bruera, Macmillan, Hanson, & MacDonald, Reference Bruera, Macmillan, Hanson and MacDonald1989; Macciocchi, Seel, & Thompson, Reference Macciocchi, Seel and Thompson2013; Rabinowitz et al., Reference Rabinowitz, Li, McCauley, Wilde, Barnes, Hanten and Levin2015), the limited discrimination observed in neurocognitive measures between ED-recruited mTBI and trauma control groups is likely due to the nonspecific neurocognitive sequelae of recent traumatic injury (and other co-occurring factors common in civilian trauma populations that can contribute to neuropsychological impairment; e.g., Iverson, Reference Iverson2005). Of course, comparable data on an orthopedically-injured athlete sample would be needed to confirm this conjecture.
To put the overall level of performance of the two study samples into context, both the mTBI and trauma control participants in this ED-recruited sample scored, on average, 0.3 SDs below the acutely concussed athletes and 1.5 SDs below the healthy control athletes of the sports sample (using ImPACT metrics at the most acute post-injury time point for illustrative purposes; Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016). Thus, irrespective of injury type, trauma patients perform cognitively similar to (or slightly worse than) adolescent and young adult athletes with sport-related mTBI and significantly worse than healthy athletes.
Whereas CNT performance was equivalent between mTBI and trauma control groups, SCAT3 symptom severity differentiated better between groups (i.e., was higher in mTBI vs. control subjects; 72-hr d=−.78), with this effect diminished but still more robust than neurocognitive measures during the follow-up period (day 8, 15, and 45 d=−.60, −.49, and −.35, respectively). This is consistent with findings from the sports arm of the study of stronger sensitivity of self-reported symptoms than CNT performance to mTBI (24-hr symptom d=−1.53 vs. mean neurocognitive Cohen’s ds in the .5 to .7 range; Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016). Taken together, these findings underscore the relevance of measuring mTBI symptoms in the clinical assessment of mTBI across patient populations and injury settings.
That said, mTBI symptoms are also not specific to mTBI, a fact supported by prior studies (e.g., Landre, Poppe, Davis, Schmaus, & Hobbs, Reference Landre, Poppe, Davis, Schmaus and Hobbs2006; Smith-Seemiller, Fow, Kant, & Franzen, Reference Smith-Seemiller, Fow, Kant and Franzen2003) and these data [i.e., differences between mTBI and control groups in Project Head to Head were somewhat smaller in the ED (d=.76) vs. the sport samples (d=1.53)]. Similar to neurocognitive measures, “mTBI” symptom ratings in our sample probably reflect both acute effects of injury in general as well as the high prevalence of mTBI-like symptoms in community samples (Iverson & Lange, Reference Iverson and Lange2003). Although the slightly longer time interval between injury and first assessment in the ED sample (<72 hr vs. <24 hr; M time from injury to first assessment=39 vs. 19 hr) could also contribute to weaker effects sizes on clinical measures (see McCrea et al., Reference McCrea, Guskiewicz, Marshall, Barr, Randolph, Cantu and Kelly2003), the relatively slow recovery trajectory of this ED sampleFootnote 3 suggests that this procedural difference probably does not contribute substantially to the observed differences in findings across samples.
Finally, test–retest reliability of the neurocognitive measures in the trauma controls of this ED sample were similar to those reported on the healthy control athletes of the sport sample (Nelson, LaRoche, et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016), with roughly half over .60 and a quarter over .70. That measurement stability was similar across samples suggests that the reduced sensitivity of neurocognitive measures to mTBI in the ED versus sport sample is not due to lower measurement stability in this population. Although most neurocognitive indices did not meet common standards for use in clinical practice, the similar stability of CNT measures across samples is nevertheless of interest given that the acutely injured nature of this sample was expected to diminish performance stability in comparison to what has been observed in healthy controls.
This investigation had several limitations. First, subjects were evaluated in a laboratory setting within 72 hr of injury; thus, it is possible that stronger group differences in clinical assessment measures would have been found had subjects been assessed more acutely (such as within the ED). Second, the study design (i.e., assignment of two of three CNTs to each subject) and presence of loss to follow-up (16% at 45 days post-injury) contributed to smaller sample sizes (<50) for some CNT measures and at some time points. Nevertheless, the findings are considered valuable since quantifying effect sizes was of greater interest than achieving statistical significance, because the acute (72-hr) post-injury time point (in which all subjects provided data) was of greatest interest in terms of hypothesized cognitive effects, and because there did not appear to be significant bias in the subset of the sample that completed day 15 and 45 follow-up appointments (i.e., completion of follow-up appointments was unrelated to major demographic and injury-related factors). Third, some participants were examined in the same room as other subjects, which could have affected their performance. In particular, group settings tend to negatively impact cognitive performance, although the magnitude of this effect depends on several factors such as group size and the particular cognitive domain being assessed (Eastvold, Belanger, & Vanderploeg, Reference Eastvold, Belanger and Vanderploeg2012; Lichtenstein, Moser, & Schatz, Reference Lichtenstein, Moser and Schatz2014). While group testing occurred relatively infrequently, in the future testing setting should be better controlled or documented so that the effect of environmental factors can be minimized or formally evaluated.
In summary, we found that among three CNTs (ANAM, DANA, ImPACT) administered to community patients recruited from an ED, none reliably differentiated between mTBI and trauma control groups within 72 hr of injury or later post-injury time points. While SCAT3 symptom scores were significantly elevated in mTBI versus control groups, effect sizes were modest in comparison to what has been reported in the sport-related mTBI literature, where concussed athletes have typically been compared to healthy controls. We interpret these attenuated findings as a consequence of the nonspecific factors common in trauma patients that mimic the clinical effects of mTBI. The implication of these findings is that these assessment tools, which are commonly used for assessment of sport-related mTBI, appear less valuable for the assessment of mTBI sequelae in this more complex and heterogeneous patient-based ED sample.
Alternatively, it is possible that more acute neuropsychological assessment or aggregation of neuropsychological measures with other emerging markers of mTBI (e.g., blood biomarkers, neuroimaging) would contribute to improved performance in the context of acute identification of the effects of mTBI specifically. Furthermore, these findings do not rule out the possibility that acute neuropsychological data could be informative for prognostic purposes in mTBI or other patient populations (Dischinger, Ryb, Kufera, & Auman, Reference Dischinger, Ryb, Kufera and Auman2009; McCrea et al., Reference McCrea, Guskiewicz, Randolph, Barr, Hammeke, Marshall and Kelly2013; Meehan, Mannix, Monuteaux, Stein, & Bachur, Reference Meehan, Mannix, Monuteaux, Stein and Bachur2014; Nelson, Tarima, et al., Reference Nelson, Tarima, LaRoche, Hammeke, Barr, Guskiewicz and McCrea2016). At the present time, however, the findings suggest that clinicians can continue to apply standard clinical criteria to assess patients for mTBI; neurocognitive assessment may provide supporting data on patients’ presenting concerns but is not useful for diagnostic purposes.
Acknowledgments
We thank David Gray, Ashley LaRoche, and Adam Pfaller for their contributions to data collection. This work was supported by the U.S. Army Medical Research and Materiel Command (M.M., award number W81XWH-12-1-0004) and the National Center for Advancing Translational Sciences (National Institutes of Health Grant Number UL1TR001436). Opinions, interpretations, conclusions, and recommendations expressed in the manuscript are those of the authors and are not necessarily endorsed by the U.S. Army or representative of the official views of the NIH. The authors have no conflicts of interest to report.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355617717000157
