INTRODUCTION
Every year thousands of service members (SMs) in the U.S. military are diagnosed with a mild traumatic brain injury (mTBI), also known as concussion (Defense and Veterans Brain Injury Center [DVBIC], 2016). These injuries can take place in a variety of settings due to several causes, including those similar to sports-related concussion in the civilian sector. Regardless of where or how concussion occurs, there is a need for timely and effective evaluation of an individual’s cognitive functioning (Kelly, Coldren, Parish, Dretsch, & Russell, Reference Kelly, Coldren, Parish, Dretsch and Russell2012). Assessment of cognitive abilities via neuropsychological (NP) tests is considered the cornerstone of concussion management (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvořák, Echemendia and Sills2013). However, these tests are time consuming and require particular expertise for administration and interpretation of results. In more recent years, computerized neurocognitive assessment tools (NCATs) have been increasingly used as a quicker and more feasibly administered alternative to NP tests (Friedl et al., Reference Friedl, Grate, Proctor, Ness, Lukey and Kane2007; McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvořák, Echemendia and Sills2013).
The Automated Neuropsychological Assessment Metrics 4 TBI-MIL (ANAM4) is an NCAT developed by the U.S. Army (Friedl et al., Reference Friedl, Grate, Proctor, Ness, Lukey and Kane2007) and widely used in the military (Defense Health Board, 2016). ANAM4 is regularly administered before a deployment as a means to generate a neurocognitive baseline for post-deployment and post-injury comparison (DoDi 6490.13). Despite the goal of NCATs, including ANAM4, existing evidence is inconclusive regarding the ability to identify cognitive issues following concussion (see Arrieux, Cole, & Ahrens, Reference Arrieux, Cole and Ahrens2017; Resch, McCrea, & Cullum, Reference Resch, McCrea and Cullum2013).
Typically findings from ANAM4 are based on analyses comparing post-injury scores either to individual baseline measurements or normative databases (see Haran et al., 2016; McCrea et al., Reference McCrea, Pliskin, Barth, Cox, Fink, French and Powell2008). The current analyses focus on within-person inconsistent performance, or intraindividual neurocognitive variability, within a single test session, as a metric potentially better suited to detect the cognitive effects of mTBI. Variability has been described in multiple ways, but often relates to three principles: persons, measures, and occasions (Hultsch, MacDonald, & Dixon, Reference Hultsch, MacDonald and Dixon2002). Interindividual variability, or “diversity,” often measures differences between persons or groups. An individual’s variability across multiple measures can be thought of as intraindividual differences or “dispersion.” The focus of the current analyses is intraindividual variability (IIV), or an individual’s variability on the same test across multiple occasions, referred to as “inconsistency.”
Although IIV is often viewed as noise or test error, it may in fact reflect fluctuation in cognitive processing and reveal cognitive deficits that a mean or standard score is attempting, but failing, to capture. For example, research in aging populations has shown IIV on various behavioral and neurophysiological measures to be associated with decline in cognitive performance (Fjell, Rosquist & Walhovd, Reference Fjell, Rosquist and Walhovd2009; Hultsch et al., Reference Hultsch, MacDonald and Dixon2002; Lovden, Li, Shing, & Linderberger, Reference Lovden, Li, Shing and Lindenberger2007). Although the literature base is relatively small, IIV in acute and post-acute concussion populations has been studied for more than two decades using both traditional NP and reaction time (RT) tests (Rabinowitz & Arnett, Reference Rabinowitz and Arnett2013; Sosnoff, Broglio, Hillman, & Ferrara, Reference Sosnoff, Broglio, Hillman and Ferrara2007; Stuss et al., Reference Stuss, Stethem, Hugenholtz, Picton, Pivik and Richard1989).
Using NP tests, Hill, Rohling, Boettcher, and Meyers (Reference Hill, Rohling, Boettcher and Meyers2013) analyzed IIV using means from the Meyers Neuropsychological Battery in individuals reporting a history of mTBI and found that overall performance is negatively correlated with variability. Similarly, in a study using RT-based stimulus discrimination and flanker tests, history of concussion was shown to be associated with increased IIV (Parks et al., Reference Parks, Moore, Wu, Broglio, Covassin, Hillman and Pontifex2015). Beyond behavioral measures, Segalowitz, Dywan, and Unsal (Reference Segalowitz, Dywan and Unsal1997) demonstrated for a TBI group, and not for a control group, RT variability was related to electrophysiological measures of attentional allocation and sustainment (the P300 amplitude and the preresponse component of the contingent negative variation E-Wave), supporting the idea that RT variability reflects this attentional processing.
Studies have also examined IIV in TBI using NCATs. Bleiberg, Garmoe, Halpern, Reeves, and Nadler (Reference Bleiberg, Garmoe, Halpern, Reeves and Nadler1997) demonstrated participants with mild to moderate TBI performed more inconsistently in same-day and across multiple day sessions than a healthy control group. Makdissi et al. (Reference Makdissi, Collie, Maruff, Darby, Bush, McCrory and Bennell2001) investigated a simple RT test in a different NCAT, CogState, in athletes and found greater standard deviation in RT in acutely concussed versus never concussed athletes at follow-up, although not at baseline. However, longer RT in concussed participants as compared to controls could account for greater standard deviation in RT. Sosnoff et al. (Reference Sosnoff, Broglio, Hillman and Ferrara2007) adjusted for mean RT in a group of individuals tested within 72 hr of concussion and found that after this adjustment, concussed individuals did not have greater RT standard deviation than healthy age- and gender-matched individuals.
The above studies, most of which demonstrate an ability to differentiate TBI and control group performance using IIV measures, all compare an individual’s performance on a test (i.e., measures) or whole battery across test sessions (i.e., occasions). In contrast, the present investigation explores potential differences in IIV within a single test session by comparing performance on one subtest repeated within a battery, in patients with acute concussion and healthy controls. Our approach allows examination of the use of IIV analyses within an abbreviated window and without a need for repeat testing of an entire battery. The ANAM4 is an ideal test to examine IIV in this way, as unlike most NCATs, the ANAM4 includes an identical simple RT (SRT) task at the beginning and the end of the battery.
Although the ANAM4 standard output generates the RT standard deviation on each subtest, our approach differs because it examines the standard deviation of the difference between the trial-by-trial RT data. This approach allows for a more fine-grained measure of IIV and an individual’s change in RT and RT variability over a brief period of time. In addition to looking at trial-by-trial raw RT data, the current study investigated acutely concussed individuals, as previous research suggests ANAM4 has limited clinical utility more than eight days following concussion, as well as healthy controls (e.g., Nelson et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016). We hypothesize that this alternative trial-by-trial approach to interpreting RT on ANAM4 will reveal differences in IIV (i.e., differences in “inconsistency”) across the two groups. As a secondary objective, we use interindividual differences (i.e., “diversity”) to investigate the potential impact demographic variables may have on any differences identified in IIV.
METHODS
Sample
A total sample of 356 individuals was selected from a larger study’s sample of SMs from Fort Bragg with and without mTBI where ANAM4 was administered (Cole, Arrieux, Dennison, & Ivins , Reference Cole, Arrieux, Dennison and Ivins2017; Cole, Arrieux, Ivins, Schwab, & Qashu, Reference Cole, Arrieux, Ivins, Schwab and Qashu2017). Informed consent was obtained from all subjects and data were collected in compliance with Womack Army Medical Center’s Institutional Review Board regulations and requirements. The sample included 240 healthy controls (CTRL) and 116 participants within 7 days of a medically documented mTBI. The mean time since injury was 4.8 days (range, 0–7 days). All injuries were sustained on or around Fort Bragg (i.e., no combat related injuries) with most injuries sustained due to hard landings during parachute training jumps (85.3%), and the remaining injuries (all<5%) due to motor vehicle crashes, falls, assaults, sports-related injury, or blast exposures during training exercises.
Instrumentation
The ANAM4 (CSRC, 2014) is an automated, computerized neurocognitive test battery that includes a sleepiness scale, mood scale, a self-report TBI questionnaire, and seven core subtests: Code Substitution Delayed (CDD), Code Substitution (CDS), Matching-to-Sample (M2S), Mathematical Processing (MTH), Procedural Reaction Time (PRO), Simple Reaction Time (SRT1), and Simple Reaction Time Repeated (SRT2). Due to the larger study’s procedures, an additional battery of questionnaires was administered before testing, with the seven core ANAM4 subtests administered per usual procedures following completion of questionnaires (Cole, Arrieux, Ivins, et al., Reference Cole, Arrieux, Ivins, Schwab and Qashu2017). Validity of the data was evaluated by an embedded effort index (EI), which flags atypical scores based on accuracy and discrepancy of responses (Roebuck-Spencer, Vincent, Gilliland, Johnson, & Cooper, Reference Roebuck-Spencer, Vincent, Gilliland, Johnson and Cooper2013). Specifically, the ANAM4 EI assesses accuracy and RT discrepancy on four of the battery’s subtests, which are transformed to weighted scores based on the infrequency of those scores. Weighted scores range from 0–48, and scores above 14 are considered invalid (Roebuck-Spencer et al., Reference Roebuck-Spencer, Vincent, Gilliland, Johnson and Cooper2013). For the purposes of this manuscript, only the EI and the raw data from the SRT1 and SRT2 were used in the analyses. The raw data for the SRT1 and SRT2 consisted of 40 trials for each subtest.
Data Processing
To prepare the SRT1 and SRT2 data for the analyses we first removed any participant who was deemed to have invalid performance by the ANAM4 EI. Fourteen participants, 11 in the mTBI group (9.5%) and 3 in the CTRL group (1.3%), were flagged by the ANAM4 EI and removed from the sample. The resulting sample size was 342 participants (13,680 trials) with 237 participants (9480 trials) in the CTRL group and 105 participants (4200 trials) in the mTBI group. Second, extremely fast and slow responses, potentially indicating common key press errors (e.g., accidental key presses or interruption of the task) were trimmed from the dataset per commonly used procedures (Batterham, Bunce, Mackinnon, & Christensen, Reference Batterham, Bunce, Mackinnon and Christensen2014; Dixonet al., Reference Dixon, Garrett, Lentz, MacDonald, Strauss and Hultsch2007; Garret, MacDonald, & Craik Reference Garrett, MacDonald and Craik2012; Hultsch et al., Reference Hultsch, MacDonald and Dixon2002; Hultsch, Strauss, Hunger, & MacDonald Reference Hultsch, Strauss, Hunter and MacDonald2008). Specifically, a lower bound for responses was set at 150 ms, with a total of 275 (2.0%) and 450 (3.3%) trials trimmed for the SRT1 and SRT2, respectively. An upper bound was set for each individual, with any trials exceeding a within-subject subtest mean of plus three standard deviations trimmed, for a total of 258 (1.9%) and 292 (2.1%) trials for the SRT1 and SRT2, respectively. To maintain a complete dataset, trimmed values were imputed using a linear interpolation procedure from the relationships among all trials from all participants in the dataset (Hultsch et al., Reference Hultsch, MacDonald and Dixon2002).
Statistical Analyses
Four distinct statistical analyses were conducted to address our study objectives. The first analysis investigated group differences on demographic variables and established residuals to control for any differences in subsequent between-groups analyses. The second analysis aimed to identify differences between the CTRL and mTBI groups on SRT1 and SRT2 performance (i.e., diversity). The third analysis investigated if there were differences between the CTRL and mTBI groups in terms of variability in RT on both SRT1 and SRT2 (i.e., dispersion). The fourth analysis, addressing the primary aim of this study, investigated if there were between-groups differences in within-persons change in RT and RT variability from SRT1 to SRT2 (i.e., inconsistency).
Group differences for demographic data were examined using Mann-Whitney U Tests and Chi-Square tests. There were minor violations of the Lilliefors test of normality for the simple reaction subtest data; however, the potential for a familywise type I error due to multiple comparisons was accounted for with sample sizes sufficient enough (i.e., n>30) for the central-limit theorem to apply.
Levene’s test was used to assess between-groups variability (i.e., diversity) in SRT1 and SRT2 performance. Any observed differences between groups on variability may be an artifact of group differences in mean performance, as larger standard deviations tend to be associated with larger means (Hale, Myerson, Smith, & Poon, Reference Hale, Myerson, Smith and Poon1988). Factors such as sex and age have been reported to confound RT mean performance and variability (Der & Deary, Reference Der and Deary2006). To control for these possible effects of sex and military rank, which is highly correlated with age, four separate linear regression procedures were used to calculate residuals for SRT1 and SRT2 (for both the CTRL and mTBI groups). The results were interpreted using the following criteria for squared association indices: recommended minimum practical effect size (r 2=0.04), moderate effect (r 2=0.25), and strong effect (r 2=0.64) (Ferguson, Reference Ferguson2009).The absolute values of the resulting residuals were used in within-persons variability (i.e., IIV) calculations.
IIV can be used to examine both dispersion on a task (i.e., across the 40 trials within each of the ANAM4 SRT subtests) and to examine inconsistency across time (i.e., change across SRT1 and SRT2). While there are numerous indices that can be used to compute IIV, the simplest is the intraindividual standard deviation (ISD). ISDs are standard deviations of RT variability within each of SRT1 and SRT2, calculated for each individual using the purified residual scores.
To examine group differences in ISD dispersion, two separate general linear model (1 × 2) analyses [analysis of variance (ANOVA)] were performed, one each for SRT1 and SRT2, with group membership (two levels) as the between-subjects variable. Effect size (ES) for group differences was calculated using the partial eta squared (ηp 2) and the results were interpreted using the aforementioned criteria for squared association indices.
To examine group differences in ISD inconsistency, a general linear model (2×2) ANOVA with repeated measures was performed, with group membership (two levels; CTRL vs. mTBI) as the between-subjects variable and subtest (two levels; SRT1 vs. SRT2) as the within-subjects variable. Pairwise comparisons were conducted to follow-up significant main effects. Significant interaction effects were explored using post hoc comparisons (one-way ANOVAs with repeated measures). ES were calculated using the partial eta squared (ηp 2) and the results were interpreted using the aforementioned criteria for squared association indices.
All analyses were performed with Matlab 2015b (Mathworks, Natick, MA) and SPSS Version 22 (IBM, Armonk, NY).
RESULTS
There were significant differences between the CTRL and mTBI groups for all demographic variables (Table 1). These differences are believed to be due to the larger number of officers in the CTRL group. That is, there are more female officers than enlisted personnel, and officers are generally older and more educated than the enlisted population. It is believed that officers were over-represented in the CTRL group due to their greater ability to control and dictate their daily schedules, allowing them to take time off to volunteer in a research study. Sex and rank, likely accounting for all other demographic differences, were controlled for in subsequent analyses, as described above.
Participant characteristics for CTRL and mTBI groups.
Abbreviations: CTRL=control group; mTBI=mTBI group.
a Two-Tailed Mann-Whitney U.
b Chi-square test
c Data missing for one mTBI participant
Table 2 reports the standard deviations for SRT1 and SRT2 as a function of injury status. It should be noted that the standard deviations for the mTBI group were nearly two and nearly three times those for the CTRL group for SRT1 and SRT2, respectively. Levene’s test for the homogeneity of variance indicated significant group differences in variability for both the CTRL (F (1,13,678)=848.65; p<.0001) and mTBI (F (1,13,678)=1,815.71; p<.0001) groups. Taken together these results indicate increased diversity in the RT subtest performance with injury status.
Standard deviations and means of raw reaction time (RT) and RT intraindividual standard deviation (ISD).
Abbreviations: CTRL=control group; mTBI=mTBI group; SRT1=Simple Reaction Time; SRT2=Simple Reaction Time Repeated
Table 3 reports the results of the regression analyses. For SRT1, there was a significant linear trend for rank and sex in both groups indicating that increased diversity for females and increased diversity with higher rank. For SRT2, there was a significant linear trend for rank, but not sex, in both the CTRL and mTBI groups, indicating increased diversity with increasing rank. In general, the magnitude of the significant trends were modest, all were less than 2% of the variance, and none reached the recommended minimum practical effect size for squared association indices.
Summary table for regression of residuals on linear trends.
Abbreviations: CTRL=control group; mTBI=mTBI group; SRT1=Simple Reaction Time; SRT2=Simple Reaction Time Repeated
The results of the ANOVA performed to examine group differences in dispersion for SRT1 revealed that there was a significant main effect for group membership (F (1,340)=10.00; p=.002; ηp 2=.03), although it did not reach the recommended minimum practical effect size. For SRT2, the ANOVA revealed a significant main effect that did exceed the recommended minimum practical effect size for group membership (F (1,340)=30.72; p<.001; ηp 2=.08). Pairwise comparisons revealed that the ISD mean for the CTRL group was significantly lower than the ISD mean for the mTBI group.
The results of the ANOVA performed to examine group differences in inconsistency (i.e., IIV) revealed that there was a significant interaction of group and time (F (1,340)=15.87; p=.001; ηp 2=.03). The main effects for group (F (1,340)=23.75; p=.001; ηp 2=.07) and time (F (1,340)=15.87; p=.001; ηp 2=.05) were also significant and exceeded the recommended minimum for a practical effect. The post hoc one-way ANOVA with repeated measures revealed a significant main effect for time that exceeded the recommended minimum for a practical effect for the mTBI group only (F (1,340)=11.49; p=.001; ηp 2=0.10). Pairwise comparisons revealed that the ISD mean for the CTRL group was significantly lower (i.e., less IIV) than the ISD mean for the mTBI group, and within the mTBI group the mean ISD for SRT1 was significantly lower than the mean ISD for SRT2.
DISCUSSION
The current study investigated differences in mean RT and RT IIV between CTRLs and those with acute concussion using raw trial-by-trial RT data from the repeated SRT subtests in the ANAM4. This approach was relatively unique as most previous studies have focused on the use of standardized scores and cognitive efficiency metrics (e.g., ANAM4 throughput scores) to investigate group differences. Moreover, prior studies examining differences in RT variability have almost exclusively done so across test sessions rather than using a subtest repeated within the same battery and test session. Our hypotheses were largely supported, as those with acute concussion had slower RTs and greater RT IIV than CTRLs.
While it is not surprising that there were differences in the RT means and IIV between the CTRL and mTBI groups, of interest, the CTRL group demonstrated improved (i.e., faster) RT as well as less IIV in SRT2 compared to SRT1. Although the mTBI group demonstrated longer, but relatively stable mean RT across subtests, the most important finding was that those in the mTBI group had more IIV in both subtests than controls, with IIV actually increasing in the second, repeated subtest. This suggests an apparent practice effect observed in the CTRL group that is not observed in the mTBI group. The lack of a demonstrated practice effect in those recovering from neurological insult could be clinically meaningful when a practice effect is otherwise expected (Lezak, Howieson, Bigler, & Tranel, Reference Lezak, Howieson, Bigler and Tranel2012).
Additionally, RT and its variability have been shown to provide information about the allocation of attentional resources in those with neurological insult such as mTBI. Specifically, it is thought that attention allocation can be measured by RT latency in healthy controls, whereas in those with mTBI attention allocation is more related to RT variability than RT latency (Bleiberg et al., Reference Bleiberg, Garmoe, Halpern, Reeves and Nadler1997; Segalowitz et al., Reference Segalowitz, Dywan and Unsal1997). As such, the current finding of greater IIV in ANAM4 SRT performance, with increasing IIV across trials, in an acute mTBI group provides additional evidence to the body of literature.
In general, these results reveal greater trial-to-trial fluctuations in performance for the mTBI group as compared to the CTRL group. Based on the central tendency theory, these fluctuations are often viewed as noise, instability, or error. However, they may be indicative of low scores due to the acute effects of concussion that may otherwise be missed by more traditional metrics. That is, analyses of variability in raw RT and ISD of RT trials appear to be a potentially valuable alternative metric for NCATs. That is, these alternative metrics may offer greater clinical utility than metrics commonly used in cognitive testing. Given the computerized nature of NCATs, metrics such as raw RT and RT ISD can be more quickly and feasibly calculated. However, inclusion of these score calculations would require changes to the NCAT’s scoring output, as otherwise the responsibility to calculate would fall on the clinician.
Additionally, norms with clinically meaningful cut points would need to be established before such metrics could be applied to clinical decision making. Lastly and of note, ANAM4 presents a conceivable advantage over other NCATs by including a repeated simple reaction time test, allowing comparison of RT and RT variability across time although still within one testing session, potentially tapping into “cognitive fatigue.” Other NCATs may benefit from a repeated RT subtest in their battery.
Limitations
The current study was derived from data from a larger study, and, therefore, procedures not relevant to the current analyses surrounded the collection of the data used in this study. These procedures sometimes included other testing before the ANAM4, which could have increased fatigue. However, any potential fatigue would be relatively equitable across groups and relatively controlled for by comparing SRT2 to SRT1, which occurred within the same testing session. Additionally, recent studies demonstrated that, when administering multiple NCATs in one session, performance was not affected by the order of administration (Cole, Arrieux, Dennison, et al., Reference Cole, Arrieux, Dennison and Ivins2017; Nelson et al., Reference Nelson, LaRoche, Pfaller, Lerner, Hammeke, Randolph and McCrea2016).
The sample included in the current analyses was primarily male and mostly enlisted, especially in the mTBI group; moreover the CTRL group was not a matched CTRL group. We attempted to control for potential differences in group composition in our analyses. However, as with any study involving cognitive assessment, there are many additional factors that could potentially impact testing results, and specifically IIV, for which we did not control. Such factors include premorbid functioning, sleep, emotional status (e.g., acute stress reaction or post-traumatic symptoms), the nature of injury, time since injury, ongoing symptomatology, potential medication with cognitive side effects (e.g., stimulants or sedatives), among others.
We believed it was beyond the scope of the current study to attempt to control for the innumerable confounding variables. Additionally, as with all studies of NCATs, there are technological and environmental considerations that could impact testing results, such as the hardware and software configurations and the testing environment. All efforts were taken to administer the tests with a computer platform as close to the ANAM4 manual specifications. Additionally, testing was done in a quiet room with a trained test proctor, in an environment similar to how baseline or post-injury testing would likely occur, likely rendering the results ecologically valid despite the potential for other sources of error.
CONCLUSIONS AND FUTURE DIRECTIONS
The results from this study support a small but growing body of literature that raw RT scores and RT variability may be much more sensitive to the cognitive decline seen during the acute period after concussion. The findings suggest mTBI participants can temporarily perform similarly to normal controls on RT latency, but repeated RT assessments at multiple time points throughout a battery demonstrate a lack of improvement in RT and increased variability. Interpreting these metrics rather than the traditionally reported standardized scores (e.g., throughput), where effects may otherwise be “washed out, appears to hold promise for the use of ANAM4 in acute concussion populations.
However, additional work is needed to fully clarify the clinical utility (e.g., diagnostic and prognostic capabilities) of these metrics and to determine if they do indeed offer advantages over traditional metrics obtained from traditional NP tests and NCATs, especially when controlling for other potential confounding variables. There is some existing evidence that shorter ANAM4 SRT is predictive of recovery in those acutely concussed (Norris, Carr, Herzig, Labrie, & Sams, Reference Norris, Carr, Herzig, Labrie and Sams2013). Thus, it may be that improved (i.e., faster) raw RT from SRT1 to SRT2, less RT variability within both subtests, and stable RT variability across SRT1 and SRT2 could be predictive of faster and/ or better recovery after concussion and, therefore, incorporated into return to duty or return to play decisions. Given the military and many sports leagues’ baseline testing procedures, it will also be important to determine if baseline assessments are valuable with regard to such metrics for diagnostic and prognostic purposes.
ACKNOWLEDGMENTS
The authors thank Karen Schwab, Brian Ivins, Felicia Qashu, Mary Alice Dale, James Wes McGee, Katie Toll, and Alex Fender for their contributions to this research. This material is published by permission of the Defense and Veterans Brain Injury Center, operated by General Dynamics Information Technology for the U.S. Defense Health Agency under Contract No.W91YTZ-13-C-0015. The authors have no financial interests to disclose. The views expressed in this manuscript are those of the author(s) and do not necessarily reflect the official policy or position of the Department of the Navy, Department of the Army, Department of Defense, or the U.S. Government.
