Introduction
The incidence of sports-related concussion at all levels is an ongoing concern. Most cases of concussion are expected to have transient effects, but there remains potential for repeated brain injuries to have longer term neuropsychiatric sequelae (Noble & Hesdorffer Reference Noble and Hesdorffer2013). Despite efforts directed at primary prevention, concussion will occur and practical and valid approaches to management are necessary (Harmon et al. Reference Harmon, Drezner, Gammons, Guskiewicz, Halstead, Herring and Roberts2013). The current approach to managing the concussed athlete mandates a multifaceted assessment protocol (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013). This includes diagnosis using neurological assessment including mental status examination and symptom reporting. Following a diagnosis of concussion, athletes should have a period of rest and graduated return to participation. Altered cognition and its measurement had previously been referred to as the ‘cornerstone’ for objectively monitoring effects of concussion using individualised baseline testing for all athletes (McCrory et al., Reference McCrory, Meeuwisse, Johnston, Dvorak, Aubry, Molloy and Cantu2009). Following the most recent Zurich meeting the potential role of NP testing was significantly reduced (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013). The Zurich group recommended that all concussed athletes undergo clinical neurological assessment, which can include computerised NP testing, with formal NP testing reserved for more exceptional cases. It is unclear when formal NP testing should be considered, with suggestions that such assessment may play a role in managing athletes with persistent symptoms (>10 days) (Makdissi, Cantu, Johnston, McCrory, & Meeuwisse, Reference Makdissi, Cantu, Johnston, McCrory and Meewisse2013). The primary role of NP testing in concussion management has been to assist in return-to-play decisions, and should be utilised in conjunction with a comprehensive medical assessment (Echemendia et al. Reference Echemendia, Iverson, McCrea, Broshek, Gioia, Sautter and Barr2012b). It was recognised that many studies have demonstrated short-term deficits on both traditional and computerised NP protocols. It has been argued that at present, there is insufficient evidence to recommend the widespread routine use of baseline NP testing as it has not been shown to alter outcome (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013; Randolph & Kirkwood, Reference Randolph and Kirkwood2009; Randolph, Reference Randolph2011). However, there is little evidence for the use of NP tests in concussion management in the absence of baseline testing.
Barth and coworkers pioneered the use of baseline NP assessment of athletes with serial retesting following concussion (Macciocchi, Barth, Alves, Rimel, & Jane, Reference Macciocchi, Barth, Alves, Rimel and Jane1996). This has been referred to as the baseline paradigm and has been extensively replicated across many athletic groups with various testing protocols (Echemendia et al., Reference Echemendia, Iverson, McCrea, Broshek, Gioia, Sautter and Barr2012b; Karr, Areshenkoff, & Garcia-Barrera, Reference Karr, Areshenkoff and Garcia-Barrera2014). Analysis can be performed as a within-subjects or mixed design, with baseline and post-injury performance as levels of the repeated factor. Yet obtaining baseline data in a large at-risk population of athletes is a time, labour, and resources intensive methodology. In contrast, if an athlete were to be tested for the first time following concussion their scores would be compared to a control or normative group (herein referred to as the normative paradigm). The normative paradigm has been more extensively used in many other fields where baseline or pre-insult testing is not feasible, such as general population TBI. As the individual is compared to a group, this is a between-subjects methodology. This is clearly a more economical approach, yet in fact has not been well investigated in the sports-related concussion literature. From a broader analytic perspective, the baseline paradigm implements a repeated-measures design, which statistically removes variance due to individual differences. Consequently the error term is reduced, making it more powerful or sensitive to differences between scores (Howell, Reference Howell2010). The power of the within-subjects design relies on the correlation between pre and post scores. The normative paradigm attempts to reduce the effect of individual differences by comparing an individual to a group of suitably similar individuals not exposed to the event of interest, in this case concussion. Nonetheless, when two sets of scores are correlated the within-subjects analysis will be more sensitive than the between-subjects analysis, all other factors being equal (Howell, Reference Howell2010).
To date, three studies have sought to compare a baseline paradigm to a normative paradigm in setting of sports-related concussion, and brief consideration to each is given. Schmidt, Register-Mihalik, Mihalik, Kerr, & Guskiewicz (Reference Schmidt, Register-Mihalik, Mihalik, Kerr and Guskiewicz2012) presented retest data for 132 concussed and 38 controls athletes on the Automated Neuropsychological Assessment Metrics (ANAM), Sensory Organization Test, and Graded Symptom Checklist. A repeated measures design was used with all athletes completing baseline assessment and subsequent retesting within 10 days post concussion or two weeks to four months in controls. Under the baseline paradigm concussed athletes were compared to their pre-injury score using a reliable change (RC) index correcting for mean practice effects seen in the control group (Iverson, Lovell, & Collins, Reference Iverson, Lovell and Collins2003). An 80% confidence interval was employed to determine impairment, equivalent to a cut-score of Z ± 1.282. The normative paradigm was analysed using the same RC index as the baseline paradigm. Rather than using the individual's baseline score, the mean from a normative group of 673 athletes was used as the comparator. The paradigms differed operationally only in the pre-injury benchmark, either the individual's score for the baseline paradigm, or the normative mean for the normative paradigm. Using McNemar's measure of agreement between related ratings, paradigms were reported as concordant. The normative paradigm was more sensitive on only one subtest (Mathematical Processing). Echemendia et al. (Reference Echemendia, Bruce, Bailey, Sanders, Arnett and Vargas2012a) retested 223 athletes pre- and post-concussion on the ImPACT protocol, most within one week of injury. Previously published two-year retest data from 95 uninjured athletes served as control data (Schatz, Reference Schatz2010). RC scores for the concussed group were calculated using the Jacobson and Truax (JT) and Gulliksen-Lord-Novick (GLN) models. The baseline paradigm was assessed using both RC indices with a 90% confidence interval, equivalent to a standardised cut-score Z ± 1.65. The normative paradigm was assessed using only baseline control data, and a simple standard Z score with cut-offs set at ±1 and ±1.5 standard deviations, yielding 68% and 87% confidence intervals respectively. The GLN model was more sensitive than the JT model on all scales. In comparing both RC models (baseline paradigm) to both Z scores (normative paradigm), sensitivity levels were marginally better than chance. Using the Kappa statistic, the GLN model demonstrated better agreement than the JT model with both normative cut-offs. The agreement between RC models and standard score models was interpreted as evidence that the baseline paradigm may be replaced by the normative paradigm.
Most recently, Louey et al. (Reference Louey, Cromer, Schembri, Darby, Maruff, Makdissi and McCrory2014) accounted for many of the shortcomings of the preceding studies. A sample of 29 symptomatic concussed athletes was assessed on the CogSport protocol. This group was compared to a 260 uninjured controls tested one year apart. The baseline paradigm utilised the RC model of Crawford and Howell (Reference Crawford and Howell1998), a regression based model that uses an individualised error term. The normative paradigm compared the concussed athlete to the baseline control data using a standard Z score. A uniform 90% CI was employed across all analyses. Chi-square analysis (without Yates correction) suggested that the baseline paradigm was more sensitive on only one of the four CogSport subtests. Yet, when considering classification of individual athletes the baseline paradigm was more sensitive than the normative paradigm, with comparable specificity levels. From the non-sporting literature, Roebuck-Spencer, Vincent, Schlegel, and Gilliland (Reference Roebuck-Spencer, Vincent, Schlegel and Gilliland2013) compared the baseline and normative paradigms in a large sample (n = 8002) of non-injured military personnel on the ANAM. Whilst they found that the overall false-positive rate was comparable for both paradigms, they demonstrated that 65.7% of individuals classified as atypical using the normative paradigm showed no change under the baseline paradigm. They went on to conclude that the baseline performance is important for minimising false-positive error.
Across the three sports-related concussion studies there are a number of conceptual, analytic, and interpretive issues to consider. First, the sensitivity of test protocols was poor for the ANAM (Schmidt et al., Reference Schmidt, Register-Mihalik, Mihalik, Kerr and Guskiewicz2012) and ImPACT (Echemendia et al., Reference Echemendia, Bruce, Bailey, Sanders, Arnett and Vargas2012a), approximating chance on many occasions. It was correctly noted by Louey et al. (Reference Louey, Cromer, Schembri, Darby, Maruff, Makdissi and McCrory2014) that when sensitivity is poor, it is likely that the two paradigms could appear equivalent. In order to improve sensitivity, Louey et al. used symptomatic concussed athletes on the CogSport protocol. It is difficult to isolate what factors contributed to the improved sensitivity aside from using symptomatic athletes. Reliability of CogSport in the control sample was quite good, compared with the low to moderate estimates seen in the ANAM and ImPACT protocols (Resch, McCrea, & Cullum, Reference Resch, McCrea and Cullum2013; Schatz, Reference Schatz2010). Reduced reliability is known to reduce test sensitivity (Cohen, Swerdlik, & Sturman, Reference Cohen, Swerdlik and Sturman2012). Second, the confidence levels (or corresponding type 1 error rate) were not equivalent in the Echemendia, Bruce et al. study, with the baseline paradigm assessed at 90%, and the normative paradigm at 68% and 87%. The Schmidt et al. study used a uniform 80% confidence interval, whereas the Louey study used a more typical 90% confidence interval. An equivalent significance or confidence level is essential to derive valid comparisons. Third, the operationalisation of the normative model was potentially flawed in the Schmidt et al. study, where the control mean was used as the comparator in a RC model. RC models have been derived to compare an individual's scores to themselves (Hinton-Bayre, Reference Hinton-Bayre2010). As such, it is unclear whether the distribution tested under the normative paradigm in the Schmidt et al. study is valid. Moreover, the Echemendia et al. (2012a) study used two RC models (Jacobson & Truax, Reference Jacobson and Truax1991; Speer, Reference Speer1992) neither of which account for practice effects. Examination of the control ImPACT data suggested one scale demonstrated a significant mean practice effect (Visual Motor), and two scales demonstrated significant differential practice (Visual Motor & Reaction Time) (Schatz, Reference Schatz2010). Differential practice refers to a statistically significant difference between the standard deviation of pretest and post-test scores (Maassen, Reference Maassen2004). Moreover, the normative paradigm had post-concussion scores compared to baseline control data, the interpretation of this is compromised in the presence of practice effects. The Louey et al. study implemented the RC attributed to Crawford and Howell (Reference Crawford and Howell1998), which will be less sensitive than the RC McSweeny (Hinton-Bayre, Reference Hinton-Bayre2010). Fourth, the analytic approach to determining equivalence has been based on dichotomous data, impaired or not. Each of the three studies has approached this differently. The Schmidt et al. study used the McNemar test for matched pairs, which was designed for comparing dichotomous outcomes in paired samples. The Echemendia study used the Kappa coefficient. Both of these techniques measure degree of agreement, and test against chance as the null hypothesis. In the present setting, establishing a difference between the paradigms is arguably more important if wishing to determine diagnostic accuracy. To this end, the Louey et al. study employed a Chi-square analysis, which does make assumptions of independence of variables (Howell Reference Howell2010). When comparing baseline to normative paradigms, the data were derived from the same sample, which may compromise the assumption of independence.
Considering the Schmidt et al. (Reference Schmidt, Register-Mihalik, Mihalik, Kerr and Guskiewicz2012) and Echemendia et al. (Reference Echemendia, Bruce, Bailey, Sanders, Arnett and Vargas2012a) studies, the methodology employed appeared to suffer from poor test–retest reliability, poor sensitivity, unaccounted practice effects, inconsistent confidence intervals, and non-comparable or unsubstantiated analytical techniques. The suggestion that the normative paradigm was comparable to the baseline model requires further clarification. The Louey et al. (Reference Louey, Cromer, Schembri, Darby, Maruff, Makdissi and McCrory2014) study with greater analytic rigor and sensitivity clearly demonstrated that the baseline paradigm was more sensitive than the normative paradigm, particularly when focused on determining impairment in individual athletes. One of the unique elements of NP assessment in sports-related concussion is the potential to collect baseline data. This paradigm was widely adopted as it was thought to provide an individualised and thus more sensitive approach to the detection of change post-trauma. If athlete safety is paramount, then rigorous research is required before abandoning the well-established baseline paradigm. The normative paradigm is attractive as it is far less time and resource intensive, but does not yet have the research base to establish its use in sports-related concussion. If a potentially less sensitive, and therefore potentially more risky, paradigm is to be adopted then a clear understanding of its limitations is required. In this vein, a recent study evaluated the diagnostic accuracy of the ImPACT and CogSport in concussed athletes who had not undergone baseline testing (Gardner, Shores, Batchelor, & Honan, Reference Gardner, Shores, Batchelor and Honan2012). Using logistic regression, NP test scores did not significantly add to the prediction of classification (concussed versus control) after demographic variables had been entered, thus offering limited support for the normative paradigm.
The present study aimed to further compare the baseline and normative paradigms in relation to cognitive performance following sports-related concussion. While previous studies have focused on computerised protocols, the present study implemented more traditional paper-and-pencil tests. Bearing in mind, the Zurich consensus statement (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013) suggested formal NP testing may be indicated in certain circumstances, which might involve the types of measures examined in this study. The present study will also compare outcomes using more stringent and relaxed confidence intervals to assess potential impact on test results. Moreover, outcome will be examined on a continuum (standard scores) as well as a dichotomy (impaired versus not) to provide a potentially more sensitive evaluation of paradigm differences.
Method
Participants
A group of 194 professional male rugby league players were assessed preseason. Twenty-seven players were diagnosed with concussion during the competitive season and were subsequently retested within 24–48 hour of sustaining a concussion. Concussions were diagnosed and referred by a certified sports medicine physician. A group of 26 uninjured rugby players volunteered as controls and were retested over a similar time frame. The groups were comparable in terms of age (Concussed M = 21.2, SD = 3.5; Controls M = 19.9, SD = 2.5), estimated full scale IQ (based on National Adult Reading Test – Revised – errors; Concussed M = 94.3, SD = 7.0; Controls M = 94.8, SD = 8.2), years of educations (Concussed M = 12.2, SD = 1.3; Controls M = 12.1, SD = 1.0) self-reported number of previous concussions (Concussed M = 2.0, SD = 1.6; Controls M = 2.5, SD = 1.6), ethnicity (Indigenous/Islander; Concussed 22%, Controls 19%; the remainder identified themselves as Caucasian), and interval in weeks between baseline and retest (Concussed M = 19.2, SD = 21.6; Controls M = 28.3, SD = 13.5). There was no reported history of learning disability, psychiatric illness, or significant medical condition in either group. All athletes spoke English as their first language.
Procedures and Outcome Measures
Athletes were tested prior to the competitive season in an office environment by trained post-graduate psychology students. Standardised administration procedures were employed to ensure consistency of assessment. At each assessment, all athletes completed a protocol of paper-and-pencil NP measures including the Stroop A (reading colour mismatched words), Stroop B (naming the colour of mismatched colour words), Speed of Comprehension test (SOCT; number of sentences correctly identified as sensible or not), Symbol Digit Modalities test (SDMT; number of symbols correctly transcribed to digits), and Digit Symbol Substitution tests (DSST; number of digits correctly transcribed to symbols). At baseline and retesting one of four standardised alternate forms was used in a counter-balanced manner, with a different alternate form used on retesting. These measures were selected based on research suggesting that information processing speed and executive functioning was most reliably affected by mild traumatic brain injury and the need for brevity of assessment (for a contemporary review, see Karr et al. Reference Karr, Areshenkoff and Garcia-Barrera2014). This study is a re-analysis of an existing data set previously described in comparative analysis of RCI models in sports-related concussion (Hinton-Bayre, Reference Hinton-Bayre2012).
Analysis
The baseline paradigm was assessed using the standard regression based RC index (RCSRB; McSweeny, Naugle, Chelune, & Luders, Reference McSweeny, Naugle, Chelune and Luders1993), which accounts for mean practice, differential practice, and adjusts for imperfect test–retest reliability. Mean practice refers to a significant change in mean scores in the control group from baseline to retest. Differential practice refers to a significant difference in baseline and retest standard deviations. The RCSRB uses linear regression to predict a post-injury score (Y′) from a baseline score (X). The difference between the actual (Y) and expected (Y′) retest scores is then divided by the standard error of estimate (SEE). The RCSRB score is interpreted as a standard score (Z or t). The normative paradigm will be assessed by comparing the post injury score of the concussed athletes to the retest mean for the controls (MY ), to account for prior exposure. A standard score will be derived by dividing the deviation score (Y-MY ) by the retest control standard deviation (SY ). Confidence levels were set at 90% (Z ± 1.645) and 68% (Z ± 1.0). Agreement between baseline and normative paradigms will be assessed through ICCs separately for 90% and 68% intervals. A two-way fixed effects ICC for agreement on single scores considering systematic differences was used (model 2, 1; Weir Reference Weir2005). Comparative diagnostic accuracy of baseline and normative paradigms for each test was examined with Receiver Operating Characteristic (ROC) curves.
Results
Baseline Comparisons
Groups did not significantly differ on preseason assessment for any of these measures using independent groups t-tests; Stroop A t(51) = 1.76, Stroop B t(51) = 1.85, SOCT t(51) = 1.36, SDMT t(51) = 0.17, DSST t(51) = 0.01. However, statistical power to detect differences between small samples is limited, and comparability of samples is important for both paradigms. The baseline test scores of the concussed athletes were thus standardised according to the control baseline MX and SX . Given that impairment for the normative paradigm will subsequently be assessed at –1SD, the percentage of concussed athletes below this cut-off at baseline should approximate 16%. By this criterion, the percentage of ‘unusual’ baseline values for concussed and control athletes were as follows: Stroop A (7.4% & 11.5%), Stroop B (0% & 15.4%), SOCT (7.4% & 15.4%), SDMT (29.6% & 19.2%), and DSST (22.2% & 15.4%). None of these proportions were statistically different, with Stroop B only approaching significance (Fisher's exact, p = 0.051). Using a z-score of Z ≤ 1.65, the proportion of individuals at baseline exceeding the value did not differ between concussed and control athletes.
Test–retest Analysis
There were significant mean practice effects for Stroop B and DSST in the control data (Hinton-Bayre, Reference Hinton-Bayre2012). Significant differential practice (SX≠SY ) was observed for three measures, with increased variance on retesting for Stroop A, and decreased variance on Stroop B and SOCT. Test–retest reliability coefficients were calculated through Pearson's r; Stroop A (0.76), Stroop B (0.86), SOCT (0.90), SDMT (0.59), and DSST (0.65).
Normative versus Baseline Paradigms
Table 1 presents sensitivity estimates (Percentage of concussed athletes showing impairment) for each scale based on 68% and 90% confidence intervals for both the Normative and Baseline paradigms. As expected, sensitivity was consistently greater using the 68% versus the 90% confidence interval for all scales. Using a 90% confidence interval, although more concussed individuals appeared to be classified as impaired under the baseline paradigm compared to the normative paradigm, these proportions were not significantly different using Fisher's exact test. A similar pattern of results was seen when classifying impairment using a 68% confidence interval.
Concussed athletes demonstrating significant impairment following injury (%, n = 27)
aEquivalent to 16% two-tail, or ± 1 SD, bequivalent to 5% two-tail, or ± 1.65 SD.
CI = confidence interval.
Percentages presented reflect those demonstrating significant impairment.
Rather than comparing dichotomous data, the continuous data from which it was derived was analysed. The standard scores from the baseline (RCSRB) and normative (Z) paradigms were compared (see Table 2). Using repeated measures t-tests, the baseline paradigm was significantly more sensitive on average than the normative paradigm, i.e., more negative standard scores, on four of the five scales and a composite score, with the final scale (SDMT) trending in the same direction. ICC (2, 1) coefficients suggested inadequate agreement on two measures (Stroop A, SOCT), but reasonable agreement on the other three measures (Stroop B, SDMT, DSST). Of note, reasonable agreement did not preclude the finding of a statistically significant difference between models (e.g., Stroop B & DSST).
Descriptive and inferential analyses of difference and agreement for concussed athletes for normative and baseline paradigms
*p < 0.05, ***p < 0.01 two-tailed.
RM = repeated measures, ICC = intra-class correlation.
To better evaluate the overall classification accuracy of the paradigms, ROC curve analyses comparing area under the curve were performed (see Table 3). There was no difference in area under the curve for Stroop A and B, SDMT, or DSST. The SOCT and a Composite score (average of the five scales) did show a significant difference, with better overall accuracy for the baseline paradigm observed. Figure 1 demonstrates the ROC curves for both paradigms for the composite score.
Receiver operator characteristic (ROC) area under the curve (AUC) analyses for normative versus baseline performance
AUC = area under the curve, CI = confidence interval, SE = standard error.
*p < 0.05, ***p < 0.001.
ROC curve of neuropsychological composite scores for Baseline and Normative paradigms.
Discussion
NP assessment has been considered an important component in the management of sports-related concussion. The use of baseline data is possible in the sports setting and provides an estimate of an individual's ability prior to concussion. However, the time and cost to obtain such data makes assessment unattainable for all but collegiate and professional level competition. As such, attempting to standardise and validate normative assessment protocols is a worthy objective. The present study re-examined an existing data set to determine whether a normative paradigm could be comparable to a baseline paradigms, being mindful of the sensitivity/specificity trade off that occurs when using different confidence intervals. The more common baseline paradigm was assessed using the RCSRB which tends to be the more sensitive of contemporary RC indices (Hinton-Bayre, Reference Hinton-Bayre2012).
The normative paradigm was indexed through comparing post-concussion scores to retest control parameters to yield a Z score. When using a dichotomous outcome of ‘impaired’ versus ‘not impaired’ the differences between paradigms was not significant for any of the measures irrespective of confidence interval. However, when using the continuous data the baseline paradigm was significantly more sensitive than the normative paradigm for four of the five measures. Overall classification accuracy was better for the baseline paradigm for one measure (SOCT) and the Composite score.
From a methodological perspective, a within-subjects methodology, such as that used by the baseline paradigm, will be more sensitive than the between-subjects methodology of the normative paradigm (Howell, Reference Howell2010). Nonetheless, the within-subjects methodology does require careful control of the potential confounding influence of practice effects. Practice effects were seen even in the control data despite use of alternate forms. The relatively short retest interval provided variable reliability coefficients. Longer retest intervals have been associated with a reduction in practice effects, but often with a reduction in reliability (Echemendia et al., Reference Echemendia, Bruce, Bailey, Sanders, Arnett and Vargas2012a; Elbin, Schatz, & Covassin, Reference Elbin, Schatz and Covassin2011). Statistically controlling for practice is far less compromising for sensitivity than adjusting for poor reliability. Any change measure needs to consider the effects of practice, as the RCSRB did in the present study. RC indices have only been validated with the baseline paradigm, not the normative paradigm. The error estimates were based on the likely error in measurement in an individual score, not a group mean, as such, an undefined distribution was considered by Schmidt et al. (Reference Schmidt, Register-Mihalik, Mihalik, Kerr and Guskiewicz2012). The present data set has demonstrated the importance of concussed athletes matching the control or normative group. As exemplified on the SDMT, group based comparisons may be nonsignificant, particularly in smaller sample sizes. Of the eight athletes found to be ‘impaired’ following concussion four would have been classified as ‘impaired’ before their injury. In the absence of baseline scores without adequate control matching, there is considerable potential to overestimate or underestimate the effects of concussion.
A central limitation to the present study also affects all previous studies comparing baseline to normative paradigms (Echemendia et al., Reference Echemendia, Bruce, Bailey, Sanders, Arnett and Vargas2012a; Louey et al., Reference Louey, Cromer, Schembri, Darby, Maruff, Makdissi and McCrory2014; Roebuck-Spencer et al., Reference Roebuck-Spencer, Vincent, Schlegel and Gilliland2013; Schmidt et al., Reference Schmidt, Register-Mihalik, Mihalik, Kerr and Guskiewicz2012). The present and prior studies have all utilised a repeated-measures design, with concussed and control athletes completing baseline and post-concussion assessments. To assess the normative paradigm, only post-concussion data was examined. Interestingly, prior studies compared the post-concussion score to the baseline scores of the normative sample. In the presence of practice effects, this might underestimate the sensitivity of the normative paradigm. Under the true normative paradigm, a concussed athlete's first assessment should occur after injury, and controls should also be test naive at initial assessment, as this reflects the actual clinical situation. Obviously in this situation, one cannot assess the normative and baseline paradigms in the same cohort. If separate cohorts are then used for each paradigm, questions regarding the equality of concussed individuals will always make interpretation problematic. Therefore, approaches such as those explored in the present study offer some evidence to assist in the validation of NP assessment of sports concussion.
The present study is limited by the small sample size for both concussed and control athletes. Prior studies comparing paradigms have had either small concussed samples (Louey et al., Reference Louey, Cromer, Schembri, Darby, Maruff, Makdissi and McCrory2014) or small control samples (Schmidt et al., Reference Schmidt, Register-Mihalik, Mihalik, Kerr and Guskiewicz2012), or a large non-injured sample of military personnel (Roebuck-Spencer et al., Reference Roebuck-Spencer, Vincent, Schlegel and Gilliland2013). Prior studies have focused on computerised assessment protocols (e.g., ANAM, Cogsport, and ImPACT), which have been favoured over paper-and-pencil assessment for the large scaled baseline assessments required at the collegiate and professional level. Aside from the reliability issues noted with these protocols, as mentioned earlier, formal NP assessment still relies on more traditional testing methods. In more complicated cases of concussion (repeated trauma, protracted symptoms) or more severe TBI in an athlete more formal assessment should be considered (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013). There is at present no clear guideline as to what testing protocol should be used in such situations, let alone whether general or athlete-specific norms should be used. Concordantly, there is still a role for the validation and standardisation of more traditional non-computerised testing protocols in sports-related concussion.
The principles considered in the present study with respects to baseline versus normative paradigms extend beyond the testing protocol employed. Practice effects, reliability, and the relevance of reliable change apply to all forms of assessment measured on a continuum, including traditional NP testing, computerised NP testing, balance testing and symptom checklists which are also used in concussion assessment. In other areas of human health normative paradigms for the assessment of abnormality are well defined where deviations from ‘normal’ are associated with physiologic sequelae (e.g., hyperglycaemia, hypertension, reduced cardiac ejection fraction). However, with respect to cognitive performance where the variability of what is considered normal is large, extensive efforts have been made to evaluate base-rates of ‘abnormal’ performance and estimate of pre-morbid functioning, as has been seen in the Wechsler intelligence and memory scales (Holdnack, Drozdick, Weiss, & Iverson, Reference Holdnak, Drozdick, Weiss and Iverson2013). It seems somewhat incongruous to abandon the baseline paradigm for cognitive assessment following sports-related concussion, based on the argument that there is no evidence that it influences outcome. Yet, it has been argued that there is limited evidence that concussion management protocols influence important outcomes, such as reducing or preventing repeated concussion, impaired athletic performance, impaired scholastic or occupational functioning, chronic neuropsychiatric problems, or catastrophic outcome (Harmon et al. Reference Harmon, Drezner, Gammons, Guskiewicz, Halstead, Herring and Roberts2013). This should not be seen as a need to abandon the process, more as a reminder that more evidence is required.
The discussion thus far has centred on comparison of paradigms in the detection of impairment following sports-related concussion. If a baseline paradigm were to be adopted, this would have implications for the assessment of recovery of cognitive function. Once a concussed athlete has demonstrated impairment, recovery should ideally be demonstrated prior to return to participation. Under the baseline paradigm, this has been achieved by showing that the retest score is not significantly different to an adjusted baseline score. It is important to recognise that the actual baseline score is only reasonable to use if there is no concern regarding practice effect or regression to the mean. The adjusted baseline score is the predicted retest score to which the actual retest score is compared in RC indices. This method works only if the control and concussed athletes are comparable pre-injury. Under a normative paradigm, the concussed athlete's performance is below the normative group. Retest in the short term (usually days to weeks) will likely see practice effects on retesting, necessitating retest normative data. Recovery of function under the normative paradigm has also not been validated in sports concussion, and requires further consideration. For example, is a shift from ‘impaired’ to ‘normal’ category sufficient? What if the concussed athlete falls just in the ‘impaired’ range (e.g., Z = −1.67), but on retesting has a score just within the ‘normal’ range (e.g., Z = −1.60)? Should a significant improvement in scores also be demonstrated? Moreover, if large scale short-interval retest normative data are collected for normative paradigm, what comparison data should be used for a concussed athlete who has participated in the retest normative data collection and thus effectively has baseline data? Clearly, a baseline approach would be sensible for the assessment of this individual's post-concussion score.
Given the complications highlighted regarding cognitive assessment in sports-related concussion, it is a poignant time to consider that such assessment forms only one component in the overall management of the injured athlete and that cognitive testing should not be the sole determinant on whether to withhold or return an athlete to participation. In this regard, a graduated return to participation has been advocated (Makdissi et al., 2013). Once a diagnosis of concussion has been established, key determinants of fitness to play has centred on absence of symptoms at rest and subsequently with progressive exertion. Such symptoms can include headache, memory loss, attention difficulties, fatigue, and other non-specific neurologic symptoms. Indeed, it has been advocated that NP assessment only be conducted once symptoms attributed to concussion have resolved. Moreover, recent sports medicine literature has suggested that NP testing only need be reserved for more complicated cases of concussion (McCrory et al., Reference McCrory, Meeuwisse, Aubry, Cantu, Dvorak, Echemendia and Turner2013). The role of NP assessment of concussion in sport continues to be controversial.
The present paper sought to compare normative versus baseline NP assessment in the sport-related concussion. Using previously published data, it was shown that the baseline paradigm was more sensitive to the effects of concussion when comparable confidence intervals were used. While not using test naive athletes to assess impairment in the normative paradigm, it was shown that agreement with the baseline paradigm was at times good. However, a statistically more negative score was seen with the baseline compared to the normative paradigm. A between-subjects comparison will always be less sensitive than a within-subjects comparison when retest scores are correlated and type error rates are comparable. When baseline data collection is not available or feasible, a normative paradigm can be considered. It might also then be considered that a more relaxed type error rate be employed to improve sensitivity, being mindful that some athletes will possibly be unnecessarily withheld from participation. A RC calculator in the form of a spreadsheet is available from the author on request.
