INTRODUCTION
The NIH Toolbox (NIHTB) for Assessment of Neurological and Behavioral Function was commissioned as part of the NIH Blueprint for Neuroscience Research initiative to provide a common metric among outcome measures in clinical research in the neurosciences (Gershon et al., Reference Gershon, Wagster, Hendrie, Fox, Cook and Nowinski2013). The NIHTB Cognition Battery (NIHTB-CB) is a set of computer-based tests of cognitive functions selected based on their relevance to daily functioning and important health outcomes across the lifespan. The computer-based format offers brief yet comprehensive assessment of multiple domains that are typically included in a neuropsychological test battery, relative ease of administration and scoring, testing in English or Spanish, normative data for individuals ages 3–85 years, and increased sensitivity to the full range of functioning and minimization of ceiling and floor effects (Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013).
Performance on the individual tests of the NIHTB-CB is summarized into composite scores representing fluid and crystallized cognitive abilities. The fluid cognition composite includes tests of cognitive abilities that are sensitive to aging and neurologic disease (e.g., processing speed, memory, executive functioning), while the crystallized cognition composite is composed of tests that are similar to existing measures of premorbid intellectual functioning (e.g., oral reading recognition), where performance is expected to remain relatively stable across the lifespan (Mungas et al., Reference Mungas, Heaton, Tulsky, Zelazo, Slotkin, Blitz and Gershon2014). Although the NIHTB-CB was not developed as a clinical or diagnostic tool, the co-normed crystallized composite may provide an individualized estimate of premorbid ability to which one’s fluid abilities may be compared, aiding in the clinical determination of cognitive decline in the presence of neurodegenerative disease or acquired brain injury (Holdnack et al., Reference Holdnack, Tulsky, Slotkin, Tyner, Gershon, Iverson and Heinemann2017).
The advantages that the NIHTB-CB offers over traditional neuropsychological test batteries has led to increased interest in its clinical utility and validity in a number of clinical populations including stroke, traumatic brain injury, and spinal cord injury (e.g., Carlozzi et al., Reference Carlozzi, Tulsky, Wolf, Goodnight, Heaton, Casaletto and Heinemann2017; Holdnack et al., Reference Holdnack, Tulsky, Slotkin, Tyner, Gershon, Iverson and Heinemann2017; Tulsky et al., Reference Tulsky, Carlozzi, Holdnack, Heaton, Wong, Goldsmith and Heinemann2017; see a special issue of Rehabilitation Psychology, 2017, volume 62, issue 4). These studies supported the construct validity of the NIHTB-CB in rehabilitation settings and demonstrated its ability to discriminate acquired brain injury from controls, concluding that the NIHTB-CB may be useful clinically in a screening capacity. Moderate associations with neuropsychological tests of preclinical Alzheimer’s disease (Buckley et al., Reference Buckley, Sparks, Papp, Dekhtyar, Martin, Burnham and Rentz2017) and hippocampal volumes (O’Shea et al., Reference O’Shea, Cohen, Porges, Nissim and Woods2016) have been reported and interpreted as preliminary support for its clinical utility with older adults. In a recent investigation of the NIHTB-CB in a memory clinic setting, Hackett and colleagues (Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018) found that the NIHTB-CB could differentiate cognitively normal adults from those with Alzheimer’s disease dementia, but not from individuals with mild cognitive impairment. Although the authors acknowledge that the NIHTB-CB was not designed to replace comprehensive neuropsychological evaluations, they concluded that it may represent a “key part of the clinical diagnostic evaluation” in memory clinics. Additional studies have established the utility of the NIHTB-CB in neurosurgical evaluations for deep brain stimulation (Loring et al., 2018), cancer treatment (Sinhaet al., Reference Sinha, Wong, Kallogjeri and Piccirillo2018), and brain tumor resection (Lang et al., Reference Lang, Cadeaux, Opoku-Darko, Gaxiola-Valdez, Partlo, Goodyear and Kelly2017), the latter study concluding that the NIHTB-CB represents a “feasible alternative to current neuropsychological batteries,” citing length, cost, and variability between test batteries as barriers to traditional neuropsychological evaluations.
Although these studies demonstrate feasibility and criterion validity, more research is needed prior to assuming the equivalence of NIHTB-CB to established measures, and therefore, its clinical utility. That is, if the NIHTB-CB is to be used in clinical practice alongside or interchangeably with traditional neuropsychological tests, it is necessary to examine how closely these measures agree before their scores can be interpreted in the same way. Most validation studies have relied on Pearson’s correlations, which are often misinterpreted as indices of agreement or interchangeability, as they only measure the linear relationship between two variables (Y = aX + b). By this metric, two measures could be perfectly correlated but not yield a single pair of scores that are equivalent due to systematic differences by either an additive (b) or multiplicative (a) amount.
In contrast, the interchangeability of measures is best assessed through agreement statistics. Agreement can be calculated using a “consistency” definition that accounts for differences in variances but not means [i.e., additivity (Y = X + a)], or an “absolute” definition, which takes into account differences in means but also differences in variances and degree of linearity. Absolute agreement describes the degree that two measures on the same scale produce identical scores (X = Y). While absolute agreement is indeed a high standard to attain for two measures obtained through different methods (i.e., computerized versus pen-and-paper), within clinical settings the resultant standard scores derived from either method will nonetheless be interpreted similarly. Analyzing absolute discrepancies between individual pairs of scores therefore offers more clinically relevant information than comparing average performances as a whole, including identifying whether one method yields systematically higher or lower scores. No studies to date have examined the degree of absolute agreement between the NIHTB-CB and analogous neuropsychological tests.
Thus, the purpose of this study was twofold. First, we sought to investigate the psychometric properties of the NIHTB-CB in a sample of cognitively intact older adults. Specifically, we examined test–retest reliability and construct validity. In addition to evaluating convergent and discriminant validity using neuropsychological tests, we also used measures that do not share common-method variance such as demographic (i.e., age, education) and imaging variables (i.e., cerebral volumes) that are pertinent to aging and dementia. With the advent of commercially available metrics of overall and focal cerebral atrophy (Azab et al., Reference Azab, Carone, Ying and Yousem2015), such multi-method validity studies are increasingly possible without the need for neuroimaging analysis expertise. We focused on the NIHTB-CB fluid and crystallized cognition composites and composites derived from “gold standard” (GS) neuropsychological tests, but also report results from individual tests for comprehensiveness. Second, we examined absolute agreement between age-adjusted scores on the NIHTB-CB and GS tests using methods that are novel to the NIHTB-CB literature. We aimed to evaluate the degree to which the NIHTB-CB and GS tests produce equivalent scores to determine whether their resultant scores can be interpreted similarly or used interchangeably, which has direct implications for the potential clinical use of the NIHTB-CB.
METHODS
Participants and Procedures
This sample included older adults ages 60–80 years who were recruited via community advertisements as part of a larger study using advanced MRI for the early detection neurodegenerative disorders. All participants were screened by phone for eligibility criteria, which included speaking English as a primary language, absence of history of significant neurologic disease, serious mental illness, or self-reported cognitive complaints beyond expectations for age. Of the 65 participants enrolled between 2013 and 2015, two were excluded due to meeting actuarial neuropsychological criteria for mild cognitive impairment (Bondi et al., Reference Bondi, Edmonds, Jak, Clark, Delano-Wood, McDonald and Salmon2014) and two were excluded due to missing NIHTB-CB data. As funding for this study was extended, participants were invited to return to undergo identical procedures to address research questions regarding longitudinal changes in MRI measures. Each participant at follow-up was re-screened to ensure there had been no major changes in medical conditions or incident cognitive decline. Of the remaining 61 participants, 37 returned for follow-up after 15.03 ± 3.11 months. None of these participants met actuarial neuropsychological criteria for MCI.
Participants completed study procedures on the same day, which included the neuropsychological battery of the National Alzheimer’s Coordinating Centers’ Uniform Data Set (Shirk et al., Reference Shirk, Mitchell, Shaughnessy, Sherman, Locascio, Weintraub and Atri2011; Weintraub et al., Reference Weintraub, Salmon, Mercaldo, Ferris, Graff-Radford, Chui and Morris2009), followed by the NIHTB-CB, and brain MRI. A licensed clinical neuropsychologist or a trained research assistant under supervision administered all tests and oversaw all scans. Participants were not allowed to take sedating or anxiolytic medications for MRI. All scans were reviewed by the PI, research staff, and MRI technologists, and any incidental findings were sent to a neuroradiologist for review. None had incidental findings that required clinical follow-up. This study was approved by the Institutional Review Board.
Measures
NIH Toolbox cognition battery
Participants completed all seven core measures of the NIHTB-CB on a computer that conformed to hardware and software specifications per NIHTB guidelines. Five measures of attention/inhibitory control, executive functioning (i.e., set shifting) working memory, processing speed, and episodic memory comprise the fluid cognition composite, and two tests of language (i.e., vocabulary comprehension and oral reading decoding) make up the crystallized cognition composite (Gershon et al., Reference Gershon, Cook, Mungas, Manly, Slotkin, Beaumont and Weintraub2014). Scores from measures of inhibitory control and set shifting are derived from both accuracy and response speed, while the remaining three fluid test scores reflect total correct responses. The two language tests are administered using computerized adaptive testing and scored using item response theory. The NIHTB-CB generates three types of scores for each subtest and composite: uncorrected standard scores, age-corrected standard scores, and fully adjusted scores that account for age, education, gender, and race/ethnicity. Only age-adjusted and unadjusted standard scores, which both have a mean of 100 and standard deviation of 15, were used in the current analyses in the interest of having consistent comparisons with criterion variables. Initial studies indicated that these composites have strong test–retest reliability (i.e., ICCs ranging from .78 to .99) and moderate to strong convergent validity (r’s ranging from .48 to .93) with traditional neuropsychological tests (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014; Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013).
“GS” neuropsychological test battery
Table 1 shows each NIHTB-CB measure matched with its corresponding GS measure of the same underlying construct. GS measures largely correspond to those used in the initial NIHTB-CB validation studies (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014; Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013). We analyzed age-adjusted scores to facilitate comparison with NIHTB-CB scores, as fully adjusted normative data were not available for all GS tests. Age-adjusted scaled scores and T-scores were derived from the appropriate normative database or test manual (Ivnik et al., Reference Ivnik, Malec, Smith, Tangalos and Petersen1996; Lucas et al., Reference Lucas, Ivnik, Willis, Ferman, Smith, Parfitt and Graff-Radford2005; Schmidt et al., Reference Schmidt1996; Wechsler, Reference Wechsler1997) and then converted to standard scores (M = 100, SD = 15) to be on the same scale as NIHTB-CB scores. The GS fluid composite score was calculated by averaging the standard scores on five well-established tests of processing speed, working memory, attention, executive functioning, and memory: Digit Symbol Substitution subtest from the Wechsler Adult Intelligence Scale – Third Edition (WAIS-III; Wechsler, Reference Wechsler1997), Color-Word trial from the Stroop Color and Word Test (Golden, Reference Golden1978), Digit Span subtest from the Wechsler Memory Scale-Revised (WMS-R; Wechsler, Reference Wechsler1987), Trail Making Test – Part B (Reitan & Wolfson, Reference Reitan and Wolfson1985), and total learning score from the Rey Auditory Verbal Learning Test (RAVLT; Rey, Reference Rey1964). The American National Adult Reading Test (AmNART), an oral reading test commonly to estimate premorbid intellectual functioning, served as the validation measure for both NIHTB-CB crystallized composite subtests. The GS battery did not include a measure of receptive vocabulary to serve as a criterion measure for the NIHTB-CB Picture Vocabulary Test, so correlations provided here are exploratory.
NIH toolbox cognition battery tests with corresponding GS tests
Notes: RAVLT = Rey Auditory Verbal Learning Test; WMS-R = Wechsler Memory Scale-Revised; WAIS-III = Wechsler Adult Intelligence Scale, Third Edition; AmNART = American National Adult Reading Test.
Neuroimaging
Participants underwent brain MRI using a 3 Tesla Siemens TIM Trio system, including 3D T1-weighted MPRAGE images that were analyzed using the automated segmentation program NeuroQuant 1.0 (Cortechs, San Diego, CA, USA). Combined bilateral cerebral volumes, in cm3, normalized to percent intracranial volume, for the hippocampi, lateral ventricles, and inferior lateral ventricles were provided in NeuroQuant’s Age-related Atrophy Report. These regions are associated with aging, neurodegenerative disease, and cognitive decline (Nestor et al., Reference Nestor, Rupsingh, Borrie, Smith, Accomazzi and Wells2008; Raz et al., Reference Raz, Lindenberger, Rodrigue, Kennedy, Head, Williamson and Acker2005). We then computed the hippocampal occupancy score (HOC; hippocampal volume/hippocampal volume + inferior lateral ventricle volume) as an index of medial temporal lobe atrophy, which has been shown to be superior to standard hippocampal volume estimates in predicting conversion from mild cognitive impairment to Alzheimer’s disease dementia (Heister et al., Reference Heister, Brewer, Magda, Blennow and McEvoy2011).
Data Analyses
Test–retest reliability
Test–retest reliability and stability were evaluated using Pearson’s correlations and concordance correlation coefficients (CCC; Lin, Reference Lin1989) with 95% confidence intervals, using an absolute definition of agreement. Practice effects were evaluated with paired t-tests and effect sizes, computed as scores at follow-up (time 2) minus scores at baseline (time 1), divided by the standard deviation of scores at time 1 (Cohen, Reference Cohen1992).
Construct validity
First, the relationships between unadjusted NIHTB-CB scores with relevant demographic variables (i.e., age and education) and MRI metrics of cerebral volumes were investigated with Pearson’s correlations. In testing convergent validity, we expected fluid cognition composite scores to correlate with age and cerebral volumes, and crystallized cognition composite scores to correlate with education. Lower correlations between crystallized scores and cerebral volumes, and between fluid scores and educational attainment, were hypothesized to indicate discriminant validity. These analyses were then repeated with GS scores for comparison. Unadjusted standard scores (M = 100, SD = 15) were used for these analyses, as both neuropsychological and MRI variables are similarly influenced by age.
Second, the relationships between age-adjusted standard scores on NIHTB-CB and GS measures were investigated with Pearson’s correlations to test their convergent and discriminant validity. In line with initial NIHTB-CB validation papers (e.g., Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Slotkin and Gershon2014), correlations with criterion measures below .4 were considered poor, .4 to .6 were adequate, and .6 or greater were good evidence of convergent validity. Evidence of discriminant validity consisted of significantly lower correlations between dissimilar measures (e.g., NIHTB-CB fluid with GS crystallized), which was confirmed with a Steiger’s z (Reference Steiger1980) statistic.
Agreement
We used several methods to evaluate agreement between age-adjusted scores on the NIHTB-CB and corresponding GS measures. First, we calculated CCC’s with 95% confidence intervals as indices of absolute agreement (i.e., X = Y) to characterize any systematic differences in scores that otherwise would not be captured using a consistency or linear definition. Like Pearson’s r, the CCC can range from −1 to +1, but it cannot exceed r, as perfect agreement is only reached when scores on the two measures are equal for each person. The CCC is preferred over Pearson’s r for studies that aim to determine whether scores on two different measures agree sufficiently to be used interchangeably. Barnhart et al. (Reference Barnhart, Haber and Lin2007) and Carrasco and Jover (Reference Carrasco and Jover2003) demonstrated that the CCC is identical to the intraclass correlation coefficient (ICC; McGraw & Wong, Reference McGraw and Wong1996) when assumptions for the latter are met; however, the CCC is more appropriate when comparing two measures that may have different variances. CCCs may be interpreted using the benchmarks proposed by Altman (Reference Altman1991): poor agreement (.20), weak (.21–.40), moderate (.41–.60), good (.61–.80), and very good agreement (.81–1.00).
Root mean squared differences (RMSD) were calculated to measure the average discrepancy between individual pairs of scores (Barchard, Reference Barchard2012). Using an absolute definition of agreement, RMSD values represent the square root of the average squared difference score and are interpreted in the metric of the original scores. Thus, an RMSD of 7.5 would indicate that the average discrepancy is 7.5 standard score points (i.e., 0.5 SD). This value was established as the maximum acceptable disagreement, as larger discrepancies may represent a clinically meaningful difference. CCC and RMSD values were calculated using the Excel spreadsheet provided in the supplementary materials by Barchard (Reference Barchard2012).
We also provide base rates for large discrepancies (> 0.5 and 1 SD) and Bland–Altman plots of the NIHTB-CB and GS composite scores, using NCSS version 12 statistical software (NCSS, Kaysville, UT, USA). Altman and Bland and Bland (Reference Altman and Bland1983) plots visually depict the differences between measurements by plotting the average of paired scores from each method on the x-axis against the difference of each pair of scores on the y-axis. This illustrates the presence of any constant differences (i.e., differences that are consistent across the entire range of scores), proportional differences (i.e., non-constant differences that are proportional to the obtained score), and degree of imprecision (i.e., random error). Precision can be estimated with the 95% limits of agreement, or limits between which 95% of observations in the population would be expected to lie (i.e., ±1.96 SD of the mean difference).
RESULTS
Characteristics of the sample are shown in Table 2. The sample was predominantly Caucasian (90%; 10% were African American) and had an average of 16 years of education. There were no significant differences on any demographic or cognitive variables between the total sample and the subsample that returned for follow-up (data not shown). MRI data were missing on five participants due to MRI safety concerns or claustrophobia (n = 4) and significant motion artifact (n = 1). All variables were examined and found to meet the statistical assumptions of the parametric analyses reported here. Descriptive statistics and histograms for age-adjusted NIHTB-CB and GS test scores are provided in Supplementary Table 1 and Figure 1 to facilitate comparisons of the distribution of scores on each battery.
Sample characteristics at baseline visit
a No statistically significant differences on demographic variables or cognitive test scores between total sample and subsample returning for follow-up (all p > .05).
b Medical history includes any lifetime history of diagnoses that participants endorsed on self-report questionnaires.
Test–Retest Reliability
NIHTB-CB crystallized and fluid cognition composites demonstrated strong test–retest reliability (CCC’s = .92 and .73, respectively) over the follow-up interval (Table 3). The reliability of individual measures varied from .46 for List Sorting Working Memory to .87 for Picture Vocabulary. Dimensional Change Card Sort was the only test showing a small but statistically significant practice effect (+3.22 points, ES = 0.33). In comparison, reliability estimates for GS measures were similar or slightly higher for the GS fluid cognition composite (CCC = .85); however, this composite and four of its individual measures showed consistent evidence of practice effects (+3.88 points, ES = 0.38).
Test–retest reliability and practice effects (mean interval 15.03 months, SD = 3.11)
Notes. CCC = concordance correlation coefficient. p values reflect significance of two-tailed t-tests for differences in paired means. Magnitudes of practice effects are shown with Cohen’s d effect sizes, with values in bold representing significantly higher scores at follow-up (alpha > .05).
n = 37 unless stated otherwise. All values reflect age-adjusted standard scores (M = 100, SD = 15).
*p < .05; **p < .01; ***p < .001.
Construct Validity
Demographic and neuroimaging correlates
Table 4 shows correlations between the cerebral volumes of interest with NIHTB-CB and GS tests. As expected, unadjusted standard scores on the NIHTB-CB fluid cognition composite were associated with medial temporal lobe volume (HOC r = .41, p = .002), lateral ventricular volume (r = −.35, p = .008), and age (r = −.51, p < .001), but not years of education (r = .07, p = .617). Only a subset of the individual NIHTB-CB measures (i.e., Dimensional Card Sort, Picture Sequencing Memory, and Pattern Comparison Processing Speed) correlated significantly with cerebral volumes. Conversely, crystallized cognition composite scores significantly correlated with years of education (r = .42, p = .001), but not age (r = −.14, p = .296) or cerebral volumes (p’s = .840 and .084 for lateral ventricular and medial temporal volumes, respectively), providing evidence of discriminant validity.
Pearson’s correlations between cognitive test scores and cerebral volumes
Notes. LAV = lateral ventricles; HOC = hippocampal occupancy score. Both LAV and HOC values reflect combined bilateral volumes, in cm3, normalized to percent intracranial volume.
Values in bold are statistically significant at p < .05.
In comparison, correlations between GS fluid cognition composite and cerebral volumes were marginally significant and slightly lower in magnitude (see Table 4), although differences between these correlations were not statistically significant for medial temporal (Steiger’s z = 1.09, p = .277) or lateral ventricular volumes (z = 1.11, p = .268). Of the individual GS tests, Digit Symbol, Stroop Color-Word, and RAVLT demonstrated significant correlations of comparable magnitude to NIHTB-CB tests.
Correlations with GS measures
Adequate to strong correlations were found between the two fluid composites (r = .58) and crystallized composites (r = .84), as shown in Table 5 and in the scatter plots on the top row of Figure 1A. Convergent validity coefficients for individual measures were much more variable, ranging from poor (r = .10) to very strong (r = .85). Correlations between tests of executive functioning were lowest and were not statistically significant. Evidence of discriminant validity was supported by significantly lower correlations between NIHTB-CB Fluid and GS crystallized cognition composite scores (r = .32, p = .018; Steiger’s z = −2.41, p = .027), and between NIHTB-CB crystallized and GS fluid cognition composite scores (r = .43, p = .001; Steiger’s z = −4.71, p < .001). Intercorrelations for all individual NIHTB-CB and GS tests are provided in Supplementary Table 1.
Correlations and agreement between age-adjusted standard scores (M = 100, SD = 15) on NIHTB-CB and GS measures
Notes. CCC = concordance correlation coefficient; RMSD = root mean squared difference; DCCS = Dimensional Change Card Sort Test; FICA = Flanker Inhibitory Control and Attention Test; Stroop C/W = Stroop Color-Word trial; PSMT = Picture Sequence Memory Test; RAVLT = Rey Auditory Verbal Learning Test; LSWM = List Sorting Working Memory Test; PCPS = Pattern Comparison Processing Speed; PVT = Picture Vocabulary Test; ORRT = Oral Reading Recognition Test; AmNART = American National Adult Reading Test.
*p < .05; **p < .01; ***p < .001.
a Values are provided as reference only.
Agreement between age-adjusted NIHTB-CB and GS composite scores. (A) Scatter plots with regression lines and shading for 95% CIs of age-adjusted NIHTB and GS crystallized (left) and fluid (right) cognition composite scores. (B) Solid horizontal lines in Bland–Altman plots represent mean differences and upper and lower limits of agreement, illustrating where 95% of differences are expected to fall in the general population. Shading reflects 95% CIs around the mean difference. Dashed lines represent the zero line of perfect agreement. (C) Frequency distributions of discrepancy scores approximate normal distribution.
Agreement
The NIHTB-CB and GS crystallized cognition composites demonstrated good agreement (CCC = .70), although the average pairwise discrepancy approached one standard deviation (RMSD = 12.15; see Table 5), indicating that individual pairs of scores differed by 12.15 points on average. The Bland–Altman plot in Figure 1B (left panel) shows that the NIHTB-CB (M = 124.33, SD = 14.98) consistently generates higher Crystallized scores than the GS (M = 115.49, SD = 14.19), mean difference = 8.84 [SD = 8.41, 95% CI (6.68, 10.99)]. A one sample t-test confirmed that this differs significantly from zero (t = 8.21, df = 60, p < .001). As shown in the frequency distribution of the differences (Fig. 1C, left panel), only 34% of the sample obtained acceptable discrepancy scores within +-7.5 points, while 61% obtained NIHTB-CB scores that were over 7.5 points higher than their GS score. Twenty-eight percent obtained scores that differed by over 15 points (all NIHTB-CB > GS). As described in the Supplementary Material, exploratory analyses indicated that no particular participant subgroup was driving this discrepancy, although the sample sizes are too low to reliably detect demographic differences in this homoegenous sample. Furthermore, while the two oral reading tests (i.e., NIHTB-CB Oral Reading Recognition Test and AmNART) yielded comparable scores, the NIHTB-CB Picture Vocabulary test yielded scores that were 8–9 standard score points higher than the two reading tests.
The NIHTB-CB and GS fluid cognition composites demonstrated moderate agreement (CCC = .55). The proximity of the CCC to Pearson’s correlation (r = .58) reflects the fact that the two methods have nearly equivalent means, NIHTB-CB = 109.93 (SD = 14.02) and GS = 108.93 (SD = 10.38), mean difference = 1.00 [SD = 11.65, 95% CI (−1.98, 3.99), t = 0.67, df = 60, p = .504]. This is confirmed by the Bland–Altman plot and frequency distribution (Fig. 1B and 1C, right panel) showing an approximately equal number of cases above and below the mean difference. However, individual pairs of scores differed by an average of 11.60 points. Fifty-one percent of the sample obtained discrepancy scores greater than 7.5 points, and 23% obtained scores that differed by 15 or more points. The limits of agreement show that 95% of scores on the two fluid composites are expected to differ between −21.83 and +23.84 points, indicating poor precision but no systematic bias (i.e., over or underestimation). Additionally, the scatter in the Bland–Altman plot appears to trend upward at higher values. As described in the Supplementary Material (Supplementary Fig. 2), we explored the presence of proportional bias using non-parametric regression analysis (Passing & Bablok, Reference Passing and Bablok1983), which confirmed that the NIHTB-CB overestimates performance for individuals with high scores and underestimates it among individuals with lower scores. Furthermore, the NIHTB-CB appears to overestimate fluid composite scores in younger participants and underestimate it among African-American participants, although again these results are regarded as preliminary at best given the demographic homogeneity and modest size of the current sample.
DISCUSSION
This study evaluated the psychometric properties of the NIHTB-CB and its agreement with traditional neuropsychological tests in healthy older adults. We found evidence supporting its test–retest reliability over approximately 1 year and convergent and discriminant validity with criterion variables with which the NIHTB-CB does and does not share method variance. Despite these promising results, our findings nonetheless show that scores from the NIHTB-CB and GS tests are not equivalent, warranting caution should the NIHTB-CB be used for clinical purposes.
First, we found strong test–retest reliability for both fluid (CCC = .73) and crystallized (CCC = .92) composite scores. These estimates were very similar to those reported in previous validation studies with much shorter follow-up intervals (r’s = .86 and .92 for fluid and crystallized; Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014). Reliability of individual tests comprising the NIHTB-CB fluid cognition composite were generally lower and more variable than their respective GS tests, but displayed minimal evidence of practice effects over the approximate 15-month test–retest interval. This contrasts with the GS fluid measures, which consistently produced slightly higher scores at follow-up, and suggests that the adaptive format of the NIHTB-CB may be less susceptible to practice effects.
Second, our findings support the construct validity of the fluid and crystallized cognition composites, based on distinct patterns of associations with age, education, cerebral volumes, and standard neuropsychological tests. Our data suggest that the NIHTB-CB fluid cognition composite tests are at least as sensitive as GS tests in detecting cerebral volume loss that occurs in healthy aging (Dodge et al., Reference Dodge, Zhu, Harvey, Saito, Silbert, Kaye and Albin2014) and Alzheimer’s disease dementia (Heister et al., Reference Heister, Brewer, Magda, Blennow and McEvoy2011). The crystallized and fluid cognition composites also showed strong convergent and discriminant validity with neuropsychological measures, although convergent validity for individual fluid tests ranged from poor to adequate. Curiously, although the Dimensional Change Card Sort Test was the strongest correlate of cerebral volume loss, it was poorly correlated with other measures of executive functioning. This likely reflects the multifaceted nature of executive functioning tests and the disparate paradigms that were used to measure cognitive flexibility.
Finally, agreement between NIHTB-CB and GS crystallized cognition cComposites was good, but the NIHTB-CB systematically overestimated performance by 9 standard score points. In contrast, the two fluid cognition composites yielded roughly equivalent mean scores but had large pairwise differences that reflect poor precision and proportional differences, with increasing discrepancies at the tails of the distributions (NIHTB-CB > GS at higher scores and NIHTB-CB < GS at lower scores). It is possible that this overestimation reflects ceiling effects of the GS tests that are not present in the NIHTB-CB, given that the neuropsychological battery used by UDS was primarily designed for individuals with suspected cognitive impairments and may be less useful for evaluating cognition at the higher end of functioning (Mathews et al., Reference Mathews, Abner, Kryscio, Jicha, Cooper, Smith and Schmitt2014). Follow-up exploratory analyses suggested that investigation of sub-samples in future studies may yield insights into possible demographic differences driving these discrepancies.
CLINICAL IMPLICATIONS
Our findings are consistent with extant literature supporting the NIHTB-CB crystallized cognition composite as a clinically useful estimate of premorbid intellectual ability that is relatively robust to aging and cerebral volume loss, and may aid in the determination of decline in fluid abilities. The NIHTB-CB fluid cognition composite also appears to be a valid measure of cognitive functioning in older adults, which relates to non-cognitive factors (e.g., cerebral atrophy) that are pertinent to aging and neurodegenerative disease.
However, caution should be taken when interpreting the NIHTB-CB crystallized cognition composite as a measure of premorbid functioning, given the tendency to overestimate performance relative to standard tests. It is possible that a conversion factor could be applied to equate performances on these methods, because the discrepancy is constant across the entire range of scores and demographics included in this sample. Exploratory analyses suggest that the Picture Vocabulary test is the primary source of inflation of the NIHTB-CB crystallized cognition composite, yielding scores that were on average 8–9 points higher than the two oral reading tests. Thus, clinicans using the NIHTB-CB may elect to use the oral reading test alone rather than the crystallized cognition composite as a proxy for premorbid IQ.
Caution is also warranted when interpreting the NIHTB-CB fluid cognitioncomposite, insofar as age-adjusted scores are concerned. Several factors appeared to influence the size and direction of the discrepancy, including age, ethnicity, and obtained score, indicating that the NIHTB-CB may overestimate fluid cognition at higher scores and underestimate it at lower scores although lower performance in this sample is still largely within the average range. These findings suggest that additional demographic adjustments may be required when making intra-individual comparisons in clinical settings, particularly with more diverse populations.
Limitations
The current sample was relatively homogenous, well educated, and obtained higher scores than those reported in prior validation and normative studies (Casaletto et al., Reference Casaletto, Umlauf, Beaumont, Gershon, Slotkin, Akshoomoff and Heaton2015). This may limit the degree to which these results generalize to other normative and clinical populations. Future studies should further investigate whether race, education, or other characteristic disproportionately affects performance on one of the two test batteries. We chose to use age-adjusted normative data on both test batteries to evaluate construct validity and agreement, because the full range of demographic adjustments was not available for all GS tests. Additionally, we did not account for order effects on performance as the administration order was fixed per the study protocol. We also did not apply any Type-I error adjustments in our statistical analyses in line with other psychometric papers (e.g., Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014; Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013), as we intended to interpret the overall pattern of associations between the two batteries. Another caveat relates to the selection of GS tests, which do not directly correspond to the administration formats of the NIHTB-CB tests and also have imperfect reliability and validity. Thus, it is important to note that low agreement between NIHTB-CB and GS measures does not indicate the superiority of one measure over the other per se, but rather that the two measures often generate different interpretative results.
CONCLUSION
This study contributes to the literature by providing clinically relevant information on the comparability of NIHTB-CB and traditional neuropsychological test scores. Agreement or comparability is frequently discussed but rarely systematically examined in the neuropsychological literature. Although the methods used in this study are common in other fields, only a handful of studies in the neuropsychological literature have made use of these methods to assess agreement between scores on a continuous scale (e.g., Berg et al., Reference Berg, Durant, Banks and Miller2016). The current practice is to rely on estimates of linear agreement (e.g., Pearson’s correlations) which fail to capture constant differences, or mean differences (e.g., t-tests), which capture constant differences but not random or proportional differences, as large measurement error will result in the incorrect conclusion that the methods are equivalent. Comparability is a question of both random deviation (i.e., error) and systematic deviation (i.e., constant and proportional) and no single statistic can estimate both. Although it would be very unlikely for two different tests to agree perfectly, the degree of agreement has critical implications for clinical practice. For example, if a measure of crystallized abilities systematically overestimates premorbid IQ, large discrepancies may be misinterpreted as evidence of decline and ultimately lead to misdiagnoses.
Our findings underlie the need to incorporate both descriptive and statistical analyses of agreement when evaluating new measures against a reference standard to complement classical psychometric methods and enhance the interpretation of validation studies. In conclusion, these findings suggest that the NIHTB-CB appears to be a valid tool for cognitive assessment in aging, but that it may generate different interpretative results from existing measures commonly used in clinical evaluations. Thus, these two methods should not be considered interchangeable.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355617719000614
ACKNOWLEDGEMENTS
The authors thank Brianna Jones, Deborah Nelson, Danae Montgomery, and Clifford Chan for their assistance with participant recruitment and research coordination. They also gratefully acknowledge the study participants for generously offering their time and data for this work. This research was supported by the National Institutes of Health (NCATS UL1TR000062/KL2 TR000060 and NIA K23 AG044434 to A.B.).
CONFLICTS OF INTEREST
The authors have nothing to disclose.
