INTRODUCTION
Solid and convenient cognition measures are beneficial for research on Alzheimer’s disease (AD) to help characterize the associated longitudinal trajectory of cognitive decline and identify mild cognition change at the preclinical stage. The NIH Toolbox Cognition Battery (NIHTB-CB) provides a standardized set of measures to assess multiple domains of cognitive function and serves as a common currency for cross-study comparisons (Gershon et al., Reference Gershon, Wagster, Hendrie, Fox, Cook and Nowinski2013). As shown in Table 1, the adult version (age ≥ 18) of the battery includes seven tests and measures multiple cognitive domains (Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013). The NIHTB-CB has many advantages as it is: (1) applicable across the lifespan, (2) brief, (3) non-proprietary, (4) based on state-of-the-art test theories and technology, and (5) available in both English and Spanish versions (Mungas et al., Reference Mungas, Heaton, Tulsky, Zelazo, Slotkin, Blitz and Gershon2014). The validity and utility of the battery has been shown in cognitively unimpaired adults (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014; Mungas et al., Reference Mungas, Heaton, Tulsky, Zelazo, Slotkin, Blitz and Gershon2014; Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013), and clinical samples with spinal cord injury (Carlozzi, Goodnight, et al., Reference Carlozzi, Goodnight, Casaletto, Goldsmith, Heaton, Wong and Tulsky2017; Cohen et al., Reference Cohen, Tulsky, Holdnack, Carlozzi, Wong, Magasi and Heinemann2017), traumatic brain injury (TBI; Carlozzi, Goodnight, et al., Reference Carlozzi, Goodnight, Casaletto, Goldsmith, Heaton, Wong and Tulsky2017; Nitsch et al., Reference Nitsch, Casaletto, Carlozzi, Tulsky, Heinemann and Heaton2017; Tulsky, Carlozzi, et al., Reference Tulsky, Carlozzi, Holdnack, Heaton, Wong, Goldsmith and Heinemann2017; Tulsky, Holdnack, et al., Reference Tulsky, Holdnack, Cohen, Heaton, Carlozzi, Wong and Heinemann2017), stroke (Carlozzi, Goodnight, et al., Reference Carlozzi, Goodnight, Casaletto, Goldsmith, Heaton, Wong and Tulsky2017; Carlozzi, Tulsky, et al., Reference Carlozzi, Tulsky, Wolf, Goodnight, Heaton, Casaletto and Heinemann2017; Nitsch et al., Reference Nitsch, Casaletto, Carlozzi, Tulsky, Heinemann and Heaton2017; Tulsky, Holdnack, et al., Reference Tulsky, Holdnack, Cohen, Heaton, Carlozzi, Wong and Heinemann2017), and intellectual disabilities (Hessl et al., Reference Hessl, Sansone, Berry-Kravis, Riley, Widaman, Abbeduto and Gershon2016). In addition, a study including adults with varied cognitive statuses provided supportive findings for the validity of NIHTB-CB in assessing neurocognitive domains related to dementia (Hackett et al., Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018). Furthermore, for the cognitively unimpaired adult population, uncorrected, age-corrected, and fully demographically corrected normative standards have been developed for both English and Spanish versions (Casaletto et al., Reference Casaletto, Umlauf, Beaumont, Gershon, Slotkin, Akshoomoff and Heaton2015, Reference Casaletto, Umlauf, Marquine, Beaumont, Mungas, Gershon and Heaton2016). All of its unique features provide support that NIHTB-CB could potentially be a promising instrument in measuring cognition for AD research.
Tests of the NIH Toolbox Cognition Battery (NIHTB-CB) adult version (age ≥ 18)
Note. Score type = the name of score type exported from the iPad; IRT = Item Response Theory. For Flanker and DCCS, an accuracy score and a reaction time score are first calculated. The accuracy score is calculated as 0.125 × the number of correctly answered trials. The reaction time score is calculated as a function of the log (base 10) of the median reaction time using only correct trials with time length between 100 ms and 3 SD away from the participant’s mean time. The computed score is equal to the accuracy score if the participant has an accuracy rate ≤ 80%, and is the sum of the accuracy score and the reaction time score otherwise. For Processing Speed and Working Memory, the raw score is the number of items correctly answered. For Episodic Memory, Vocabulary, and Reading, scoring is based on the Item Response Theory (IRT). IRT models the probability of a correct response to an item given the underlying latent cognitive ability. The theta score represents the latent cognitive ability level (NIH & Northwestern University, 2006–2016). Higher score indicates better performance for each test. Two supplemental tests were also provided for the NIHTB-CB, including Auditory Verbal Learning Test (AVLT; Rey) Immediate Recall (trials 1, 2, 3) measuring learning, and Oral Symbol Digit Test measuring processing speed. However, these two tests were not included in this study.
Understanding the latent factor structure of NIHTB-CB specifically for its application to AD research samples is necessary for correct interpretation of findings from AD studies using the battery. A factor analysis of NIHTB-CB on cognitively unimpaired adults identified five factors, including executive function/processing speed (EF-PS), working memory, episodic memory, vocabulary, and reading. A subsequent second-order factor analysis on the same data showed the differentiation of fluid cognition (EF-PS, working memory, and episodic memory) versus crystalized cognition (vocabulary, reading) (Mungas et al., Reference Mungas, Heaton, Tulsky, Zelazo, Slotkin, Blitz and Gershon2014). The five-factor model has been validated in clinical samples with acquired brain injuries (TBI and stroke; Tulsky, Holdnack, et al., Reference Tulsky, Holdnack, Cohen, Heaton, Carlozzi, Wong and Heinemann2017). Both studies included other neuropsychological tests considered as gold standard measures in addition to NIHTB-CB, as their goal was to define convergent and discriminant validity of the battery based on its similarity and difference in factor loading patterns compared to the standard measures. Another factor analysis (Hackett et al., Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018) which included the NIHTB-CB battery only was conducted on a mixed sample of participants who were cognitively unimpaired, with subjective cognitive decline, mild cognitive impairment (MCI), and dementia due to AD. Episodic Memory and Working Memory tests were excluded because they were too challenging for participants with cognitive impairment and yielded low completion rates. However, the supplemental Auditory Verbal Learning Test (AVLT) Immediate Recall and Symbol Digit tests were included. Two factors were identified capturing fluid and crystalized cognition. However, when AD participants were excluded from the analysis, tests measuring the fluid cognition instead loaded on two separate factors capturing EF-PS and memory, respectively, while the crystalized cognition factor remained unchanged. Based upon the literature, the first goal of this study was to investigate the factor structure of NIHTB-CB in its application to a typical AD research sample with a wide range of cognition status including the cognitively unimpaired, MCI, and dementia. Different from the previous studies, the factor analysis was performed on the NIHTB-CB battery only including all seven tests listed in Table 1. The focus was to understand the underlying structural relations of multiple cognition domains and capture cognitive processes as a related and organized neuropsychological system.
The evaluation of measurement invariance between the cognitively unimpaired versus impaired is important for defining the utility of a battery for AD research. Variant factor structure would imply qualitative changes in the underlying neuropsychological system as the disease progresses, whereas invariant factor structure would suggest a quantitative decline in the same cognitive spectrum (Hayden, Plassman, & Warren, Reference Hayden, Plassman and Warren2011). Although measurement invariance of NIHTB-CB has not been tested, the finding of different numbers of factors between the analyses including versus excluding AD participants by Hackett et al. (Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018) suggested the possibility of variant factor structure across clinical diagnosis. The second goal of this study was to evaluate measurement invariance of NIHTB-CB between cognitively unimpaired versus impaired groups including MCI and dementia.
Previous studies have found demographic differences in cognitive performance, including differences across age, sex, race/ethnicity groups, and education level, either using NIHTB-CB (Casaletto et al., Reference Casaletto, Umlauf, Beaumont, Gershon, Slotkin, Akshoomoff and Heaton2015, Reference Casaletto, Umlauf, Marquine, Beaumont, Mungas, Gershon and Heaton2016; Flores et al., Reference Flores, Casaletto, Marquine, Umlauf, Moore, Mungas and Heaton2017) or other cognition measures (Collie, Shafiq-Antonacci, Maruff, Tyler, & Currie, Reference Collie, Shafiq-Antonacci, Maruff, Tyler and Currie1999; Norman, Evans, Miller, & Heaton, Reference Norman, Evans, Miller and Heaton2000; Norman et al., Reference Norman, Moore, Taylor, Franklin, Cysique and Ake2011). Measurement invariance across these key demographic variables is necessary to confirm that differences in the cognition test scores truly represent demographic differences in the cognition abilities being tested (Dowling, Hermann, La Rue, & Sager, Reference Dowling, Hermann, La Rue and Sager2010). Moreover, measurement tools that allow fair comparison across different demographic groups are fundamental to address health equity issues (Victorson et al., Reference Victorson, Manly, Wallner-Allen, Fox, Purnell, Hendrie and Gershon2013). Previous research on demographic invariance in cognition measures for older adults has mainly focused on race/ethnicity and related culture or language factors (Mungas, Widaman, Reed, & Tomaszewski Farias, Reference Mungas, Widaman, Reed and Tomaszewski Farias2011; Siedlecki et al., Reference Siedlecki, Manly, Brickman, Schupf, Tang and Stern2010; Tuokko et al., Reference Tuokko, Chou, Bowden, Simard, Ska and Crossley2009). Research is significantly lacking in testing measurement invariance across multiple demographic variables or specifically for an AD research sample. Furthermore, in our literature review, the only demographic measurement invariance testing on NIHTB-CB for adults was age invariance in the cognitively unimpaired (Mungas et al., Reference Mungas, Heaton, Tulsky, Zelazo, Slotkin, Blitz and Gershon2014). The third goal of this study was to test measurement invariance of NIHTB-CB across four major demographic variables, including sex, race/ethnicity, age, and education, in its application to an AD research sample. These four demographic variables were employed in deriving the fully demographically corrected normative standards for NIHTB-CB (Casaletto et al., Reference Casaletto, Umlauf, Beaumont, Gershon, Slotkin, Akshoomoff and Heaton2015, Reference Casaletto, Umlauf, Marquine, Beaumont, Mungas, Gershon and Heaton2016), which implies the significance of these variables in cognition variability. Findings of the invariance testing will be informative for the application of NIHTB-CB and its norms to AD research, and the interpretation of demographic differences in measured cognitive abilities found in a study.
In summary, this study investigated the factor structure of NIHTB-CB and its measurement invariance across clinical diagnosis groups and key demographic variables for a mixed sample of older adults with unimpaired cognition, MCI, and dementia. Findings will help evaluate the battery’s utility for AD research.
METHOD
Participants
The study included 411 participants from the Wisconsin Alzheimer’s Disease Research Center (ADRC). ADRC participants were recruited from memory diagnostic clinics and community. Women and men aged 45 and older with decisional capacity were eligible for enrollment. Exclusion criteria included major medical conditions (e.g., advanced congestive heart failure, kidney failure, severe untreated sleep apnea, and HIV/AIDS), major neurologic disorders (e.g., significant ischemic or hemorrhagic stroke, multiple sclerosis, and history of brain surgery), major psychiatric conditions (e.g., major Axis I disorder or addictive disorder), or lack of a study partner. Table 2 summarizes the sample demographics.
Sample characteristics (n = 411)
Note. Age ranged 45–94 years, with M = 66.3, SD = 9.8. All four participants with Hispanic ethnicity had white race.
ADRC Visit and Test Administration
The ADRC participants undergo annual or biennial clinical and cognitive assessment at an academic medical center in Madison, Wisconsin. (Visit frequency was based on age and clinical diagnosis.) For the purposes of this study, we used cross-sectional data collected at a single time visit. The National Alzheimer’s Coordinating Center (NACC) Uniform Data Set (UDS) (Besser et al., Reference Besser, Kukull, Knopman, Chui, Galasko, Weintraub and Morris2018) was collected at each visit. Between March 14, 2016 and March 8, 2017, the iPad English version of NIHTB-CB was administered at one visit immediately after completion of the NACC UDS neuropsychological battery version 3 (Weintraub et al., Reference Weintraub, Besser, Dodge, Teylan, Ferris, Goldstein and Morris2018). The study protocol was approved by the University of Wisconsin Institutional Review Board. Informed consent was obtained from each participant prior to the study.
Clinical Diagnosis
Following each ADRC visit, a clinical diagnosis was made at the Consensus Diagnosis Conference by a multidisciplinary team of geriatricians, neurologists, and neuropsychologists with expertise in dementia following NIA-AA criteria (Albert et al., Reference Albert, DeKosky, Dickson, Dubois, Feldman, Fox and Phelps2011; McKhann et al., Reference McKhann, Knopman, Chertkow, Hyman, Jack, Kawas and Phelps2011). The diagnosis was based on the comprehensive clinical and cognitive assessment results acquired at the visit and was not determined by biomarkers. Cognition measures independent from NIHTB-CB were used for diagnosis, including the NACC UDS neuropsychological battery and AVLT (Schmidt, Reference Schmidt1996). The sample included 77.1% unimpaired and 22.9% impaired individuals with varied severity levels and causes (Table 2).
Statistical Analyses
Evaluating the factor structure of the whole sample
Exploratory factor analysis (EFA) with the oblique geomin rotation was first performed on the whole sample with a focus on identifying the number of factors (Yates, Reference Yates1987). Given seven tests, a maximum of three factors can be extracted (Muthén & Muthén, Reference Muthén and Muthén2009). The number of factors was chosen based on the following criteria: (1) the number of eigenvalues greater than one, (2) good model fit, (3) the model solution having a clear factor structure with each test loaded on a single factor, that is, the test had a significant and high loading on one factor, but low loading(s) on the other factor(s), (4) clinical meanings, and (5) model parsimony (Fabrigar & Wegener, Reference Fabrigar and Wegener2012). Confirmatory factor analysis (CFA; Bollen, Reference Bollen1989) was next applied to further refine and confirm the factor structure identified by EFA with a focus on the relations between the tests (i.e., observed indicators) and the latent factors.
Testing factorial invariance across groups
As summarized in Table 3, invariance was tested in five dimensions, across clinical diagnosis, sex, race/ethnicity, age, and education, respectively, by comparing two groups in each dimension. The CFA model confirmed on the whole sample was first tested for each group separately. If the CFA fit well for both groups split by a specified variable, factorial invariance was next tested with a hierarchical series of nested two-group CFA models in the following order: (1) Configural invariance requires that the two groups have the same pattern of freely estimated and fixed at zero parameters, whereas all freely estimated parameters are allowed to differ across groups. Confirmed configural invariance serves as the baseline model and implies that the same latent constructs are measured for both groups. (2) Based on configural invariance, metric (weak) invariance requires that the factor loadings, that is, slopes or regression coefficients of the tests on the latent constructs, are equal across groups. Under confirmed metric invariance, latent factor variances and covariances are comparable across groups, and group difference in the ratios of factor variances and the correlations of latent factors are thus interpretable. (3) Scalar (strong) invariance additionally requires equal indicator intercepts, that is, difficulty levels of the tests. Under confirmed scalar invariance, latent factor means are also comparable and group difference in the latent factor means is thus interpretable. (4) Residual variance (strict) invariance additionally requires equal indicator residual variances. Under confirmed strict invariance, the unique factors contribute equally across groups, and thus, the group differences in the means and variances of the indicators are entirely attributable to the group differences in the latent factors. Based on strict invariance, (5) factor variance–covariance invariance and (6) factor mean invariance were further tested in order (Meredith, Reference Meredith1993; Meredith & Teresi, Reference Meredith and Teresi2006; Vandenberg, Reference Vandenberg2002; Vandenberg & Lance, Reference Vandenberg and Lance2000; Widaman & Reise, Reference Widaman, Reise, Bryant, Windle and West1997). Models (1–4) test measurement invariance and evaluate whether the relations between the tests and the latent constructs are same across groups. Scalar invariance is required to confirm measurement invariance and allows meaningful comparison in the latent constructs between groups. Strict invariance is more desirable but is usually difficult to achieve. Models (5, 6) test structural invariance and evaluate group differences in the variabilities, correlations, and levels of the latent constructs being measured (Byrne, Shavelson, & Muthén, Reference Byrne, Shavelson and Muthén1989; Vandenberg & Lance, Reference Vandenberg and Lance2000).
Classification of clinical diagnosis and demographic groups for invariance testing
Note. Because of the limited sample size in each impaired group, dementia and MCI due to all causes were combined into one group, whereas the impaired not MCI was excluded from the invariance testing. Due to a similar consideration, race/ethnicity groups were classified as underrepresented groups (URG) versus non-URG. Following the NIH definition (NIH Diversity in Extramural Programs, 2019), a participant was classified as URG if s/he self-reported primary, secondary, or tertiary race as African American, American Indian or Alaska native, Native Hawaiian or other Pacific Islander, or self-reported Hispanic ethnicity. A participant was classified as non-URG if s/he self-reported only White or Asian in primary and secondary races and self-reported No to Hispanic ethnicity. A participant was classified as URG unknown and not included for the invariance testing, if s/he self-reported other or unknown in race or ethnicity. Age was classified as a binary variable, <65 versus ≥65, since around 65 is commonly considered as the start of late adulthood. Because age is the biggest risk factor for dementia/MCI, and in the current sample age was highly associated with the incidence rate of dementia/MCI, 6.8% for participants < 65 versus 31.5% for those ≥65, p < .0001 (Fisher’s exact test), age invariance was tested only for cognitively unimpaired participants. Education level was classified into low (without bachelor’s degree) versus high (with bachelor’s degree) education groups, as these two groups would likely have access to different occupations, involving different cognitive demands and leading to different social economic status.
a 72 out of the 82 dementia/MCI participants had AD as a cause.
Model estimation
Analyses were performed on raw scores (Bowden, Cook, Bardenhagen, Shores, & Carstairs, Reference Bowden, Cook, Bardenhagen, Shores and Carstairs2004). These were the “computed” scores for Flanker and Dimensional Change Card Sort (DCCS), “raw” scores for Processing Speed and Working Memory, and “theta” scores for Episodic Memory, Vocabulary, and Reading. (Explanation of these scores is provided in the note under Table 1) (NIH & Northwestern University, 2006–2016). Two extremely high scores (22.7 and 35.7) for Vocabulary and one (36.1) for Reading were excluded from the analysis, because the tests may not reliably measure these individuals’ abilities, given lack of items with high difficulty levels. Such items are needed to appropriately assess the highest functioning individuals. Models were tested with Mplus version 8 (Muthén & Muthén, Reference Muthén and Muthén1998–2017) using the full information maximum likelihood (FIML) sandwich estimator with robust standard errors (MLR) which handles missingness and nonnormality (Enders, Reference Enders2010; Wang & Wang, Reference Wang and Wang2012; Yuan & Bentler, Reference Yuan, Bentler, Sobel and Becker2000). The description of model identification and sample Mplus codes are provided in the Supplemental Material.
Assessing model fit
Model fit was evaluated based on multiple indices in order to make best use of the available data and draw the most robust conclusion. Overall model fit was assessed using fit indices including the comparative fit index (CFI; Bentler, Reference Bentler1990), the root mean squared error of approximation (RMSEA) with 90% confidence interval (Steiger & Lind, Reference Steiger and Lind1980), and the standardized root mean squared residual (SRMR; Bentler, Reference Bentler1995). Model fit was considered adequate by meeting the following criteria: CFI ≥ 0.95, RMSEA ≤ 0.08, and SRMR ≤ 0.08 (Browne & Cudeck, Reference Browne and Cudeck1992; Hu & Bentler, Reference Hu and Bentler1998, Reference Hu and Bentler1999). Misfit in individual parameters was evaluated using model modification indices (MI), which are the amount of reduction in the model χ 2 if a parameter fixed at zero or constrained equal across groups were freely estimated (Steiger, Shapiro, & Browne, Reference Steiger, Shapiro and Browne1985). A parameter was freed by using the threshold MI > 10 as a start (Wang & Wang, Reference Wang and Wang2012). However, parameters with MI close to 10 were also freed if the model fit needed further improvement and the freed parameter had an estimate sufficiently different from zero. For factorial invariance testing with nested two-group CFAs, a more restricted invariance model was selected if the overall model fit was acceptable, and it was similar in model fit compared with the less restricted invariance model it nested within. Model fit difference was assessed using the Satorra-Bentler (SB) scaled correction χ 2 difference test (Satorra & Bentler, Reference Satorra and Bentler2001) and change in CFI. Because the χ 2 test can be overly sensitive for sample sizes above 150 (Dowling et al., Reference Dowling, Hermann, La Rue and Sager2010) and to adjust for inflated type I error rate associated with multiple comparisons (i.e., five model comparison pairs across the six invariance levels), a more conservative significance level of p < .01 (i.e., 0.05/5) was adopted. Insignificant χ 2 difference tests (i.e., p ≥ .01) and ΔCFI ≤ 0.01 (Cheung & Rensvold, Reference Cheung and Rensvold2002) were considered as the criteria for similar model fit. Partial invariance (Byrne et al., Reference Byrne, Shavelson and Muthén1989; Millsap & Kwok, Reference Millsap and Kwok2004) was examined by allowing part of the constrained parameters to differ across groups, if this was suggested by large MIs and led to improved model fit. Under partial invariance, at least two invariant indicators per factor were required to confirm measurement invariance and meaningful comparisons across groups (Dowling et al., Reference Dowling, Hermann, La Rue and Sager2010; Mungas et al., Reference Mungas, Widaman, Reed and Tomaszewski Farias2011).
RESULTS
Descriptive Statistics
Univariate descriptive statistics and Pearson correlations of the tests are provided in Tables 4 and 5 for the whole sample and each diagnosis group, and provided in the Supplemental Tables S1 and S2 for each demographic group. The dementia/MCI group generally had higher missing rates, lower averages, greater variabilities, and lower correlations than the cognitively unimpaired group.
Means, standard deviations, and ranges of the test scores for the whole sample and each diagnosis group
Note. Reasons for missingness included the following: (1) The participant was unable to complete the test because of limited cognitive abilities or other limitations such as poor vision or hearing; (2) The test was automatically skipped if the participant failed on the sample items before the test; (3) The participant refused the test; (4) There was lack of time to administer the test. In addition, two extremely high scores (22.7 and 35.7) for Vocabulary and one (36.1) for Reading were excluded from the analysis, because the tests may not reliably measure these individuals’ abilities, given lack of items with high difficulty levels. Such items are needed to appropriately assess the highest functioning individuals. The Vocabulary and Reading tests are administered with the Computer Adaptive Testing (CAT) format and are scored based on the Item Response Theory (IRT). With CAT, the next item an individual receives depends on her/his response to the previous item. CAT allows that a test is tailored to an individual’s ability level, and thus, the battery can be applicable to individuals with a broad range of ability levels, which, in turn, can reduce the chances in the floor and ceiling effects. However, lack of items with very high (or low) difficulty levels for individuals with extremely high (or low) abilities would result in unreliable scores for these individuals.
Pearson correlations of the test scores for the whole sample and each diagnosis group
Note. Insignificant correlations (p > .05) are underscored. Sample sizes are included in the parentheses under the correlations. The dementia/MCI group had much smaller sample sizes than the cognitively unimpaired group. Thus, the comparison should be based on the effect size of the correlations rather than the p-values. In addition, the pairwise missing rate was consistently higher for the dementia/MCI group than the cognitively unimpaired group. The missing rate was similar between different correlation coefficients (i.e., different pairs of tests) for the cognitively unimpaired. However, it varied for the dementia/MCI group and was most substantial for the correlations that involved memory tests. This missing pattern implied a systematic restriction in the samples such that only the relatively less impaired in the dementia/MCI group was included in the correlation estimation and comparison, and this restriction was most severe for correlations that involved memory tests. As a result, different subsamples of the dementia/MCI group were being compared between different correlations.
EFA and CFA for the Whole Sample
EFA yielded two eigenvalues (3.98, 1.13) greater than one. Supplemental Figure S1 provided the scree plot of all eigenvalues. As shown in Table 6, the one-factor solution had unacceptable model fit, whereas standard errors could not be computed for the three-factor solution due to model identification issues. In contrast, the two-factor solution yielded good model fit and a clear fluid-crystalized cognition factor structure as depicted in Figure 1 (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014). This model was next confirmed by CFA, as evidenced by its excellent overall model fit, all MIs < 10, and all factor loadings being large (0.60–0.90), positive, and significant (Table 7).
Factor loadings, χ 2 test, and model fit indices for the exploratory factor analyses with geomin rotation for the whole sample (n = 411)
Note. Insignificant factor loadings (p > .05) are underscored. Factor loadings > .50 are in boldface. CFI = comparative fit index; RMSEA = root mean squared error of approximation; SRMR = standardized root mean squared residual. Model fit is considered adequate by meeting the following criteria: CFI ≥ 0.95, RMSEA ≤ 0.08, and SRMR ≤ 0.08. The three-factor solution is not reported, because standard errors could not be computed due to model identification issues.
Factor structure of the whole sample and all groups except for the group with age < 65.
Standardized parameter estimates, χ 2 test, and model fit indices for confirmatory factor analysis final models for the whole sample and each group
Note. For the whole sample and each group except for the group with age < 65, a two-factor model fit well, including (1) fluid cognition (F1) measured by Flanker, DCCS, processing speed, working memory, and episodic memory, and (2) crystalized cognition (F2) measured by vocabulary and reading. For the group with age < 65, the two-factor model did not fit well, χ 2 (df = 13) = 52.761, p-value < 0.001, CFI = 0.846, RMSEA (90% CI) = 0.136 (0.099, 0.175), SRMR = 0.077. However, a three-factor model fit well, including (A) executive function/processing speed (FA) measured by Flanker, DCCS, and processing speed, (B) memory (FB) measured by working memory and episodic memory, and (C) language (FC) measured by vocabulary and reading. Factor variances are one under standardized solution. Factor means are fixed at zero to satisfy model identification. Insignificant parameter estimates (p > .05) are underscored. CFI = comparative fit index; RMSEA = root mean squared error of approximation; SRMR = standardized root mean squared residual. Model fit is considered adequate by meeting the following criteria: CFI ≥ 0.95, RMSEA ≤ 0.08, and SRMR ≤ 0.08.
a Factor loadings in “()” are cross-loadings on the crystalized cognition factor (F2).
b Fixed at zero to satisfy model identification.
c The factor covariance equals correlation because factor variances are one under standardized solution.
d RMSEA was slightly higher than the cutoff value. With small sample sizes (n < 200), RMSEA tends to be too high and over rejects the true population model (Curran et al, Reference Curran, Bollen, Chen, Paxton and Kirby2003; Hu & Bentler, Reference Hu and Bentler1998). Given this limitation of RMSEA and the other model fit indices CFI and SRMR both being adequate, the factor model was still concluded to be acceptable.
CFA for Each Group
Except for the group with age < 65, the two-factor fluid-crystalized cognition CFA (Figure 1) fit well for all groups with a few minor variations: (1) Working Memory had small cross-loadings on the crystalized cognition factor for the cognitively unimpaired (0.24) and non-under-represented groups (non-URG) (0.19), and (2) the residual variance of Reading was fixed at zero for model identification needs for the dementia/MCI group. Differently, the group with age < 65 had three factors, including EF-PS, memory, and language (Figure 2). More detailed results are summarized in Table 7.
Factor structure of the group with age < 65.
Two-group CFAs for Invariance Testing
Following the results of the single-group CFAs, two-group CFAs were next tested for factorial invariance across diagnosis, sex, race/ethnicity, and education, but not across age. Results are summarized in Table 8.
Summary of factorial invariance testing final models with two-group confirmatory factor analysis
Note. m = number of parameters allowed to differ across groups; CFI = comparative fit index; RMSEA = root mean squared error of approximation; SRMR = standardized root mean squared residual; Δχ 2 = the Satorra-Bentler (SB) scaled correction χ 2 difference statistic; Δdf = change in df; ΔCFI = change in CFI. Model fit is considered adequate by meeting the following criteria: CFI ≥ 0.95, RMSEA ≤ 0.08, and SRMR ≤ 0.08. Violation of invariance is considered under the following criteria: significant Δχ 2 (p < .01), ΔCFI > 0.01.
a χ 2 difference test and ΔCFI are not applicable because the two models are identical.
Across diagnosis: cognitively unimpaired versus dementia/MCI
The results showed that (1) the configural invariance model fit well except that Working Memory was cross-loaded on the crystalized cognition factor for the cognitively unimpaired group only. (2) The metric invariance model had a small deviation from meeting the criteria for similar model fit compared against the configural invariance model, p = .007 for χ 2 difference test and ΔCFI = 0.015. Given that the model had good overall fit and there were no large MIs to indicate misfit in individual parameters, the model was considered acceptable. Partial invariance was allowed such that Episodic Memory differed across diagnosis and yielded a greater loading for the cognitively unimpaired than dementia/MCI, which suggested that the test was more sensitive in detecting individual difference in the underlying latent fluid cognition construct for the unimpaired. (3) With similar justification, the scalar invariance model was considered acceptable with partial invariance. Working Memory and Episodic Memory yielded higher indicator intercepts for the cognitively unimpaired, which implied that these tests were more difficult and less favorable for individuals with dementia/MCI. (4) Four tests had residual variances different across diagnosis, including DCCS, Working Memory, Episodic Memory, and Vocabulary, which indicated that the group difference in some unique factors also contributed to the group difference in the observed scores of these tests in addition to the fluid and crystalized cognition constructs. (5) The two factors had greater variances and lower means for individuals with dementia/MCI than the cognitively unimpaired, which suggested greater individual variabilities and lower levels in the cognition constructs for this group.
Across sex: male versus female
(1) Testing across sex achieved configural, full metric, and close to full scalar invariance, except that Episodic Memory had a slightly higher intercept for females than males, which implied that the test was easier and more favorable for females. (2) All tests had residual variances invariant across sex, which suggested that sex similarity or difference in the test scores can be fully attributable to sex similarity or difference in the underlying fluid and crystalized cognition constructs. (3) The two sexes also had equal factor variances, covariance, and means, which indicated sex similarity in the variabilities, correlation, and average levels of the cognition constructs.
Across race/ethnicity: URG versus non-URG
(1) The two race/ethnicity groups generally had configural invariance, except that Working Memory was cross-loaded on the crystalized cognition factor for non-URG only. (2) All tests had invariant factor loadings, except that Processing Speed had a greater loading for non-URG, which suggested that the test was more sensitive in detecting individual difference in the fluid cognition ability for non-URG. (3) All tests had invariant intercepts, which indicates that the tests had comparable difficulty levels across groups. (4) Invariant residual variances were observed for all tests, except for URG being larger in Reading, which implied that some unique factors contributed more to the Reading scores for URG, and thus contributed to group difference in the scores. (5) The two groups had equal factor variances and covariance, which indicates group similarity in the variabilities and correlation of the cognition constructs. (6) The two groups also had equal means in the fluid cognition factor; however, URG had a lower mean in the crystalized cognition factor.
Across education: low versus high
(1) Testing across education achieved configural, full metric, and full scalar invariance, which implied that all tests had comparable discrimination abilities and difficulty levels for the two groups. (2) Three tests had unequal residual variances, including Flanker, DCCS, and Episodic Memory, which indicated that some unique factors contributed differently to the scores of these tests across education. (3) The two groups had equal variance in the fluid cognition; however, the low education group had a greater variance in the crystalized cognition and a higher correlation of the two factors. (4) The high education group had higher means for both factors.
DISCUSSION
Factor Structure of the Whole Sample
The two-factor fluid-crystalized cognition structure was confirmed for the whole sample and for each group except for the group with age < 65. This factor structure was consistent with previous factor analyses on NIHTB-CB (Hackett et al., Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018; Mungas et al., Reference Mungas, Heaton, Tulsky, Zelazo, Slotkin, Blitz and Gershon2014). These findings support using fluid and crystalized cognition composites for AD research. Fluid abilities are “used to solve problems, think and act quickly, and encode new episodic memories” (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014, p. 2) and are mostly influenced by biological processes. They grow rapidly through childhood, reach a peak at early adulthood, and decline afterward. These abilities tend to be more sensitive to changes in brain structure and functions associated with aging and neurological disorders. Thus, fluid cognition composite could be a sensitive measure to detect cognitive impairment associated with AD. Crystalized abilities “represent an accumulated store of verbal knowledge and skills” (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen and Gershon2014, p. 3) and are influenced by experience, education, and cultural exposure. They develop rapidly during childhood, continue to improve slightly into middle adulthood, and remain stable at late adulthood. Thus, crystalized cognition composite may serve as an efficient measure for cognitive reserve (Hackett et al., Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018). A study (McDonough et al., Reference McDonough, Bischof, Kennedy, Rodrigue, Farrell and Park2016) found that cognitively unimpaired adults whose fluid cognitive ability was worse compared to crystalized cognitive ability measured using factor scores showed evidence of early AD neuropathology evaluated using structural MRI and PET imaging. Larger discrepancy in the fluid and crystalized cognitions was associated with greater beta-amyloid deposition and cortical thickness in AD-vulnerable brain regions. The finding suggested that this discrepancy may be a marker of preclinical AD and highlighted the importance of the distinction between these two cognition constructs.
Different Factor Structure across Age
The two-factor fluid-crystalized cognition structure was held for individuals with age ≥ 65. However, for individuals with age < 65, the fluid cognition factor was separated into two factors: EF-PS and memory. This was aligned with the finding by Hackett et al. (Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018) about the separation of EF-PS and memory into two factors when excluding AD participants, given that AD participants were much older than the rest of the sample on average. Previous research showed that age affects cognitive domains differently (Heaton, Ryan, & Grant, Reference Heaton, Ryan, Grant, Grant and Adams2009; Tulsky et al, Reference Tulsky, Saklofske, Chelune, Heaton, Ivnik, Bornstein, Prifitera and Ledbetter2003). Therefore, a possible reason is that memory may decline at a later age or at a different rate compared to EF-PS, and thus, the two constructs may be more divergent during the transition period from middle to late adulthood. In addition, Flanker, DCCS, and Processing Speed tests all involve reaction time in scoring, whereas the other tests do not. This might have also contributed to age differences in the factor structure given that reaction speed might differ significantly between the two age groups. In total, researchers should exercise caution in the analysis and interpretation of longitudinal changes measured using the fluid cognition composite. Separate composites for EF-PS and memory could be considered for the age population under 65, and individual component tests might be preferred for longitudinal trajectories spanned across 65.
Partial Measurement Invariance across Diagnosis
Configural invariance across diagnosis was confirmed, such that the fluid and crystalized cognition factors were found for both cognitively unimpaired and impaired groups. Partial metric and scalar invariance was found: Episodic Memory was less sensitive in detecting individual difference for the group with dementia/MCI, and Episodic Memory and Working Memory were more difficult and less favorable for this group. Relatedly, higher missing rates were observed for these two tests, which was consistent with the low completion rates found on these tests by Hackett et al. (Reference Hackett, Krikorian, Giovannetti, Melendez-Cabrero, Rahman, Caesar and Isaacson2018). Given that the majority of this group had AD as a cause, these findings highlight two things: (1) impairment in memory is a salient feature in AD dementia and (2) the tests are too challenging for individuals with AD and insensitive at the lower end of memory function, suggesting potentially limited utility for this population. Additional factors could have also contributed to refusal or incompletion, including fatigue associated with immediately administering NIHTB-CB after completion of the NACC UDS 3 battery and unfamiliarity with electronic testing. Both cognition factors had greater variances for the dementia/MCI group than the cognitively unimpaired group, and the correlation of the two factors for the former (0.22) was only about half of the size for the latter (0.42). This suggested more heterogeneity in cognitive abilities for the impaired, which was likely due to the heterogeneity in their disease severity. Nonetheless, lower means for dementia/MCI than the unimpaired found on both factors supported the validity of these factors in distinguishing between clinical diagnoses.
Measurement Invariance across Sex, Race/Ethnicity, and Education
Measurement invariance was generally confirmed across sex, race/ethnicity, and education at the scalar invariance level, allowing meaningful comparisons of latent factor means, variances, and correlation and identification of demographic differences in these factor properties. URG had a lower mean level in crystalized cognition, which could have resulted from cultural differences and historical injustice in the exposure to the contents of test items. Moreover, these factors might have played different roles for each included URG subgroup. The high education group had higher mean levels in both cognition constructs, highlighting the positive influence of education on cognitive function and reserve.
Conclusions
To our knowledge, this is the first study that evaluated factor structure and tested measurement invariance of NIHTB-CB including all seven tests on an AD research sample. Its utility in AD research is supported by the confirmed fluid-crystalized cognition factor structure and its measurement invariance across sex, race/ethnicity, and education. Nonetheless, partial invariance was found across clinical diagnosis, highlighting the potential challenges in measuring memory of individuals with AD. Different factor structures were identified across age, suggesting the possible longitudinal variation in the underlying meaning of fluid cognition.
Limitations and Future Directions
Sample size
In this study, sample sizes (n) for individual impaired diagnoses and minority race/ethnicity groups were small. Small samples tend to have greater probability in model non-convergence and improper solutions, inflated type I error rates, and reduced statistical power for detecting the violation of invariance (Chen, Reference Chen2007; Jorgensen, Kite, Chen, & Short, Reference Jorgensen, Kite, Chen and Short2018; Marsh, Hau, & Wen, Reference Marsh, Hau and Wen2004; Meade, Johnson, & Braddy, Reference Meade, Johnson and Braddy2008). Small ns in these groups also led to unbalanced ns in the two-group CFAs. Unbalanced ns are associated with reduced power, which becomes more severe as the ratio of group ns increases (Brace & Savalei, Reference Brace and Savalei2017; Chen, Reference Chen2007; Yoon & Lai, Reference Yoon and Lai2018). To address these issues, we combined dementia and MCI, and combined more than one race/ethnicity into one group. Such grouping is admittedly problematic, because subgroups are not monolithic. If subgroups have different factor structures, the combined group would represent the largest membership, masking unique pattern(s) of the smaller subgroup(s). We recommend several strategies to address small or unbalanced ns for future invariance testing. (1) Increase efforts to recruit more participants with impaired diagnosis or from minority race/ethnicity groups. (2) The impact of limited ns can be alleviated with greater factor over-determination and higher communalities, which, for example, can be achieved by including more reliable indicators for each factor (MacCallum, Widaman, Zhang, & Hong, Reference MacCallum, Widaman, Zhang and Hong1999; Meade & Lautenschlager, Reference Meade and Lautenschlager2004; Meade & Bauer, Reference Meade and Bauer2007). (3) Two approaches adopted in this study may help yield more robust findings. One is to test CFA on each group separately to first ensure the same factor structure between groups before pooling them together for the two-group CFA. The other is to draw conclusions based on evaluating multiple test indices, including overall model fit indices, change in fit indices between nested models, and MI for individual parameters. (4) The subsampling method, which repeatedly samples a subset of the larger group to have the same n as the smaller group, may provide a solution to achieve adequate power under severe unbalanced ns (Yoon & Lai, Reference Yoon and Lai2018).
Missing data
The two memory tests were too challenging for participants with dementia or MCI and led to high missing rates. In addition, three unreliably extremely high scores on Vocabulary and Reading were excluded given lack of items to appropriately assess the highest functioning individuals. These findings implied limited utility of the battery for such populations. Logistic regression analyses showed that performance on other tests predicted missingness for each situation with high predictive power (c-statistic ranged from 0.87 to 0.97, Supplemental Table S3 and Figure S2). This supported that the data could be missing at random (MAR) if such prediction completely accounted for the missingness. However, if missingness was additionally related to the missing score itself, missing not at random (MNAR) could have occurred, which unfortunately was not testable. We used the FIML estimator to handle missing data. FIML provides unbiased parameter estimates under MAR but biased estimates under MNAR, although the bias tends to be isolated to a subset of model parameters (Enders, Reference Enders2010). The potential bias could possibly include omission of non-invariance or underestimation of difference in factor means across diagnosis.
Biomarker profile
Following the new NIA-AA research framework toward a biological definition of AD based on biomarkers (Jack et al., Reference Jack, Bennett, Blennow, Carrillo, Dunn and Haeberlein2018), the next research steps could be the evaluation of factor structure and measurement invariance across different AT(N) biomarker profiles and brain changes. Findings would help further define the utility scope of NIHTB-CB in AD research.
ACKNOWLEDGEMENTS
This work was supported by the National Institutes of Health (NIH) National Institute on Aging (NIA) (P50AG033514, P30AG062715, R01AG054059) and the Clinical and Translational Science Award (CTSA) program through the NIH National Center for Advancing Translational Sciences (NCATS) (UL1TR002373). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. We thank the Geriatric Research Education and Clinical Center (GRECC) at the William S. Middleton Memorial Veterans Hospital for providing support and resources for this project. This is GRECC manuscript number 004-2020.
CONFLICT OF INTEREST
Dr. Cynthia Carlsson receives grant support from NIH/Lilly, NIH, Veterans Affairs, and Bader Philanthropies. Dr. Sterling Johnson previously served on the advisory board for Roche Diagnostics. Dr. Sanjay Asthana serves as a site PI for pharmaceutical trials funded by Merck Pharmaceuticals, Lundbeck, NIH/UCSD, EISAI, and Genentech Inc.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355617720000922
