Introduction
The Cognitive Abilities Screening Instrument (CASI) was developed to facilitate cross-cultural studies of cognitive aging and dementia in Japanese and Japanese-American elderly individuals (Teng et al., Reference Teng1994). It combined elements from the most commonly used tests of global cognitive functioning in the U.S.A. and in Japan. It has been used in numerous studies in the Pacific Rim; among these are the Kame Project (Graves et al., Reference Graves1996) and the Honolulu-Asia Aging Study (HAAS) (White et al., Reference White1996).
CASI development involved careful consideration of linguistic and neuropsychological content to ensure that Japanese and English versions tapped the same cognitive domains (Teng et al., Reference Teng1994). These are essential steps toward effective cross-cultural comparisons, analogous to exercises carried out across different sites to standardize dementia diagnosis methodology and to harmonize instruments across cultures (Larson et al., Reference Larson1998). Additional steps need to be taken to ensure these preparatory steps are successful (Teresi et al., Reference Teresi, Stewart, Morales and Stahl2006). Here we analyze CASI data from two large studies of Japanese-American elderly individuals to investigate how well the CASI performs across language groups. We focus on differential item functioning (DIF).
DIF is present in a test item if people from different demographic groups have unequal probabilities of item success when controlling for the ability measured by the test (Camilli and Shepard, Reference Camilli and Shepard1994). DIF assessment is a first step in determining whether test items may be biased (Camilli and Shepard, Reference Camilli and Shepard1994). Next, the impact of DIF should be considered. Epidemiologists may not care as much about how many items have DIF as they do about DIF's impact on groups. Could group differences be caused (or masked) by items with DIF? For clinicians, the primary interest may be the DIF's impact on individual scores. If DIF has a large impact, scores that account for DIF should be used. See Crane et al. (Reference Crane2007) for further discussion.
Previous studies have found DIF related to language in the Mini-mental State Examination (Folstein et al., Reference Folstein, Folstein and McHugh1975) when comparing Spanish versus English test-takers (Teresi et al., Reference Teresi, Golden, Cross, Gurland, Kleinman and Wilder1995; Marshall et al., Reference Marshall, Mungas, Weldon, Reed and Haan1997; Crane et al., Reference Crane, Gibbons, Jolley and van Belle2006; Dorans and Kulick, Reference Dorans and Kulick2006; Edelen et al., Reference Edelen, Thissen, Teresi, Kleinman and Ocepek-Welikson2006; Jones, Reference Jones2006; Morales et al., Reference Morales, Flowers, Gutierrez, Kleinman and Teresi2006). In addition, tests of cognitive function are often found to have DIF related to education and other demographic factors (Teresi et al., Reference Teresi, Golden, Cross, Gurland, Kleinman and Wilder1995; Reference Teresi, Kleinman and Ocepek-Welikson2000; Jones and Gallo, Reference Jones and Gallo2001; Reference Jones and Gallo2002; Crane, Reference Crane2006; Crane et al., Reference Crane, van Belle and Larson2004; Reference Crane, Gibbons, Jolley and van Belle2006). None of these studies evaluated DIF's impact on individuals or groups.
The primary goal of this study is to examine CASI items for DIF in Japanese-Americans to determine how successful the test developers were in developing an instrument that has equivalent measurement properties in Japanese and English speakers. Secondary goals are to investigate the CASI for DIF related to other covariates, including age, sex, and education. We are interested in DIF's presence and its impact on groups and individuals.
Methods
Overview
We used item-level CASI data from the Kame Project and HAAS to evaluate items for DIF within each study. We used a hybrid ordinal logistic regression – item response theory (IRT) approach for DIF detection. We analyzed data from the Kame Project regarding language of testing; there were too few Japanese test-takers in HAAS (n = 120) to address this question. We analyzed CASI items from both studies for DIF related to age, sex and educational attainment. We determined DIF presence and its impact on groups and individuals.
Data source: the Kame Project
A schematic representation of the samples analyzed in this study is shown in Figure 1. A cohort of 3,045 eligible individuals aged 65 and over, 96% of whom were of 100% Japanese origin, was identified in a census of Japanese Americans in King County in November 1991. Study census details are described elsewhere (Graves et al., Reference Graves1996). Of those eligible, 1,985 (65.2%) participated in the baseline evaluation (1992–94).
Schematic representation of study designs and covariates examined for DIF.
The CASI was administered to all participants. Participants were stratified by CASI score and age and sampled into the clinical and neuropsychological evaluation phase of the prevalence study. Of 1,985 individuals participating in CASI screening, 450 were sampled from three cognitive strata and five age strata, with individuals scoring <81 or age ≥80 sampled with 1.0 probability, and younger and higher scoring individuals sampled at lower frequencies. Results of the clinical and neuropsychological evaluations were reviewed in consensus conferences, and dementia diagnoses were made using DSM-IIIR criteria (American Psychiatric Association, 1987). Data analyzed here come from the baseline examination 1992–94 for the 1,836 non-demented participants. Trained interviewers rated CASI scores as invalid for participants with limitations in hearing, eyesight, or motor control, leaving 1,708 evaluated here.
Data source: HAAS
The Honolulu Heart Program (HHP) cohort included 8,006 Japanese-American men born from 1900 to 1919 living on the island of Oahu, Hawaii, at enrollment in 1965 (Syme et al., Reference Syme, Marmot, Kagan, Kato and Rhoads1975). Three midlife examinations conducted in 1965–68, 1968–70, and 1971–74 included the collection of clinical and demographic information. At the fourth examination (1991–93), HAAS began as an extension of the HHP. At this examination, 3,734 HHP cohort members (80% of survivors) participated and took the CASI. A multi-step procedure was used to identify individuals with dementia (for details, see White et al., Reference White1996). Dementia diagnoses were made using DSM-IIIR criteria (American Psychiatric Association, 1987). At HHP Exam 4 (the baseline HAAS exam), there were 3,508 dementia-free participants of whom 3,148 had valid CASI scores; these data are analyzed here.
HAAS wives of Japanese ancestry
A probability sampling based on age and CASI score determined which men were selected for further evaluation in the second stage of the dementia assessment. Those selected were asked to identify a caregiver or potential caregiver; this person was invited for a proxy interview. In most cases, this was the participant's wife. CASI scores were available for 489 wives of HAAS participants, 477 of whom were of Japanese ancestry and were included in our analyses of DIF related to sex. We thus examined HAAS CASI items for DIF related to sex on 477 + 3,148 = 3,625 participants.
Measure: CASI
The CASI is a 40-item test of global cognitive functioning (Teng et al., Reference Teng1994). Items were identical or similar to ones used in the Hasegawa Dementia Screening Scale (Hasegawa, Reference Hasegawa, Bergener, Lehr, Lang and Schmitz-Scherzer1983), the MMSE (Folstein et al., Reference Folstein, Folstein and McHugh1975) and the Modified MMSE (Teng and Chui, Reference Teng and Chui1987). A new item on judgment was added. The CASI was developed in parallel in English and Japanese at three workshops in which items were scrutinized for cultural equivalence, back-translated and pilot-tested.
Item response theory
We scored CASI items using IRT so that scores accounting for DIF could be formed, as explained below. We used PARSCALE (Muraki and Bock, Reference Muraki and Bock2003), with Samejima's graded response model (Samejima, Reference Samejima1969), a polytomous extension of the two-parameter logistic model. The mean score was zero with a standard deviation of one.
DIF detection methods
We have developed an approach to DIF assessment that combines ordinal logistic regression and IRT. Details of this approach have been outlined previously (Crane, Reference Crane2006; Crane et al., Reference Crane, van Belle and Larson2004; Reference Crane, Gibbons, Jolley and van Belle2006). We used the Stata (StataCorp, 2007) program “difwithpar” (Crane et al., Reference Crane, Gibbons, Jolley and van Belle2006) to detect DIF and obtain IRT scores accounting for DIF. The program is available by typing “ssc install difwithpar” at the Stata command prompt.
We examined three ordinal logistic regression models for each item:
In these models, cut represents the cutpoint(s) for each level in the proportional odds ordinal logistic regression model, θ (theta) is the IRT estimate of cognitive ability, and “group” is the indicator for the demographic covariate we are assessing. In model 3, β3 is the coefficient for the ability-group interaction term.
Two types of DIF are identified (Crane et al., Reference Crane, Gibbons, Jolley and van Belle2006). In items with non-uniform DIF, demographic interference between ability and item responses differs at varying ability levels. In items with uniform DIF, the interference is the same across all ability levels. To detect non-uniform DIF, we compared the log likelihoods of models 2 and 3 to test the significance of the interaction term. We used an alpha level of 0.002, based on Bonferroni adjustment for the 27 items with enough discordance to analyze in Kame. For comparability, we used 0.002 for all other non-uniform DIF assessment. To detect uniform DIF, we determined the relative difference between parameters associated with θ [β1 from equations 1 and 2] using the formula |(β1(equation 2)-β1(equation 1))/β1(equation 1)|. If the relative difference was >10%, group membership interfered with the expected relationship between θ and item responses.
We accounted for DIF by using items free of DIF as anchors and estimated group-specific item parameters for items with DIF. The need for group-specific item parameters required that continuous covariates be categorized. The resulting ability estimates (θ) were generated from all of the items, but only DIF-free items were used in common across groups.
Spurious false-positive and false-negative results may occur if ability scores used for DIF detection includes many items with DIF (see Millsap, Reference Millsap2006). We used the θ score which accounted for DIF as the ability level for DIF detection, and re-ran models 1–3. We compared the lists of items found with DIF using original and modified θs. If the same items were identified, we concluded our findings were not related to spurious DIF. If different items were identified, we used the most recent findings to generate new θ estimates and repeated these steps until the same items were identified on successive runs. The final θ values account for DIF related to that covariate.
Individual and group-level DIF impact
We determined DIF's impact for each covariate for individuals by subtracting unadjusted IRT scores from IRT scores accounting for DIF. If DIF made no impact the difference would be 0. We needed a reference point for understanding DIF's impact. Since minimal clinically important score differences (Guyatt et al., Reference Guyatt, Osoba, Wu, Wyrwich and Norman2002) have not been specified for the CASI, we used the median standard error of measurement; individual differences related to DIF larger than this value have a salient difference related to DIF (Crane et al., Reference Crane, Gibbons, Jolley and van Belle2006). We performed analogous calculations to compare group mean scores when accounting for and when ignoring DIF to determine group-level DIF impact.
Covariates assessed for DIF
A schematic representation of covariates assessed for DIF is shown in Figure 1. We analyzed Japanese vs. English language testing in the Kame Project; too few HAAS participants took the test in Japanese to analyze DIF related to language. We analyzed data from the studies separately for DIF related to self-reported proficiency reading or writing Japanese, age, sex, and self-reported educational attainment. Categories are as shown in Table 1, except that the two lowest educational groups were combined for the Kame participants.
Demographic characteristics of participants in the Kame Project and HAAS
*477 wives of Japanese descent who completed the CASI were included in the analyses of DIF related to sex.
Other statistics
Demographic features were compared using Kruskal-Wallis tests for continuous variables and χ2 tests for categorical variables.
Ethical approval
The Kame Project was approved by the University of Washington institutional review board. HAAS was approved by institutional review boards of Kuakini Medical Center (Honolulu, Hawaii) and the Honolulu Department of Veterans Affairs. All participants in both studies gave written informed consent.
Results
Demographic characteristics
Study cohort demographic characteristics are shown in Table 1. On average, Kame participants were more likely to be tested in and to read or write Japanese, were younger, and had more years of education than HAAS participants (all p values < 0.001). HAAS wives were slightly younger (53% were ≤75; p < 0.001) and had fewer years of education (43% had ≤ 8 years; p < 0.001) than HAAS men.
DIF
In these community-dwelling participants without dementia, some of the items were answered correctly by almost everyone (meaning they were too easy for those participants) and hence did not provide enough variability to be analyzed for DIF. This was particularly the case among Kame participants, where only 27 of the 40 items had enough variability to be assessed for DIF. HAAS had more participants, and all items had enough variability to be analyzed for DIF related to at least one covariate.
Language
In Kame, seven CASI items had DIF related to language of testing (Table 2), and four of these had DIF related to self-reported proficiency in reading or writing Japanese (see Table S1 published as supplementary material online attached to the electronic version of this paper at www.journals.cambridge.org/jid_IPG). The group-level impact of DIF related to language of testing was minimal, with mean group changes close to zero (see Figure S1 published as supplementary material). On the individual level, minimal changes due to DIF occurred in both the positive and negative directions in both Japanese and English speakers, with only one participant experiencing salient DIF. In HAAS, no items had DIF related to self-reported proficiency reading or writing Japanese.
Item-level findings of differential item functioning related to language of testing (Kame), age, and education*
*Non-uniform DIF is present (bold) if p <0.002. Uniform DIF is present (bold) if the % change in β1 is > 10.
**Age categories were 64–74, 75–79 and 80–98 years.
***Education categories were 0–11, 12, and 13+ years in Kame and 0–8, 9–11, 12, and 13+ years in HAAS.
–Indicates an item did not have enough discordance to analyze with these subgroups.
Age, sex and education
In both studies, few items had DIF related to age (Table 2), and no items had DIF related to sex. There were more items with DIF related to educational attainment identified in Kame (13 of 27 items) than in HAAS (3 of 35 items) (Table 2). However, the impact of this DIF was minimal. Figure 2 shows the impact of DIF related to education among Kame Project participants. Only 27 Kame participants (1.5%) and 1 (0.2%) HAAS participant had salient DIF related to education. Other DIF impacts were smaller (see Figure S1 published as supplementary material online at www.journals.cambridge.org/jid_IPG). Mean group changes were 0.1 point or less.
Minimal impact of DIF with respect to educational attainment among Kame Project participants, operationalized as the difference between the IRT CASI score accounting for DIF and the naive IRT CASI score. In the box-and-whiskers plots, the box spans the 25th to 75th percentiles, with the median indicated. The whiskers define 1½ times the inter-quartile range; individual observations more extreme than this are indicated with dots. The bold vertical lines indicate the median value of the standard error of measurement for the population, the threshold for salient DIF. HS = high school.
Discussion
The CASI was designed to facilitate measurement of cognitive functioning in Japanese and Japanese-American elderly participants, irrespective of whether they were more comfortable in Japanese or English. Careful traditional methods were employed to increase comparability of CASI scores across the languages. This paper represents the first attempt to quantify how successful these efforts have been in producing a test that is valid for assessing cognitive functioning in Japanese- and English-speaking Japanese-American participants.
The CASI was better targeted to participants in the HAAS study than the Kame Project. Many CASI items could not be analyzed for DIF in Kame due to a high proportion of correct scores. Few HAAS participants took the test in Japanese. These facts made a thorough investigation of the CASI for DIF related to language of testing impossible. The evaluations we were able to perform found a few items with DIF in Kame, though that DIF appeared to have minimal impact on individuals or groups. This suggests that effort directed at the development of the CASI (Teng et al., Reference Teng1994) was worthwhile; the test appears to have minimal DIF related to language.
The Japanese-English language analyses were unique to this paper, but findings were similar to prior studies comparing Spanish versus English in the MMSE (Teresi et al., Reference Teresi, Golden, Cross, Gurland, Kleinman and Wilder1995; Marshall et al., Reference Marshall, Mungas, Weldon, Reed and Haan1997; Crane et al., Reference Crane, Gibbons, Jolley and van Belle2006; Dorans and Kulick, Reference Dorans and Kulick2006; Edelen et al., Reference Edelen, Thissen, Teresi, Kleinman and Ocepek-Welikson2006; Jones, Reference Jones2006; Morales et al., Reference Morales, Flowers, Gutierrez, Kleinman and Teresi2006), which also found several items with DIF related to language of testing. We also examined DIF related to self-reported proficiency in reading or writing Japanese. There was no DIF in HAAS and little in Kame, and DIF's impact on individuals or groups was minimal in Kame.
DIF related to other demographic covariates has been examined for several global cognitive tests (Teresi et al., Reference Teresi, Kleinman and Ocepek-Welikson2000; Crane, Reference Crane2006; Crane et al., Reference Crane, van Belle and Larson2004; Reference Crane, Gibbons, Jolley and van Belle2006; Jones and Gallo, Reference Jones and Gallo2001; Reference Jones and Gallo2002). Consistent with this literature, we found few items had DIF related to sex or age. A few items had DIF related to education, though this DIF had minimal impact on scores of individuals or groups. This finding is different from the extant literature on DIF's impact related to education in global cognitive tests; in general, individual and group impact of DIF related to educational attainment is much greater than that related to other covariates.
Several limitations should be kept in mind when interpreting our results. While we performed sensitivity evaluations of our DIF findings using other criteria, there is no universal agreement on DIF detection techniques. Different methods may have produced different results, though in broad terms most approaches seem to have similar findings when applied to the same data sets (Millsap, Reference Millsap2006). It should also be noted that the women included in the HAAS analysis of DIF related to sex were wives of HAAS participants who were more likely to have dementia and are not representative of all women of Japanese ancestry on the island of Oahu. In the Kame Project, the study census enumerated >90% of the Japanese-American population living in King County in 1990, and thus the Kame population can be said to represent the Japanese-American population of King County, Washington (Graves et al., Reference Graves1996).
It is of some interest that despite item-level data on some 3,700 Japanese-American participants across two studies we were unable to evaluate fully the CASI for DIF related to language of testing. There are several reasons for this. First, an extremely high proportion of HAAS participants took the CASI in English; thus, despite having a sample roughly twice the size of Kame, roughly a third as many HAAS participants took the CASI in Japanese. Second, many CASI items had 100% correct responses in Kame. While methods for DIF analyses of small sample sizes exist (Lai et al., Reference Lai, Teresi and Gershon2005), these methods fail when there is a discrepancy between ability levels and item difficulties.
A further complication of any analysis of DIF related to language of test administration is differences in covariates across language groups. In HAAS, for example, 9% of the Japanese speakers had a high school education, compared with 51% of their English-speaking counterparts. A naive analysis of DIF related to language of test administration may uncover DIF that is, in fact, due to education. Methods exist for assessing DIF related to language when other covariates (such as education) are very different across language groups, but for the reasons outlined above we were unable to perform such analyses here.
A more powerful way to determine whether there was DIF in the CASI related to language of test administration is to perform targeted, stratified data collection. Hundreds of Japanese-speaking (and English-speaking) elderly Japanese-Americans would need to be sought and examined with the CASI. Sample sizes for DIF analyses are discussed in Crane et al. (Reference Crane, Gibbons, Jolley and van Belle2006). Ideally both English-speaking and Japanese-speaking cohorts would have broad and overlapping distributions of educational attainment and other demographic characteristics. While this stratified sample would not represent the general population, IRT has important invariance properties such that item parameters are invariant across samples within a linear transformation if model assumptions are met.
To the extent that we could assess DIF, the CASI appeared to meet the goal of measuring cognitive function equivalently in Japanese and English test takers. We found many items with DIF related to the covariates we analyzed, but DIF impact was minimal, producing small and clinically insignificant differences in scores. Targeted data collection would be needed to confirm this conclusion. Methods such as these should be used in other studies with multiple language groups, to confirm that measures function equivalently or, if not, form scores that account for DIF.
Conflict of interest
None.
Description of authors' roles
L. Gibbons contributed to the conception and design of the paper and the drafting of the manuscript, and conducted all the statistical analyses. S. McCurry and K. Rhoads made critical revisions of the manuscript for important intellectual content. K. Masaki contributed to data collection and made critical revisions of the manuscript for important intellectual content. L. White, A. Borenstein and E. Larson obtained funding, contributed to data collection and made critical revisions of the manuscript for important intellectual content. P. Crane obtained funding and contributed to the conception and design, interpretation of data, and drafting of the manuscript.
Acknowledgments
These analyses were supported by an Alzheimer's Association Investigator Initiated Research Grant (P. Crane). Data collection was supported by NIA N01-AG-4-2149, NIA U01-AG-1-9349-01, NIA 1RO1 AG19349-01, NHLBI N01-HC-05102, and NIH R01 09769. Programming to detect differential item functioning was supported by NIH P50 AG05136 (Raskind). Dr. Larson was supported by NIH U01 AG06781. The funding sources had no role in how the study was conducted nor in the decision to publish the results.
Supplementary materials (attached to the electronic version of this paper, available online at www.journals.cambridge.org/jid_IPG )
• Table S1 contains item-level findings of differential item functioning related to self-reported proficiency reading or writing Japanese for the Kame Project and for the Honolulu-Asia Aging Study.
• Figure S1 shows differential item functioning impact for other covariates analyzed for DIF. This figure is a complement to Figure 2 in the text.
