Introduction
Executive function is a domain of intellectual function comprising judgment, mental flexibility, planning, initiating, sequencing, monitoring, and inhibiting complex behavior, necessary for independent, purposeful, self-serving behavior (American Psychiatric Association, 1994; Royall et al., Reference Royall2002). Impairment occurs particularly in those with damage to the frontal lobes and sub-cortical brain structures as a consequence of, for example, cerebrovascular disease or Alzheimer's disease (Almkvist Reference Almkvist1994; Chen et al., Reference Chen, Ratcliff, Belle, Cauley, DeKosky and Ganguli2000; Román et al., Reference Román2004).
Impaired executive function is prone to reduce ability to perform activities of daily living and attend to healthcare needs. In a prospective study of 7,717 elderly women living in the community, impairment of executive function, rather than global cognitive impairment, better predicted functional decline and mortality (Johnson et al., Reference Johnson, Lui and Yaffe2007). A very high prevalence (72%) of impairment in executive functioning has been found among medical and surgical patients referred to a liaison psychiatry service (Schillerstrom et al., Reference Schillerstrom, Deuter, Wyatt, Stern and Royall2003), and among medical inpatients in a general hospital (52%), where the impairment was generally not recognized by medical teams (Schillerstrom et al., Reference Schillerstrom2005). Impaired executive functioning is associated with resistance to care in older patients with dementia and with more extensive needs for care in older patients (Royall et al., Reference Royall, Cabello and Polk1998; Stewart et al., Reference Stewart, Gonzales-Perez, Zhu and Robinson1999).
Executive functioning is not well assessed by routine bedside tests of cognitive function such as the Mini Mental State Examination (MMSE) (Folstein et al., Reference Folstein, Folstein and McHugh1975), where the purpose is to screen for dementia, and the focus is principally upon memory and orientation (Schwamm et al., Reference Schwamm, Van Dyke, Kiernan, Merrin and Mueller1987; Naugle and Kawczak Reference Naugle and Kawczak1989; Faustman et al., Reference Faustman, Moses and Csernansky1990; Schillerstrom et al., Reference Schillerstrom, Deuter, Wyatt, Stern and Royall2003). Most neuropsychological tests of executive function are time-consuming and otherwise unsuitable for routine bedside assessment (Schillerstrom et al., Reference Schillerstrom, Deuter, Wyatt, Stern and Royall2003). The Executive Interview (EXIT25), an extensively validated 25-item multitask assessment of executive function, is designed for clinical bedside assessment. It can be administered and scored in approximately 15 minutes with high inter-rater reliability (Royall et al., Reference Royall, Mahurin and Gray1992). A cut-off point of 15 and more out of 50 is proposed to identify significant executive impairment; it discriminates between independent community-dwelling older people and those in care home settings (Royall et al., Reference Royall, Mahurin and Gray1992; Reference Royall1993), and those with and without decision-making capacity regarding healthcare (Holzer et al., Reference Holzer, Gansler, Moczynski and Folstein1997). Concurrent validity is demonstrated with other neuropsychological tests of executive functioning (Royall et al., Reference Royall, Mahurin and Gray1992) and with relevant abnormalities on magnetic resonance imaging (Royall et al., Reference Royall, Rauch, Roman, Cordes and Polk2001). Executive impairment detected by EXIT25 is associated with functional impairment, medication non-compliance in HIV-positive patients (Mann et al., Reference Mann, Westlake, Wise, Beckman, Beckman and Portez1999), and ability to perform activities of daily living independently in patients with schizophrenia (McGurk and Meltzer, Reference McGurk and Meltzer2000; Martinez-Aran et al., Reference Martinez-Aran2002) and depression (Kiosses et al., Reference Kiosses, Klimstra, Murphy and Alexopoulos2001). Notwithstanding these largely positive measurement characteristics, others have pointed out that EXIT25 is still longer than would be practical for routine administration in clinical settings and that it lacked ecological validity with some items requiring the administrator to behave in a socially unconventional manner (Larson and Heinemann, Reference Larson and Heinemann2010).
We aimed to introduce EXIT25 as a part of our routine clinical assessment of older people referred to our general hospital liaison service in order to improve detection of impairment of executive functioning. In so doing, we sought to abbreviate the scale according to classical psychometric scale principles while also assessing hierarchality (the extent to which items are reliably ordered by degree of difficulty). We hypothesized that it would be possible to identify a smaller subset of items with strong unidimensionality, high inter-rater and test-retest reliabilities, high correlation with the original scale score, and good concurrent and predictive validity.
Methods
Participants and procedures
We aimed to recruit 120 clinically stable medical and surgical inpatients aged 65 years and over referred to the Liaison Psychiatry Service for Older People at King's College Hospital, London, UK, for whom the assessing clinician felt that there was a clinical indication to assess executive function. We excluded patients with delirium; those lacking a sufficient grasp of written and spoken English to participate in the task; those with significant visual impairment (several items involve looking at pictures, drawing, and reading); and those with severe global cognitive impairment (an MMSE score of 10 or lower). We routinely recorded age, gender, education, ethnicity, and length of inpatient stay for all the clients of our service. We obtained verbal informed consent for administration of EXIT25 as a routine clinical assessment. We further aimed to have 30 EXIT25 assessments observed and co-rated by a second rater from the research team, and a further 50 participants were assessed for the second time (between 5 and 7 days after initial assessment) using EXIT25 to establish item and scale test-retest reliability. Written informed consent was obtained for this research activity. Full ethics approval was obtained for all aspects of the study (Ref. 08/H0722/102 – NRES Committee London, Camden & Islington Health Research Authority National Research Ethics Service).
Measurement
The EXIT25 is a brief bedside test of executive functions, with items assessing fluency, persistence, resistance to interference, echopraxia (automatic behavior), and primitive release reflexes. Abnormalities that can be detected include perseveration, imitation behavior, intrusions, frontal release signs, lack of spontaneity, and disinhibited behaviors. For each item, a score of 0 indicates no abnormality, and 1 or 2 mild or severe impairment respectively, with fully operationalized rating instructions. Higher total scores, on a range of 0 to 50, indicate greater degrees of impairment. All EXIT25 administrations were carried out by Fedza Mujic (an experienced old age psychiatry associate specialist), or Eyal Lebovich, Damian Clifford, or Maite Von Heising (psychiatry clinical trainees). Fedza Mujic and Eyal Lebovich carried out the inter-rater reliability ratings.
Permission was obtained to use the EXIT25 for this study, and to make necessary adaptations from the North American to British English cultural setting. Only item 4, “Anomalous Sentence Repetition,” of the questionnaire was adjusted to include nursery rhymes that were more familiar to people in the United Kingdom.
Statistical analysis
We first assessed individual items for floor and ceiling effects (item mean score and standard deviation), inter-rater and test-retest reliability (Kendall's τ-B), and item-total correlations. Items were selected for retention in the brief scale on the basis of adequate item-total correlation, and inter-rater and test-retest reliability. Internal consistency (Cronbach's α) was computed and compared for the original and brief scale. We carried out exploratory factor analysis, inspecting scree plots to select the number of factors for extraction, and with varimax rotation for factor loadings. Item response was assessed using the Mokken (Reference Mokken1971) non-parametric item response model to measure the hierarchical properties of items in a scale, assessing whether the items can be ordered by the degree of difficulty so that any individual who fails on a particular item will also fail all the items ranked higher in difficulty. To assess single monotonicity, we estimated the Loevinger coefficients for each item (Hi) and the whole scale (H), where values between 0.3 and 0.4 suggest weak scalability, values between 0.4 and 0.5 moderate, and values above 0.5 suggest strong scalability. We also tested formally for violations of monotonicity (using the Stata loevH monotonicity command) and non-intersection (using the Stata loevH nipmatrix command) between pairs of items (minimum violation 0.03, α = 0.05), using overall criteria values as an indication of the likelihood of assumption violation: ≤40 “satisfactory,” 40 to 79 “questionable violation,” 80 and above “strongly suggesting an assumption violation” (Molenaar and Sijtsma, Reference Molenaar and Sijtsma2000).
We assessed the construct validity of the original and brief versions of the EXIT25 by assessing independent contributions of age, sex, ethnicity, education, and dementia (η 2, derived from a general linear model), and comparing these with the same models with the MMSE score as the outcome. We explored the predictive validity of the EXIT25 by assessing the association of quarters of the score distribution with the count of days of admission, using negative binomial regression.
Results
The full sample comprised 122 older medical or surgical inpatients, 63 men and 59 women, referred to the Liaison Psychiatry Service for Older People. Mean age was 81.4 years, SD = 8.0 (range 62–99 years). Most participants were of white British or Irish ethnicity (n = 98, 80.3%), while 24 had other backgrounds, including 16 (13.1%) born in the Caribbean. The most common psychiatric diagnosis at initial liaison service assessment was dementia (n = 65, 53.3%), followed by depression (15, 12.3%), and five cases each of schizophrenia, alcohol use disorder, and adjustment disorder. Twenty-seven of those assessed (22.1%) did not merit a psychiatric diagnosis at first assessment. Median length of admission was 34 days (range 5 to 195 days). The mean MMSE score was 21.2, SD = 5.0, range 10–30. All participants were administered the full EXIT25 assessment, and all items were rated, with no missing data. The EXIT25 total scores were approximately normally distributed, with a mean of 17.0, SD = 6.3. Two-thirds of the participants (65.6%) scored 15 or more on the EXIT25, indicating likely executive dysfunction.
Psychometric properties of EXIT25
The full 25-item scale showed only moderate internal consistency (Cronbach's α = 0.72). Only 12 items had an item-total correlation of ≥0.20, and only eight items an item-total correlation of ≥0.30 (Table 1).
The main scores and scale properties of EXIT25
Note: *Items included in the final brief scale.
For several items, low correlations with the total score were likely explained by floor or ceiling effects with consequently limited item variance. Floor effects (most participants passing, with an average item score close to zero) were apparent for the Grasp Reflex (Q10), Snout Reflex (Q13), Grip Task (Q19), Utilization Behavior (Q24), and Imitation Behavior Task (Q25). A ceiling effect (most participants failing, with a mean score close to 2) was apparent for Design Fluency (Q3). Most of these items also showed low test-retest reliability (Kendall's τ-B < 0.50), this shortcoming was also evident for the Memory/Distraction task (Q6 – Kendall's τ-B = 0.35), Echopraxia I (Q16 – Kendall's τ-B = 0.15), and Counting Task I (Q23 – Kendall's τ-B = 0.24). Inter-rater reliability was adequate to high for most of the items, with the exception of those with very limited variance. Test-retest reliability for the total 25-item score was 0.86 (95% confidence interval (CI) = 0.77–0.92). Inter-rater reliability was 0.96 (95% CI = 0.92–0.98).
The Kaiser–Meyer–Olkin Measure of Sampling Adequacy (0.62) barely supported exploratory factor analysis, although Bartlett's Test of Sphericity was strongly significant (χ 2 = 537.0, df = 300, p < 0.001). The scree plot suggested a four-factor solution accounting for 35.4% of the scale variance. The first dominant factor with an eigenvalue of 3.8 (15.0% of the variance) comprised Thematic Perception (Q5), Go/No-Go task (Q15), Luria Hand Sequence II (Q18), and Serial Order Reversal Task (Q22), with weaker loadings (>0.50) from Number-Letter Task (Q1), Word Fluency (Q2), and Luria Hand Sequence I (Q17). The second, third, and fourth factors (eigenvalues: 1.55 to 1.84) were much less clearly characterized with much cross-loading, with only one item in the third factor loading at >0.50. The Mokken analysis revealed that the EXIT25, in this setting, could not be considered to be a hierarchical scale. Loevinger's H coefficient for the scale as a whole was 0.16, while for individual items Loevinger's H exceeded 0.30 for only two items. There were critical violations of non-intersection assumptions for almost all items.
Scale abbreviation
In the first phase of abbreviation, we retained items that (a) lacked prominent floor or ceiling effects, (b) had an item-total correlation >0.20, and (c) had a test-retest reliability of >0.30. We then repeated the assessment of internal consistency on the ten retained items, yielding a Cronbach's α of 0.74. However, one retained item, Automatic Behavior II (Q9) now had an item-total correlation of only 0.18. On dropping this item, the resulting nine-item scale had a Cronbach's α of 0.75. All item-total correlations exceeded 0.30; Number-Letter task (Q1, 0.59), Word Fluency (Q2, 0.40), Anomalous Sentence Repetition (Q4, 0.30), Thematic Perception (Q5, 0.42), Memory/Distraction Task (Q6, 0.31), Go/No-Go Task (Q15, 0.54), Luria Hand Sequence I (Q17, 0.41), Luria Hand Sequence II (Q18, 0.49), and Serial Order Reversal Task (Q22, 0.43). The abbreviated nine-item scale was a weak but adequate Mokken scale, with Loevinger's H of 0.33. Only two items had a Loevinger's H of <0.30. There were no significant violations of montonicity, and neither critical non-intersection violations (criteria values range from 0 to 53). Scale test-retest reliability for the abbreviated scale was 0.88 (0.80–0.93), and inter-rater reliability was 0.96 (0.92–0.98).
Construct validity of EXIT25 and Brief EXIT
The Brief EXIT correlated 0.90 (p < 0.001) with the full EXIT25 scale score. Both measures correlated significantly with the MMSE score: EXIT25 (−0.68, p < 0.001) and Brief EXIT (−0.70, p < 0.001). In multivariable models comprising the effects of age in years, sex, years of education, ethnicity, and dementia diagnosis, neither age nor sex was significantly associated with MMSE, EXIT25, or Brief EXIT scores. Years of education was positively associated with MMSE scores (η 2 = 4.9%), but not with EXIT25 (η 2 = 1.3%) or Brief EXIT scores (η 2 = 2.2%). The effect of ethnicity was also more apparent for MMSE (lower, more impaired scores for black and minority ethnic groups; mean difference = −2.7 points, 95% CI = −4.7––0.6; η 2 = 5.4%) than for EXIT25 (η 2 = 2.0%) or Brief EXIT (η 2 = 4.0%). Dementia diagnosis contributed the largest variance component for EXIT25 (15.9%), Brief EXIT (11.4%), and MMSE scores (18.6%).
Contrary to our hypothesis, neither EXIT25 nor Brief EXIT was independently predictive of longer length of hospital stay (Table 2), although the trends for the associations were in the predicted direction, with evidence that increased risk of longer stay was mainly confined to those with severe executive impairment. Those in the top quarter of EXIT25 scores (≥22) were predicted to have a 40% higher count of days in hospital (count ratio: 1.40, 95% CI = 0.91–2.15) than those in the lowest quarter of EXIT25 scores, while a score of ≥11 for Brief EXIT was associated with a 53% higher count of days in hospital (count ratio: 1.53, 95% CI = 0.96–2.43). After controlling for the EXIT25 score, neither age, sex, education nor the MMSE score was associated with the length of hospital stay. Number of medications prescribed (an index of overall illness severity) was the main independent predictor of the length of hospital stay (count ratio: 1.23, 95% CI = 1.03–1.47).
The Effect of EXIT score, MMSE score, and number of medications prescribed on length of hospital stay in days (negative binomial regression), also adjusting for age, sex, and education
Discussion
When tested among 122 older medical and surgical inpatients referred to a psychiatric liaison service, the original 25-item EXIT25 scale was found to lack desirable scaling properties either as a classical or hierarchical scale. Consistent with our hypotheses, it did seem possible to reduce the number of items to nine out of the original 25 while improving internal consistency, test-retest and inter-rater reliability, and maintaining high correlation with the original EXIT25 score, and moderate inverse correlation with the MMSE score. Furthermore, the Brief EXIT scale had superior item response hierarchical scaling properties compared with the original version, providing further evidence to support unidimensionality. Both full EXIT25 and Brief EXIT scores demonstrated relatively favorable characteristics, (a) correlating less strongly with education and ethnicity than did the MMSE, suggesting less test bias with respect to the cognitive traits assessed and (b) providing better independent prediction of length of hospital stay, indicating discriminant predictive validity.
The main strengths of our study are its naturalistic context in a busy clinical service with assessments focused upon clinical need; the completeness of the data collected; the psychometric approach to item reduction; and the linkage of executive function data to a relevant outcome, length of hospital stay. The main weakness is the lack of other neuropsychological tests of executive function to establish the concurrent validity of the EXIT25.
In a previous similar study conducted among clients of a rehabilitation service with stroke or traumatic brain injury, Larson and Heinemann (Reference Larson and Heinemann2010) also report moderate internal consistency, and frequent violation of hierarchical scaling assumptions for the full EXIT25. They used both low Rasch infit (IRT or outfit mean square values) and classical scale (low item-total correlations) theory criteria for item reduction, retaining 14 of the 25 original items in an abbreviated test called the “Quick EXIT.” All of the discarded items were also discarded in our item reduction, and all of the nine items that we retained were among those retained in the Quick EXIT. Five additional items were retained in the Quick EXIT: Design Fluency, Interference Task, Motor Impersistence, Echopraxia, and Grip Task. There were differences in sample, design, and findings. The rehabilitation clients were considered to be minimally impaired in executive function, in contrast with our sample of older liaison psychiatry clients, where two-thirds scored 15 or above on the EXIT25, perhaps because of the lack of a ceiling effect for the highest difficult item, Design Fluency, in the rehabilitation sample. Internal consistency in the rehabilitation sample was higher for the rehabilitation sample than for our liaison psychiatry sample (Cronbach's α 0.86 vs. 0.72 for the full EXIT25, and 0.88 vs. 0.75 for the abbreviated scale), perhaps reflecting the difficulty of neuropsychological testing among those with acute medical and surgical conditions. Nevertheless, both studies suggest that there is potential to abbreviate the original EXIT25, and, in so doing, improve internal consistency and hierarchical scaling properties. These two studies, conducted independently and on samples with different clinical profiles, selected a largely overlapping set of items for inclusion in the abbreviated scale.
Conclusions
Future research could usefully focus on piloting these brief measures of executive function in relevant clinical settings when administered en bloc, rather than with items interspersed in the original longer version. Their performance needs to be assessed against other candidates for brief bedside assessment of executive function, for example, the Addenbrookes’ Cognitive Examination (ACE; Mathuranath et al., Reference Mathuranath, Nestor, Berrios, Rakowicz and Hodges2000), which is now in its third edition (ACE-III), the Frontal Assessment Battery (Dubois et al., Reference Dubois, Slachevsky, Litvan and Pillon2000), and the Brief Memory and Executive Test (Brookes et al., Reference Brookes, Hannesdottir, Lawrence, Morris and Markus2012). Important criteria would be relative ease and time for assessment and scoring, and independent predictive associations with decision-making capacity, adherence with treatment and rehabilitation programs, duration of stay and failed discharge (for hospital inpatients), and increasing needs for care, including placement in residential and nursing home care settings.
Conflict of interests
None
Description of authors’ roles
Dr Fedza Mujic and Professor Martin J. Prince initiated and designed the study, with the latter supervising and advising on all aspects of the study. Dr Fedza Mujic wrote and made application to Ethics Committee for approval and subsequently submitted annual reports, end of study report, and final study summary to the same committee. Drs Fedza Mujic, Eyal Lebovich, Maite Von Heising, and Damian Clifford, did practical part of the study as described in the Methods section. Professor Martin J. Prince analyzed the data for this study and described statistical methods used. Dr Fedza Mujic wrote the first draft of the paper and all authors contributed to the final version with Professor Martin J. Prince finalizing the manuscript.
