Introduction
Delirium is characterized by disturbances in attention and awareness that come with impairment in at least one other cognitive function (e.g. memory, orientation, language, visuospatial ability, perception). These disturbances develop suddenly and fluctuate throughout the day (American Psychiatric Association, 2013). Delirium occurs in 29% to 64% of seniors in medical and geriatric wards, and up to 38% in patients in post-acute care and in nursing home residents (Inouye et al., Reference Inouye, Westendorp and Saczynski2014). The consequences of delirium include increased risk of functional and cognitive decline, dementia, institutionalization, and death (Witlox et al., Reference Witlox, Eurelings, de Jonghe, Kalisvaart, Eikelenboom and van Gool2010; Inouye et al., Reference Inouye, Westendorp and Saczynski2014). Delirium also leads to higher healthcare costs due to longer hospital stay and increased demands on nursing staff (Rizzo et al., Reference Rizzo, Bogardus, Leo-Summers, Williams, Acampora and Inouye2001). In subsyndromal delirium, studies have associated core symptoms of delirium with such negative outcomes as cognitive, functional, behavioral, and mood problems (Cole et al., Reference Cole2014). These consequences emphasize the importance of detecting delirium early in order to identify and treat its underlying causes (e.g. dehydration, infection, reaction to medication, myocardial infarction).
While some reliable and valid tools exist to help clinicians detect the presence of delirium (Adamis et al., Reference Adamis, Sharma, Whelan and Macdonald2010; Wong et al., Reference Wong, Holroyd-Leduc, Simel and Straus2010; Inouye et al., Reference Inouye, Westendorp and Saczynski2014), physicians and nurses generally fail to recognize this syndrome in 32% to 69% of patients (Flaherty et al., Reference Flaherty2007). Time constraints and lack of training or expertise in the use of the tools are major barriers to daily delirium screening (Adamis et al., Reference Adamis, Sharma, Whelan and Macdonald2010; Brummel et al., Reference Brummel, Vasilevskis, Han, Boehm, Pun and Ely2013). Consequently, researchers have proposed the use of a brief and simple test to screen for the presence of key features of delirium before resorting to longer mental health assessment, as an interesting avenue to explore (O'Regan et al., Reference O'Regan2014).
Since disturbances in attention and awareness are essential symptoms for diagnosis of delirium (American Psychiatric Association, 2013), more and more experts and researchers have been trying to find simple, short tests that would target these symptoms and be sensitive to delirium (Flaherty et al., Reference Flaherty2007; Kolanowski et al., Reference Kolanowski, Fick, Yevchak, Hill, Mulhall and McDowell2012). Validated standardized tests to capture inattention are available for use in patients with, and without, cognitive impairment, but their characteristics make them difficult to use in clinical practice (Kolanowski et al., Reference Kolanowski, Fick, Yevchak, Hill, Mulhall and McDowell2012). For example, some tests require materials (computer, pen, and paper), while neuropsychologists or well-trained clinicians must administer others. Other available tests that are more feasible for use by nurses, include the Serial Sevens Test (subtraction by 7 from 100) of the Mini-Mental State Examination (MMSE), the MOTYB (reciting the months of the year backward), and several others, such as the digit span forward and digit span backward (repeating a string of numbers forward or backward), counting the number of times the examiner claps their hands, naming the days of the week backward, or signaling when a letter is said (Kolanowski et al., Reference Kolanowski, Fick, Yevchak, Hill, Mulhall and McDowell2012; Inouye et al., Reference Inouye, Westendorp and Saczynski2014). Researchers are currently trying to gauge the usefullness of several of these bedside tests for the assessment of attention in the context of delirium. As our contribution to this effort, we used data from a previous study to explore the usefulness of ten short, simple, and objective attention tests for delirium screening in elderly adults with, and without, cognitive impairment. These tests are drawn from the concentration subscale of the HDS (Cole and Dastoor, Reference Cole and Dastoor1987).
Methods
Design, settings, and selection of participants
This paper presents secondary analyses of data collected in a large-scale study on delirium among older patients/residents in acute and LTC settings (Voyer et al., Reference Voyer2015). The study was conducted on three units (medical, cardiology, and coronary care) of an ACH and five units of a LTC facility. Data collection took place from November 2012 to November 2013. The Research Ethics Boards of each participating facility approved the study.
Patients/residents enrollment
Patients (ACH) and residents (LTC), with or without a medical diagnosis of dementia in their medical chart, were included in the study if they were aged 65 years or over and were receiving medication at least twice a day. They were excluded if they had a history of psychiatric illness (specifically psychotic disorders, bipolar disorders, and major depressive disorders with psychotic features). A research assistant (RA) compiled weekly lists of all eligible patients/residents for each study site. As a result, in ACH, some patients had just been admitted to the unit whereas others were admitted one or two weeks before entry in the study. Similarly, for recruited residents, some had been admitted to their nursing home in the previous week and others months, or even years previously. Recruitment procedures were different for individuals with cognitive impairment and those without. Presence of cognitive impairment was defined as the presence of any indicators (e.g. dementia, confusion, disorientation, etc.) in the medical chart or reported by the bedside nurse, which could affect the patient's/resident's ability to give informed consent. For individuals with cognitive impairment, institution staff sent a letter about the study to the legal guardians or, in the absence of a legal guardian, to the family member designated as guardian, and invited them to contact the RA for further information. This letter was sent to all families. Respondents who agreed to participation on behalf of another were requested to sign proxy consent forms. Each week, consenting patients/residents were randomly selected to join in the study. In addition, direct assent to participation was obtained from individuals deemed to have mild or moderate cognitive impairment (according to the RA's clinical judgment). Potential participants from among patients/residents without cognitive impairment were also randomly selected from the weekly list of all those eligible for the study. Their bedside nurse asked them if they would agree to meet with the RA to learn about the project. Then, if they agreed to take part, they were invited to sign the consent form. Up to six patients/residents were recruited each week.
Data collection and measures
The study involved two RAs per study site (ACH and LTC). The first RA used the CAM to determine the presence of delirium and the HDS to assess both attention (concentration subscale) and the severity of cognitive impairment (full scale). The first RA completed the CAM with three data sources (chart, family members, if available, and nursing staff) and also had to make three structured assessments of delirium symptoms over a 7-hour observation period. The second RA was blinded to the CAM delirium status of patients/residents and collected the participant's baseline characteristics (age, sex, date of admission, level of functional autonomy, level of comorbidity, and type of dementia, if any).
Instruments
Primary measures
Attention was assessed with the concentration subscale of the HDS (see description of the HDS in descriptive measures). This subscale includes ten attention tests. Each test comprises a brief objective task to measure attention including some that are commonly used in clinical settings and in research: Serial Sevens Subtraction; Serial Threes Subtraction; MOTYB; Days of the Week Backward; Counting 93 to 85; Counting 10 to 1; Months of the Year Forward; Days of the Week Forward; Counting 1 to 10. The tasks are organized hierarchically in decreasing order of difficulty. The score obtained on the subscale ranges from 0 to 10, and represents the most difficult attentional task that the participant completed successfully. For instance, if Item 7 is the most difficult task successfully completed, a score of 7 is given for the Concentration subscale of the HDS.
The presence of delirium symptoms was measured with the CAM, an established and widely used instrument to aid in the detection of delirium in both clinical and research settings (Inouye et al., Reference Inouye, van Dyck, Alessi, Balkin, Siegal and Horwitz1990). This instrument takes 10 minutes to complete and enables a non-psychiatrist clinician to capture the features of delirium specified in the Diagnostic and Statistical Manual of Mental Disorders (revised third edition). The CAM has been shown to be sensitive (94% to 100%) and specific (90% to 95%), compared to the diagnosis of a psychiatrist (Inouye et al., Reference Inouye, van Dyck, Alessi, Balkin, Siegal and Horwitz1990; Wei et al., Reference Wei, Fearing, Sternberg and Inouye2008) and its use is supported by evidence in a literature review on the accuracy of bedside instruments for diagnosing delirium (Wong et al., Reference Wong, Holroyd-Leduc, Simel and Straus2010). In the present study, delirium was defined as meeting the criteria for DSM-5 delirium: a disturbance in attention and awareness that develops over a short period of time and tends to fluctuate in severity over the course of the day, together with a disturbance in cognition (e.g. memory impairment, disorientation, language, visuospatial perceptual disturbances). RAs underwent training to recognize these symptoms of delirium using the CAM. The training involved clinical demonstrations, trial ratings of cases and discussion of findings until there was satisfactory agreement between the research staff and the principal investigator (PV). During the study, RAs assessed samples of patients and residents simultaneously and independently for inter-rater reliability (n = 37/193; 19% of total sample). Percentage of agreement varied from 72% to 100%, while κ values ranged from 0.30 to 1, depending of the item tested.
Descriptive measures
Cognitive impairment severity was assessed with the HDS (Cole and Dastoor, Reference Cole and Dastoor1987). This scale was devised for adaptive testing and comprises 20 subscales assessing a wide range of cognitive functions. The maximum score for the entire scale is 200 points. Cognitively intact older adults generally score the maximum number of points, or close to it (Bickel, Reference Bickel1996). The scale displays almost no floor effect; i.e. participants with severe dementia can respond correctly to some of the easiest items and obtain scores other than zero (Ronnberg and Ericsson, Reference Ronnberg and Ericsson1994). The HDS has well-established validity and reliability (Bickel, Reference Bickel1996). For example, Ronnberg and Ericsson (Reference Ronnberg and Ericsson1994) reported a test–retest reliability of 0.96 (Spearman ρ coefficients) and a concurrent validity coefficient of 0.86 between the HDS and the Mini-Mental Status Examination. Engelborghs and colleagues (Engelborghs et al., Reference Engelborghs2006) also demonstrated a significant correlation between HDS scores and biological markers of Alzheimer's disease.
Functional autonomy was measured with the Functional Autonomy Measurement System (SMAF) (Hébert et al., Reference Hébert, Carrier and Bilodeau1988). This 29-item scale measures functional ability in five areas: activities of daily living (ADL: seven items), mobility (six items), communication (three items), mental functions (five items), and instrumental activities of daily living (IADL: eight items). In this study, we used a modified 20-item version for institutionalized participants (excludes the eight IADL items and one item related to exterior mobility). Disability for each item is scored on a 5-point scale: 0 = independent, 0.5 = with difficulty, 1 = with supervision, 2 = with help, and 3 = dependent. This version of the scale has shown good test–retest and inter-rater reliability (ICC of 0.95 and 0.96, respectively) (Desrosiers et al., Reference Desrosiers, Bravo, Hebert and Dubuc1995). Higher score indicates severe autonomy impairment.
Data extracted from the participants’ medical charts included: demographic information (e.g. age, sex, and years since admission), diagnosis of dementia, and other medical diagnoses. Information on medical problems was used to compute the Charlson Comorbidity Index (Charlson et al., Reference Charlson, Pompei, Ales and MacKenzie1987), which is validated as a predictive index for survival among older patients and nursing home residents (Bravo et al., Reference Bravo, Dubois, Hebert, De Wals and Messier2002; Buntinx et al., Reference Buntinx, Niclaes, Suetens, Jans, Mertens and Van den Akker2002). The overall score ranged from 0 to 37, with higher scores indicating greater comorbidity.
Statistical analyses
Means and standard deviations were used to present the participants’ characteristics, as were frequencies and percentages to describe continuous and discrete variables, respectively.
To explore the usefulness of the ten attention tests that comprise the HDS concentration subscale in the detection of delirium, each test item was transformed into a dichotomous variable (success or failure). Because the items in the subscale are arranged in descending order of difficulty, researchers had to assume that participants had succeeded in the easier items and failed the more difficult ones, even if the RA had not necessarily administered them. For instance, a participant who correctly answered Item 7 but failed Item 8, would be rated 0 (failure) for Items 8 to 10 and rated 1 (success) for Items 0 to 7. The proportion of participants who successfully completed each test item was computed for the entire sample. Subgroup analyses were also conducted to compare participants’ performance according to their delirium status and also to the presence of indicators of cognitive impairment in the medical chart. Differences between subgroups were assessed using Fisher's exact test.
Finally, using a DSM-5 criterion-defined delirium as gold standard, we used sensitivities, specificities, as well as predictive values (positive and negative) and their confidence intervals, and calculated the concurrent validity for each item of the Concentration subscale. All analyses were carried out using SAS for Windows, version 9.4.
Results
Description of the study population
A total of 589 patients and 90 residents met the eligibility criteria (Figure 1). There were 267 (39%) invited to participate in the study and 210 (79%) consented. Of those who consented, 17 (8%) were not evaluated (seven revoked their consent, five were discharged home, and five were no longer eligible). Of the remaining 193 participants, data on the HDS Concentration subscale was missing for two participants (1%), leaving a sample of 141 patients and 50 residents for analyses. Table 1 presents the characteristics of these 191 participants. Mean age was 81 years. Almost 20% of the participants had a high level of comorbidity (score ≥8 on the Charlson comorbidity index) and 24 (12%) showed substantial deterioration in their functional autonomy (score ≥ 29 on the SMAF). For example, 24 (13%) were completely dependent for dressing, 40 (21%) were dependent for personal hygiene, and 22 (12%) were unable to walk. Participants took an average of ten medications per day. According to the medical charts, 62 (32%) had cognitive impairment indicated in their medical record; 40 (21%) had a diagnosis of dementia, and 23 (12%) met the DSM-5 criteria for delirium.
Flow Chart.
Characteristics of participants
a Indication of cognitive impairment in the medical chart.
SD: Standard deviation, HDS: Hierarchic Dementia Scale, SMAF: Functional Autonomy Measurement System, CCI: Charlson comorbidity index.
Participants’ performance for each attention test on the HDS concentration subscale
As Table 2 shows, the results we obtained varied a great deal. We found that, among the whole sample, the majority of the participants (n = 155, 81%) were able to Recite the days of the week backward, and slightly more than half (n = 109, 57%) could name the MOTYB. Only 13% (n = 24) successfully completed the most difficult test of the subscale (Serial Sevens Subtraction), while 8% (n = 15) failed all test items. Subgroup analyses comparing the performance of participants with delirium with those participants without delirium, revealed a statistically significant difference (p < 0.01) between groups for four test items (Serial Threes Subtraction, MOTYB, Days of the Week Backward, and Counting from 93 to 85). Participants without delirium were 33% to 46% more likely to succeed in these particular items than the participants with delirium. In the analyses comparing participants with indicators of cognitive impairment in their medical charts with those participants without cognitive impairment, the latter group performed significantly better (p = <0.0001) on all test items. More than 97% of participants with no indicators of cognitive impairment in their medical chart successfully completed the seven easiest test items. Performance on the most difficult test item (Serial Sevens Subtraction) was poor, regardless of subgroup.
Proportion of participants who successfully answered each item of the HDS concentration subscale for the entire group, by delirium status and by the presence of indicators of cognitive impairment in the medical chart
Additional analyses compared the performance of participants with delirium with those participants without delirium, but this time taking into account the presence of indicators of cognitive impairment in the medical chart (Table 3). In the group with no indicators of cognitive impairment, the participants without delirium performed significantly better on three test items compared to the participants with delirium (MOTYB, Days of the Week Backward, and Counting from 93 to 85). However, this was not the case among the group with indicators of cognitive impairments; no significant difference in performance was observed between individuals with delirium and those without.
Proportion of participants who successfully answered each item of the HDS concentration subscale according to the presence of indicators of cognitive impairment in the medical chart and by delirium status
HDS: Hierarchic Dementia Scale.
Concurrent validity for each attentional test of the HDS concentration subscale
Table 4 presents the concurrent validity of each attentional test compared with a DSM-5 criterion-defined delirium. Sensitivities decreased gradually from 96% to 17% with decreasing level of test item difficulty. Specificities were low for the two most difficult items (14% and 47%), but moderate to high (63% to 93%) for the other items. Positive predictive values were low for all items (from 13% to 31%), while negative predictive values were relatively high (from 89% to 96%). When considering both sensitivity and specificity, MOYTB was the best performing item. MOYTB identified 83% of the participants with delirium. Regarding specificity, MOYTB identified 63% of the participants without delirium. Positive predictive value was low: among all the participants who were unable to successfully complete MOYTB, only 23% were actually in delirium. When data were analyzed according to the presence or absence of cognitive impairment (data not shown in the Tables), this attention test item still had the best combination of sensitivity (63%, 95% CI = 24–91) and specificity (79%, 95% = CI 71–86) in those participants without cognitive deficits. Positive predictive value remained low (17%, 95% CI = 6–35). In participants with cognitive impairment, no test demonstrated both sensitivity and specificity over 50%.
Predictive performance of the ten items of the HDS concentration subscale for DSM-5 criterion-defined delirium
CI: Confidence interval; HDS: Hierarchic Dementia Scale.
Discussion
The objective of this study was to explore the usefulness of bedside attention tests for capturing an episode of delirium among older patients and residents with, or without, cognitive impairment. We investigated ten brief, objective, and simple tasks that required no materials and virtually no examiner training. These were drawn from the concentration subscale of the HDS (Cole and Dastoor, Reference Cole and Dastoor1987).
Our results suggest that, of the ten attention tests evaluated, MOYTB demonstrated the best balance between sensitivity and specificity (83% and 63%, respectively) for the entire sample. MOTYB is the third most difficult item on the HDS Concentration subscale and 57% of the participants completed it successfully. Similar sensitivity (83.8%; 95% CI [68.0–93.8] but higher specificity (89.6; 95% CI: [81.7–94.9], was reported in a study by O'Regan et al. (Reference O'Regan2014) that explored the usefulness of MOYTB in the delirium screening of older hospitalized adults. A possible explanation for this difference may relate to the criteria used to define a “successful test.” In our study, we deemed participants to have succeeded in the test if they could recite all the MOTYB without error, whereas O'Regan et al. (Reference O'Regan2014) considered that patients in their study had passed this test by reaching July without error. Using a more restrictive definition may have resulted in an increased number of false positive results, and, consequently, decreased the specificity. The results obtained by Bellilli et al. (Reference Bellelli2014) reinforce this view. In their study, they gave a score of 0 for MOYTB if the patients recited at least 7 months correctly, a score of 1, if they recited less than 7 months, or a score of 2, if they were unable to start. Interestingly, the sensitivity and specificity were 86.2% and 82.6%, respectively for a score of 2 and 93.1% and 49.8% for a score of 1.
Another important finding of our study is that the performance of the MOYTB task to capture an episode of delirium varied across subgroups. This test yielded a sensitivity and specificity of 62.5% and 79.3%, respectively in those with no medical chart indication of cognitive impairment, compared to a sensitivity and specificity of 93.3% and 19.1% in the group with indication of cognitive impairment. In this latter subgroup, the MOYTB task missed very few of the delirium cases; however, the number of false positive results was very high (81%). This result indicates that a failure at this test item could, in fact, be attributed to the presence of cognitive impairment. For instance, inability to recite the MOTYB could reflect a deficit in semantic memory and executive function related to dementia, independently of delirium. This result, plus the low positive predictive values observed for all the test items we evaluated, implies that the use of only a single test item is not be the best option for screening for delirium in older individuals with, and without cognitive impairment. For instance, in the presence of premorbid cognitive impairment, more extensive assessments of delirium may be necessary when a change in general status or mental state is observed. This view is also supported by O'Regan et al. (Reference O'Regan2014). They recommended the use of regular formal delirium screening over a brief attention test in the context of premorbid cognitive impairment. A recent study conducted in LTC settings revealed that residents with a delirium superimposed on dementia presented more behavioral symptoms than those with dementia alone (Landreville et al., Reference Landreville, Voyer and Carmichae2013). Therefore, among older adults in the later stages of dementia, it would be interesting to investigate the usefulness of screening tests for delirium, based on behavioral symptoms. This avenue could also be interesting in the presence of aphasia, such as is seen in advanced cases of dementia.
Our study is exploratory and the results must be interpreted with caution. A first limitation is that the HDS was conceived for adaptive testing. Consequently, not all test items were administered to all the participants. We had to assume that participants who failed a particular test item would have failed the more difficult ones. Although the HDS has been validated as such, it is possible that some participants would have succeeded in, or failed, some of the test items that were not administered. Accordingly, a study where all the test items would be administered independently and in a different order, is needed to confirm the results we observed in this study. Another limit is that participants were deemed to have delirium if they presented the DSM-5 criteria for delirium at any of the three assessments conducted using the CAM over the 7-hour observation period. As a subscale of the HDS, the attention test items were administered only once during this period. Therefore, we cannot be sure that the attention tests were actually conducted during the delirium episode, if any occurred. Furthermore, given that the MOTYB task was administered only once, the impact of repeatedly measuring patients with this test item still needs to be investigated. Indeed, if the MOYTB task is administered several times a day over several days, improvement may occur due to repetition and not because of any actual change in functioning. Small sample sizes also limited the analyses that could be performed and the results presented should be replicated in a larger scale study. Finally, we did not take into account the education level of the participants for the analysis of their performance on the concentration tasks. This oversight may have influenced their results and consequently, the results we obtained.
Conclusion
Our results revealed that the performance of these attention tests varied greatly and no one test can be designated to become a single-item screening tool for delirium among older patients and residents with, or without, cognitive impairment. This is particularly important in the presence of premorbid cognitive impairment, where more extensive assessments of delirium may be required, especially when a change in general status or mental state is observed.
Conflict of interest
None.
Description of authors’ roles
PV conceived and supervised all phases of the study. NC, JD, PL JM, and MS contributed to the design of this study. PHC carried out the statistical analysis. AB, SR, and PV drafted the manuscript. All authors read and approved the final manuscript.
Acknowledgments
This study used data collected in a large scale study supported by a grant from the Canadian Institutes of Health Research (#221803) and the Réseau Québécois de Recherche sur le Vieillissement. Financial support was also received from the Saint-Sacrement Hospital Foundation.
