INTRODUCTION
Delirium is a usually transient neuropsychiatric syndrome characterized by an abrupt onset and fluctuating disturbances in consciousness and cognition, as well as problems in a range of noncognitive domains, including disturbances in motor behavior, emotionality, and the sleep/wake cycle, all caused by an underlying etiology (American Psychiatric Association, 2000; Trzepacz et al., Reference Trzepacz, Breitbart and Franklin1999).
Delirium is the most common psychiatric syndrome across healthcare settings (Bucht et al., Reference Bucht, Gustafson and Sandberg1999; Inouye et al., Reference Inouye, Westendorp and Saczynski2014). In patients undergoing cardiac surgery, delirium affects up to 70% of patients (Gottesman et al., Reference Gottesman, Grega and Bailey2010; Norkiene et al., Reference Norkiene, Ringaitiene and Misiuriene2007). With respect to the intensive care setting, this occurrence reaches 80% in patients on mechanical ventilation (Pun & Ely, Reference Pun and Ely2007). Furthermore, delirium often inflicts short-term (Rosen et al., Reference Rosen, Clagett and Valentine2002; Santos et al., Reference Santos, Velasco and Fraguas2004) and long-term adversities for the patients and the healthcare system (Koster et al., Reference Koster, Hensens and van der2009). Among others, these include a prolonged stay on the ICU (Ely et al., Reference Ely, Shintani and Truman2004; Ouimet et al., Reference Ouimet, Kavanagh and Gottfried2007), more frequent or prolonged mechanical ventilation (Heymann et al., Reference Heymann, Radtke and Schiemann2010), and increased rates of morbidity and mortality (Balas et al., Reference Balas, Happ and Yang2009; Heymann et al., Reference Heymann, Radtke and Schiemann2010), and, as long term-consequences, a decline in functionality and cognitive abilities (Bickel et al., Reference Bickel, Gradinger and Kochs2008) and increased rates of institutionalization (Ouimet et al., Reference Ouimet, Kavanagh and Gottfried2007).
Several instruments have been developed to improve the screening for and detection of delirium. In the intensive care setting, the Confusion Assessment Method for the ICU (CAM–ICU) (Ely et al., Reference Ely, Inouye and Bernard2001a ) and the Intensive Care Delirium Screening Checklist (ICDSC) (Devlin et al., Reference Devlin, Fong and Schumaker2007) are commonly utilized instruments.
From a review (Gusmao-Flores et al., Reference Gusmao-Flores, Salluh and Chalhub2012) including 9 studies assessing delirium with the CAM–ICU with sample sizes ranging from 22 to 181 patients, with delirium rates between 22 and 87%, the sensitivity and specificity of the CAM–ICU ranged from 46.7 to 100% and 71.4 to 100%, respectively. The metaanalysis by Gusmao-Flores et al. (Reference Gusmao-Flores, Salluh and Chalhub2012) indicates a sensitivity and specificity of 80 and 95.9%, respectively, the accuracy of which was considered excellent.
From this same review, the ICDSC was evaluated in four studies that included 59 to 126 patients. Delirium rates ranged from 16 to 38.6%, and the sensitivity as well as specificity ranged from 42.9–95.7 to 72.6–94.7%, respectively. The metaanalysis indicated a sensitivity and specificity of 74 and 81.9%, respectively, and the accuracy was considered good (Gusmao-Flores et al., Reference Gusmao-Flores, Salluh and Chalhub2012). From this review (Gusmao-Flores et al., Reference Gusmao-Flores, Salluh and Chalhub2012), the CAM–ICU was considered to be superior to the ICDSC overall.
However, from different studies, the sensitivity of the CAM–ICU and ICDSC were reported at lower rates, ranging from only 18 to 28% for the CAM–ICU and at 47% for the ICDSC (Neufeld et al., Reference Neufeld, Hayat and Coughlin2011; Reference Neufeld, Leoutsakos and Sieber2013).
Thus, although the CAM–ICU and ICDSC have been evaluated in various settings, some inconsistencies remain, particularly with studies documenting lower sensitivity toward delirium (Neufeld et al., Reference Neufeld, Leoutsakos and Sieber2013; van Eijk et al., Reference van Eijk, van Marum and Klijn2009; Reference van Eijk, van den and van Marum2011). Clinical observation actually supports these findings with respect to subsyndromal and milder forms of delirium. In order to confirm this hypothesis, in the following, these scales were evaluated versus the delirium diagnosis of the DSM–IV–TR (American Psychiatric Association, 2000), which was established in a large sample.
METHODS
Patients
All patients in this prospective, descriptive cohort study were recruited at the University Hospital of Zurich, a level one trauma center, with nearly 900 beds and 39,000 admissions yearly. The cardiovascular surgical patients in our study were recruited on a 12-bed intensive care unit between May of 2013 and April of 2015. The inclusion criteria were (1) being an adult, (2) being able to consent, and (3) being under intensive care management for more than 18 hours. The exclusion criteria were; (1) not being able to consent or (2) a history of substance use disorder, aiming to exclude delirium caused by withdrawal.
Procedures
All of our participants were informed of the rationale and procedures of the study, and an initial attempt to obtain written informed consent was made. In those patients unable to provide written consent at that time—either due to more severe delirium, their medical condition and sedation, or frailty—proxy assent from the next of kin or a responsible caregiver was obtained instead. After their medical condition improved, consent from these patients was obtained. Patients were excluded when participation and consent at the initial attempt or after improvement were refused.
The assessment of delirium was performed by four raters trained in the application of the DSM–IV–TR criteria, and interrater reliability was achieved.
The aim of our study was to evaluate the concurrent validity, sensitivity, and specificity, as well as the positive and negative prediction of the CAM–ICU and ICDSC, versus the DSM–IV–TR diagnostic criteria.
The baseline assessment included several steps. At first, the patient was interviewed. Second, the presence of delirium was determined according to DSM–IV–TR criteria (American Psychiatric Association, 2000). Third, nurses and doctors specifically trained in the administration of the ICDSC (Devlin et al., Reference Devlin, Fong and Schumaker2007) and CAM–ICU (Ely et al., Reference Ely, Inouye and Bernard2001a ) completed these scales.
If required, the assessment was completed by obtaining collateral information from the nursing and medical-surgical staff, the electronic medical record system (Klinikinformationssystem, KISIM, CisTec AG, Zurich), and family or caregivers.
Measurements
Diagnostic and Statistical Manual, 4th ed., Text Revision (DSM–IV–TR)
The diagnosis of delirium was determined using the DSM–IV–TR (American Psychiatric Association, 2000), including four criteria: (1) disturbance of consciousness (i.e., reduced clarity of awareness of the environment) with reduced ability to focus, sustain, or shift attention; (2) a change in cognition (such as memory deficit, disorientation, language disturbance) or the development of a perceptual disturbance that is not better accounted for by a preexisting, established, or evolving dementia; (3) the disturbance develops over a short period of time (usually hours to days) and tends to fluctuate during the course of the day; and (4) there is evidence from the history, physical examination, and laboratory findings that: (a) the disturbance was caused by the direct physiological consequences of a general medical condition, (b) the symptoms in criterion (a) developed during substance intoxication or during or shortly after a withdrawal syndrome, or (c) the delirium had more than one etiology.
The Confusion Assessment Method for the Intensive Care Unit (CAM–ICU)
The CAM–ICU (Ely et al., Reference Ely, Inouye and Bernard2001a ) is based on the CAM (Inouye et al., Reference Inouye, van Dyck and Alessi1990), reflecting the DSM–III–R criteria (American Psychiatric Association, 1987), and designed for patients with limited communication abilities. This scale contains four features with two levels (absent and present): (1) acute onset and fluctuating course, (2) inattention, (3) altered level of consciousness, and (4) disorganized thinking. Feature 1 scores as absent or present; feature 2 includes “recognizing letters” scores as number of errors (more than two scores as present); feature 3 scores the Richmond Agitation Assessment Scale (RASS) other than alert and calm (RASS = 0) as present; and feature 4 includes simple questions and instructions, with a combined number of errors of more than one scored as present. Features 1 plus 2 and either 3 or 4 scored as present indicates presence of delirium. The nonverbal items achieve a lower sensitivity than the verbal items. Interrater reliability ranges from 0.79 to 0.95 (McNicoll et al., Reference McNicoll, Pisani and Ely2005).
The Intensive Care Delirium Screening Checklist (ICDSC)
The ICDSC (Devlin et al., Reference Devlin, Fong and Schumaker2007) is a screening instrument that includes eight items based on DSM–IV TR criteria specifically designed for the intensive care setting, and it has two score points: absent and present. This scale was designed for patients with limited communication abilities, such as intubated patients. The items include the assessment of (1) consciousness (comatose, soporose, awake, or hypervigilant); (2) orientation; (3) hallucinations or delusions; (4) psychomotor activity; (5) inappropriate speech or mood; (6) attentiveness; (7) sleep/wake cycle disturbances; and (8) fluctuation of symptomatology. The maximum score is 8, and scores of more than 3 indicate the presence of delirium. Each item is rated based on the patient's behavior over the previous 24 hours, and interrater reliability between intensive care staff was considered adequate (Bergeron et al., Reference Bergeron, Dubois and Dumont2001).
Statistical Analysis
All statistical procedures were conducted using the Statistical Package for Social Sciences (SPSS, v. 22). Descriptive statistics were implemented for characterization of the study sample with respect to their sociodemographic and clinical variables, including a comparison of the included versus excluded patients. The inclusion criterion was performance of the CAM–ICU, ICDSC, and DSM–IV–TR in diagnosis.
In the next step, patients with delirium were compared to those without delirium using the same procedures. For the determination of differences between those with and without delirium, Student's t test was employed for variables on a continuous scale (such as the age of patients). For items on categorical scales (such as the presence of items on the CAM–ICU or ICDSC), Pearson's χ2 was determined.
The interrater reliability with respect to a DSM–IV–TR diagnosis was determined by its corresponding Fleiss's κ, with perfect agreement defined as >0.80 (DeVellis, Reference DeVellis2012).
The concurrent validity of the CAM–ICU and ICDSC was calculated versus the DSM–IV–TR diagnosis of delirium, and, in addition, the validity of the CAM–ICU and ICDSC were determined. All scales represented two score levels, indicating the absence or presence of delirium. Cohen's κ was then determined as a measure of concordance. Agreement was defined as moderate (0.41–0.60), substantial (0.61–0.80), or perfect (>0.80) (DeVellis, Reference DeVellis2012). Further, the respective sensitivity and specificity, as well as the corresponding positive and negative predictive values (PPV and NPV), were calculated, and their confidence intervals were determined as exact Clopper–Pearson confidence intervals.
The significance level of Cronbach's α was set at p = 0.05 for all of the implemented tests.
RESULTS
Characteristics of the Study Sample
Patients were predominately male and on average in their mid-60s. In total, 42% were diagnosed with delirium according to the DSM–IV–TR, versus 23.3% diagnosed with the CAM–ICU and 28.7% with the ICDSC (Table 1).
Sociodemographic and medical variables of all patients assessed, included, and excluded patients
DSM–IV–TR = Diagnostic and Statistical Manual, 4th ed., Text Revision); CAM–ICU = Confusion Assessment Method for the ICU; ICDSC = Intensive Care Delirium Screening Checklist; SD = standard deviation.
a Student's t test. b Pearson's χ2.
Characteristics of Included versus Excluded Patients
Excluded patients were not different from included patients. Between those patients included versus those excluded, neither age, gender distribution, nor day of assessment were different. Also, no differences were found with respect to the rates of delirium diagnosis as determined with the ICDSC or DSM–IV–TR (Table 1).
Characteristics of Patients with Delirium versus Those Without
The patients with delirium were older and proportionately more likely to be women, and they were assessed at a later time in their hospitalization (Table 2). In those with delirium as diagnosed by the DSM–IV–TR, the CAM–ICU detected delirium in only every other patient (46.7%). In contrast, the ICDSC detected delirium in two of three (63%). Conversely, both the CAM–ICU and ICDSC had a false positive rate of 5%. All items of the ICDSC scored more frequently in delirious patients when including the trend in item 7. As expected, ICDSC total scores were higher in the presence of delirium.
Sociodemographic and medical variables of patients with and without delirium
DSM–IV–TR = Diagnostic and Statistical Manual, 4th ed., Text Revision; CAM–ICU = Confusion Assessment Method for the ICU; ICDSC = Intensive Care Delirium Screening Checklist; SD = standard deviation.
a Student's t test. b Pearson's χ2.
Interrater Reliability with Respect to DSM–IV–TR Diagnosis
With respect to DSM–IV–TR diagnosis, the overall rating agreement between psychiatrists' assessments of delirium was almost perfect (Cohen's κ = 0.89, CI 95% = 0.69–1.1, p < 0.001), and with respect to the presence and absence of delirium perfect (Cohen's κ = 0.97, CI 95% = 0.69–1.1, p < 0.001; Cohen's κ = 0.93, CI 95% = 0.69–1.1, p < 0.001).
The CAM–ICU and ICDSC versus DSM–IV–TR and CAM–ICU versus ICDSC
The following evaluations of concurrent validity are listed in Table 3 and 4.
Allocation of the presence and absence of delirium: CAM–ICU and ICDSC versus the DSM–IV–TR diagnosis
DSM–IV–TR = Diagnostic and Statistical Manual, 4th ed., Text Revision; CAM–ICU = Confusion Assessment Method for the ICU; ICDSC = Intensive Care Delirium Screening Checklist.
1. The CAM–ICU versus DSM–IV–TR
Agreement was moderate between the CAM–ICU and DSM–IV–TR, and half of the cases of delirium were not detected, as evidenced by the sensitivity of less than 50%. Conversely, the specificity and PPV nearly reached 95% and the NPV was 70%. Although the CAM–ICU was not a very sensitive instrument, it was very specific, with high predictive values with respect to presence or absence of delirium.
2. The ICDSC versus the DSM–IV–TR
Agreement between the DSM–IV–TR and ICDSC in terms of detection of delirium was also within the moderate range. The ICDSC was able to detect two of three patients with delirium, as evidenced by its sensitivity of 63%. In addition, the ICDSC proved to be a highly specific instrument, with its specificity and PPV exceeding 90% and the NPV reaching nearly 80%. Altogether, the ICDSC proved to be a sensitive and highly specific instrument.
3. The CAM–ICU versus the ICDSC
Finally, the concurrent validity between the CAM–ICU and ICDSC was also only moderate. The sensitivity reached almost 80%, the specificity exceeded 80%, the NPV exceeded 90%, and the PPV reached nearly 60%.
DISCUSSION
Summary of Main Findings
From these findings, the CAM–ICU and ICDSC were found to be useful and very specific instruments within the daily clinical routine in the detection of delirium in an intensive care setting. However, the concurrent validity as well as the sensitivity of these instruments versus the psychiatric assessment and application of DSM–IV–TR criteria was only moderate. Between the CAM–ICU and ICDSC, the latter reached greater accuracy in the diagnosis of delirium versus the DSM–IV–TR. Similarly, the sensitivity of these scales was only moderate to substantial, whereas the specificity was substantial to perfect. The advantage of prediction of delirium with these scales versus the DSM–IV–TR was substantial.
Comparison with the Existing Literature
The findings presented in this study supported previous studies which indicated that both the CAM–ICU and ICDSC did not reach the previously assumed sensitivity. This concern has been raised in two studies that included more than 300 patients (van Eijk et al., Reference van Eijk, van Marum and Klijn2009; Reference van Eijk, van den and van Marum2011), as well other more recent studies that included almost 200 patients (Neufeld et al., Reference Neufeld, Hayat and Coughlin2011; Reference Neufeld, Leoutsakos and Sieber2013). The first two studies conducted on medical-surgical and general ICU settings documented the sensitivity of the CAM–ICU at 46.7 and 64.3%, while reaching a specificity of 98.1 and 88.8%, respectively. The latter studies were conducted on medical oncology and post-anesthesia care units, with documented sensitivities of 18 and 28%, respectively, while specificity remained high.
Similarly, with respect to the ICDSC, two studies documented lower sensitivities toward delirium (43 to 47%) and high specificities (>94%) (Neufeld et al., Reference Neufeld, Leoutsakos and Sieber2013; van Eijk et al., Reference van Eijk, van Marum and Klijn2009).
Conversely, a number of studies indicated that both the CAM–ICU and ICDSC are very sensitive and specific instruments. For the CAM–ICU, three studies documented nearly perfect sensitivities (96–100%) (Ely et al., Reference Ely, Inouye and Bernard2001a ; Reference Ely, Margolin and Francis2001b ; Lin et al., Reference Lin, Liu and Wang2004), while two other studies (Luetz et al., Reference Luetz, Heymann and Radtke2010; Mitasova et al., Reference Mitasova, Kostalova and Bednarik2012) documented sensitivities ranging between 79 and 80%. With respect to the specificities of the CAM–ICU, all of these studies exceeded 88%, while reaching perfect specificity.
Although studies evaluating the ICDSC are fewer in number, the sensitivities and specificities usually reached 95% (Bergeron et al., Reference Bergeron, Dubois and Dumont2001; Gusmao-Flores et al., Reference Gusmao-Flores, Salluh and Dal Pizzol2011),
Disagreement Between the CAM–ICU and ICDSC versus the DSM–IV TR
Naturally, the improper administration of the CAM–ICU and ICDSC had to be considered as a potential source of error. However, all personnel administering these tests had been rigorously trained within a standardized, mandatory, hospital-wide program. Thus, this source of error appeared less likely and was therefore excluded. Moreover, overdiagnosis of delirium by expert raters was excluded by the perfect agreement between their assessments. Furthermore, the DSM–IV–TR (American Psychiatric Association, 2000) has been recognized to be more sensitive in detection of delirium, particularly the subsyndromal and milder forms (Laurila et al., Reference Laurila, Pitkala and Strandberg2003). In contrast, the CAM–ICU is based on DSM–III–R criteria (American Psychiatric Association, 1987), which have been recognized to be more restrictive in the diagnosis of delirium (Laurila et al., Reference Laurila, Pitkala and Strandberg2003). Thus, the underlying diagnostic criteria likely contributed to the moderate sensitivity of the CAM–ICU. Similarly, in another study, the original CAM versus the DSM–IV–TR reached only moderate agreement (Meagher et al., Reference Meagher, O'Regan and Ryan2014). With respect to the sensitivity of the ICDSC, the diagnostic criteria were not responsible since they were the same (DSM–IV–TR). However, the cutoff score for this scale has been debated, and decreasing it from 4 to 3 actually increased the sensitivity (George et al., Reference George, Nair and Ebenezer2011).
Notwithstanding, with these conflicting results and later reports indicating lower sensitivities, the question remains as to whether the CAM–ICU or ICDSC detect delirium sufficiently when compared with experts' assessments. Most likely, these instruments will suffice in most instances and when implemented as screening instruments will enhance the detection of delirium. However, these instruments are not perfect, and, as our results and parts of the literature suggest, they might not fully suffice in detecting every type of delirium, particularly the subsyndromal and milder forms. Thus, potentially, negative CAM–ICU or ICDSC scores do not necessarily refute the presence of delirium, irrespective of whether it is mild or subsyndromal.
STRENGTHS AND LIMITATIONS OF THE STUDY
This study has several strengths; however, a number of limitations should be noted. Almost 300 patients were prospectively screened and rated for delirium using the CAM–ICU and ICDSC versus the DSM–IV–TR criteria. Eventually, 210 patients with corresponding assessments were included. These patients were not different from those excluded. With respect to diagnosis of delirium with the DSM–IV–TR, interrater agreement was perfect. The limitations included the high prevalence of hypoactive delirium, which was due to the critical care population studied, and the absence of baseline cognitive recording owing to the prospective nature of the study. Thus, preexisting cognitive disorders could not be excluded despite screening the medical record for them. Moreover, our study was cross-sectional, and further longitudinal studies of the concurrent validity of these scales, as well as the impact of unrecognized delirium, are required. Without a doubt, further studies assessing the sensitivity and specificity as well as positive and negative prediction of delirium with the CAM–ICU and ICDSC are called for.
CONCLUSIONS
In summary, neither the CAM–ICU nor ICDSC proved to be highly accurate instruments. Altogether, both instruments reached only moderate agreement and sensitivity, although the ICDSC proved to be somewhat more accurate. Conversely, both instruments proved to be highly specific, and the positive as well as negative prediction of delirium was substantial. Although both these instruments are very useful in an intensive are setting, nondetection by these scales does not necessarily refute the presence of delirium.
CONFLICTS OF INTEREST
The authors hereby declare that they have no conflicts of interest to declare.
Agreement, sensitivity and specificity, positive and negative predictive value (NPV and PPV) of CAM–ICU and ICDSC versus the DSM–IV–TR diagnostic criteria
CI 95% = 95% confidence interval; DSM–IV–TR = Diagnostic and Statistical Manual, 4th ed., Text Revision; CAM–ICU = Confusion Assessment Method for the ICU; ICDSC = Intensive Care Delirium Screening Checklist; PPV = positive predictive value; NPV = negative predictive value.
