Introduction
Delirium is an acute, severe neuropsychiatric syndrome characterized by fluctuating disturbances in attention, arousal, and cognition. It is highly prevalent in older hospitalized patients and is associated with adverse outcomes including functional decline, new institutionalization, persistent cognitive impairments including dementia, and higher mortality (Siddiqi et al., Reference Siddiqi, House and Holmes2006; Witlox et al., Reference Witlox, Eurelings, de Jonghe, Kalisvaart, Eikelenboom and van Gool2010).
The core diagnostic feature of delirium is “inattention,” defined as a “reduced ability to direct, focus, sustain, and shift attention” in the DSM-5 criteria for delirium (American Psychiatric Association, 2013). The evidence suggests that several aspects of attention are likely to be affected in delirium, including the basic orienting response, focusing, sustaining, and dividing attention (Tieges et al., Reference Tieges, Brown and Maclullich2014). The extent to which each of these aspects of attention is affected in delirium is poorly understood, though deficits in sustained attention (i.e. the ability to maintain attention to stimuli over time) have been implicated (O’Keeffe and Gosney, Reference O’Keeffe and Gosney1997; Brown et al., Reference Brown, Fordyce, Zaghdani, Starr and MacLullich2011; Tieges et al., Reference Tieges, Brown and Maclullich2014).
A range of methods is used in research and in clinical practice to ascertain inattention. These include subjective assessments based on interview and clinical observation, and objective assessments using brief neuropsychological tests (Hall et al., Reference Hall, Meagher and MacLullich2012). Objective tests likely offer some advantages over subjective methods, including standardization of instructions, and greater reliability and reproducibility. In addition, objective tests are less likely to be operator dependent and reliant upon clinical experience than subjective assessments. Several neuropsychological tests are currently used to assess inattention in delirium. These include digit span, spatial span, months of the year and days of the week backwards, vigilance “A” and serial 7s (O’Regan et al., Reference O’Regan2014; Tieges et al., Reference Tieges, Brown and Maclullich2014). Generally these tests perform well in detecting delirium. However, the available data show that most existing tests discriminate poorly between delirium and other mental disorders including dementia (Morandi et al., Reference Morandi2012). Indeed, many studies show that patients with dementia show significant deficits in tests often used to assess inattention in delirium, such as spatial span (Meagher et al., Reference Meagher, Leonard, Donnelly, Conroy, Saunders and Trzepacz2010), serial 7s (Bronnick et al., Reference Bronnick, Emre, Lane, Tekin and Aarsland2007) and months of the year backwards (Katzman et al., Reference Katzman, Brown, Fuld, Peck, Schechter and Schimmel1983). Further, existing tests of attention have not been validated for severity grading in delirium, which is useful in monitoring delirium progress over time and providing a more fine-grained measure of delirium than dichotomous scoring methods.
To help address the lack of robust, objective assessments for inattention in delirium, we previously developed a novel attentional test implemented on a custom-built computerized device entitled the EDTB (Brown et al., Reference Brown, Fordyce, Zaghdani, Starr and MacLullich2011; Tieges et al., Reference Tieges, McGrath, Hall and Maclullich2013). The EDTB tasks require participants to count and verbally report how many times either one or two slowly presented lights on the device illuminate. The lights are illuminated for 1 sec, and the inter-stimulus intervals are 1–4 sec. The design of this test was motivated by the need for a task that was cognitively simple whilst placing demands on attentional functioning. Prior research had found that the ability to sustain attention in a simple counting task may be relatively spared in Alzheimer's disease (Lines et al., Reference Lines, Dawson, Preston, Reich, Foster and Traub1991; Perry et al., Reference Perry, Watson and Hodges2000; Morandi et al., Reference Morandi2012). The EDTB differs from these prior tasks, in that the stimuli are visual instead of auditory, and distracting stimuli are presented in some of the trials in order to increase task difficulty. In our work with the EDTB, we found that patients with delirium showed marked deficits on these tasks, whereas patients with dementia and cognitively unimpaired controls performed at or near ceiling level. The ROC areas under the curves were between 0.80 and 1.00, indicating good to excellent accuracy of the EDTB in discriminating delirium from dementia and control groups (Brown et al., Reference Brown, Fordyce, Zaghdani, Starr and MacLullich2011). We concluded, therefore that these new objective attentional tests offered potential utility in detecting delirium and differentiating it from dementia. More broadly, validated, objective neuropsychological assessments potentially offer a more robust approach to the measurement of inattention in delirium than presently available methods.
We recently developed a software application for detecting deficits of visual sustained attention in delirium (“DelApp”), which is based on our EDTB sustained attention tasks. The rationale is that although the EDTB showed good performance, tests on devices that are universally available, such as smartphones, are more readily applicable in research and clinical settings. Smartphone-based applications are increasingly being used as a method for assessing cognitive function in older individuals (Brouillette et al., Reference Brouillette2013).
In this preliminary study, we first conducted a feasibility study to assess acceptability of the DelApp for the assessment of attention in older hospitalized patients. We then proceeded with a pilot case-control study to compare performance on the DelApp in a selected sample of older hospitalized patients with dementia, delirium, or no cognitive impairment.
Methods
Study 1
Design
This study was designed to assess the feasibility and acceptability of the DelApp in hospitalized older patients, and to compare the EDTB and DelApp. Formal diagnostic assessment of each patient was not performed as this was not required to meet the aims of Study 1. A within-subjects design was used. Patients were studied on a single occasion and underwent EDTB and DelApp tests. Semi-structured interviews with patients were conducted to assess their opinions of these tests. Study 1 and Study 2 were approved by the Scotland A Research Ethics Committee.
Participants
Patients aged 60 years and older who were able to communicate in English were eligible. All patients had to demonstrate capacity to provide written, informed consent; this was because the ethics committee ruled that for Study 1 it was unnecessary to involve patients requiring proxy consent. Exclusion criteria were: severe sensory impairments or severe acute illness that would impede testing and interview, and where clinical staff considered that participation would adversely affect the patient's care.
The researchers (AS, KSc, and KSu) recruited 20 patients from Medicine of the Elderly and orthopedic wards at the Royal Infirmary of Edinburgh, Scotland. Patients meeting eligibility criteria were identified through consultation with the clinical care team. Forty-three patients were identified, and of these nine patients declined to participate, three patients were not available for testing because they were discharged or moved by the time the researcher approached them, and eleven were deemed unsuitable by the researcher due to lack of capacity (cognitive impairment, reduced arousal) or severity of illness. This recruitment process yielded the final target sample size of 20 patients.
Measurement and procedures
Participants first undertook a brief visual acuity test to ensure that they could perceive the test stimuli. The visual acuity test comprised six short trials in which participants were asked to (1) identify a change in the color of a white circle (5 cm diameter) presented on the smartphone screen (white-to-grey and grey-to-white); (2) identify a change in the shape of the stimulus (circle-to-star and star-to-circle); and (3) name two letters presented on the screen one at a time (B and E) (Figure 1(A)). Testing proceeded only when the participants could reliably perceive the stimuli.
(A) The DelApp visual acuity test. The stimulus in trials 1–4 changed 5 sec after trial onset and total trial duration was 10 sec. Letters (trials 5 and 6) were presented on screen for 8 sec. The outer edges of these stimuli represent the outer edges of the smartphone's screen. (B) The DelApp attention task comprised seven trials of increasing difficulty presented in a fixed order. Participants were instructed to count centrally presented circles with a duration of 1 sec each, sometimes in the presence of distracting triangles (500 ms each; Levels 2 and 3). Number of circles to be counted ranged from 2 to 8. The inter-stimulus interval between circles increased from 1–2 sec in Levels 1 and 2 to 3–4 sec in Level 3. Patients were asked to verbally report the total number of counts per trial.
Patients then completed visual sustained attention counting tasks on the EDTB and DelApp (www.edinburghdelirium.ed.ac.uk) in counterbalanced order (see Figure S1, available as supplementary material attached to the electronic version of this paper at www.journals.cambridge.org/jid_IPG). The EDTB Mark 2 is a purpose-built cuboidal, grey plastic box (13 × 21.5 × 7.5 cm) with two protruding circular illuminable buttons (5 cm diameter), presented to participants in landscape orientation. Each button contains concealed (when not illuminated) light-emitting diodes (LEDs) and is surrounded by four LEDs concealed under a translucent white cover. The box further contains a concealed central 7 × 7 matrix of LEDs to display distracting stimuli such as checkered patterns (Tieges et al., Reference Tieges, McGrath, Hall and Maclullich2013).
The DelApp task was presented on the 9.2 × 5.9 cm display of a Samsung Galaxy S2 running the Android 3.2 operating system. The brightness was set automatically based on the ambient light. The target stimulus was a single large white circular illumination (5 cm diameter) presented in the center of the screen against a black background. On some trials, this was surrounded by pseudo-random subsets of eight small downward-pointing triangles (0.3 × 0.3 × 0.4 cm), each presented for 500 ms, to provide distraction (Figure 1(B)). The smartphone was held in portrait orientation at a distance of approximately 50 cm from participants.
All EDTB tasks involved presentation of one light only, instead of the two lights used in the original EDTB studies, so as to more closely resemble the DelApp and thus provide a more direct comparison. Participants were asked to count and then verbally report at the conclusion of the sequence (as indicated by the researcher) the number of times they saw the illuminations (i.e. illuminations of the button on the EDTB or circles presented on the smartphone screen).
Both the EDTB and DelApp counting tasks comprised seven trials of increasing difficulty, presented in a fixed order. This served to minimize floor and ceiling effects. The illuminations lasted 1,000 ms, and distracting stimuli lasted 500 ms each. The number of illuminations increased from 2 to 8 across trials. The inter-stimulus interval ranged between 1,000 ms and 2,000 ms in Level 1 (trials 1–3) and Level 2 (trials 4–5), and between 3,000 ms and 4,000 ms in Level 3 (trials 6–7). Distracting stimuli were added for Level 2 and approximately twice as many distracting stimuli were shown in Level 3. Trial one was a practice trial, which was repeated a number of times if necessary. Trials were scored as correct or incorrect. If no answer was given, this was scored as incorrect. Thus, we did not distinguish between incorrect responses and omissions. Likewise, patients who could attempt only a subset of the trials still received a score (i.e. total number of correct completed trials) and were included in the statistical analyses.
The end of each trial in the DelApp task was signaled to the researcher by a short vibration generated by the phone.
The DelApp counting task was preceded by a brief assessment of level of arousal (LoA). This was included because some patients with abnormal LoA cannot engage with cognitive testing, and these patients are considered as showing severe inattention (see the guidance notes in the DSM-5 diagnostic criteria for delirium (American Psychiatric Association, 2013)). To allow for some grading of the DelApp test scores in patients unable to perform the attention task, the arousal assessment comprised the following three items: (1) Can the patient keep his/her eyes open for 10 sec? (2) Can the patient say his/her name? (3) Can the patient track an object (e.g. badge, phone) for 5 sec? The arousal assessment was combined with the seven-item counting task to provide an overall score out of ten. The full DelApp assessment generally took less than five minutes to complete.
Following the EDTB and DelApp tests, participants underwent a semi-structured interview in which they were asked a series of scripted open-ended questions relating to their experience of the DelApp in comparison to the EDTB. Interviews were recorded and transcribed. The whole testing and interview assessment typically lasted around 15 minutes.
Statistical analysis
Comparisons between test scores on the DelApp and EDTB tasks were analyzed using a Wilcoxon signed-rank test. The threshold for statistical significance was p ≤ 0.05. A qualitative thematic analysis was conducted on verbatim transcriptions of recorded interviews (Braun and Clarke, Reference Braun and Clarke2006). All quantitative analyses were carried out in R version 3.0.1 (R Core Team, 2014).
Study 2
Design
This study employed a between-subjects design, comparing performance on the DelApp test in three groups: patients with delirium, dementia, and no severe cognitive impairment. This was a case-control study and so we deliberately aimed to have groups that were clear-cut clinically.
Participants
Inclusion and exclusion criteria were as for Study 1. Proxy consent was sought for patients lacking capacity to provide consent for themselves. Patients were recruited from Medicine of the Elderly and orthopedic wards at the Royal Infirmary of Edinburgh, and acute and rehabilitation wards in Liberton Hospital, Edinburgh.
A total of 274 patients were initially identified by the clinical staff and researchers as potentially appropriate for the study. Of these, 27 patients declined, two patients were not available for testing because they had been discharged or moved, and two patients were participating in a similar study and were therefore excluded. Thirty-six patients were not suitable, due to severe cognitive impairments, severely reduced LoA such that no engagement was possible, severity of illness or distress. Twenty-five patients who required proxy consent or asked researchers to come again at a different time could not be approached again due to researcher unavailability. Proxy consent was sought but could not be obtained for one patient. A sample of 181 patients underwent assessment in the study. However, 25 patients were excluded after assessment because they could not be allocated to any of the pre-defined study groups. The final study sample size was 156 (delirium: N = 50; dementia: N = 52; control: N = 54).
Measurement and procedures
Cognition and delirium status were assessed on one occasion. All cognitive and delirium assessments were conducted by graduate psychologists (AS, KSc, and KSu) who had been fully trained by a geriatrician (AM) and a postdoctoral psychologist (ZT). This training process included: regular supervisor meetings; ward round observations; role play and mock assessments; a supervisor-led teaching session on extracting information from case notes to inform categorization; and a certified course on Good Clinical Practice. Students followed detailed Standard Operating Procedures for the delirium and cognitive assessments and they were closely monitored throughout the recruitment and data acquisition stage by their supervisors (AM and ZT). All tests were administered at the patient's bedside, with the curtain surrounding their bed closed.
The presence of delirium was assessed using the Confusion Assessment Method (CAM; Inouye et al., Reference Inouye, van Dyck, Alessi, Balkin, Siegal and Horwitz1990). Patients who met CAM criteria for delirium were included in the delirium group irrespective of their current or prior level of cognition. Severity was measured using the DRS-R98 (Trzepacz et al., Reference Trzepacz, Mittal, Torres, Kanary, Norton and Jimerson2001). Scoring of the DRS-R98 was based on the preceding 24 hours. Higher scores indicate greater severity and amount to a total score of 39 (severity subscale). Discussions with the clinical staff and scrutiny of the case notes provided additional diagnostic and severity information.
Patients in the dementia group either had a prior formal diagnosis, or clearly met DSM-IV criteria for dementia (using information from case notes and informants) as determined by a consultant geriatrician. Patients with dementia did not have delirium at the time of assessment. Patients were included in the control group only if they demonstrated normal cognition on a general cognitive measure described below, and had no documented history of chronic cognitive or functional impairment.
The two letter trials of the visual acuity assessment were discontinued as they were considered redundant and related to a separate task involving letter sequences. The remaining four trials and DelApp procedures were the same as for Study 1. These were accompanied by additional measures of attention, arousal and cognition, as follows.
The short OMCT (Katzman et al., Reference Katzman, Brown, Fuld, Peck, Schechter and Schimmel1983) was used to assess the overall level of cognition. This test is a validated measure of cognitive impairment (Katzman et al., Reference Katzman, Brown, Fuld, Peck, Schechter and Schimmel1983). To facilitate analyses we scored the OMCT such that higher scores indicate better performance. The suggested scores for categorization are: 24–28 = normal cognition; 19–23 = questionable impairment, ≤18 = dementia (Morris et al., Reference Morris1989). A set of brief attention tests (Marcantonio, Reference Marcantonio2008) established as a method of assessing inattention in delirium and here termed as the brief attention tests (BAT) was used to provide a standardized measure of inattention. The BAT comprised: digit span forwards (3–5 digits), digit span backward (3–4 digits), and days of the week and months of the year in reverse order. Each correct item equated to one point and scores of 5 (out of 7) or below were considered indicative of inattention. The BAT informed scoring of the inattention item on the CAM and served as a comparison for the DelApp.
The Observational Scale of Level of Arousal (OSLA; www.edinburghdelirium.ed.ac.uk) (Tieges et al., Reference Tieges, McGrath, Hall and Maclullich2013) was included to provide a quantitative measure of LoA and to aid assessment of delirium. It was developed in-house and designed specifically to characterize abnormalities in LoA in patients with delirium. The OSLA comprises four graded items: eye opening, eye contact, posture, and movement. Higher scores indicate greater abnormality in LoA (maximum score is 15).
The order of attention assessments (DelApp, BAT) and cognitive assessment (OMCT) was systematically counterbalanced across patients using a Latin squares design. The DRS-R98, OSLA, and CAM were completed following the assessments. All assessments including DelApp and reference diagnosis for delirium were performed by single raters.
Following the assessment of delirium and cognition (including the DelApp), some patients showed patterns of results on the cognitive tests that suggested the initial group assignment (based on case notes and discussions with ward staff) was incorrect. For example, patients who were initially deemed as potential controls by ward staff sometimes had cognitive impairment as determined by the more detailed cognitive testing performed as part of the present study. For such patients the most appropriate grouping was decided blind to DelApp results by a consultant geriatrician (AM) based on all the other available information (i.e. cognitive test scores, information taken from medical notes and conversations with ward staff). Some patients could not be allocated to any group, because their test scores deviated considerably from group medians and/or they did not present a symptom profile that was clearly characteristic of any one group. These patients were excluded, again without knowledge of the DelApp scores, since for the purposes of this study we aimed to select patients who could be classified as being part of one of the predefined clinical groups.
Statistical analysis
Non-parametric statistical tests were used as the majority of the data were non-normally distributed and there was heterogeneity of variance across groups. Kruskal–Wallis and Mann–Whitney U tests were used for between-group analyses. Continuity corrections were applied to Mann–Whitney U tests. Holm corrections were applied to multiple comparisons. For these comparisons the 95% confidence interval (CI) limits are expressed as differences in mean rank between each pair of groups. Kendall's tau was used for correlations as there were many ties in the data (Field et al., Reference Field, Miles and Field2012). Pearson's χ2 tests were used for proportional data. Linear regressions were carried out using a conservative threshold for significance of p ≤ 0.01, due to non-normality. ROC curves were compared using DeLong's method (DeLong et al., Reference DeLong, DeLong and Clarke-Pearson1988). χ2 tests were used to explore differences in performance between subsequent levels of task difficulty. Odds ratios are also reported.
Medium to large effect sizes were found in a similar study (Brown et al., Reference Brown, Fordyce, Zaghdani, Starr and MacLullich2011). Therefore, taking a relatively conservative estimate of effect size as 0.35, where p = 0.05 and N = 141, all statistical tests used in the present analysis will have > 95% power. This calculation defined the minimum acceptable sample size.
Results
Study 1
Twenty patients, aged 66–93, were recruited. No significant difference was found between scores on the DelApp (median = 10, range 8–10) and EDTB (median = 10, range = 8–10; p = 0.41). Qualitative thematic analysis indicated that responses were centered around four themes: Physical presentation of the DelApp; ease of DelApp task; further development of the application; and device preference (EDTB or DelApp). All patients stated that they had no difficulty perceiving the stimuli, or completing the tests on the DelApp. Three participants mentioned that the peripheral distractors of the DelApp were not as distracting as those on the EDTB. When asked which device they preferred, nine out of 20 patients had no preference, seven preferred the DelApp and four preferred the EDTB.
Study 2
Patient characteristics
Age differed significantly among groups (p < 0.001; Table 1). Delirium and dementia groups were overall older than controls (delirium vs. control: p < 0.001, 95% CI = (13.0, 52.27); dementia vs. control: p < 0.001, 95% CI = (23.93, 62.81)). There was no difference in age between dementia and delirium groups.
Demographics and pairwise comparisons between groups. Demographics and test scores for each group are presented as median (inter-quartile range) unless otherwise specified. Statistical comparisons between groups were performed using Kruskal–Wallis tests. Pairwise comparisons were carried out using Mann–Whitney U tests. Holm corrections were applied to control for type I error. The 95% confidence interval (CI) limits for these group comparisons represent the differences in mean rank between each pair of groups
Note. Calculations represent percentages of each group in which diagnoses are present. Exhaustive comorbidity data were not recorded for all participants, therefore calculations are subject to the availability of data. COPD = Chronic obstructive pulmonary disease; CON = controls; DEM = dementia; DEL = delirium; CI = confidence intervals; OSLA = Observational Scale of Level of Arousal; OMCT = Orientation-Memory-Concentration Test.
General cognition, as measured with the OMCT, differed between groups (Table 1). Scores in the delirium and dementia groups were significantly lower compared to controls (delirium vs. control: p < 0.001, 95% CI = (64.78, 91.04); dementia vs. control: p < 0.001, 95% CI = (58.59, 84.58)). There was no difference between dementia and delirium groups.
Between-group differences in attention
DelApp:
Of the 156 patients included in the study, ten participants (eight from the delirium group and two from the dementia group) were unable to provide answers on some or all trials on the counting task. Where this occurred trials were scored as incorrect, as this was considered the result of severe inattention, and the overall scores for these patients were included in the statistical analyses. DelApp scores differed among groups (p < 0.001; Table 1 and Figure 2). Patients with delirium scored significantly lower than patients with dementia (p < 0.001, 95% CI = (50.58, 74.38)) and controls (p < 0.001, 95% CI = (69.25, 92.84)). The dementia group scored significantly lower than controls (p < 0.001, 95% CI = (6.90, 30.25)).
Boxplots illustrating the DelApp results of each group. The median is represented by the thick horizontal bar. The interquartile range is represented by the height of the inner boxes. The position of vertical bars represents the value of the most distant scores that are not considered to be outliers. Outliers are represented by open circles.
Age was not a significant predictor of DelApp score (β = −0.05, t (154) = −2.33, p = 0.021, R 2 = 0.034, 95% CI = (8.88, 15.86)).
BAT:
BAT scores also differed between groups (p < 0.001; Table 1). The delirium and dementia groups scored lower on the BAT than controls (delirium vs. control: p < 0.001, 95% CI = (51.43, 83.86); dementia vs. control: p < 0.001, 95% CI = (20.85, 52.96)). Scores were also lower in delirium compared to dementia (p < 0.001, 95% CI = (14.37, 47.1)).
DelApp and abnormal LoA
OSLA scores reflecting LoA differed between groups (p < 0.001; Table 1). As expected, the delirium group scored higher than patients with dementia (p < 0.001, 95% CI = (60.2, 78.4)) and controls (p < 0.001, 95% CI = (66.67, 84.7)), indicating greater abnormality in LoA in patients with delirium, while there was no difference between dementia and control groups. There was a moderate negative correlation between OSLA and DelApp scores (τ = −0.61, p < 0.001), indicating that a more abnormal LoA was associated with worse inattention. A total of seven patients, all from the delirium group, scored less than 3 on the DelApp arousal sub-scale.
ROC analysis
ROC analysis was performed on the DelApp with delirium diagnosis as a reference. The area under the curve (AUC) was 0.96 (p < 0.001, 95% CI = (0.93, 0.995)), with 98% sensitivity and 93% specificity for detecting delirium in the study sample as a whole using a cut-off of 8 (out of a maximum possible score of 10). The seven CAM-negative participants who scored at or below this cut-off were all from the dementia group and had severe cognitive impairments (median OMCT score = 5, range = 0–9). Analysis based on the delirium and control groups alone revealed an AUC of 0.99 (p < 0.001, CI = (0.97, 1)), with 98% sensitivity and 100% specificity for detecting delirium. Analysis based on the delirium and dementia groups alone returned an AUC of 0.93 (p < 0.001, CI = (0.88, 0.99)), with 98% sensitivity and 87% specificity using the same cut-off of 8.
By comparison, the AUC of the BAT for detecting delirium in the entire study sample using a cut-off of 5 (out of 7) was 0.82 (p < 0.001, 95% CI = [0.75, 0.89]), with 76% sensitivity and 72% specificity. Direct comparison of the discriminative ability of these tests found that the AUC was significantly higher for the DelApp compared to the BAT (z = 3.8, p < 0.001, r = 0.37).
The DelApp as a measure of severity of attentional deficits
There was a moderate negative correlation between DelApp and DRS-R98 severity scores (τ = −0.60, p < 0.001). A 1-point decrease in DelApp score reflecting more inattention was associated with a 4.8-point increase in DRS-R98 severity score (β = −0.208, t (154) = −15.23, p < 0.001, R 2 = 0.62, CI = (9.66, 10.3); Figure 3).
Scatterplot demonstrating the relationship between DelApp and DRS-R98 severity scores. The diagonal black line is the overall regression line. Each group is represented separately using the shapes listed in the legend above. Data points have been jittered. There was a moderate negative correlation between DelApp and DRS-R98 severity scores (τ = −0.60, p < 0.001).
The proportion of correct responses on the DelApp test given by patients in the delirium group decreased as the level of difficulty increased. Based on odds ratios, the odds of giving the correct response was 2.2 times higher for level 1 compared to level 2 (p = 0.003, 95% CI = (1.26, 3.86)). Similarly, the odds of giving the correct response was 1.33 times higher for level 2 compared to level 3, though this was not statistically significant. No effects of task level were found for the dementia group or controls. Frequencies of responses are presented in Table 2.
Distribution of responses for each group on the DelApp across three levels of difficulty
Discussion
This preliminary study showed initial feasibility and validity of a new smartphone-based test of attention for use in delirium assessment. The DelApp was effective in objectively measuring inattention associated with delirium in a selected sample of older hospitalized patients. The DelApp was 98% sensitive and 93% specific to delirium in the sample as a whole, which is similar to the diagnostic performance of the original EDTB tasks (Brown et al., Reference Brown, Fordyce, Zaghdani, Starr and MacLullich2011). Furthermore, patients with delirium performed poorly on this test compared to cognitively normal patients and patients with dementia, even though delirium and dementia groups did not differ with respect to overall cognitive impairment as assessed by a standard test. This is important because symptom profiles in delirium and dementia commonly overlap, yet these disorders have radically different etiologies and treatments.
Patients with delirium performed more poorly on both the DelApp and BAT, which includes digit span, than patients with dementia. However, there was a substantial overlap in BAT scores between delirium and dementia groups. Direct comparison of the discriminative ability of DelApp and BAT tests indicated that the DelApp was more effective in detecting delirium and discriminating delirium from dementia. This is not surprising because digit span is not only employed as a test of attention in delirium, but also as a measure of IQ, working memory, and executive functioning (Iverson, Reference Iverson2001; Buchanan et al., Reference Buchanan, Heffernan, Parrott, Ling, Rodgers and Scholey2010; Greneche et al., Reference Greneche, Krieger, Bertrand, Erhardt, Maumy and Tassi2011). Moreover, studies have shown that patients with dementia perform poorly on the digit span backward (Meagher et al., Reference Meagher, Leonard, Donnelly, Conroy, Saunders and Trzepacz2010) and that the digit span forward does not distinguish between delirium and dementia (O’Keeffe and Gosney, Reference O’Keeffe and Gosney1997). In contrast, the DelApp attention assessment appears to provide a purer measure of attention, placing low demands on other aspects of cognition.
The broad range of DelApp scores in the delirium group supports the notion that this test potentially provides a graded measure of the severity of inattention in delirium. Further, scores on the DelApp and the DRS-R98 were moderately well correlated. Thus, the DelApp may help with severity grading of delirium. However, linear regression revealed large residuals in the delirium group, which may be explained by the range of symptoms and different time frames covered by each test. Specifically, the DelApp assesses current LoA and inattention, whereas the DRS-R98 accounts for a broad range of delirium symptoms occurring in the preceding 24 hours.
The DelApp test appears to have a number of strengths including its simplicity, ease-of-use, objectivity, and portability. Controlled presentation of stimuli and automated scoring make the DelApp less prone to experimenter bias or error, and it also requires little training and does not strongly rely on clinical judgment. As such, the DelApp may be of particular value when used by non-experts (Kean and Ryan, Reference Kean and Ryan2008). Our preliminary results suggest that the DelApp could provide a valid measure of delirium severity and as such may be useful in tracking change in delirium presence and severity over time.
Although abnormal LoA as measured with the OSLA was found in delirium, in line with previous findings (Tieges et al., Reference Tieges, McGrath, Hall and Maclullich2013), only seven patients (all with delirium) had abnormal scores on the arousal assessment of the DelApp. In the present study, this subscale therefore made a relatively small contribution to the overall diagnostic performance of the DelApp. This finding may indicate that the DelApp arousal assessment was insensitive to subtle abnormalities in LoA. However, spectrum bias may have played a role here, because ward staff generally did not recommend patients who were asleep or very drowsy, or extremely agitated, and so these patients were mostly not approached for recruitment. Importantly, in clinical practice 10% of all patients may have abnormalities in LoA as detected by routine assessments (Prytherch et al., Reference Prytherch, Smith, Schmidt and Featherstone2010). Therefore, the utility of the DelApp arousal subscale needs to be addressed in future studies with consecutive, unselected patients.
This study has several limitations that must be acknowledged. The DelApp assessment and reference diagnosis of delirium were both carried out by single raters. Although the raters did not consider the DelApp assessment when completing the CAM and DRS-R98 (which always followed the DelApp assessment), it cannot be stated with certainty if DelApp scores may have influenced delirium diagnosis to some extent or not. This may have resulted in incorporation bias. It should be noted however that a diagnosis of delirium and/or dementia was ascertained through communication with the clinical team responsible for the care of a patient and by examination of the case notes. Further, the clinical profile of most patients, including information from case notes and neuropsychological tests (blind to the DelApp) was discussed with an experienced geriatrician to validate the diagnosis clinically. Another study limitation is the use of a selected patient sample, which may affect the generalizability of the present findings. The DelApp requires further validation by confirming diagnostic test accuracy and potential for clinical applicability in a representative population of consecutive, unselected patients with blinded raters who perform the reference standard and index tests independently. Assessment of global cognition was done using the OMCT (Katzman et al., Reference Katzman, Brown, Fuld, Peck, Schechter and Schimmel1983). Whilst this measure is validated and has the advantages of brevity and ease-of-use, it does not provide a fine-grained measure of cognitive impairment. Future studies should consider using a more comprehensive assessment of cognitive impairment and also include a retrospective informant questionnaire such as the Informant Questionnaire of Cognitive Decline in the Elderly (IQCODE; Jorm and Jacomb, Reference Jorm and Jacomb1989) in order to obtain more accurate information about the possible presence of pre-morbid cognitive impairment prior to admission. Finally, grouping of delirium and co-morbid delirium-dementia precluded an investigation of the DelApp's ability to diagnose delirium superimposed on dementia. Our reason for grouping these patients was that the majority of older patients with delirium had some degree of underlying cognitive impairment or (often undiagnosed) dementia and therefore it was hard to recruit a “pure” delirium group with the means available to us at the time. Nonetheless, it would be interesting in future studies to assess differences in DelApp performance between delirium groups with and without dementia.
Feedback from patients indicated that the distractor lights on the hardware box were experienced as more distracting than the distractor triangles in the DelApp task. Future versions of the DelApp should therefore use distracting stimuli with a modified appearance (larger size, brighter colour, and/or more frequent presentation) to make them stand out more. A related issue is that, though patients with delirium performed worse on the third level of the attention task compared to previous levels, this performance difference did not reach significance. Thus, the attentional load of task level 3 could be increased in future versions of the DelApp. In sum, further development of the DelApp is required, including optimisation of test parameters, user interface and data acquisition and display. Also, in order for the DelApp to be readily applied as a screening tool in routine clinical practice, the duration of the DelApp assessment needs to be further shortened. This could be done by reducing the number of trials, the length of counting sequences and/or the inter-stimulus delays, and also by adopting a scoring system whereby the test is terminated following a specified number of repeated errors. Further, a more detailed characterization of type and severity of dementia is required to determine if our method can distinguish moderate to severe forms of dementia from milder forms of delirium. Finally, performance on the DelApp should be compared across different subtypes of dementia, including dementia with Lewy bodies which is associated with prominent attentional deficits (Metzler-Baddeley, Reference Metzler-Baddeley2007).
Conclusions
The current study provides the first evidence for the feasibility and validity of a smartphone-based assessment of attention in older hospitalized patients with delirium, including those who are frail or acutely unwell. The findings also support and extend previous observations that patients with delirium have specific impairments of sustained attention. The DelApp shows promise as a sensitive, specific, and valid tool to assist identification and severity grading of delirium in research and clinical practice. In this regard, the DelApp could potentially be integrated into current delirium assessment tools by providing a robust and validated assessment of inattention. Further development of the DelApp is now required to make it suitable as a screening tool for delirium in routine clinical practice. Finally, studies are needed to assess validity in representative, unselected patients using independent raters.
Conflicts of interest
AM holds patents on computerized devices and tests for measuring attention in delirium.
Description of authors’ roles
Alasdair MacLullich (AM) and Zoë Tieges (ZT) designed the study and provided supervision. Antaine Stíobhairt (AS), Klaudia Suchorab (KSu), and Katie Scott (KSc) carried out patient recruitment, data collection and data analysis (with help of ZT), and also contributed to study design. The manuscript was drafted by AS and ZT; AM, KSc and KSu assisted with writing the manuscript. The DelApp software was developed by Alexander Weir and Stuart Parks in collaboration with ZT and AM. Susan Shenkin helped with patient recruitment, and provided input into the project plan and manuscript.
Acknowledgments
This study was funded by a Medical Research Council Centenary Early Career Award to Z. Tieges. Funding from the Biotechnology and Biological Sciences Research Council, the Engineering and Physical Sciences Research Council, the Economic and Social Research Council, and the Medical Research Council is gratefully acknowledged. The authors also thank the patients and staff from the Medicine of the Elderly and acute orthopedic wards of the Royal Infirmary of Edinburgh and the acute and rehabilitation wards of Liberton Hospital in Edinburgh.
