Introduction
Visuospatial constructive cognition is the ability to observe and reproduce a visual image, including its components. Traditionally, this ability is tested through tests of drawing and functional tasks such as buttoning shirts, constructing models, making a bed and assembling objects such as furniture (Giannakopoulos et al. Reference Giannakopoulos, Duc, Gold, Hof, Michel and Bouras1998). Visuospatial function is commonly impaired in major neurocognitive disorders (Meagher et al. Reference Meagher, Moran, Raju, Gibbons, Donnelly, Saunders and Trzepacz2007, Reference Meagher, Leonard, Donnelly, Conroy, Saunders and Trzepacz2010; Brown et al. Reference Brown, McGrory, McLaren, Starr, Deary and Maclullich2009) and is tested in the two most popular bedside instruments for assessment of cognitive function – the Montreal Cognitive Assessment (MoCA) (Nasreddine et al. Reference Nasreddine, Phillips, Bédirian, Charbonneau, Whitehead and Collin2005) which includes the clock drawing test (CDT), and Mini Mental State Examination (MMSE) (Folstein et al. Reference Folstein, Folstein and McHugh1975) which includes the intersecting pentagon test (IPT). These tests are relatively easy to administer, sensitive to cognitive changes with good predictive validity for significant cognitive disorder, and correlate significantly with other standardised screening tests of cognition such as the Cambridge Cognition Examination and the MMSE (Bourke et al. Reference Bourke, Castlenden, Stephen and Dennis1995; Shulman, Reference Shulman2000).
However, the CDT has multiple versions with different scoring methods such that there is a lack of consistency in testing and interpretation. Moreover, its validity is impacted upon by issues such as language, culture and level of education of the subject being assessed (Pinto & Peters, Reference Pinto and Peters2009). Price et al. (Reference Price, Cunningham, Coronado, Freedland, Cosentino and Penney2011) found that both inter and intra-rater reliability of the CDT were low without repeated training. Similarly, the IPT is also complicated by inconsistency in scoring methods (Helmes, Reference Helmes2013).
The letter and shape drawing (LSD) test is a novel test developed by the Cognitive Impairment Research Group (CIRG) at the University of Limerick Medical School. It aims to provide a test of visuospatial ability that is simpler than existing bedside tests (e.g. clock drawing) and that can be readily presented via computer-assisted devices to allow for convenient and reliable testing. Moreover, it is also expected that it would be brief; easy to administer; acceptable to older persons; minimally affected by education, sex and age, with high sensitivity and specificity for major neurocognitive disorders (Borson et al. Reference Borson, Scanlan, Peijun and Ganguli2003).
The objective of this pilot study was to investigate how the LSD test performs in comparison with conventional tests of visuospatial function (specifically the CDT and IPT) as well as generalised cognition (as measured on the MoCA) in medically unwell hospital inpatients over the age of 65.
Methods
This was a cross-sectional study carried out at University Hospital Limerick, Ireland between December 2013 and February 2014.
The subjects of the study comprised hospitalised elderly medical patients aged 65 years and above who were receiving care in the general wards of University Hospital Limerick (Ireland). The patients were consecutively referred by the medical team as deemed well enough to participate. They were specifically assessed for dementia (evident from clinical case-notes or collateral history) and all participants were assessed for possible delirium assessed using the confusion assessment method (CAM) (Inouye et al. Reference Inouye, van Dyck, Alessi, Balkin, Siegal and Horwitz1990).
Measures
All participants were assessed with the following series of tests; the CAM, MoCA, IPT, CDT and the LSD. All tests were administered in a single session by the same assessor and in the order listed. The LSD test was presented last in order to minimise any possible advantage due to less fatigue that might be conferred by being presented earlier in the series of tests.
CAM
The presence of possible delirium was assessed using the CAM (Inouye et al. Reference Inouye, van Dyck, Alessi, Balkin, Siegal and Horwitz1990). The CAM is the most widely used screening tool for delirium (Wei et al. Reference Wei, Fearing, Sternberg and Inouye2008). All CAM assessments were performed by a researcher (D.W.) who was formally trained in its use.
MoCA
The MoCA is a screening tool for the detection of general cognitive impairment (Nasreddine et al. Reference Nasreddine, Phillips, Bédirian, Charbonneau, Whitehead and Collin2005). It assesses attention and concentration, executive function, memory, language, visuo-constructional skills, conceptual thinking, calculation and orientation. Visuospatial ability is assessed using a clock-drawing task (three points) and a three-dimensional cube copy (one point). The total possible score is 30 points; a score of ⩾26 is considered normal but other work has suggested a cut-off score of 20 or less points as indicative of significant cognitive impairment (Waldron-Perrine & Axelrod, Reference Waldron-Perrine and Axelrod2012). The MoCA was administered in keeping with the Montreal Cognitive Assessment Administration and Scoring Instructions (available at www.mocatest.org). The sensitivity and specificity of the MoCA for significant cognitive impairment are high (both >85%), it has high inter-rater and test–retest reliability and, in contrast to the MMSE, is less prone to ceiling effects as performance is normally distributed (Nasreddine et al. Reference Nasreddine, Phillips, Bédirian, Charbonneau, Whitehead and Collin2005; Koski, Reference Koski2013).
IPT
This is a geometric copying test derived from the original Bender Gestalt test (Reference Bender1938). The subject is presented with a copy of two intersecting pentagons drawn at angles to one another producing a diamond shape where they overlap. The subject is then requested to copy the design on the blank half of the page. For scoring, we applied the six-point hierarchical scoring scale developed by Bourke et al. (Reference Bourke, Castlenden, Stephen and Dennis1995), where 6 represents a perfect reproduction and 1 represents the poorest effort (see Table 1). A score of <4 is considered to reflect a failed performance. This scale allows for sensitive and objective assessment of the range of possible performances.
Scoring systems for the interlocking pentagon test (IPT) and clock drawing test (CDT)
CDT
The CDT examines visuospatial abilities as well as receptive language, numerical knowledge, working memory and executive functions (both motor and cognitive) (Freedman et al. Reference Freedman, Leach, Kaplan, Winocur, Shulman and Delis1994). It is widely used in geriatric practice, where it can provide a brief ‘cognitive scan’ (Lezak et al. Reference Lezak, Howieson and Loring2004). In this study, subjects were provided with a pre-drawn circle onto which the participant was requested to place all the numbers and the large and small hands on the clock face to show the time ‘ten past eleven’. We used the scoring method of Sunderland et al. (Reference Sunderland, Hill, Mellow, Lawlor, Gundersheimer and Newhouse1989) which ranges from 0 to 10 according to spatial representation of the numbers and hands of the clock (see Table 1). A score of <6 is considered a failed performance.
LSD test
The LSD test is a novel instrument developed by the Cognitive Impairment Research Group at the University of Limerick. It is designed to emphasise visuospatial abilities over other functions that are relevant to performance on tests of constructional abilities (e.g. fine motor skills). The initial design was piloted for acceptability and coverage in a series of trials in a nursing home population with varying degrees of cognitive impairment. The test includes a series of 15 designs that link 1 cm spheres arranged in increasingly complex 3×3, 4×4, 5×5 and 6×6 grids that the subject copies to an adjacent blank grid (e.g. see Fig. 1). Each grid occupies one half of an A5 page. A correct performance requires that all relevant spheres are connected to complete the required shape. Omissions (but not commissions) are rated as errors. As such, performance can be precisely determined as a correct reproduction of the letter or shape or not. Subjects are permitted a single trial of each of the 15 items. The test typically takes 5 minutes to complete. The current version is presented in a pen and paper format but is designed to allow for ease of transfer to a touch sensitive grid on an android tablet platform. Performance was measured by totalling the number of correctly completed grids (scoring range 0–15).
Sample items from the letter and shape drawing test.
Data collection
Scoring of the CDT, IPT and LSD was achieved through a consensus group comprised of members of the CIRG (D.W., O.W., M.L., D.M.). The typical completion times for the full test battery were 30 minutes.
Ethical approval
The procedures and rationale for the assessments were explained to all participants but because many patients had possible delirium and/or dementia at entry into these studies it was presumed that some would not be capable of giving informed written consent. The University Hospital Limerick Research Ethics Committee approved an approach whereby patient verbal assent was augmented with proxy consent from next of kin (where possible) or a responsible caregiver. This is in accordance with the Helsinki Guidelines for Medical research involving human subjects (World Medical Association, 2004). These assessment procedures did not have any identified significant risks but the patient and family/carers were informed that they could withdraw participation at any stage
Statistical analysis
Statistical analysis was conducted using the statistical software package SPSS for Windows Version 19. Continuous variables were tested for normality and reported as means (standard deviation) or median (range), while categorical variables are reported as counts and percentages. Spearman’s correlation coefficient was used to measure the strength of association between total LSD score and the CDT, IPT, MoCA and individual neuropsychological domains of the MoCA. The optimal cut-off score for the LSD for identifying impaired performance on the MoCA (score ⩽20) was explored using receiver operated curve analysis. Areas under the receiver operated curve (AUC) were generated and compared using the ‘pROC’ version 1.8 R package (Robin et al. Reference Robin, Turck and Hainard2011). The accuracy of the LSD, CDT and IPT for identifying significant cognitive impairment was compared according to sensitivity, specificity, positive and negative predictive values for MoCA score ⩽20.
Results
General clinical and demographic characteristics
We assessed a total of 40 elderly medical patients (mean age 81.0±7.71; 21 female). The median MoCA score was 15.5 (mean 14.9±7.9; range 1–29), with the majority (n=35; 88%) scoring below 26 (i.e. the threshold for significant cognitive impairment). Four of these patients were CAM positive for delirium while 12 had documented dementia.
Comparison of performance on the LSD, CDT and IPT
The correlation between the scores on the three tests of visuospatial ability were as follows: CDT and IPT (r=0.42); total LSD score and CDT (r=0.56), and total LSD score and IPT (r=0.71).
For patients who scored 15 on the LSD (the maximum score) the range of CDT scores was 5–10. There were variable scores for the LSD in those subjects who scored highly (7 or greater) on the CDT. For patients who scored 15 on the LSD the range of scores on the IPT was 4–6. These findings indicate correlation that varies from moderate to strong across the various tests of visuospatial ability.
Comparison of LSD, CDT and IPT in terms of total MoCA score
The correlation between Total MoCA and LSD scores was 0.91 showing a strong positive relationship between the scores. This was higher than the correlation between the total MoCA score and the IPT or CDT (both cases = 0.67).
Correlation between the total LSD score and individual domains of the MoCA
Table 2 shows the correlation between LSD score and the seven domains of the MoCA. There was high correlation with all domains (ranging from 0.54 to 0.86) and especially for the domains of orientation, attention, delayed recall and visuospatial function.
Correlation of letter and shape drawing (LSD) test with Montreal Cognitive Assessment (MoCA) domains
a All correlations statistically significant at 1% level of significance.
Identification of significant cognitive impairment on the MoCA by visuospatial tests
We examined the ability of the tests of visuospatial function to identify those patients with significant general cognitive impairment as evidenced by scores of 20 or less on the MoCA. The most accurate cut-off score in terms of combined sensitivity and specificity values for the LSD was 2 or more incorrect responses (i.e. pass if ⩾14) (AUC=0.99). This cut-off score identified 26 of the 29 cases with MoCA ⩽21 (90%) and without any false positives. The equivalent identification for the CDT (applying the cut-off score of <6) was 17/29 (59%), while for the IPT (applying the cut-off score of <4) correct identification was 16/29 (55%). The sensitivity, specificity, positive and negative predictive values for performance on the tests of visuospatial tests for significant cognitive impairment as per a score of ⩽20 on the MoCA is shown in Table 3.
Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for total Montreal Cognitive Assessment (MoCA) score ⩽20 according to the clock-drawing test (CDT), interlocking pentagon test (IPT) and letter and shape drawing (LSD) test
CI, confidence interval.
Comparison of AUCs (LSD v. CDT and LSD v. IPT)
By using a cut-off point of ⩾14 as equating with a pass on the LSD, we compared the areas under the curve of the CDT and IPT (Hanley & McNeil, Reference Hanley and McNeil1983). For the first comparison (LSD v. CDT) the area under the curve was 0.77 (95% CI: 63.03–91.64%) with a power of 0.84 at a significance level of 0.05. For the second comparison (LSD v. IPT) the AUC was 0.81 (95% CI: 67.61–94.2%) with a power of 0.94. Although the accuracy of the IPT was higher than for the CDT for LSD performance, a comparison of the two tests indicated that this was not statistically significant (z=−0.37, p=0.36).
DISCUSSION
This pilot study indicates that the LSD test can be readily applied to the testing of cognitive performance in elderly medical inpatients. The capacity of the LSD to engage visuospatial abilities was supported by its agreement with the IPT and CDT, which are commonly used bedside tests of visuospatial function. In addition, we found that the LSD correlated very highly with total MoCA scores and to a substantially greater extent than either the CDT or IPT. Errors on the LSD were indicative of significant generalised cognitive impairment such that we were able to identify a cut-off score with high sensitivity and specificity for patients who scored in the range for significant cognitive impairment with the MoCA.
The CDT and IPT are the most commonly used bedside tests of visuospatial function but have difficulties in terms of the consistency of presentation, testing procedures and interpretation. The LSD test is designed to minimise these shortcomings with a simple presentation using a grid system to optimise consistency. The scoring of performance is simple and precise with minimal subjectivity since subjects must correctly connect the spheres on the grid. This presentation was developed with the potential to apply it to touch-sensitive devices where the presentation and interpretation can be even more consistent. This would also readily allow for dimensional adjustments (e.g. in diameter of the circles on the grid or connector line thickness) according to subject characteristics (e.g. increased diameter for those with impaired visual acuity) but such calibration will require further studies comparing performance on different versions of the test in subjects with varying visual and/ or fine motor skills. This ability to adjust testing procedures according to subject characteristics can be a significant advantage of the LSD test over the CDT and IPT.
Rather than conceptualising visuospatial constructive cognition as a unified construct, many observers categorise tests according to their content as: assembly versus graphomotor, copy versus draw-to-command, and complex versus simple tasks. Some tests (e.g. CDT) traverse these boundaries due to the availability different versions (Lezak et al. Reference Lezak, Howieson and Loring2004). In a study of visuoconstructional testing Somerville-Ruffalo (Reference Somerville-Ruffalo2004) concluded that the most useful distinction was in terms of complexity level. Complex visuospatial constructive cognitive tasks tend to be more heterogeneous in their underlying neuropsychological domains and are more demanding of executive skills. In contrast, simple tests are more homogenous in function, mostly assessing visuospatial and perceptual skills. However, they also concluded that visuospatial function was the primary determinant of performance on all of the tests they included, both simple and complex. For our work, we used a version of the CDT that was intermediate in complexity (i.e. a pre-drawn circle but without numbers or hands). Our findings suggest that the LSD is a simpler tests than either the CDT or IPT as reflected in the numbers of patients who successfully completed each as well as correlation in performance across the tests.
There was high correlation between the LSD test and the MoCA, including each of its individual domains. As we expected, there was a high correlation with the visuospatial element of the MoCA but performance on the LSD also correlated very highly with the orientation item of the MoCA. These findings suggest that while emphasising visuospatial abilities, performance on the LSD also engages other cognitive abilities. One implication is that the LSD may be useful for testing for conditions where disorientation is a prominent feature, such as delirium. Moreover, the concordance between impaired performance on the LSD test and significant cognitive impairment on the MoCA suggest that the LSD test could be useful as a brief test of general cognition that is simple to administer and interpret while incorporating highly consistent presentation and administration procedures. These possibilities warrant further investigation.
Given the variety of neuropsychological processes that underlie performance on tests of visuospatial abilities, using a single test may limit the accuracy of assessment in populations with heterogenous cognitive abilities (Lezak et al. Reference Lezak, Howieson and Loring2004). The LSD is designed to traverse a wide range of performance abilities from simple copying of an ‘X’ in a 3×3 grid to copying a complex shape in a 6×6 grid. As such, the various sections of the LSD may engage both simple and complex functions within a single test and may allow for more accurate assessment of real world clinical populations where abilities typically vary considerably both across patients as well as for individual patients over time. To this end, our study identified aspects of the current version of the LSD test that may warrant amendments. Performance on the LSD test indicated that the majority of subjects could readily complete most of the 15 items without error. While this has positive implications for the ability of the test to engage clinical populations, the findings also suggest that the test could be readily abbreviated. However, many patients who were unable to complete the other tests without error achieved almost maximum scores on the LSD test. This suggests that the level of difficulty of the LSD may need to be increased and that additional more complex items could improve the sensitivity of the test in higher functioning subjects. A major shortcoming of many existing bedside tests of cognition (including both the CDT and IPT) is that the level of cognitive function required to register minimal engagement produces bottoming out effects in clinical populations where subjects with, for example, delirium cannot engage even minimally with the test (Bellelli et al., Reference Bellelli, Morandi, Davis, Mazzola, Turco and Gentile2014; European Delirium Association, 2014). It is hoped that the simplicity of the initial items in the LSD test can allow for better coverage of performance in those with severe cognitive impairment. The subjects included in this study were mostly in the higher range of performance on the MoCA such that further studies in patients with more severe cognitive difficulties can examine this issue in more detail. Overall, this study suggests that the LSD could be usefully amended to include fewer items but with some additional complex items.
This work identified that two or more errors on the LSD test had high sensitivity and specificity for identifying those subjects with impaired performance on the MoCA. These preliminary findings should be examined in a larger study that includes subjects from across a more complete range of cognitive abilities. In addition, the accuracy of performance on the LSD for identifying specific neurocognitive disorders (e.g. delirium) can be explored in future work where more precise diagnoses are established. Never the less, the capacity of the LSD test to allow for rapid and highly consistent cognitive testing at the bedside supports the possibility that it can be adapted for a useful role in everyday clinical practice.
Limitations
This pilot study included a relatively small number of subjects and did not allow for assessment of the possible impact of various confounding factors such as sensory abilities, frailty and educational background on performance (Hong et al. Reference Hong, Yoon, Shim, Cho, Lee and Kim2011). In addition, the series of tests were administered in a specific sequence for all subjects such that we could not assess the possible impact of practice effects and/or fatigue with testing. In an effort to reduce any possible advantage for the LSD test it was presented last in the series so that it would be most prone to any negative effects of fatigue. Future studies can examine this issue by applying testing sequences that are randomised.
Conclusion
The LSD test is a novel test of cognitive function that emphasises visuospatial abilities. It allows for simple and highly consistent testing. Performance on the LSD test correlates highly with existing gold standard tests for bedside testing of visuospatial function such as the CDT and IPT, and also with more generalised cognitive function measured on the MoCA. The usefulness of the LSD test may be enhanced by amending the number and complexity of items. Further larger studies can examine the reliability and acceptability of the LSD test in different clinical populations to clarify how it can be used in the identification of clinically significant cognitive impairment and specific neurocognitive disorders in everyday practice.
Acknowledgements
The authors thank the staff at University Hospital Limerick for their support with this work. Dr Maeve Leonard (RIP) provided much of the inspiration and wisdom that underpins the content of this report.
Financial Support
This work was supported by a research project grant from the Health Research Board (HRA 2011/48).
Conflicts of Interest
None.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committee on human experimentation with the Helsinki Declaration of 1975, as revised in 2008. The study protocol was approved by the institutional review board of each participating institution. Written informed consent was obtained from all participating patients.
