Introduction
As life expectancy continuously lengthens, the prevalence of chronic conditions associated with age increases. Dementia, including that caused by Alzheimer's disease (AD), is mostly diagnosed in patients above 60 years of age, and its prevalence is growing in socioeconomically depressed regions (Prince et al., Reference Prince, Bryce, Albanese, Wimo, Ribeiro and Ferri2013). Dementia involves progressive cognitive impairment, typically starting with impairments in declarative memory and daily living. Although no treatment can currently control these cognitive symptoms, pharmacological (Santo et al., 2013) and non-pharmacological (Yamaguchi et al., Reference Yamaguchi, Maki and Yamagami2010) interventions may improve the patient's cognitive, behavioral, and functional symptoms, in addition to improving the patient's quality of life. In this sense, the earlier a diagnosis is made, the sooner the patient's treatment will commence, and the sooner the consequent benefits will occur.
For diagnosing dementia, at least two cognitive domains must be impaired compared to the patient's previous level of performance. These cognitive domains include memory, reasoning, visuospatial abilities, language, and the mood/personality/behavior complex (McKhann et al., Reference McKhann2011). Visuospatial abilities describe a cognitive domain involved in several dementia subtypes, such as Parkinson's disease dementia, Lewy body dementia, and AD (Salmon and Bondi, Reference Salmon and Bondi2009). In patients with AD, this impairment usually is reflected in a patient's performance on visuoconstructional tests, such as drawing and object assembly (Tippet and Black, Reference Tippett and Black2008). Acceptable performance on these paradigms requires that specific elements of the stimuli must be integrated and transformed along goal-oriented motor outputs; this process demands the association of different brain regions including, the occipital and parietal lobes (Guérin et al., Reference Guérin, Ska and Belleville1999; Tippet and Black, Reference Tippett and Black2008). In AD, the functional integration of these regions is usually altered, which may underlie the visuospatial dysfunction (Prvulovic et al., Reference Prvulovic2002; Tippet and Black, Reference Tippett and Black2008).
Cognitive impairment can be objectively assessed by means of neuropsychological assessment – i.e. using structured tests to make inferences about the patient's actual cognitive integrity. Because the concept of cognitive decline is pivotal to making a dementia diagnosis, educational background and other demographic factors (e.g. age and gender) should be considered when comparing present and past functional levels (Salmon and Bondi, Reference Salmon and Bondi2009). The role of educational background in neuropsychological assessment is crucial with respect to differential diagnosis issues. This is of particular relevance in those cases of cognitive complaints among elderly individuals with low educational levels. Because most neuropsychological assessment tasks are influenced by the educational background of the patient, those tasks that are less dependent on formal education should preferentially be used.
The Rey-Osterrieth Complex Figure Test (Rey, Reference Rey1941) is among the most traditional measures of measuring constructional praxis. However, as highlighted in a review by Rosselli and Ardila Reference Rosselli and Ardila(2003), this task exhibits a ceiling effect among healthy adults and older adults with 12 or more years of education (i.e. a performance between 30 and 36 points), whereas illiterates’ scores usually exhibit less efficient performance (i.e. between 15 and 20 points). In this sense, test interpretation is markedly influenced by educational level, reducing its validity for neuropsychological assessment, especially when we consider criterion-related validity. Despite the predominance of drawing tasks in the assessment of constructional praxis, some evidence suggests that classical neuropsychological tasks such as the Rey-Osterrieth Complex Figure Test seem to be highly related to formal education (Rosselli and Ardila, Reference Rosselli and Ardila2003). Even providing specific population norms for subjects with low educational levels requires special consideration of the important peripheral effects of administering such paper-and-pencil tasks; for example, patients may experience negative emotional reactions while performing the tasks, which may impair their performance (Guruje et al., Reference Guruje1995; Unverzagt et al. Reference Unverzagt1999). Thus, although drawing tasks are usually applied in the assessment of visuospatial abilities, in contexts where the subject shows little familiarity with paper-and-pencil skills, model assembly procedures could be an interesting alternative for patient evaluation.
Baiyewu et al. Reference Baiyewu(2005) adapted a constructional praxis test using sticks for the assessment of dementia. The Stick Design Test was proposed as an alternative for drawing tests that would be suitable for elderly patients with low formal education. They developed and validated the test in a sample (N = 724) with an average age just over 78 years, a predominance of females and little education (i.e. less than 10% of the participants had any formal schooling). The results for the Stick Design Test showed significant effects for gender (viz., females did more poorly than males), education (viz., better scores were elicited from participants with any education compared to those without formal education), and age (viz., better scores for younger participants). The Stick Design Test correctly classified 58% of the Dementia group and detected cognitive dysfunction of any type for 36% of a combined group of participants with cognitive impairment without dementia and patients with dementia. This task presents, therefore, a potential solution to the assessment of elderly patients with low educational levels. Nevertheless, Baiyewu et al. study (Reference Baiyewu2005) did not explore the relationship between the test scores and specific measures of functional performance. Their study focused solely on criterion-related validity and did not explore aspects related to construct validity – the cognitive processes being assessed by the test – and ecological validity – the relationship of the test with everyday functioning.
The present study aims to evaluate the use of the Stick Design Test in elderly Brazilian patients with low formal education. We intend to investigate its psychometric properties related to construct, criterion-related and ecological validities, as well as to assess the influence of formal education on test performance. Based on the report of Baiyewu et al. Reference Baiyewu(2005), we hypothesize that the diagnosis of dementia will be met with a moderate level of accuracy.
Methods
Participants
In the present study, 128 elderly persons from the city of Belo Horizonte, Brazil, were assessed in an outpatient public health unit by a multidisciplinary team composed of gerontologists, neuropsychologists, occupational therapists, and others. The patients were divided into two groups: healthy controls and patients with mild AD, diagnosed according to DSM-IV (Diagnostic and Statistical Manual of Mental Disorders, 4th Edition) (APA, 2000), and NINDS-AIREN (McKhan et al., Reference McKhann2011) criteria. Participants with significant sensorial limitations, positive psychotic symptoms or without a close caregiver at the interview were not invited for participation. Diagnosis involved an unstructured clinical interview performed by the geriatrician and objective cognitive assessment, including the Mini-Mental State Examination (Folstein et al., Reference Folstein, Fosltein and McHugh1975), the Mattis Dementia Rating Scale (Mattis, Reference Mattis1988) and subtests from the CERAD Neuropsychological Battery (Morris et al., Reference Morris1989). All of the neuropsychological assessment was performed by clinical neuropsychologists specialized in elderly populations; age and education-corrected scores and cutoffs were used. Functional status was assessed in the clinical interview. The Clinical Dementia Rating was used to stage AD patients; only participants with no dementia (CDR = 0, control group) or mild dementia (CDR = 1, AD group) were invited for participation.
Neuropsychological assessment
The Stick Design Test proposed by Baiyewu et al. Reference Baiyewu(2005) was used in this study. The test consists of four different models to be constructed using four wooden matches. The models vary and are scored based on general configuration, orientation of the whole figure and orientation of the wooden matches’ global aspects. The stimuli vary on different spatial aspects: open/closed, geometrical/non-geometrical, canonical/non-canonical position, and right/oblique angles. The reader should refer to the Baiyewu et al. Reference Baiyewu(2005) paper for the original stimuli. Three other neuropsychological measures were used. The Mini-Mental State Examination (Folstein et al., Reference Folstein, Fosltein and McHugh1975) served as a general measure of cognitive functioning. For testing the Stick Design Test's construct validity, we adopted another visuospatial measure, the Clock Drawing Test (Shulman, Reference Shulman2000) – scored from 0 (worst) to 5 (best) – as a measure of convergent validity. The Digit Span Forward Test (de Paula et al., Reference de Paula2010) – corrected by the product of the total trials and the maximum span – was used for the assessment of divergent validity due to its being a measure of verbal working memory.
Statistical procedures
Due to the non-normal distribution of the data, non-parametric tests were adopted. For assessing the Stick Design Test's construct validity, a factor analysis procedure was performed with the test variables, aiming to characterize the test's latent structure. Principal axis factoring was used for factor extraction and, if more than one component were found, an orthogonal rotation (viz., Varimax) was selected. Bartlett's test for sphericity and the Kaiser–Meyer–Olkim test of sample adequacy were used for assessing the viability of the procedure.
Convergent validity and divergent validity were assessed through correlations between the neuropsychological measures. Because the Stick Design Test is supposed to assess visuospatial abilities, we computed Spearman rank-order correlations between its total score and the Clock Drawing Test, a classical neuropsychological measure related to visuospatial processing. Moderate to high correlations among these measures were expected to suggest that a similar construct was being assessed. The same procedure was applied between the Stick Design Test and the Digit Span Forward Test – a measure of the phonological loop of working memory. Here, weak or non-significant correlations between these measures were expected to suggest that different constructs were being measured. The magnitudes of the correlation coefficients were compared through Fisher r-to-Z transformations. The same procedures were adopted to assess the influence of formal education on the performance of the two visuospatial ability measures.
Criterion-related validity was assessed through a receiver operating characteristic (ROC) curve analysis, which evaluated the Stick Design Test's capacity for distinguishing healthy controls from AD patients, while using the Clock Drawing Test as a parameter. To assess which test performed better for diagnostic purposes, we compared the area under the curve of the two measures, using the MEDCALC software (MedCalc for Windows, version 12.4.0.0–64-bit, MedCalc Software, Mariakerke, Belgium). A secondary analysis tested a potential synergistic effect of the Stick Design Test with the Mini-Mental State Examination for the correct classification of the participants. This procedure was performed through logistic regression, with one model (Model 1) containing only the Mini-Mental State Examination and the other (Model 2) adding the Stick Design Test. These procedures were established because the Mini-Mental State Examination is usually adopted in the screening or diagnosis of dementia, and the addition of other tests in the patient's assessment should not be redundant. Explained variance was estimated by the Negelkerke method. Once again, the same procedure was applied to the Clock Drawing Test on a third model (Model 3) as a comparison parameter.
Finally, ecological validity, conceptualized as the relationship between neuropsychological tests and the everyday functioning of the patient (Chaytor and Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003), was assessed. This property was evaluated through Spearman rank-order correlations between the Stick Design Test and a Activities of Daily Living (ADLs) inventory based on the scales proposed by Katz et al. (Reference Katz, Ford, Moskowitz, Jackson and Jaffe1963) and Lawton and Brody (Reference Lawton and Brody1969); designed to assess self-care ADLs and domestic and complex components of instrumental ADLs (de Paula et al., in press A). We expected higher correlations with Instrumental ADLs because these more complex activities are usually more strongly associated with cognitive functioning in this population (de Paula and Malloy-Diniz, Reference de Paula and Malloy-Diniz2013). The statistical procedures, unless stated differently, were performed using SPSS 19.0. (IBM Corp, 2010) Statistical significance was established at 0.05.
Results
The sociodemographic description of the participants and neuropsychological tests performance are shown in Table 1. No significant differences regarding age (U = 2729.00, p = 0.573), education (U = 2522.00, p = 0.177) or proportion of men and women (χ2 = 2.27, p = 0.152) were found. The groups differed on all cognitive tests (p < 0.001), with effect sizes ranging from small to large. Differences in functional status were significant for Instrumental Domestic and Complex ADLs (p < 0.001) but not for Personal Care ADLs (U = 2556.00, p = 0.061).
Participants’ description and groups’ comparison on neuropsychological and functional measures
*p < 0.001.
AD = Alzheimer's disease; SE = standard error; Pc = percentile; ADL = activities of daily living; SE = standard error of the mean; Pc = percentile; MW U = Mann–Whitney U test; Z = difference between standard scores, r = effect size for group comparisons.
Considering the factor structure of the Stick Design Test, the significance of Bartlett's test of sphericity (χ2 = 173.02, p < 0.001) and the results of the Kaiser–Meyer–Olkim test of sample adequacy (0.779) indicated that sufficient sample size existed for the procedure. Only one factor was extracted (eigenvalue: 2.60), accounting for 64% of the test variance. The factor loadings for the four test stimuli were 0.737, 0.823, 0.726, and 0.633. These factor loads can be considered to be large.
Correlations between the neuropsychological measures, age, education, and ADLs are shown in Table 2. The Stick Design Test was moderately correlated with the Clock Drawing Test (ϱ = 0.544, p < 0.001), whereas it weakly correlated with the Digit Span Forward Test (ϱ = 0.255, p = 0.002). When these two correlation coefficients were compared, a significant difference was found (Z = 2.84, p = 0.004). These results suggest the existence of a stronger association between the Stick Design Test another measure of visuospatial processing (convergent validity) than with a measure of verbal working memory (divergent validity), thereby providing evidence of its construct validity. When testing for the correlations between formal education and the two visuoconstructional praxis tests, significant associations were found for both the Clock Drawing Test (ϱ = 0.503, p < 0.001) and the Stick Design Test (ϱ = 0.211, p = 0.009). However, this factor was smaller in magnitude in the case of the second test (Z = 2.68, p = 0.007), suggesting that the Stick Design Test was significantly less dependent on formal education than was the Clock Drawing Test for this population.
Correlations between sociodemographic, neuropsychological, and functional measures
*p < 0.05, **p < 0.01. ADL: activities of daily living.
The assessment of criterion-related validity using ROC curve analysis is shown in Figure one. The area under the curve of the Stick Design Test (0.756, SE = 0.03, p < 0.001) can be considered moderate in magnitude. For the Clock Drawing Test, the area under the curve was also moderate in magnitude (0.840, SE = 0.03, p < 0.001). When the areas under the curve of these two independent analyses were compared, a significant difference was found (Z = –1.98, p = 0.048), suggesting greater accuracy when the Clock Drawing Test was used by itself. The ROC analysis for these two measures is shown in the Figure 1.
ROC curve analysis for the diagnosis of mild Alzheimer's disease.
With respect to the logistic regression models, Model 1, including the Mini-Mental State Examination, was significant (χ2 = 91.87, p < 0.001) and correctly classified 82% of the participants. The addition of the Stick Design Test (Model 2) was significant (p < 0.001) and improved the overall prediction rate to 84%. The use of the Mini-Mental State Examination and the Clock Drawing Test (Model 3) also showed significant results (χ2 = 100.91, p < 0.001) and, similar to the previous analysis, classified 84% of the participants correctly, indicating no differences in the accuracies of Models 2 and 3. A description of the models is given on Table 3.
Logistic Regression Models for participant's classification
Model differences: 1 < 2, 1 < 3, 2 = 3.
AD = Alzheimer's disease.
For estimating the ecological validity of the Stick Design Test, correlations were computed between the test and the systematic reports of functionality by the participants’ caregivers (Table 2). Moderate correlations were found between the test and aspects of domestic and complex ADL.
Discussion
The development of instruments that could be comparable across educational levels or involving a little-studied minority is, in fact, a challenge in the context of cross-cultural studies of dementia. Analyzing the influence of formal education on neuropsychological assessment is important not only in clinical diagnosis and prognosis but also in the explanations of cognitive processes that underlie specific abilities and their neurobiological correlates (Ardila et al., Reference Ardila2010). These findings support the idea that the acquisition of reading and writing skills may radically change cognitive strategies (Rosselli and Ardila, Reference Rosselli and Ardila2003). Moreover, educational level most likely explains some of the inconsistencies among cognitive screening studies (Jacqmin-Gadda et al., Reference Jacqmin-Gadda, Fabrigoule, Commenges, Letenneur and Dartigues2000).
Visuospatial tasks involving drawing are markedly difficult for older adults with little or no formal education (Baiyewu et al., Reference Baiyewu2005; Ardila et al., Reference Ardila2010; de Paula et al., Reference de Paula2010). An alternative visuospatial task, such as arranging sticks in a particular pattern, is feasible, as demonstrated in a previous study (Baiyewu et al., Reference Baiyewu2005) and by our present results. We assessed the influence of formal education on test performance and found that the Stick Design Test is less influenced by this variable than the Clock Drawing Test – a test that, according to a Brazilian review (Vasconcelos, Reference Vasconcelos, Brucki and Bueno2007), is preferentially used for the assessment of visuospatial abilities. Our analyses show that the Stick Design Test is a valid instrument for assessing visuospatial abilities in older adults with low education levels, as suggested by lines of evidence supporting its construct validity (viz., a single-factor structure, with convergent and divergent correlations with other neuropsychological measures), criterion-related validity (viz., a significant and moderate-magnitude area under the curve, with significant additive effects in combination with the MMSE) and ecological validity (viz., significant correlations with domestic and complex instrumental ADLs). Together, these results show the adequacy of an instrument that is less influenced by formal education in the assessment of visuospatial abilities.
As suggested by Dansilio and Charamelo Reference Dansilio and Charamelo(2005), the loss of perspective is the most frequent error committed by illiterates when copying figures, which includes difficulties in arranging oblique parallels and some steps in the tracing of 3D drawings. This may explain discrepant results when literate and illiterate subjects' performance in drawing tests is compared (Hong et al., Reference Hong2011), because these tasks contain these features. On the other hand, in samples for which subjects have relatively high educational levels, it becomes difficult to identify impaired patients across selective dementia level groups using traditional visuoconstructional tasks (Cecato et al., Reference Cecato, Fiorese, Montiel, Bartholomeu and Martinelli2012). In this regard, it is assumed that different educational groups need approaches specific for assessing this construct.
Our study found significant differences between the diagnostic powers of the Stick Design Test and the Clock Drawing Test. When we consider the instruments alone in the context of our sample, the Clock Drawing Test showed better criterion-related validity, as measured by the area under the curve in a ROC analysis, although the differences between the curves were discrete. This might be an artifact of our study's sample, which – despite predominantly including subjects with little formal education – did not show as high a proportion of illiterates as did the original study (Baiyewu et al., Reference Baiyewu2005). However, when the two instruments were used along with the Mini-Mental State Examination, no significant differences were found with respect to diagnostic accuracy (i.e. both instruments correctly classified 84% of the participants). In this sense, when used as a screening method conjointly with the MMSE, the Stick Design Test is as accurate as the Clock Drawing Test in determining a participant's classification. Because the Stick Design Test is usually better accepted for patients in the context of a comprehensive evaluation, it is as effective as the Clock Drawing Test for diagnostic purposes and offers the advantage of eliciting less negative emotional reaction and lowering anxiety levels during a patient's assessment.
Despite the usual high sensitivity and specificity (i.e. approximately 85% each) found for the Clock Drawing Test in first-world elderly patients, where educational background usually exceeds eleven years (Shulman, Reference Shulman2000), Lourenço et al. (Reference Lourenço, Ribeiro-Filho, Moreira, Paradela and Miranda2008) found that the Clock Drawing Test, despite showing acceptable reliability, does not appear to be valid for detecting cognitive impairment in low-educated older adults. Those authors argued that educational level did not appear to interfere with its stability but instead strongly limited its capacity to correctly classify clinical (viz., AD) and control groups. Similar results have been reported by other Brazilian studies using drawing-based visuospatial ability measures, with systematically low or moderate power for the detection of cognitive impairment and dementia (Bertolucci et al., Reference Bertolucci, Okamoto, Brucki, Siviero, Neto and Ramos2001; Porto et al., Reference Porto, Fichman, Caramelli, Bahia and Nitrini2003).
Nevertheless, when adopted as a single measure of cognitive functioning, the Clock Drawing Test was slightly superior to the Stick Design Test in the classification of AD. A previous report from our group (de Paula et al., in press B) investigated which cognitive abilities underlie the Clock Drawing Test in older adults with low formal education. That study suggested that executive functions, semantic processing, global cognitive status and visuospatial abilities were significant predictors of test performance. Executive functions and global cognitive status, especially, were the factors most related to performance in the Clock Drawing Test. In this sense, the Stick Design Test may be a more specific measure of visuospatial abilities, whereas the Clock Drawing Test might be better suited as a general measure of cognitive status or a global (non-specific) measure of executive functioning in this population. This hypothesis is corroborated by the changes in the Wald χ2 values of Models 2 and 3 in the logistic regression analysis. The Clock Drawing Test version adopted for our studies uses a pre-drawn circle, reducing its visuospatial demands. The shared variance between the two measures (30%) suggests that most of the specific variance may be due to other cognitive factors.
Additional ways that our study contributes to the research are evident in a comparison to the original paper on the Stick Design Test (viz., Baiyewu et al., Reference Baiyewu2005). Our study focused specifically on the test's psychometric properties and feasibility for diagnosing AD patients. We found criterion-related validity for the Stick Design Test similar to that found by Baiyewu and colleagues, reflecting stable properties across different populations. However, we also analyzed both construct validity (concerning the cognitive processes assessed by the test) and ecological validity (i.e. the relationship between the test and everyday functioning). As presented above, our results reinforce the notion that the Stick Design Test is a good measure of visuospatial processing and offers good associations with the daily activities of life. Future studies should investigate the reliability of the Stick Design Test both in terms of internal consistency and stability over time.
Some important limitations of the present study should be addressed. Our sample size is relatively small and may not be representative of older Brazilian adults, so our results might not be fully generalizable to other populations. The version of the Clock Drawing Test adopted in this research, although useful for diagnostic purposes (Shulman, Reference Shulman2000), may not be a well-suited measure of visuospatial abilities due to the involvement of other cognitive processes. Moreover, the Stick Design Test has a strong ceiling effect (viz., the cutoff is 1 point below the total score), which may have reduced the test variance in our sample and influenced the statistical procedures adopted in our study. Last, the Stick Design Test might be an interesting tool for the assessment of impairment in visuospatial abilities and patients’ diagnoses when used alongside other neuropsychological tests. However, its standalone accuracy for diagnosis is limited.
Our results indicate that the Stick Design Test has adequate psychometric properties and serves as a potential tool for the assessment of visuospatial abilities in older adults with low formal education. The data obtained suggest that the test is sensitive to cognitive impairment and that this sensitivity is maintained across different cultures. Furthermore, its usage in conjunction with other screening tests such as the MMSE should enhance the assessment of cognition in elderly people. Future studies should replicate the current findings and aim to test the measure's applicability to the diagnosis of other clinical conditions and the assessment of visuospatial abilities.
Conflicts of interest
None.
Description of authors’ roles
J. J. de Paula designed the study, analyzed the data, and wrote the paper. Costa MV, Bocardi MB, and Cortezzi M helped with data collection, manuscript writing, English review, and approved the final version of the manuscript. Moraes EN supervised the geriatric assessment of the participants and approved the final version of the manuscript. Malloy-Diniz LF helped to design the study, reviewed the statistical analysis and the manuscript, approving its final version.
Acknowledgments
Grant INCT-MM (FAPEMIG: CBB-APQ-00075-09/CNPq 573646/2008–2) and FAPEMIG (APQ-01972/12-10, APQ-02755-10).
