Introduction
Dementia is a global public health priority with an estimated 300 million people currently affected (Prince et al., Reference Prince, Guerchet and Prina2013b). Current epidemiological data and projections suggest that the majority of affected individuals reside in low- and middle-income countries (LMICs) and that this proportion will rise to over 71% of the world total by 2050 (Prince et al., Reference Prince, Bryce and Ferri2013a). Identification of dementia allows early intervention and can improve outcomes (Prince et al., Reference Prince, Guerchet and Prina2013b). In LMICs, scarcity of healthcare and human resources has resulted in a lack of trained clinicians able to correctly identify dementia (Saxena et al., Reference Saxena, Thornicroft, Knapp and Whiteford2007). The World Health Organization's (WHO) mental health strategy addresses this resource gap through training non-specialist clinicians to carry out tasks normally completed by specialists in better resourced areas (Dua et al., Reference Dua2011). This approach requires well validated and accurate screening tools to assist non-specialists.
Worldwide, over 700 million people are illiterate (UNESCO, 2015). Illiteracy and low rates of formal education are more common in LMICs, particularly in older adults (UNESCO, 2015), but can also be found in some HICs due to similar historical difficulties with resources and conflicts affecting the older population as well as recent population shifts due to globalization and migration.
Illiteracy and low educational level are associated with increased risk of dementia and have been hypothesized to account for some of the observed differences in reported prevalence (Caamano-Isorna et al., Reference Caamano-Isorna, Corral, Montes-Martinez and Takkouche2006; Meng and D'Arcy, Reference Meng and D'Arcy2012). Conversely, rates of cognitive impairment and dementia may be overestimated in LMIC populations due to poor performance in cognitive screening tests which require some degree of literacy (Paddick et al., Reference Paddick2015). Illiterate and low-educated adults can perform poorly on many conventional cognitive tests, without evidence of cognitive or functional decline on clinical history and examination. Most cognitive screening tests for dementia in frequent use, even in LMICs, were developed and validated in HIC settings with higher levels of formal education than those found amongst older adults in many LMICs. These differences in performance are not simply restricted to those aspects of cognitive screening traditionally taught in schools such as reading, writing or calculation, but include visuospatial and language ability (Rosselli et al., Reference Rosselli, Ardila and Rosas1990; Ostrosky-Solis et al., Reference Ostrosky-Solis, Ardila, Rosselli, Lopez-Arango and Uriel-Mendoza1998). Performance differences have been attributed to simple lack of confidence or familiarity with test situations, but evidence exists of differences in cortical organization and development in non-demented individuals without formal schooling (Castro-Caldas et al., Reference Castro-Caldas, Petersson, Reis, Stone-Elander and Ingvar1998; Ardila et al., Reference Ardila2010). These differences have been noted between individuals with just one year of formal schooling in comparison with those without formal schooling (Ostrosky-Solis et al., Reference Ostrosky-Solis, Ardila, Rosselli, Lopez-Arango and Uriel-Mendoza1998).
Difficulties in screening and diagnosis of dementia in low-literacy settings are widely recognized. Epidemiological studies of dementia from sub-Saharan Africa (SSA) report illiteracy rates of 47–97% in older adults (Guerchet et al., Reference Guerchet2009; Yusuf et al., Reference Yusuf, Baiyewu, Sheikh and Shehu2011). Similarly, the majority of participants in an international dementia prevalence study from LMICs including China, India, and South America reported either no formal education or incomplete primary education (Sosa et al., Reference Sosa2009). In rural Tanzania, our research group has reported that only 7% of females and 18% of males aged 70 years and over had 4 years of education or greater (Paddick et al., Reference Paddick2014).
The effect of lower levels of formal education on cognitive screening performance has been demonstrated in HIC settings, but the overall level of education in these studies is generally higher than that found in older populations in LMICs (Crum et al., Reference Crum, Anthony, Bassett and Folstein1993). More recently, a cross-cultural protocol for dementia diagnosis has been developed for use in low-literacy LMIC settings, but this is lengthy and not suitable for routine use by non-specialists (Prince et al., Reference Prince, Acosta, Chiu, Scazufca and Varghese2003). Existing dementia prevalence studies in SSA hospital settings often utilize tools designed and validated in HICs such as the Mini-Mental State Examination (MMSE), suggesting that these tools are common in routine practice in LMICs (Uwakwe, Reference Uwakwe2000; Touré et al., Reference Touré2008; Napon et al., Reference Napon, Traore, Niakara, Ouango, Ouango and Kabore2009; Ramlall et al., Reference Ramlall, Chipps, Pillay and Bhigjee2013).
We aimed to systematically review existing literature on brief, non-specialist cognitive screening tools with criterion validity against a clinical dementia diagnosis in older adults with very low or no formal education or literacy ability. We wished to determine whether existing brief cognitive screening tools can be used in populations of older adults with low or no formal education with similar levels of accuracy to those reported in HIC settings.
Methods
This systematic review followed published guidelines on reporting of systematic reviews of diagnostic accuracy studies, including PRISMA (Moher et al., Reference Moher, Liberati, Tetzlaff and Altman2009).
Search strategy
We searched four major databases Medline (1946–), Embase (1981–), Psychinfo (1806–), and Cinahl (1981–) for studies published prior to December 01, 2014. Included search terms were dementia (subject headings: dementia, cognitive impairment/defect/disorders, Alzheimer's disease (AD), memory disorders), questionnaires (subject headings: questionnaires, brief cognitive screening; keyword – assess*, screen*), low literacy (subject headings: educational status, educational attainment level, educational background, illiteracy; keyword – illitera* and low literacy), and validity (subject headings: validity, test validity; keyword – valid).
Reference lists of review articles and included studies were hand searched, alongside online databases of LMIC based literature including SCIELO (Central and South America) and African Journals online in order to identify non-indexed peer reviewed articles and ensure a global perspective. Results were not restricted to English language, but only abstracts available in English, Spanish, French, Italian, or Portuguese were reviewed due to resource availability.
Eligibility/inclusion criteria
Abstracts and titles were initially screened by S-MP and WKG separately, with full text articles requested for further assessment where it was not clear whether or not studies met the inclusion criteria. Disagreements were resolved through discussion.
Diagnostic accuracy studies of any cognitive measure, assessment, instrument, or test for face-to-face cognitive screening of dementia were included if they measured criterion validity against a gold standard diagnosis of dementia in individuals aged 45 years and over in a low-literacy setting, and were suitable for non-specialists to use in routine care. Self-assessment, telephone, and informant measures were excluded. We defined gold standard diagnoses as those using standard criteria including ICD or DSM, or clinical diagnoses made by a specialist clinician such as a neurologist, psychiatrist, or geriatrician. Studies simply reporting correlation between a cognitive screening tool and a more established screening test were excluded. The age cut-off of 45 years was selected based on previous epidemiological literature from LMICs. Low-literacy was defined as four years of formal education or less. Illiteracy was defined by self-report or less than one year of formal education. Suitability for use by non-specialists was determined by test duration (20 minutes or less) and ability to be performed without specialist training. In determining whether included screening tools met this criterion, information was sought from the index article or reference lists. Where unavailable, likely duration and eligibility were determined based on clinical experience of the researchers. Detailed neuropsychological batteries and tests for dementia subtype were excluded, as were studies from specialist settings, such as post-head injury or stroke. Illiteracy and educational background based on self-report were acceptable, as existing large epidemiological studies from LMIC settings have also used self-report and standardized literacy assessments are not routinely available (Sosa et al., Reference Sosa2009).
Since research interest in this area is relatively recent, we wished to accurately reflect the current knowledge base and data quality. Both case–control (stage one diagnostic studies) and cross-sectional studies were therefore eligible for review.
Where studies described a low-education population, but data on individuals with 4 years of education or less were not presented, authors were contacted directly and requested to provide this data. In the absence of this additional data, only those studies where a minimum of 25% of participants were illiterate or had no formal schooling were included. Studies including combined cognitive screening and informant questionnaires were included if cognitive screening accuracy could be separately extracted from presented data. Although cultural differences may impact on test performance, consideration of such differences was beyond the scope of this review. Therefore, studies from any geographical location were included if eligibility criteria were met.
Data extraction
Data were extracted by one author (S-MP) and a selection checked by another author (WKG). Demographic data extracted included geographical location and clinical setting, screening test used, age and gender balance of cohort, dementia prevalence, overall median education, and proportion of individuals with less than one year of education or illiteracy. Diagnostic accuracy data including sensitivity, specificity, AUROC, and cut-off used were also extracted for each screening tool. Cross-sectional community (lower prevalence), primary/secondary care (higher prevalence), and case–control studies were considered separately. Cognitive domains assessed in each screening test were determined, and test subsequently categorized into single domain or multiple domain tests.
For meta-analysis, true positive, true negative, false positive, and false negative screening results were extracted or calculated where possible from published data or original data supplied by authors. Where possible, these data were extracted separately for the illiterate/no formal education subgroup.
Quality assessment
Study quality was evaluated using the Quality Assessment of Diagnostic Accuracy Studies version 2 (QUADAS-2) tool (Whiting et al., Reference Whiting2011). QUADAS-2 consists of 4 domains and 14 items assessing the risk of bias, applicability, and quality. Each item is scored as “yes,” “no,” or “unclear.” Two reviewers (SMP and WKG) independently evaluated each of the included studies using agreed criteria based on the QUADAS-2 core questions and applicability to the study question. Core criteria included: (1) method of patient selection (consecutive or random selection, study design and inclusion/exclusion criteria); (2) index test used and blinding to results of the reference standard assessment; (3) use of an appropriate reference standard independent of the screening test result; and (4) flow and timing (appropriate interval between index test and reference standard, all participants assessed with the same reference standard, all participants included in the analysis).
Studies meeting all four core criteria were rated good quality. Fair quality studies failed to meet one core criterion but were otherwise robust. Case–control studies were assessed separately and acknowledged not to meet criterion 1.
Meta-analysis
A bivariate diagnostic random-effects meta-analysis was conducted using the mada open source package for R software (Doebler and Holling, Reference Doebler and Holling2012). Bivariate meta-analysis is recommended in studies of diagnostic tests due to threshold effects and underlying between studies variability. A random-effects model was used to adjust for heterogeneity. Only cross-sectional studies with at least three individuals with dementia in a low-literacy population were included in meta-analyses. Studies of insufficient quality on QUADAS-2 assessment were not included in meta-analysis. This included those with a case–control design and where inadequate blinding in screening test administration or gold standard diagnosis resulted in significant bias.
Combined sensitivity, false positive rate, and diagnostic odds ratios (DOR) were used to summarize screening test accuracy. Forest plots were used to summarize the variability in the thresholds used. Between study heterogeneity was assessed using both the I 2 statistic and Cochran's Q with p <0.10 considered statistically significant. A hierarchical summary receiver operating characteristic (HSROC) curve was constructed to present summary estimates of sensitivity and specificity alongside 95% CI and prediction region in each subgroup. Area under curve (AUC) and partial AUC statistics corrected for observed data and false positive rates were also calculated.
Accuracy was assessed using this method for all screening tools in low-literacy groups in both community or low prevalence and clinic-based or higher prevalence settings. There is evidence that both sensitivity and specificity can vary with disease prevalence alongside positive predictive value, possibly due to the differing spectrum of patients assessed in each setting (Leeflang et al., Reference Leeflang, Rutjes, Reitsma, Hooft and Bossuyt2013). Studies in these different settings were therefore analyzed separately. All screening tools were initially analyzed together, in order to assess the accuracy of cognitive screening methods. This approach has been used in previous meta-analyses of screening tools with a focus on HIC settings (Mitchell and Malladi, Reference Mitchell and Malladi2010). Where possible, accuracy was evaluated for illiterate/no education groups only. It was intended to analyze performance of specific tests where sufficient data were available.
Results
Included studies
The study selection process is shown in Figure 1. A number of cognitive assessment tools developed for use in low-educated and illiterate populations did not meet our inclusion criteria. These included the Community Screening Instrument for Dementia (CSI-D) used in dementia prevalence studies across LMIC countries. The CSI-D requires an informant, has a duration of over 30 minutes and utilizes a computer algorithm for risk stratification therefore cannot be used by non-specialists (Prince et al., Reference Prince, Acosta, Chiu, Scazufca and Varghese2003). A brief CSI-D has been modeled and internally validated but not evaluated in clinical practice (Prince et al., Reference Prince2011). Similarly the Literacy Independent Cognitive Assessment (LICA) and shorter version (S-LICA) were felt to be too complex and/or lengthy for routine use (Choi et al., Reference Choi2011). The Rowland Universal Dementia Scale (RUDAS) was designed for cross-cultural assessment (Basic et al., Reference Basic2009), but we were unable to identify a low-literacy validation study of the RUDAS for inclusion in this review. A total of 45 studies meeting our study criteria were identified. A summary of included studies is presented in Tables 1–3.
Prisma flowchart of study selection.
Summary of screening tools validated in low-education settings – population and community (low prevalence) studies
Note: CIND: cognitive impairment no dementia, MCI: mild cognitive impairment.
Summary of screening tools validated in low-education settings – clinic (higher prevalence) studies
Note: CI: illiterate control, OPD: out patient department.
Summary of screening tools validated in low-education settings – case–control studies
Note: AD: Alzheimer's disease, NC: normal control C-patient control, TR: total recall, DR: delayed recall, CI: illiterate control, DI: illiterate dementia, CL: control literate, DL: dementia literate, OPD: out-patient department.
A total of 27 different screening tests were evaluated, of which four were cultural adaptations of the MMSE. A total of 14 tests (12 multi-domain and 2 single domain) were specifically developed for use in low-literacy settings. A summary of the cognitive domains assessed is presented in Table 4. A majority of studies are related to multi-domain, rather than single domain, cognitive tests. Of those tests evaluated, only the MMSE, cognitive abilities screening instrument (CASI), Eurotest, Fototest, and KICA were validated in more than one good or fair quality study. The low-literacy validations identified for each study are also included in Table 4.
Characteristics of cognitive tests included in the review
Note: F + C: free and cued (recall), Del: delayed (recall), TMT: trail making test, CDT: clock drawing test, S: stage (command), HDS: Hasegawa Dementia Scale, CMT: Chula Mental Test, VSID-P: Vellore Screening Instrument for Dementia-Patient, S-LICA: Literacy Independent Cognitive Assessment (short form), KICA: Kimberley Indigenous Cognitive Assessment, CDT: clock drawing test (Sutherland scoring), CDT: clock drawing test (Shulman scoring), 7MS: seven minute screen, PCL: Prueba Cognitiva de Leganes, SPMSQ: short portable mental status questionnaire, MOCA: Montreal Cognitive Assessment, COST: cognitive state test, CASI-S: cognitive abilities screening instrument (short version), SDT: stick design test, Mini-Cog: Mini Cognitive Screen, MMSE: Mini-Mental Status Examination, FOME: Fuld Object Memory Test, B-IMC: Blessed Roth Memory Concentration test, CASI: cognitive abilities screening instrument memory alteration test (M@T)*: domain tested, ?: unclear. NB Filho Mini-Cog article only includes registration and recall items, not the clock test score in scoring.
The geographical distribution of good or fair quality studies is shown in Figure 2. Although the majority originated in LMICs, a significant number did not. Twelve studies were conducted in low-literacy populations in Spain, and two in minority Aboriginal populations in Australia. Similarly, three of the most-validated tests, such as the Eurotest, Fototest, and KICA, were developed in low-literacy settings in HICs.
Map of good and fair quality low-literacy screening tool validation studies.
Population and community studies
Fourteen cross-sectional community or population-based studies were identified and are summarized in Table 1. Geographical location of these studies included China (five studies), India (two studies), Spain (two studies), Australia (two studies), Brazil, Turkey, and Nigeria. A total of 12 screening tools were validated in this setting. Sample size ranged from 455 to 8,411 in population-based studies and 101–490 in smaller community based studies. One small study from India included both community and outpatient clinic cohorts, reported separately, and another in Australia used community health contacts to oversample for cognitive impairment. Reported rates of illiteracy or no formal education ranged from 25% to 91%. Dementia prevalence was generally low, ranging from 3% to 21% with the exception of one study reporting a prevalence of 34% in a population over-sampled for cognitive impairment. All studies utilized DSM-IV, DSM-III, or DSM-III-TR criteria by consensus or specialist clinician, with the exception of one study utilizing an algorithm-based DSM-IV diagnosis. Two studies also applied research criteria for dementia subtypes.
Prevalence/clinic settings
Thirteen screening tests were evaluated over 16 studies conducted in clinic settings. The majority of studies took place in Spain (eight studies) with four taking place in Brazil, two in India, and one each in Thailand and China. These studies are summarized in Table 2. Prevalence of dementia ranged from 10.4% in primary care (Baos Sánchez et al., Reference Baos Sánchez, Ruiz Muñoz, Bernal López, Ballester Rodríguez and Ángora Mazuecos2007), 17.8% in nursing home residents (Jitapunkul et al., Reference Jitapunkul, Lailert, Worakul, Srikiatkhachorn and Ebrahim1996), and 19–33% in more specialist clinics. Illiteracy rates in these studies varied from 5.3% (Verghese et al., Reference Verghese2012) to 65% (Martínez de la Iglesia et al., Reference Martínez de la Iglesia, DueñasHerrerob, Carmen Onís Vilchesa, Aguado Tabernéa, Albert Colomerc and Luque Luquec2001) with the majority quoting rates from 25% to 40%. However those studies reporting lower illiteracy levels quoted rates of none or minimal schooling of 41.4% to 49% (Carnero Pardo et al., Reference Carnero Pardo2007; Saez-Zea et al., Reference Saez-Zea, Carnero-Pardo and Gurpegui2008) amongst participants. Included studies where less than a quarter of the sample were illiterate or had no formal studies are those where supplementary information on a low-literacy subgroup was obtained from the study authors. Most studies used DSM-III, DSM-III-TR, or DSM-IV dementia criteria, with two studies utilizing dementia subtype diagnoses.
Case–control studies
Case–control designs are recommended for use only in the first stage of assessment of a diagnostic test, and the majority of these studies related to initial validations of newly developed tests, or initial studies of well-established tests in a low-literacy setting, see Table 3. These studies were nevertheless relevant to our study question with level of illiteracy ranging from 20% to 100%. Identified case–control studies validated 16 different cognitive tests (including two cultural adaptations of the MMSE) over 15 studies. The majority of studies took place in Brazil. Other studies were located in Spain, Greece, China, Korea, Hong Kong, Senegal, and Turkey.
Sample size in these studies ranged from 60 to 968 with most studies including around 200 participants. Dementia prevalence ranged from 16.75 to 63%, with most studies aiming to include 50% of participants with dementia and 50% control participants. Two studies employed consecutive sampling, but subsequently included only normal controls and individuals with mild/moderate dementia in the analysis. All but two took place in clinic/high prevalence settings, and the majority utilized NINCDS-ADRA criteria for AD alongside DSM-III or DSM-IV dementia criteria with some studies only including patients with AD dementia. Sensitivity and specificity for each included screening test are summarized in Table 3.
Quality assessment
A total of 24 of the 45 included studies were of good or fair quality, see Table 5. Of those rated as being of poorer quality, a significant number (15) utilized a case–control design. Additional spectrum bias was evident amongst case–control studies with most including only individuals with mild dementia alongside healthy controls. Six cross-sectional studies were found to have inadequate blinding procedures in administration of the index screening test, or gold standard reference diagnosis or screening tests applied to persons with established diagnoses. Most community studies used population samples from cross-sectional or longitudinal studies. Others utilized cluster sampling or a stratified sample from two stage community-based studies. Two studies oversampled for cognitive impairment resulting in potential spectrum bias. The majority of clinic-based studies sampled consecutive or randomized referrals (eight studies) (Almeida, Reference Almeida1998; Xu et al., Reference Xu, Meyer, Huang, Du, Chowdhury and Quach2003; Carnero-Pardo et al., Reference Carnero-Pardo2006; Baos Sánchez et al., Reference Baos Sánchez, Ruiz Muñoz, Bernal López, Ballester Rodríguez and Ángora Mazuecos2007; Saez-Zea et al., Reference Saez-Zea, Carnero-Pardo and Gurpegui2008; Carnero-Pardo et al., Reference Carnero-Pardo2011b; Carnero-Pardo et al., Reference Carnero-Pardo, Sáez-Zea, De la Vega Cotarelo and Gurpegui2012). Others utilized convenience sampling (Lourenço et al., Reference Lourenço, Ribeiro-Filho, Moreira, Paradela and Miranda2008; Ribeiro Filho and Lourenço, Reference Ribeiro Filho and Lourenço2009), oversampled for cognitive impairment (Martínez de la Iglesia et al., Reference Martínez de la Iglesia, DueñasHerrerob, Carmen Onís Vilchesa, Aguado Tabernéa, Albert Colomerc and Luque Luquec2001) or supplemented additional unimpaired controls (Carnero Pardo et al., Reference Carnero Pardo2007; Verghese et al., Reference Verghese2012). Generalizability of both community and clinic-based studies was limited by exclusion of individuals with “severe” (De Yebenes et al., Reference De Yebenes, Otero, Zunzunegui, Rodriguez-Laso, Sanchez-Sanchez and Del Ser2003; Del Ser et al., Reference Del Ser, Sanchez-Sanchez, Garcia de Yebenes, Otero and Munoz2006; Lourenço and Veras, Reference Lourenço and Veras2006; Stanley et al., Reference Stanley2009) or “significant” sensory impairment (Stanley et al., Reference Stanley2009) and neurological or psychiatric disorders (Stanley et al., Reference Stanley2009; Lu et al., Reference Lu2011; Verghese et al., Reference Verghese2012).
QUADAS-2 assessment of study quality
Note: CC: case control design, +: substantial risk of bias present, −: no substantial risk of bias, ?: unclear from text, DX: diagnosis.
Three studies provided insufficient information about study design most commonly in the domain of patient flow and timing; these were rated “fair” if no information in the manuscript led reviewers to suspect serious methodological flaws.
Meta-analyses
Studies were excluded from meta-analyses due to insufficient quality, as assessed by the QUADAS-2 tool; all case–control studies were excluded for this reason. A full description of the reasons for exclusion from meta-analyses is listed in Table 5. Four separate meta-analyses were conducted.
Community or low prevalence studies
Population and community studies included in the meta-analysis included twelve cohorts from eleven primary studies with a total of 15,939 participants of whom 1,376 were diagnosed with dementia (8.6%). Screening tests included variations of the MMSE (four cohorts), and single studies of the HDS, KICA, MOCA, CASI, 7MS, B-IMC, PCL, and VSID-P. All are multi-domain cognitive tests. One community study of a single-domain test (Stick design, Nigeria) was excluded from the meta-analysis after failing to meet the QUADAS-2 quality standard as the screening test formed part of the diagnostic assessment. Two other studies where participants were oversampled for cognitive impairment were considered more appropriately analyzed with high prevalence studies.
There was non-significant heterogeneity between studies when measured by Cochran's Q: (17.704 (11 df, p = 0.089)), but moderate heterogeneity when measured with Higgins' I2, a measure of the percentage of total variability due to between studies variability (37.9%).
Combined data in a bivariate random-effects model resulted in a sensitivity estimate of 0.869 (95% CI = 0.791–0.921) and specificity of 0.886 (0.823–0.923). The combined DOR for the model was 50.529 and a HSROC was plotted with an AUC of 0.937and partial AUC (restricted to observed FPRs and normalized) of 0.901, see Figure 3. The most accurate screening tests were 7MS, PCL, and KICA-Cog in Australia. The least accurate were the Hindi MMSE and VSID-P, although the VSID is designed to be used alongside an informant history.
Results of meta-analysis. Forest plots of diagnostic odds ratio (DOR) and hierarchical summary ROC (HRSROC) curves.
Higher prevalence or clinic-based studies
A total of 13 clinic-based or higher prevalence primary studies including 17 validation cohorts were suitable for inclusion in meta-analysis. Other studies were excluded where low-literacy data could not be extracted or supplied by study authors. Higher prevalence studies included 3,487 individuals of whom 899 were diagnosed with dementia (25.8%). Tests evaluated were the Spanish MMSE (MEC) (two studies) Chinese MMSE, Brazilian MMSE (two studies) the VSID-P, KICA-Cog (two studies), CASI (2 cohorts from one study), Fototest (two studies), Eurotest, PMIS, MAT, CMT, and SPMSQ.
There was little heterogeneity between studies (Cochran's Q: 16.235; p = 0.437, I 2 1.449%).
Pooled sensitivity was estimated at sensitivity of 0.845 (0.817–0.869) and specificity of 0.847 (0.805–0.882). The combined DOR was 35.681 and the HSROC resulted in an AUC of 0.881 (partial AUC 0.843). The least accurate test was the Brazilian MMSE, and the most accurate were the CMT and PMIS in Thailand and India albeit with wide confidence intervals. All are multi-domain tests with the exception of the PMIS, which is a picture-based recall test.
Studies of illiterate individuals
Studies including only illiterate individuals or where data for illiterate individuals could be extracted included 2,098 individuals (392 with dementia) across seven primary studies. These included three clinic-based studies, three community studies, and one combining memory clinic and population study participants. Screening tests were all multi-domain and included Brazilian and Chinese variants of the MMSE, the KICA (Aboriginal population in Australia), the SPMSQ (Spain), the CASI (China),and the BR-IMC (China). One large study of the MOCA was excluded as diagnostic accuracy data for illiterate individuals related to cognitive impairment rather than dementia. Similarly, studies of insufficient quality on the QUADAS-2 were also excluded, a large proportion of those identified.
There was little heterogeneity between studies (Cochran's Q: 6.734; p = 0.346, I 2 −10.9%). Combined sensitivity estimate was 0.818 (0.769–0.859) and specificity was 0.801 (0.745–0.848). The combined diagnostic odds ratio was 18.753 and the HSROC resulted in an AUC of 0.869 (partial AUC 0.812). The least accurate tests were the Brazilian MMSE and B-IMC in China with similar performance for the MMSE in another Brazilian study and for the Chinese MMSE. The most accurate tests were the KICA-Cog and SPMSQ.
Validation studies of the MMSE
A total of eight primary studies (nine separate cohorts) validated variants of the MMSE in a low-education setting. These included MMSE-Brazil, Hindi MMSE, MMSE-Turkish, Chinese MMSE, and MMSE Spanish version (MEC). These are analyzed together because despite cultural and educational variations, all retain the structure of the original and cover similar cognitive domains. A total of 3,559 individuals were included of which 397 had dementia (11.2%). Cut off points for dementia screening varied widely and ranged from 14 to 20 for those who were illiterate or without formal education and 17 to 23 in those with at least one year of education. Most studies utilized different cut off points for those with and without formal education.
There was no significant heterogeneity between studies (Cochran's Q: 9.894; p = 0.273, I 2 19.147%). Combined sensitivity estimate was 0.828 (0.789–0.862), and specificity was 0.817 (0.717–0.887). The combined DOR was 22.981 and AUC was 0.853 (partial AUC 0.819). The MMSE demonstrated the most accuracy when used in low-literacy cohorts in Spain and Brazil, but performed poorly in some of the lowest literacy settings in LMICs.
Discussion
The overall diagnostic accuracy of cognitive screening tests for dementia in low-literacy settings is similar to that reported in reviews of diagnostic accuracy in HICs. A meta-analysis of brief cognitive screening tools for identification of dementia in HICs reported a pooled sensitivity and specificity of 72% and 88.2%, respectively, in community settings (dementia prevalence 16%), 88.9% and 88.4%, respectively, in secondary/specialist care (dementia prevalence 28%; Mitchell and Malladi, Reference Mitchell and Malladi2010). In comparison, we calculated a sensitivity of 86.9% and 84.5% and specificity of 88.6% and 84.7% in community and clinical studies, respectively. Although this degree of accuracy might be acceptable in a HIC setting, with higher levels of access to specialist clinicians, in LMIC settings higher sensitivity and specificity are necessary to improve dementia case-finding and screening, especially in primary care or community settings.
In lower prevalence or community studies, the most accurate tests were the 7MS, PCL, and KICA-Cog conducted in Spain and Australia, and in higher prevalence or clinical studies, the most accurate were the CMT and PMIS. In illiterate individuals, the best-performing tests were the KICA-Cog and SPMSQ. In all three settings, the worst-performing tests were adaptations of the MMSE, alongside the VSID-P in the community and the B-IMC in illiterate individuals. However, the VSID was designed to be utilized alongside an informant interview, and we have analyzed only the performance of the cognitive screening tool. No studies of hospital inpatients were identified that met our study criteria.
Overall, we found the diagnostic accuracy of cognitive screening tools to be best in community or population settings compared to clinic settings. This may be related to the spectrum of participants seen. Some community studies excluded individuals with significant sensory impairment, which may have artificially increased the diagnostic accuracy of the screening test. Clinic-based studies taking place in geriatric medicine clinics might also be expected to have a higher proportion of individuals with general frailty or medical conditions adversely affecting performance on a cognitive test. Performance of screening tools included within these studies is therefore representative of routine clinical practice conditions.
Diagnostic accuracy was reasonable (DOR 18, AUC 0.869) in studies only including illiterate individuals, although, in general, screening test performance was lowest in this group. A few good or fair quality studies were identified in this group, with a significant proportion utilizing a case–control design.
Heterogeneity between studies was generally low to moderate, and highest in community settings. This may be partially due to the fact that we selected studies relevant to a specific low-literacy population only, but is interesting in that these studies took place in many different geographical locations and where cultural background would be likely to differ.
Many briefer screening tools focused on recall only. Recall is affected early in AD dementia but less so in other dementia subtypes. There is evidence that vascular dementia is more common in LMIC settings, although overall knowledge of dementia subtypes in these regions is limited. Tools focusing on AD-type pathology only may therefore be of limited utility. Similarly, most good or fair quality studies did not report or assess for dementia subtypes, and many case–control studies focused only on individuals with AD.
Screening tool length was variable, with tests ranging from 1 to 20 minutes in length. As expected, longer tests generally assessed a wider range of cognitive domains. Evidence from HICs suggests that screening tools should be of 5 minutes duration or less in order to be practical for use in primary care (Brodaty et al., Reference Brodaty, Low, Gibson and Burns2006). Of the screening tools included in this study, only the COST, Eurotest, Fototest, and VSID included most cognitive domains within 4–7 minutes and might therefore have potential for use in primary care. Of these, only Eurotest and Fototest have been independently validated, but not in a substantial low-literacy cohort, independent validation studies of these tests in low-literacy LMIC settings have yet to take place. These tests were also not among the most accurate identified.
In resource-poor settings, cost is also a consideration. The MMSE is subject to license restrictions and its use now requires payment. Most but not all of the tests reviewed are in the public domain or free to use with permission, see Table 4.
A significant proportion of the identified studies utilized a case–control design, the majority of which took place in urban clinical settings, frequently teaching hospitals, and university clinics. These therefore had limited generalizability. Similarly, a large number of validation studies initially identified did not meet our study criteria as the overall level of education among participants was relatively high. The majority of these studies appeared to have been carried out in teaching hospital clinics and may not be representative of the older population in these settings.
The majority of screening tools identified in this study, whether developed for a low-literacy setting or validation of an established screening tool in a low-literacy setting, had been validated only once. Few had an identifiable independent validation with the exception of the MMSE variations, Eurotest, Fototest, KICA-Cog, and CASI. Only for the MMSE, were sufficient studies identified to separately analyze diagnostic performance.
Performance of the MMSE overall in this review resulted in a combined sensitivity estimate of 82.8% and specificity of 81.7%. In comparison, a recent meta-analysis including 149 studies of the MMSE reported similar findings with combined sensitivity of 81% and specificity of 89% (Tsoi et al., Reference Tsoi, Chan, Hirai, Wong and Kwok2015). When compared to other screening tests included in this review, the MMSE performed poorly as a screening test for dementia despite inclusion of culturally relevant adaptations. Better performance was seen in HIC settings or settings with relatively higher formal education and worst in those with the lowest literacy, reflecting the well-evidenced educational bias of the instrument.
Variations of the MMSE are therefore least likely to be accurate and should be avoided for dementia screening in low-literacy settings. Despite this, the MMSE remains the most widely used dementia screening test worldwide (Shulman et al., Reference Shulman2006), including in low-literacy LMIC settings.
The cut-off values utilized for the MMSE varied widely. Some studies utilized different cut-off values for participants with and without formal education, frequently based on normative data. Although a bivariate method of meta-analysis takes threshold effects into account (Reitsma et al., Reference Reitsma, Glas, Rutjes, Scholten, Bossuyt and Zwinderman2005), the approach of varying screening cut-off values in different populations has been criticized for reducing validity and this is also likely to reduce generalizability to other populations. This approach is also likely to lead to considerable ceiling effects in those with better cognitive performance (Kraemer et al., Reference Kraemer, Moritz and Yesavage1998).
Many screening tests included in this review were developed both for a low-literacy population and specific to the cultural and geographical setting in which the study was carried out. Examples include the KICA-Cog test developed for the Aboriginal minority group in Australia, VSID in India, and Chula Mental Test (CMT) in Thailand. Therefore, although the CMT and KICA-Cog were among the best-performing screening tests, performance cannot be generalized to, or recommended in other settings. Other tests are more easily transferable such as the picture recall tests of the PMIS and Fototest, but these currently lack independent validation in a low-literacy cohort in a LMIC.
Limitations
This review assessed cognitive screening in low-literacy populations regardless of geographical location. Cultural differences are likely to affect performance on cognitive testing, and this issue has not been the focus of this review.
It is possible that other articles meeting our search criteria exist, but were not identified using our search criteria if they included a low-literacy population and did not refer to this in the article abstract or search terms. Since we also reviewed reference lists of included studies, articles not identified are likely to be a few. We only reviewed articles in English, Spanish, French, Italian, or Portuguese, and therefore we may have missed studies published in other languages. Nevertheless, all articles identified through database search had an English abstract and were excluded through not meeting the study criteria. Although we attempted to contact study authors for additional data, for many older studies this was not possible and therefore studies had to be excluded from meta-analysis. This may have led to bias in the study.
Conclusion
This systematic review and meta-analysis of brief cognitive screening tools validated in illiterate and low-literacy settings demonstrated that cognitive screening tools have a diagnostic accuracy in low-literacy or illiterate populations similar to that reported in meta-analyses of screening tool performance in identification of dementia in higher literacy settings. This accuracy is likely to be inadequate to allow accurate case identification by non-specialists in LMIC settings. Published high quality studies are few, and few screening tools have been independently validated in a low-literacy setting. Differing cut-off values further reduce generalizability. At present, although existing screening tools demonstrate reasonably good accuracy in low-literacy settings, the available evidence is inadequate to allow recommendation of any one particular test. We recommend that further high-quality independent validations of tests designed for low-literate populations should take place in LMIC settings in order to establish diagnostic utility in these settings.
Conflict of interest
None.
Description of authors’ roles
S. Paddick and W. Gray designed the study and completed data extraction and assessment of study quality. S. Paddick wrote the first draft and conducted statistical analysis supervised by W. Gray. J. Richardson and J. McGuire conducted and advised on the literature searches. Professor Walker and C. Dotchin designed the study and supervised S. Paddick. All co-authors reviewed the final version of the study prior to publication.
