Introduction
Dementia affects a substantial proportion of older Aboriginal and Torres Strait Islander peoples living in remote communities (12% aged over 45 years; Smith et al., Reference Smith2008) and urban areas (13% aged over 60 years; Radford et al., Reference Radford2015), impacting greatly the lives of their family and community members (Smith et al., 2011). Emerging dementia studies with other indigenous populations, such as the First Nations people in Canada and the Melanau in Malaysia, are reporting similar results (Jacklin et al., 2013; Pu'un et al., Reference Pu'un, Othman and Drahman2014). Unfortunately, the ability to identify Aboriginal Australians with dementia and provide the necessary community care and support had until fairly recently been hindered by the lack of valid cognitive assessment tools.
The Kimberley Indigenous Cognitive Assessment (KICA) was developed to assess dementia in older rural and remote-living Aboriginal and Torres Strait Islander peoples (LoGiudice et al., Reference LoGiudice2006; Smith et al., Reference Smith2009; www.wacha.org.au/kica.html). The KICA comprises a number of components, including formal cognitive testing (KICA Cog) and an informant-based cognitive questionnaire (KICA Carer). The optimum cut-point for the KICA Cog is 33/34 out of 39 with a sensitivity of 93.3% and a specificity of 94.8% (Smith et al., Reference Smith2009). The KICA Cog is now widely utilized across rural and remote Australia. As yet, the KICA Carer, the informant cognitive section of the KICA, had not been validated with Aboriginal Australians.
Informant cognitive interviews contain questions related to person's cognition, function, and behavior to be answered by their caregiver, family member, or friend who knows the patient well enough to be aware of change over time. It has been reported that informant interviews can increase the effectiveness of cognitive assessments, and minimize the education, language, and cultural bias often present in cognitive tests (Prince et al., Reference Prince, Acosta, Chiu, Scazufca and Varghese2003; Jorm, Reference Jorm2004). A frequently used informant questionnaire is the Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) (Jorm and Jacomb, Reference Jorm and Jacomb1989), which when validated with a sample of an aged care assessment team (ACAT) clients had a sensitivity of 79% and specificity of 78% (Flicker et al., Reference Flicker, Logiudice, Carlin and Ames1997), and at an Australian memory clinic it had a sensitivity of 94% and a specificity of 47% (Knafelc et al., Reference Knafelc2003).
Informant-based cognitive questionnaires can be used on their own when cognitive testing cannot be conducted, but are usually used in conjunction with cognitive testing (Jorm, Reference Jorm2004). When informant questionnaires and cognitive testing are used in combination, there are two key methods for determining whether the overall result is positive (the person has cognitive impairment). The first method of reading the test results is in series. This is the case where the results of both tests (formal cognitive and informant) need to be positive before the overall result is positive (“and” rule). Reading of test results in parallel is possible when either test is positive for the overall result to be positive (“or” rule; Flicker et al., Reference Flicker, Logiudice, Carlin and Ames1997; Mackinnon and Mulligan, Reference Mackinnon and Mulligan1998). Mackinnon and Mulligan (Reference Mackinnon and Mulligan1998) also describe a third method for the combination of the test scores of the Mini-Mental State Examination (MMSE; Folstein et al., Reference Folstein, Folstein and McHugh1975) and IQCODE using a graphical method, the DemeGraph (http://www.biostats.com.au/Demegraph). When looking at two tests in combination, the correlation of the tests should also be reviewed. When two tests used in combination are not strongly correlated, the positive predictive value (PPV, probability that a positive test result will be correct; Loong, Reference Loong2003) is higher as they are collecting different information (Flicker et al., Reference Flicker, Logiudice, Carlin and Ames1997).
This paper describes the development and adaptation of the KICA Carer tool and the psychometric properties of the KICA Carer, both used individually and in combination with the KICA Cog. The practical implications for the utilization of these tests will also be discussed.
Methods
Development
The KICA Carer was developed in the Kimberley region of Western Australia as part of the study to develop complete KICA tool (LoGiudice et al., Reference LoGiudice2006; Smith et al., Reference Smith2007). Commonly used informant questionnaires were reviewed, including the IQCODE and the Cambridge Mental Disorders of the Elderly Examination (CAMDEX) (Roth et al., Reference Roth1986), and informant cognitive questions from the Community Screening Interview for Dementia (CSI'D') (Hall et al., Reference Hall, Gao, Emsley, Ogunniyi, Morgan and Hendrie2000) were reviewed and adapted as required. We consulted specialists, a range of local health professionals, and Aboriginal community members as detailed in Smith et al., (Reference Smith2007). This resulted in the development of an initial KICA Carer survey tool comprising 13 questions.
Inter-rater reliability
The reliability of the KICA Carer was determined while investigating the KICA Cog reliability, and this process has been described elsewhere (LoGiudice et al., Reference LoGiudice2006).
Adaptation process
The KICA Carer was translated into Walmajarri, and back translated to English by the interpreters of the Kimberley Language Resource Centre to determine the questions that could be interpreted successfully (LoGiudice et al., Reference LoGiudice2006, Smith et al., Reference Smith2007, Reference Smith2009). The KICA Carer inter-rater reliability and back-translation results were reviewed to determine the questions that would be included in the final version of the KICA Carer tool. This review process led to the deletion of five questions and adaptation of one question in the tool (see results for details), with the final KICA Carer tool containing eight of the original 13 questions.
Validity
The KICA Carer validation was completed from data from the later dementia prevalence study (Smith et al., Reference Smith2008). Informants completing KICA Carer were not necessarily caregivers (the majority of the participants did not require a caregiver), but were identified by participants as someone who knew them well. The KICA Carer results were compared with consensus diagnosis using Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) and International Classification of Diseases (ICD-10) criteria, based on a cognitive review, which was completed by a specialist psychiatrist or geriatrician blinded to KICA results.
The methodology used in the prevalence study to determine the properties of KICA Cog are described in Smith et al. (Reference Smith2009). There are two changes done in validity analyses in this paper, with the inclusion of participants with a diagnosis of cognitive impairment not dementia (CIND) and the exclusion of 14 participants who did not have an informant available to complete KICA Carer, yielding 349 participants. CIND participants were included in the sample to ensure that it was more representative of true population, thus limiting spectrum bias.
Statistical analysis
The Statistical Package for Social Sciences (SPSS) version 21 was used for data analysis. The Bland and Altman (Reference Bland and Altman1999) method for repeatability and Intra-Class Coefficient (ICC) were used to determine inter-rater reliability. Factor analysis was used to ensure spread of questions over a number of domains. CIND participants were paired with “No cognitive impairment” in the analyses so that the ability of KICA Carer to differentiate between participants with dementia versus all other participants could be analyzed. Sensitivity and specificity values and cut points were determined with a Receiver Operator Characteristic (ROC) curve, and calculation of the area under curve (AUC). Discriminant Function Analysis was used to determine the ability of each question to differentiate between people with and without dementia.
Sensitivity and specificity values for KICA Carer and KICA Cog in both series and parallel were determined at various cut points. We then calculated PPV and negative predictive values (NPV) individually and in combination, based on the dementia prevalence of 12.0%.
Ethics approval
Remote Aboriginal community, Aboriginal medical service (KAMSC), and ethics approval (UWA, WAAHIEC) were obtained prior to the commencement of the study. Participants provided informed written consent, and had assistance from interpreters whenever required.
Results
Inter-rater reliability
The inter-rater reliability component of the KICA Carer study was completed with 14 informants. The ICC for the 13 questions ranged from poor (ICC = 0.07) to excellent (ICC = 0.93; see Table 1). The total KICA Carer score showed no differences between raters (mean difference of 0.07, SD = 1.77). The total score for one of the 14 informants differed significantly between two raters. The Bland–Altman coefficient of repeatability was 3.54, limits of agreement were from −3.47 to 3.61, and range was from −3 to 4.
Inter-rater reliability results for KICA Carer
Notes: †Question is no longer in the present tool.
* Unable to calculate because of empty cells.
Question 5 – response identical for 14/14 participants.
Question 8 – response identical for 14/14 participants.
Question 10 – response identical for 13/14 participants.
The draft KICA Carer initially comprised 13 questions. Four questions were removed (forgets where keeps things, trouble in finding words, talks about long time ago, and changing story) because of poor acceptance by informants. Another question was deleted on advice of linguists, as it could not be translated accurately (Smith, Reference Smith2008). The question with poor inter-rater reliability, “Does s/he forget what happened yesterday,” was revised with the following additional explanatory sentence: “Would (name) remember today if s/he had been to the store yesterday?”
The final version of KICA Carer comprises eight questions, with each question being scored out of two (0 = no, 1 = sometimes, 2 = all the time), giving a total possible score of 16 (see Table 2).
KICA Carer final eight questions
Psychometric properties of KICA Carer individually
This study was completed with 349 informants, and with the final eight-question KICA Carer tool. The majority of the informants were women (76%), and were primarily adult children of the participants (36%). Formal caregivers made up 5% of those interviewed, and 57% of those interviewed lived with the participants.
The mean age of the 349 participants (who had an informant to complete KICA Carer) was 60.8 years, with an age range of 45–96 years, and 55% were women. Forty-two participants (12%) had dementia, 28 (8%) had CIND, and 279 (80%) had no cognitive impairment. CIND participants had the DSM-IV diagnoses of cognitive disorder not otherwise specified, amnestic disorder due to general medical condition, mental disorder due to general medical condition, mild mental retardation, and alcohol-related disorder NOS and schizophrenia residual type. Participants came from 20 different Aboriginal language groups.
The internal consistency of KICA Carer was 0.85. Discriminant function analysis determined that three of the eight KICA Carer questions best classified dementia versus no cognitive impairment. The following three questions correctly classified 91.7% of all dementia diagnoses: “Forgets what happened yesterday,” “forgets where he is now,” and “forgets pension/pay week.”
A KICA Carer cut point of 2/3 (a score of 3 or more is positive) out of a possible score of 16 gave the optimum sensitivity (76.2%) and specificity (81.4%) (see Table 3). The area under the ROC curve (AUC) was 0.89 (95% CI = 0.85, 0.94). The KICA Cog cut-off score of 33/34 (using only the results of the 349 people who had caregivers) gave a sensitivity of 92.9% and specificity of 89.9%, with an area under the ROC curve of 0.96 (95% CI = 0.94, 0.98). The AUCs for two tests were different (p < 0.001). The KICA Carer and KICA Cog correlation yielded a Spearman's ρ = −0.35.
Sensitivities and specificities for KICA Carer and KICA Cog
Psychometric properties of KICA Carer and KICA Cog in combination
The sensitivity and specificity of KICA Carer and KICA Cog in both series and parallel were determined (Table 4). The optimum scores were seen in series, with KICA Carer 1/2 and KICA Cog either 33/34 or 34/35, or in parallel with KICA Carer 3/4 and KICA Cog either 32/33 or 33/34.
Sensitivities and specificities for KICA Carer and KICA Cog in series and parallel
When the predictive values of the tests were determined using the dementia prevalence of 12.0%, the optimum use of the tests in combination was found to be in series; using a cut point of KICA Cog as 33/34 and that of KICA Carer as 1/2, giving a PPV of 65.5% and NPV of 98.6% (see Table 5). Individually the KICA Cog had a PPV of 55.6% and NPV of 98.9%, and the KICA Carer had a PPV of 35.8% and NPV of 96.2%.
Properties of KICA Carer and KICA Cog in series, in parallel, and individually for the differentiation of dementia (n = 42) from no dementia (n = 307)
Discussion
Quality screening tools are vital for rural and remote health services to improve the accuracy of clinical diagnoses, as there is often limited or no access to medical equipment, such as a CT scan, or specialist clinicians. The KICA Carer was developed through extensive consultation with Aboriginal communities, specialists, and local service providers. The questions were readily accepted by informants, and the questionnaire elicited valuable information about cognitive change, with good sensitivity and specificity scores. The KICA Carer is a brief tool, taking less than 5 min to administer. It comprises eight questions with a maximum possible score of 16 (see Table 2). With an area under the ROC curve as 0.89, KICA Carer is a good test for screening Aboriginal Australians for dementia. The KICA Carer can be freely downloaded along with full KICA from www.wacha.org.au/kica.html.
This validation study was completed as a component of a wider prevalence study. The PPV of KICA Carer is based on a prevalence of 12.0%. However, unless it is used as part of a general wellness check, in practice the dementia prevalence of patients screened with KICA Carer would be higher, increasing the PPV. Assuming similar sensitivity and specificity, when the higher dementia prevalence of Aboriginal patients seen by the Kimberley ACAT is used (40%, n = 119; Bruce et al., Reference Bruce, Baird, Saddler and Goldswain1998), the KICA Carer alone has a PPV of 73.2% and a NPV of 83.7%.
The PPV of 35.8% for KICA Carer when used on its own is similar to that of the IQCODE reported by Mackinnon et al. (Reference Mackinnon, Khalilian, Jorm, Korten, Christensen and Mulligan2003) in a large Australian community sample with a dementia prevalence of 5.2% (cut point < 3.6/≥3.6, PPV 36%, NPV 98%, sensitivity 67%, specificity 93%). Although a higher PPV of 81% for the same cut point on IQCODE has been documented (Knafelc et al., Reference Knafelc2003), this came at the expense of specificity (47%), and was with a group of patients at a memory clinic with a dementia prevalence of 71%.
The KICA Cog is an excellent tool for screening dementia (AUC 0.96). Although using KICA Carer and KICA Cog in series improves the PPV performance of KICA Cog (55.6 KICA Cog, 65.5 in series), this method does not significantly add to the ability to accurately screen people for dementia, and as such it does not justify recommending the use of two tests in combination. It is recommended that KICA Carer be used in cases where it is not appropriate to conduct formal cognitive assessment, such as when there is no interpreter available, or disability (e.g. limited speech) affects the ability to participate in the KICA Cog assessment.
Ready et al. (Reference Ready, Ott and Grace2004) state that research based on informant report should ideally involve the informants that live with the participants, to increase the validity of the information and decrease selection bias. Although 85% of informants were family members of the participants, only 57% of informants lived with participants. Unfortunately, no specific age data were collected on informants, however relationships with the participants included children, spouses, siblings, grandchildren, niece/nephews, cousins, friends, and formal caregivers. The inter-rater reliability findings were based on 14 informants. The precision of ICC may have been improved with more participants. Another possible limitation includes the lack of a “gold standard” cognitive assessment for comparison in the Indigenous Australian population, although the DSM-IV criteria utilized by the panel of experts are widely accepted (Hogervorst et al., Reference Hogervorst, Bandelow, Combrinck, Irani and Smith2003).
This validation was completed as part of a prevalence study with informants that were not necessarily “caregivers,” and dementia prevalence was lower than seen in usual health and community care practice. As such, a validation study of KICA Carer and KICA Cog with a sample of caregivers of community clients of an ACAT, or caregivers of hospital inpatients, would be useful to determine the tool's validity with people referred for assessment.
Although we envision that KICA Carer would be appropriate for use with Aboriginal Australians living in remote Australia, we acknowledge wide diversity in Aboriginal and Torres Strait Islander peoples. This data were only collected from Aboriginal informants living in one region of Western Australia. The KICA Cog has been recently modified and validated for urban-living Aboriginal Australians (Radford et al., Reference Radford and Mack2012), and previously for Torres Strait Islander communities (LoGiudice et al., Reference LoGiudice2011; Russell et al., Reference Russell, Flicker, LoGiudice, Smith, Helmes and Strivens2013). Further research is required to determine whether KICA Carer needs to be adapted for these populations.
For best clinical practice to assess the cognition of older Aboriginal Australians living in remote regions, the KICA Cog should be used where possible, with an interpreter as required. However, we recognize that this will not always be possible in practice. The KICA Carer can be utilized on its own when patients cannot be assessed directly with KICA Cog.
In conclusion, the KICA Carer informant questionnaire is an important subcomponent of the KICA tool for patients where it is not possible to utilize KICA Cog. A KICA Carer score of ≥3/16 indicates that a medical review should be completed to rule out other possible causes of cognitive impairment prior to dementia diagnosis.
Conflict of interest
None.
Description of authors’ roles
D. LoGiudice, L. Flicker, O. Almeida, and N. Lautenschlager formulated the research questions. D. LoGiudice, L. Flicker, and D. Atkinson provided supervision to K. Smith in carrying out the study. D. LoGiudice and K. Smith developed the KICA Carer. K. Smith and J. Thomas performed inter-rater testing. K. Smith and A Dwyer carried out validity testing. D. Atkinson, A. Dwyer, and K. Smith assisted with advice, ensuring that appropriate consultation and feedback were provided to communities. D. LoGiudice, L. Flicker, and N. Lautenschlager performed the specialist assessments, and L. Flicker/N. Lautenschlager and O. Almeida completed the consensus diagnoses. K. Smith wrote the paper and performed statistical analyses. D. LoGiudice, L. Flicker, D. Atkinson, O. Almeida, and N. Lautenschlager assisted with editing the paper.
Acknowledgments
In memory of Gordon Oscar and Edward Mungulu (named with permission from family), with sincere thanks to other community research assistants, Judith Brooking, Darryl Isaac, Gabriella Dolby, Brenda Green, Wayne Milgin, Kevin Nungatcha, Charlene Carrington, Laurie Churnside, Lorna Hudson, Ronald Mosquito, and Melissa Sunfly. Sincere thanks are extended to the participating community members and community councils from Ardyaloon, Looma, Mowanjum, Junjuwa, Wirrimanu, Warmun, and Derby. Grateful assistance is acknowledged from Kimberley Aged and Community Services, Kimberley Interpreting Service, Kimberley Aboriginal Medical Service Council, Dr. Nicolette De Zoete, and WA Country Health Service – Kimberley.
