Introduction
Increased incidence of dementia has seen an increase in long-term care provision, and the trend for those being cared for is predicted to rise in the context of an aging society (Fahy and Livingstone, Reference Fahy and Livingstone2001). Historically for dementia care staff, dementia-related training has been limited (Henwood, Reference Henwood2001) and levels of knowledge poor (Thurmond, Reference Thurmond1999). Within the United Kingdom, knowledge of dementia has come under increasing scrutiny, and the issue of raising understanding and awareness has become a key focus of the recent National Dementia Strategy (UK Department of Health, 2009). This highlighted pervasive gaps in knowledge of dementia within health and social care settings, and a need for clear objectives regarding the importance of training and education. Often, care staff are not adequately trained to manage the complex needs and behaviors that they encounter, and studies have shown that poor knowledge can hinder staff from working effectively with people with dementia, irrespective of years of experience in the caring role (Ayalon et al., Reference Ayalon, Ayalon, Arean, Bornfeld and Beard2009).
There is a paucity of research that focuses on the impact of knowledge of dementia with respect to care staff. This might be partly attributable to the lack of existing standardized tools for this group. Past knowledge of dementia measures have generally been developed using samples of students and healthcare professionals (Dieckmann et al., Reference Dieckmann, Zarit, Zarit and Gatz1988; Barrett et al., Reference Barrett, Haley, Harrell and Powers1997) and family caregivers (Gilleard and Groom, Reference Gilleard and Groom1994; Kuhn et al., Reference Kuhn, King and Fulton2005). In a recent review of dementia knowledge measures, Spector et al. (Reference Spector, Orrell, Schepers and Shanahan2012) highlight the over-representation of items relating to cognitive symptoms and biomedical aspects of dementia knowledge within published tools. They recommend a contemporary measure which incorporates “biopsychosocial” (Spector and Orrell, Reference Spector and Orrell2010) and “person centered” (Kitwood, Reference Kitwood1997) models of care, particularly in the context of evaluating psychosocial interventions such as dementia education programs in a standardized way.
This study aims to develop a knowledge of dementia measure with good psychometric properties (reliability, validity, and utility) that represents the domains most central to the work of care staff. The measure will have a unique focus by representing both factual knowledge of dementia and, important for care staff, knowledge of dementia care. Items will represent knowledge of using different psychosocial strategies to manage situations or behaviors perceived as “difficult” in the day-to-day delivery of person-centered dementia care.
Good face and content validity are expected, this will be evaluated by a group of experts in the field of dementia. Construct validity will be established by measuring convergent validity (i.e. the measure will be correlated with scales measuring similar constructs in which positive associations would be predicted). It is anticipated that greater knowledge would be associated with more person-centered attitudes, and levels of subjective competence, as such links have been made in past research. (Zimmerman et al., Reference Zimmerman2005; Hughes et al., Reference Hughes, Bagley, Reilly, Burns and Challis2008). Greater knowledge is also expected to lead to greater job satisfaction, as measured by the Job Satisfaction Index (Firth-Cozens and Hardy, Reference Firth-Cozens and Hardy1992), as past findings have related greater sense of work satisfaction with those who perceive themselves to be better trained in dementia care (Zimmerman et al., Reference Zimmerman2005). Evidence for construct validity will be provided in the measure's ability to detect known group differences in a priori levels of knowledge.
The measure is expected to demonstrate acceptable reliability, established through a number of different forms of reliability testing: (a) test-retest reliability will be examined to assess the tool's temporal stability across two administrations, and (b) internal consistency will be calculated to assess homogeneity of the scale. This paper will outline the conceptual development and assess the above-mentioned psychometric properties of the measure.
Methods
Developmental stages
The independent steps of measure development followed within this study are informed by a model of test construction developed by Allen and Yen (Reference Allen and Yen1979). See Figure 1 for illustration of the development process.
Flowchart outlining the measure development process.
Literature search
Information was extracted from a number of sources, including government/social strategies and guidelines for dementia (Skills for Care, 2005; National Institute for Health and Clinical Excellence, 2006; UK Department of Health, 2006; 2009) to determine the domains of the construct of knowledge of dementia. The literature was explored in depth and important and reoccurring themes were drawn out, including choice, consent, capacity, pharmacological treatment, dementia and mental health problems, behavioral and psychological symptoms of dementia, psychosocial interventions, and person-centered care.
Next, we conducted a detailed analysis of past dementia knowledge measures and instruments of related constructs, such as aging and mental health, to explore the domains under each construct in question, looking at how they were defined during measure development.
Focus group
A focus group was held comprising seven frontline dementia care staff that explored elements of knowledge important in their role. The dominant themes that emerged from qualitative analysis of the discussions were added to list gathered from the literature search. Similar themes were amalgamated and arranged under primary headings. In total 13 domains were identified and presented with a written definition of the concept subsumed under each. The domains tended to cluster under the constructs of factual-based knowledge (“knowing that”) and care-based knowledge (“knowing how”). These domains encompassed a broad range of content, reflecting both the biomedical and psychosocial aspects of dementia knowledge important for care staff.
Expert review
The domains were listed on a feedback questionnaire and distributed to five dementia experts: a clinical psychologist, a consultant psychiatrist, a research associate in nursing, a dementia research associate, and a senior dementia trainer for consultation. Experts were requested to rank the domains in order of importance and to make further comments and alterations. The resulting eight domains derived were dementia general knowledge, psychological and behavioral symptoms of dementia, personhood, communication, psychosocial interventions/activities, challenging behavior, abuse prevention and risk, and consent and decision-making. Three overarching domains emerged: dementia core knowledge, dementia care knowledge, and risk and legal knowledge.
Item generation and choice of test format
Items were then generated from the eight selected domains. It was decided that items would be presented in statement form, using multiple choice format of five possible responses, including one correct answer, three distracters, and an I don't know response option. Distracters were employed to gain more information about patterns of responding and to assess the nature of incorrect answers. The I don't know response was included to reduce and control for guessing (Courtenay and Weidemann, Reference Courtenay and Weidemann1985). Next, items were scrutinized from previously published knowledge of dementia questionnaires and selected on the basis of matching appropriateness to the domain criteria. Items from published measures have gone through a process of testing and so can prove useful in terms of being psychometrically sound (Streiner and Norman, Reference Streiner and Norman2008). When necessary these items were adapted to make appropriate for the target sample and outdated terminology was modified to incorporate more person-centered and less medical language. New items were produced to cover the domains not represented by past measures and to add to existing items under other domains.
Expert commentary
In addition, experts in particular areas (such as nursing and psychology) were approached to contribute items to represent their area of expertise. Subsequently, an item pool of 39 statements with multiple choice response options was created. Items in the dementia care knowledge domain represented aspects of care practice in operation and so answers tended to be subjective, without a definitive “right” or “wrong” response. Such items required judgment of best practice, as this is considered to be a realistic reflection of the dilemmas that arise during the daily routine of staff in a caring role. In contrast, the core knowledge domain sought to tap objective factual knowledge about dementia covering areas of knowledge important for care staff to know as recommended by dementia experts.
To undergo further content analysis, the item pool was sent to experts for comments on clarity, correctness, difficulty, and use of language. Five experts (as above) responded and subsequent revisions were made, which saw one item being dropped due to ambiguity. In addition, the questionnaire was distributed at a dementia-related conference, where feedback was attained from 18 respondents representing a wide range of healthcare professions. Items were rated according to (a) difficulty, (b) relevance, and (c) ambiguity, using a 3-point Likert-type rating scale, for example, “Is the item ‘too difficult’/‘about right’/‘too easy.’” Resulting changes and adaptations were made to wording and the pool of 38 items was prepared for pilot testing. The pilot had 38 items falling under three domains: dementia core knowledge (15 items), dementia care knowledge (17 items), and risk and legal (six items).
Pilot testing
Through convenience sampling, 45 people were recruited to pilot the measure. This included seven dementia experts from a variety of disciplines (psychiatry, psychology, and nursing) and 38 care staff. Participants were recruited from two residential care homes in London and Nottingham and from several dementia-related courses. Participants were either given or posted the questionnaires. Demographic information was not collected for the pilot study.
The results of the pilot were scrutinized according to a number of item analysis procedures: expert agreement on the correct answer, reliability analysis, and frequency of endorsement. The latter refers to the difficulty of an item, as indicated by the proportion of people getting the answer either right or wrong.
Based upon expert responses, eight items were removed from the scale due to discord as to what constituted the “right” answer. Reliability analysis was carried out on care staff responses (N = 38) using SPSS version 17.0. In total, five items were removed from the scale for reliability purposes. A final item was removed for being too easy. Overall, analysis of the pilot reduced the number of items from 38 to 24. Domain three (risk and legal knowledge) was dropped as it was reduced to three items; these were subsequently incorporated into the dementia care knowledge domain.
Following the pilot analysis, the dementia core knowledge domain consisted of 11 items covering the following sub-domains: incidence rate, symptoms, course, treatment, and diagnosis (items 1–8) along with items pertaining to the behavioral and psychological symptoms of dementia (items 9–11). The second domain dementia care knowledge consisted of 13 items which represented a range of sub-domains, including person-centered care, communication, psychosocial interventions, managing challenging behavior, risk, abuse prevention, and consent (items 12–24).
Empirical evaluation of the 24-item measure
Participants
Participants were recruited from a number of sources, including staff from care homes in North London, trainees attending cognitive stimulation therapy training events, and a London-based specialist dementia training organization. Five social care residential homes within the Greater London region and two homes from the private sector took part in the study. Researchers visited care homes to distribute surveys. All staff within the home were targeted, including care staff, senior care staff, and management teams. A confidential box was provided where completed questionnaires could be deposited. Questionnaires were also administered to dementia care staff and healthcare professionals attending various training days, such as cognitive stimulation therapy training. Stamped addressed envelopes were given to return surveys if they were not completed on the day. Three of the residential social care homes described above took part in the test-retest substudy. In total 80 surveys were administered to the test-retest group with approximately one week between administrations.
Measures
Demographic and background information sheets were attached to each survey pack, consisting of a set of questions about job title, gender, age, and years of experience in dementia care. Three measures were administered in addition to the 24-item dementia knowledge measure developed as part of this study:
The Approaches to Dementia Questionnaire (ADQ; Lintern and Woods, Reference Lintern, Woods and Phair2000), a UK-developed 19-item measure evaluating attitudes to dementia, is presented on a 5-point Likert scale, for example, “Indicate to what extent you agree or disagree with the statement. . . ‘People with dementia are unable to make decisions for themselves.’” Higher scores indicate higher level of person-centered and hopeful attitudes toward dementia. The authors reported acceptable internal consistency, construct validity and acceptable test-retest reliability. Within the current study an acceptable level of internal consistency was achieved for the total scale (Cronbach's α = 0.78), while both sub-domains of hope and person-centeredness were found to be internally consistent.
The Job Satisfaction Index (Firth-Cozens and Hardy, Reference Firth-Cozens and Hardy1992) is an 18-item measure on 7-point Likert scale, assessing staff's level of satisfaction across different domains relating to their job, including pay, relationships with colleagues and supervisors, physical conditions, recognition, and opportunities, for example, “How satisfied are you with . . . ‘The opportunities to use your abilities?.’” For the current study excellent internal consistency was achieved (α = 0.93) along with moderate test-retest reliability (Intra Class Correlation (ICC) = 0.64).
The Sense of Competence in Dementia Care – Staff questionnaire (SOCID-S; Schepers et al., Reference Schepers, Orrell, Shanahan and Spector2011) is a 17-item measure, with a 4-point response scale that assesses staff's subjective sense of competence when working with people with dementia. This measure was prepared and piloted in conjunction with the knowledge of dementia scale and evaluated on the same large scale sample. In the current study the internal consistency for the overall scale was excellent (α = 0.91), while test-retest reliability (N = 58) was found to be substantial (ICC = 0.74).
Analytic plan
Results were analyzed using SPSS version 17.0. Initial item analyses procedures were carried out to examine the strength of individual items and to expose items that would be deleterious to the total scale. Tests of homogeneity of the scale were conducted using Cronbach's α coefficient. The reliability coefficient of the total scale, and individual items, across two administrations was calculated using ICC and Cohen's κ respectively. In order to control for the number of missing values, responses left blank or skipped were re-coded as an I don't know response. Where it was clear that an entire page was not completed due to misreading of the survey, then missing values were entered.
Results
Demographics
As part of the main sample 316 surveys were administered, and 211 surveys were completed and returned giving a response rate of 67%. The main sample (N = 211) primarily constituted care staff (n = 153), followed by allied health professionals (n = 32), management staff (n = 20), and non-clinical staff (n = 6, including kitchen staff, secretaries, and domestics). With regard to ethnicity, 29% described themselves as White British, 15.6% as Black African, 16.1% as Asian, 11.4% as European, 9% as Black Caribbean, and 4.7% of “other” ethnicity. The group ranged in age from 23 to 71 years (mean age = 44.2 years, SD = 11.4), and the great majority of the sample were females (83%). The test-retest group yielded 58 completed surveys. The majority, 86.2% were females (mean age = 43.86 years, SD = 11.8). This group comprised 45 care staff, seven management staff, one allied health professionals, and five non-clinical staff.
Item analysis
Item difficulty index, based upon the main sample (N = 211), was conducted to inform the frequency of endorsement of individual items (see Table 1). Generally, only items having endorsement rates between 20% and 80% correct are used (Streiner and Norman, Reference Streiner and Norman2008), items falling outside these parameters are considered too difficult or too easy. Eight items on the 24-item scale fell above the 0.80 endorsement rate. In order to maintain content coverage, it was decided that only those items demonstrating frequency of endorsement above 0.90 would be excluded. Enders and Bandalos (Reference Enders and Bandalo1999) suggest that having certain items that are highly skewed is acceptable and may not have a significantly negative effect on the overall properties of the scale (particularly if the mean inter-item correlation is high). As a result of item analysis procedures, four items in total were excluded and 20 items retained for further analysis. It was decided to name the final 20-item scale as the DK-20 (available as supplementary material attached to the electronic version of this paper at www.journals.cambridge.org/jid_IPG). The instrument has two domains: dementia core knowledge (11 items), and dementia care knowledge (nine items). Subsumed under the dementia core knowledge domain are the sub-domains of general knowledge (eight items) and behavioral and psychological symptoms of dementia (three items). The second domain, the dementia care knowledge, has six sub-domains, including person-centeredness (three items), communication (one item), psychosocial interventions/activities (two items), managing challenging behavior (one item), risk and abuse prevention (one item), and consent and decision-making (one item). The average score for the total sample (N = 211) was calculated, producing a mean score of 14.2 (SD = 3.1). The average score for the 11-item dementia core knowledge domain was 7.2 (SD = 2.1), whereas for correct answers for the nine-item dementia care knowledge domain it was 6.9 (SD = 1.6).
Item characteristics of the 20 items retained for the final scale
Notes: aADKT = Alzheimer's Disease Knowledge Test (Carpenter et al., Reference Carpenter, Balsis, Otilingam, Hanson and Gatz2009).
bDKT = The Dementia Knowledge Test (Moniz-Cook et al., Reference Moniz-Cook, Woods and Gardiner2000; compiled from ADKT and DQ).
cDQ = Dementia Quiz (adapted; Gilleard and Groom, Reference Gilleard and Groom1994)
dMSHAKE = Mary Starke Harper Ageing Knowledge Exam (adapted; Santo-Novak et al., Reference Santo-Novak, Duncan, Grisson and Powers2001)
eMHQ = Penn State Mental Health Caregiving Questionnaire (Spore et al., Reference Spore, Smyer and Cohn1991)
Reliability analysis
Internal consistency is a test of the scale's homogeneity, referring to the extent to which each item on the scale taps the overarching construct being measured. Cronbach's α coefficient was 0.63 for the 20 items (N = 201), which suggests marginal internal consistency. Coefficient α for the dementia core knowledge domain was 0.58, while for the dementia care domain α = 0.47. The marginal level of reliability for the total scale can be partially explained by (a) the reduced number of items within the scale (where four items were removed due to item difficulty index) as increasing Cronbach's α is in part related to greater number of items within a scale (Gliem and Gliem, Reference Gliem and Gliem2003), and (b) by the heterogeneous nature of the domains within the scale due to the breadth of content coverage.
Item-total correlation: Further analysis was carried out to establish values for item-total correlation. A recommended guideline (that items should correlate with the total score above 0.20; Kline, Reference Kline1986) was used in context with the item difficulty index to weigh up the strength of each item (see Table 1). Overall, item-total correlation results were satisfactory with only four items falling under the recommended mark, suggesting that the scale is relatively homogenous. Two items (items 1 and 8) in the dementia core knowledge domain were found to have poor inter-item correlations with the rest of the items within the domain. However, these items represented important aspects of the knowledge of dementia domain (incidence and symptoms) and were therefore retained for the purpose of content validity.
Test-retest reliability analysis was performed on 58 surveys, giving a 73% response rate. Test-retest reliability for the total scale was ICC = 0.73, p < 0.001, which indicated substantial reliability of the scale across administrations. In addition, the temporal stability of individual items was scrutinized by calculating Cohen's κ for each item across the two time points (see Table 1). Based upon criteria outlined by Landis and Koch (Reference Landis and Koch1977), the majority of items fell within the fair substantial reliability mark. Although one item demonstrated poor reliability overall, the test-retest reliability of the scale was satisfactory.
Validity analysis
Face and content validity was strengthened by steps taken during the conceptual development process, including an in-depth literature review, focus groups with dementia care staff, and repeated expert consultation. Content validity was furthermore established by attaining expert consensus on what constituted the “correct” answer for the items during the pilot study.
Convergent validity, the correlation between the DK-20 scale and measures of similar and overlapping constructs was assessed. Results demonstrated a significant correlation between the DK-20 scale and the ADQ calculated using Spearman's one-tailed test, r(175) = 0.36, p < 0.001. Further analysis indicated a significant association between the DK-20 and both the hope, r(183) = 0.32, p < 0.001, and the person-centered, r(188) = 0.30, p < 0.001, domains of the ADQ. As the person-centered domain of the ADQ and the dementia care knowledge domain of the DK-20 overlap in terms of both tapping the person-centered approach as outlined by Kitwood (Reference Kitwood1997), their relationship was analyzed and results showed a significant correlation, r(188) = 0.22, p < 0.001, one-tailed. A further significant association between the DK-20 and the Job Satisfaction Index was demonstrated, r(188) = 0.20, p < 0.02. Finally, contrary to the hypothesis, a non-significant correlation with the SOCID-S was attained, r(189) = –0.09, p = 0.23.
Discrimination between groups
The data distribution of the main sample was explored by conducting the Kolmogorov–Smirnov Test. Normal distribution was indicated; however, further statistical analysis demonstrated evidence of bi-modality in the distribution of the data when mapped onto a histogram. It was hypothesized that this disparity reflected the difference in scores among the care staff sample, and the remainder of the sample's knowledge scores, which would be expected to be higher based upon their job positions (i.e. care home managers and healthcare professionals). Further, the Kolmogorov–Smirnov Test was carried out on the sample of care staff alone, and results indicated normal distribution. In order to test the prediction, the sample was split according to self-reported job role into care staff (n = 132), and senior staff and professionals (n = 63). The non-parametric Mann–Whitney test was carried out to compare mean scores for each group. Results indicated a significant difference in knowledge scores between these two groups U = 2904.5, p < 0.001), with senior staff attaining a higher average score. Thus, the DK-20 is capable of discriminating between groups where difference in a priori knowledge levels is expected.
As the DK-20 is constructed using a multiple choice format, endorsement of the distracter responses provides useful information about response patterns, which indicates gaps in knowledge and areas of misinformation about dementia among the sample group. The response pattern varied for different items. For example, levels of the correct response were low in item 1, where 40% of the respondents overestimated the incidence of dementia. A significant level of misinformation emerged around the benefits of early assessment (item 4). Just over half the sample was able to discern the correct criteria for a diagnosis of dementia (item 6), and answers to items 7 and 9 suggested relative levels of misinformation about the treatment and course of dementia. Overall, respondents performed well on items tapping the person-centered domain informed by Kitwood (Reference Kitwood1997).
Discussion
This study provided preliminary results for the reliability and validity of the DK-20, a new measure of dementia knowledge aimed at unqualified frontline care staff. The DK-20 is the first knowledge scale to be developed and standardized specifically for care staff working within residential settings. The measure has good utility (15-min administration time) along with established face and content validity, acceptable test-retest reliability, and marginal levels of internal consistency. Good construct validity was demonstrated, and the scale is also capable of discriminating subgroups based upon expected differences in levels of a priori knowledge. Due to the multiple choice response options the measure can identify areas of poor knowledge, misinformation, and negative bias in responding, and could be used by managers as an indicator of staff training needs. Moreover, the DK-20 scale was found to demonstrate strong correlations with measures of related constructs, which establish it as a valuable tool in exploring relationships between knowledge and subjective staff factors, such as attitudes and hope, allowing consideration of how these variables relate to and exert influence on care behaviors.
There are several limitations. The overall level of internal consistency for the scale is marginal, which can be partially explained by the attempt to capture a relatively heterogeneous construct such as knowledge of dementia. Future development could serve to increase the coherence of the subscales with a view to improving reliability. In addition, it would be of value to assess the concurrent validity of the scale by administering the DK-20 along with another established test of knowledge. Contrary to the initial hypothesis, an association was not found between knowledge and sense of competence. This unexpected result might reflect the underlying differences between the two constructs being measured, in that knowledge of dementia is objective and factual based, whereas sense of competence refers to subjective degree of felt confidence. One might know how to do something without necessarily feeling competent in doing it, suggesting that other factors (environmental, and/or staff-related) might impact the relationship between these two variables.
Avenues for further examination and development of the DK-20 are recommended. It would be valuable to assess divergent validity by administrating the DK-20 with a measure of an unrelated construct where a low association would be expected. Capturing changes in knowledge following educational interventions is a potential for further use of this measure. A UK-based dementia training organization has used the DK-20, and early reports suggest that knowledge results can improve after training. However, further research is required to assess the DK-20 scale's ability to detect change following education.
Like any knowledge-based test, the DK-20 will require regular information updates, to keep pace with the emergence of new knowledge and facts about dementia, in particular around issues such as medication and treatment.
This study presents the preliminary findings for the reliability and validity of a new scale assessing knowledge of dementia. The DK-20 fills a previous gap in providing a tool designed specifically for assessing knowledge levels and identifying educational needs among dementia care staff. There is scope for further evaluation, such as a large pre-post research study, to assess sensitivity and to delineate discrete cut-off points for high and low levels of knowledge, indicating need for tailored educational interventions. Overall, findings generally support the usage and continued development of the measure. Widespread use of the tool in a research context and within clinical settings will help further establish its psychometric properties as well as its functional impact.
Conflict of interest
None.
Description of author's roles
Niamh Shanahan was the main researcher responsible for all parts from design to publication writing. Astrid K Schepers was involved in study design, data collection, and write-up. Aimee Spector and Martin Orrell provided expert advice, practical support in study design and planning, data collection, data analysis, and write-up.
