Introduction
Conceptual models of dementia
Purely medical models of dementia (Lyman, Reference Lyman1989) explain symptoms and behavior in dementia as a result of organic change. These have been increasingly superseded by approaches that encompass psychosocial aspects (Kitwood, Reference Kitwood1990; Downs et al., Reference Downs, Clare, Anderson, Woods and Clare2008). Such models promote a more positive approach to dementia care by not simply ruling behavior or mood as an inevitable consequence of dementia but as something that may be amenable to change.
More recently, Spector and Orrell (Reference Spector and Orrell2010) created a “Biopsychosocial model of dementia,” an amalgamation of prominent theories and intended to be a useful clinical tool to help formulate interventions and identify potential issues in practice. The model recognizes that the level of actual functioning is a combination of the severity of impairment plus the interaction of a number of limiting factors, which restrict a person's potential level of functioning. The difference between “possible” and “actual” functioning can be thought of as “excess disability” that may be amenable to change with the consideration of the limiting factors that are “tractable.” These tractable factors include psychosocial factors (such as mental stimulation, mood, and social psychology) and biological factors (such as physical health and sensory impairment).
Psychosocial outcome measures for care staff
An obstacle in the successful evaluation of training is inadequate outcome measures. Measures exploring constructs around person centered approaches or biopsychosocial factors are minimal (Spector et al., Reference Spector, Orrell, Schepers and Shanahan2012). Vignettes are descriptions of people or circumstances that can be real or fictional, and are used in research as a tool to investigate participant responses in relation to a construct or idea. They may be used to explore beliefs and attitudes or evaluate practice (Lapatin et al., Reference Lapatin2012) and have been shown to be one of the most effective ways of assessing clinician competence (Peabody et al., Reference Peabody, Luck, Glassman, Dresselhaus and Lee2000). This is due to a number of strengths: they are reasonably cheap and quick to administer, they provide a realistic case study from which to work that has no issues with ethics (Hughes and Huby, Reference Hughes and Huby2002), and they allow natural answers rather than forcing responses or testing passive knowledge.
Development of vignette-based measures
Whilst there is no established framework for developing vignettes, previous studies have outlined recommendations. Lapatin et al. (Reference Lapatin2012) broke the process down into a number of “iterative steps” including (1) identification and prioritization of content and format, (2) draft of vignettes by suitable team, and (3) reliability test and calibration process. Barter and Renold (Reference Barter and Renold1999) published an overview of recommendations: vignettes must be plausible, related to target audience, and contain sufficient depth, as well as considering format and relative rareness of examples with aims in mind. There are a number of approaches to scoring vignettes including qualitative analysis or use of a standardized scoring scheme, with the development of the scoring scheme largely depending on the aim of the measure. Standardized scoring structures can include asking for interpretation of the vignettes in the form of multiple choice answers (Kitchener and Jorm, Reference Kitchener and Jorm2002) or levels, for example one study awarded marks by rating formulations on six levels representing increasing depth and relevance (McClain et al., Reference McClain, O'Sullivan and Clardy2004).
Current study
Whilst a number of measures for dementia care staff exist, none specifically tap into the skill of thinking about biological, psychological, and social factors, nor do they demonstrate one's application of knowledge to real world scenarios. This study aimed to create a biopsychosocial vignette for case conceptualization in dementia care and explore its inter-rater reliability, test–retest reliability and construct validity. For the purposes of this study, “thinking in a biopsychosocial way” refers to the ability to distinguish issues in dementia care beyond symptoms or biological factors, to create a wider picture of issues faced, and therefore potential actions.
Regarding construct validity, it was hypothesized that scores after training in the biopsychosocial model would be higher (over practice effects explored by test–retest with no intervention). Regarding convergent validity, it was hypothesized that those with more background experience, greater knowledge of dementia and more person centered and hopeful attitudes to dementia would score higher.
Methods
The process of development involved a number of sequential phases, each addressing different aspects of reliability and validity. Ethics were approved by University College London (UCL) ethics.
Phase 1: Vignette development
Review of existing measures
A review of previous vignette measures within dementia and vignettes in clinically related fields was conducted. This led to indications that the measure; (1) should be relatively brief in order to increase ease of application; (2) should include sufficient content to be able to draw multiple points with a variation in difficulty; (3) should present fairly typical issues, in order for healthcare professionals to relate them to current clients and problems; (4) should include examples from two different settings (care home and community setting) and contain different features, e.g. age and gender, which would increase generalizability.
Development of preliminary vignettes
Two fictional vignettes were created: “Mary” and “John” (see Figure 1). The biopsychosocial model (Spector and Orrell, Reference Spector and Orrell2010) was used to create a brief outline of possible problems, which were then built upon to create a coherent “story.” Resources used at this stage included researchers own professional experience, and expert texts including the Camberwell Assessment of Needs for the Elderly (CANE) manual (Orrell and Hancock, Reference Orrell and Hancock2004). For both cases, issues were presented in a way that would require varying levels of interpretation to identify them. Preliminary vignettes were presented to two psychologists and one dementia services manager. These were refined following their feedback.
A Biopsychosocial vignette for case conceptualization in dementia; an extract.
Phase 2: Field testing
AIM
To check general stylistic issues and language suitability, and generate maximum content validity.
Procedure
Using convenience sampling, 12 people were approached and invited to complete the vignette measure online using Qualtrics (2001), an online survey platform. Participants were additionally asked for basic demographic data and to provide feedback (in the form of open ended questions with prompts, for example, “how realistic did you find these case studies?”).
Outcome
Nine people completed the exercise; one community psychiatric nurse (CPN), one dementia adviser, one postdoctoral researcher, one registered nurse, one senior clinical lecturer, two trainee clinical psychologists, and two research assistants. Feedback from responses was positive; with most people indicating that they felt the exercise was “realistic.” Those who expressed that they had found the exercise difficult related this to their lack of experience within a particular setting or client group.
The content and instructions of the vignette were refined according to feedback. First, the answers generated indicated that all of the factors were identifiable because they were each referred to at least once; however they were not consistently identified by each respondent. This suggested that there might be variation in the ease of reporting or interpreting factors within the vignettes, resulting in adequate variation in responses. In terms of content, it was suggested that the vignettes could be made more succinct to increase readability. This was done with minor alterations to the issues included. Small changes to the instructions to add clarity to questions were included, for example “Please list as many as you can think of” was added.
Phase 3: Pilot
Procedure
A convenience sample was generated using established connections with health professionals known to researchers. This included current colleagues, those presently attending training and previous attendees of a training course who had given permission to be contacted about future research. Inclusion criteria were that people (1) currently worked with people with dementia and (2) could speak English. Potential participants were approached via email or in person to complete the exercise.
Measures
The VIG-Dem consisted of two small fictional case studies named “John” and “Mary” of approximately 200 words each. They were both preceded by a set of instructions and each of the vignettes were presented in turn and accompanied by two questions. The aim of the questions was to allow participants to demonstrate their knowledge by interpreting the vignettes in terms of “issues” they may contain and suggesting appropriate “interventions.” The final scoring scheme structure contained a list of potential items that could be awarded either one or two marks depending on the level of interpretation and could be summed to give an overall score with a maximum score of 56. These items are arranged into four sections “part one: issues” and “part two: interventions” for both “Mary” and “John” (see Figures 2 and 3).
Mary scoring scheme.
John scoring scheme.
Participants were given the option of completing it as a paper-based measure or online via Qualtrics (2001). The content was identical in each format. The word “Biopsychosocial” was deliberately omitted from all materials so that respondents would not be cued into eliciting a particular style of response.
The Approaches to Dementia Questionnaire (ADQ) (Lintern et al., Reference Lintern, Woods and Phair2000) is a 19 item rating scale that was developed to indicate positive, person centered attitudes of care staff. It contains two sub scales, “Hope” and “Person-centeredness.” Statements are rated on a 5-point likert rating scale ranging from one (strongly agree) to five (strongly disagree), with higher scores indicating higher hopeful and person centered approaches. Previous validation of this scale has shown good reliability (Cronbach's α > 0.80) and validity. The ADQ was included to provide information on convergent validity.
The DK-20 (Shanahan et al., Reference Shanahan, Orrell, Schepers and Spector2013) was added after an interim analysis, to further explore convergent validity. Hence, DK-20 scores are only available for a sub-section of participants (N = 43). It consists of 20, five-option multiple-choice questions on a mixture of dementia knowledge and care attitudes. The measure has two subscales, “Dementia Core Knowledge” and “Dementia Care Knowledge.” Moderate reliability has been found for this measure (Cronbach's α > 0.60).
General demographics (age and gender) and background variables were recorded. This included number of years working with people with dementia, name of current working role, current and previous types of work settings, and highest level of education.
Training effects and test–retest reliability
A sub-sample (N = 30) included those who were attending training specifically on the biopsychosocial model. They were asked to complete the measure at two time points; before and after training, with an interval of approximately five weeks. This was to explore the measure's sensitivity to change following training. In order to assess test–retest reliability a further sample (N = 10) agreed to complete the measure at a second time point approximately five weeks after initially completing the exercise. These participants were included to assess consistency of the measure over time.
Scoring scheme development
When developing the scoring scheme, there were specific aims: achieving good inter-rater reliability and ensuring that it is user friendly, that is, not too long or difficult to mark. The process involved several steps: (i) a brief heading was created for each of the initial points that were designed to be included in (and therefore could be pulled out of) the vignettes; (ii) each pilot answer was explored by putting each statement under the most appropriate heading and a new heading created if it did not fit (i.e. expressed a new appropriate issue or intervention). This was conducted with all answers from the field test and continued with responses from the main pilot (N = 5), until it was felt that all possible relevant answers had been represented; (iii) the list of possible items were explored in terms of lower level observations and interpretations and then regrouped into meaningful categories (using the biopsychosocial factors) to form a provisional structured marking frame; (iv) two alternative marking frame structures were explored: hierarchical (i.e. levels awarded on depth and relevance) and accumulative (i.e. marks awarded for specific content). An accumulative structure was chosen, as it was the easiest to implement on the types of response generated by this exercise; (v) this scoring scheme was used to mark three pilot responses by the researcher and a second marker, in order to check the usability of the scheme as well as indicate potential inter-rater reliability; (vi) discrepancies between marks, as well as ambiguity in its interpretation were discussed and clarified, resulting in a second revision. The finalized scoring scheme was used by the researcher to mark all participant responses. Twenty-five responses were then re-marked by a second rater to explore inter-rater reliability. All researchers were blind to participant characteristics at the time of marking; (vii) the initial scoring scheme for the vignette measure had a possible range from 0–113, with participant total Scores ranging from 6–62 with a mean of 25.75 (SD 10.36). Coverage of this scoring scheme was 49% for these participants. Therefore, the scoring scheme was further developed by analyzing possible redundant items i.e. the items that were rarely awarded marks on the scoring scheme reflecting the most difficult points to identify in the vignettes. It was decided that by using an 80% cut-off, the resulting scoring scheme would have greater utility in that it would be quicker to mark, allow greater coverage and produce a normal distribution.
Results
One-hundred-thirty-one completed responses were collected; the demographics of this sample can be seen in Table 1. Due to convenience sampling, used response rates were not possible to calculate.
Participant demographics (N = 131)
Data were inputted, coded, and analyzed on SPSS version 21.0 and an alpha level of 0.05 was used for all statistical tests.
Scoring properties
The scoring scheme for the vignette measure had a possible range from 0–56, with participant total ccores ranging from 4–42 with a mean of 25.8 (SD 10.36). A Shapiro–Wilk test indicated that the distribution of total scores was normal (W (131) = 0.990, p = 0.426.
Content validity
This was ensured by a rigorous development process. Spector and Orrell (Reference Spector and Orrell2010) conducted a thorough literature review to create a biopsychosocial model summarizing factors in dementia care and therefore by using this as a framework it was ensured that a wide range of biological, social, and psychological factors were considered. The use of expert opinions and feedback from healthcare professionals ensured its applicability to real world practice.
Face validity
Targeted questions were asked during field-testing to assess people's perceptions of the exercise, while general comments were sort throughout. Field-testing indicated that people felt the vignettes were “realistic” and had positive attitudes to the exercise. Further comments throughout the study continued to show this trend; one participant commented “it was useful doing the exercise and illustrated that I need to brush up my knowledge in some areas.” Another commented “I enjoyed the exercise as it's the type of work I do.”
Effect of format
The 69 participants who completed the exercise online had a mean score of 23.2 (SD 7.08) and the 62 who completed hard copies had a mean score of 17.2 (SD 7.31). The difference between total scores (Sqrt) for these groups was significant (t (129) = 4.77, p < 0.001) indicating that those completing the exercise online scored higher on average.
Internal consistency
The mean score for “Mary” was 11.5 (SD 4.76) and “John” was 8.8 (SD 3.74). Spearman's correlation showed that scores for the two vignettes had a strong, significant correlation (ρ (129) = 0.68, p < 0.001). Cronbach's α coefficient was 0.73 for the 44 items on the scoring scheme indicating good internal consistency.
Inter-rater reliability
The intra-class correlations indicated substantial inter-rater reliability looking at absolute agreement for total score for both data sets marked by two raters (N = 25, ICC(2,k) = 0.94, p < 0.001).
Convergent validity: existing measures
Total scores (sqrt) had a positive significant correlation with the ADQ total score (ρ (127) = 0.37, p < 0.001), Hope sub-scale (ρ (127) = 0.38, p < 0.001), and Person-centeredness sub-scale (ρ (128) = 0.27, p = 0.002). For the sub-sample of DK-20 data, total scores (sqrt) had a positive significant correlation with the DK-20 total score (ρ (41) = 0.49, p= 0.001), Care Knowledge sub-scale (ρ (41) = 0.45, p = 0.002), and the Core Knowledge sub-scale (ρ (41) = 0.46, p = 0.002).
Background characteristics
The different roles of the healthcare professionals were recoded into 13 categories. Analysis of variance showed that the differences between Roles were significant for total scores (sqrt) (F(12,117) = 3.55, p = 0.001). Post hoc comparisons using the Tukey HSD test indicated that occupational therapists (M= 26, SD = 8.27) scored significantly higher than support workers (M = 15.6, SD = 7.34) and healthcare assistants (M = 12.3, SD = 6.73). Trainee psychologists (M = 24.9, SD = 6.84) scored higher than healthcare assistants; dementia workers (M = 25.9, SD = 8.45) scored better then support workers and healthcare assistants. Analysis of variance revealed a significant difference between levels of education for total scores (sqrt) (F(4,115) = 3.33, p = 0.01). Post hoc Tukey HSD test did not reveal significant relationships between these groups. Spearman's correlations found that total score was not significantly correlated to the number of years experience working in dementia care (ρ (123) = 0.10, p = 0.26), nor number of settings worked in (ρ (98) = 0.18, p = 0.07).
Construct validity: sensitivity to change
There was a significant difference between total scores (sqrt) before and after the biopsychosocial learning intervention (t (29) = 3.57, p < 0.001) with post-test scores on average 5.8 points higher (SD = 9.29).
Test–retest reliability
The mean total score for time point one was 26.3 and 25.7 for time point two. The intra-class correlations indicated excellent test–retest reliability for total Score between time points one and two (N = 10, ICC(2,k) = 0.96, p < 0.001).
Discussion
Summary of findings
This study has described the development of a vignette exercise and scoring scheme and assessed its reliability and validity as an outcome measure. The scores achieved in this study had an adequate range of 38 covering 67% of the possible scheme. Scores were skewed to the lower end of the scale; this could potentially be an artifact of this sample not possessing the knowledge to reach the higher scores, but is more likely to be a consequence of the scoring scheme design. Previous studies have shown that when scoring schemes are based on a comparison to a model answer coverage may be low (Dudley et al., Reference Dudley, Park, James and Dodgson2010). For example, Abbas et al. (Reference Abbas, Walton, Johnston and Chikoore2012) found participant scores covered only 40% of the total possible range when scoring formulations against model answers even after training. Good internal consistency for the scoring scheme was established and the scores for each of the two vignettes were significantly correlated. This suggests that the quality of answers generated by the different vignettes do not vary greatly. There was good inter-rater reliability between two raters.
A number of indicators for convergent validity were found; two existing outcome measures, the ADQ and DK-20 were shown to have a strong positive relationship with vignette scores.
Background variables were found to have significant relationships with participant scores in the expected direction; total scores were on average higher for those with more senior roles. This would be expected as an increased knowledge base of working with people with dementia, which they would be able to apply during this exercise. This adds support that the measure is testing a knowledge-based construct.
This study also identified that it is sensitive to change in knowledge as subjects scored higher on the measure after completing a training course. Positive test–retest findings suggest this is due to learning rather than practice effects.
Experience in years and number of settings worked in did not correlate with total score. Possible explanations are (1) that those with more experience may have less motivation to “test” their knowledge or (2) that some healthcare professionals who received training years ago may have been exposed to medical models and be less familiar with biopsychosocial approaches. To support this idea, Zimmerman et al. (Reference Zimmerman2005) showed that person centered and hopeful attitudes were associated with working fewer (0–2 years) rather than more years in the field.
The format of response may have influenced scores, with those completing the questionnaire online having significantly higher scores on average. The response rate for those replying online from a database was low and hence it is possible that those replying online had a stronger motivation to complete the task. Online completers also did so in a non-controlled environment where there could have been a number of extraneous variables; time to complete task, using reference texts, or discussion amongst colleagues.
Strengths and limitations
Several limitations are of note. First, a rigorous assessment of test–retest reliability was beyond the scope of this study resulting in a limited sample. While excellent reliability over time was indicated, further data collection would strengthen this conclusion. Another potential limitation was time restraints: participants commented that there was not enough time, and in some cases, it was evident that they had taken longer than requested (both on online and hard copies). Therefore, time is a potential confounding variable that was not controlled in this study. It may be that the time given to participants was simply not adequate for them to complete the exercise to a standard that represents their ability, and that increasing the time available to participants may lead to higher scores, and therefore increased coverage of the scoring scheme.
There may have been biases within the sample. Demographics of non-completers were not recorded and therefore it is not known whether there were any systematic differences between those who did and did not respond to the invitation to take part. Although the sample was predominantly female, this is a reasonable representation of the gender mix within healthcare professionals.
Implications for future directions
Whilst results suggest that the measure may be used in its current format, potential revisions are also indicated. While the scoring scheme generates a standardized score, the generated answers have value in themselves and could be explored in different ways. For instance, qualitative questions could also be included as a way of indicating certain attitudes or approaches, for example “does this response take Mary or John's wishes in to account?” Further work on the scoring scheme could extend its use.
It was originally felt that using two vignettes with different backgrounds would increase generalizability. However, as the scores for “Mary” and “John” were significantly correlated, it may be possible that the measure could be shortened by using only one vignette at one time. This could (1) allow participants more time to focus on their answer, and (2) enable each vignette to be used at a different time point, thus reducing practice effects. To establish feasibility of this approach, a number of investigations would be needed including identification of why scores for “Mary” were consistently higher. This may be the result of it being presented first, or due to easier content or scoring.
Implications for clinical practice
As this measure employs a standardized scoring scheme, it enables the assessment between and within individuals over time. The ability to approach dementia care from a biopsychosocial framework is an increasingly valued skill and is important in order to reach an acceptable standard of quality care. This measure could be used to indicate healthcare professionals’ level of understanding, for example, when recruiting staff to different roles. It could help establish adequate staff approaches to dementia care and highlight training needs. It has the potential application as an outcome measure for assessing training or other interventions for care workers aimed at increasing a holistic approach. It may also offer researchers a tool to explore other constructs related to biopsychosocial approaches in healthcare staff. As no current measure exists to explore this particular aspect of healthcare knowledge, it could have a number of important functions.
Conclusion
This study has developed what is known to be the first vignette-based measure with a standardized scoring scheme, to assess “Biopsychosocial thinking” in dementia care. Preliminary findings for reliability and validity of the measure show promise of establishing a useful and credible tool. This measure may play a useful part in assessing competence and the impact of interventions.
Conflict of interest
None.
Description of authors’ roles
Aimee Spector conceived of the idea, oversaw the whole program of research, supervised both research projects and helped to write/commented on drafts of the paper. Molly Hebditch designed the vignette and collected and analyzed data for a large proportion of the sample, for her MSc research project. She was involved in data entry, analysis, and writing drafts of the paper. Luke Gibbor collected and analyzed further data for his BSc research project. He was involved in data entry, analysis, and writing drafts of the paper. Charlotte Stoner helped devise the new scoring system and rescored and reanalyzed the full dataset, using this scoring system. She has also commented on the final draft of the paper.
Acknowledgments
The authors would like to thank the London Alzheimer's Society, Richmond upon Thames Memory Services for their support in undertaking this research. Also, to residents at Ash Court Nursing Home and Clara Nehab House and the managers of these homes: Steven Anderson and Grace Ramalho, respectively.
