Introduction
Despite the public health importance of training our workforce to provide high quality care for older adults, there is surprisingly little research focused on evaluation of continuing education (CEU) and continuing medical education (CME) in the fields of gerontology and geriatrics (Davis et al., Reference Davis1999; Bloom, Reference Bloom2005). This is an area in need of serious study at the present time. According to a recent report on workforce needs developed by experts from the Institute of Medicine (2008), the number of older adults in the U.S.A. will almost double between 2005 and 2030, yet the nation is not prepared to meet their social and health care needs. The baby boomer generation starts to turn 65 in 2011, creating multiple challenges for the health care system including the fact that the majority of older adults suffer from at least one chronic physical and/or mental health condition and increased need for health care services compared to other segments of the population. Additionally, this generation will be the most diverse the nation has ever seen with more education, increased longevity, more widely dispersed families, and more racial and ethnic diversity, making their needs very different from those of previous generations. A key issue is the dramatic shortage of all types of health care workers, especially those in long-term care settings. Moreover, the overall health care workforce is inadequately trained to care for older adults.
In order for continuing education programs to fulfill their public health mission to improve health care professionals’ knowledge and skills, it is vital that these programs are accessible and effective (Miller et al., Reference Miller2008). While evaluation is viewed as an essential component of continuing medical education programs, rigorous evaluation linking education to meaningful clinical outcomes, such as changes in physician and health provider behavior or patient outcomes, has traditionally been lacking (Davis, Reference Davis1999; Bloom, Reference Bloom2005; Mazmanian et al., Reference Mazmanian, Davis and Galbraith2009).
There are a range of possible outcomes for evaluating the impact of CEU or CME on clinical practice and patient health. These methods include participation, satisfaction, learning (e.g. clinical knowledge or skills gained), performance (e.g. changes in practice behavior), patient health, and population health (Mazmanian and Davis, Reference Mazmanian and Davis2002; Tian et al., Reference Tian, Atkinson, Portnoy and Gold2007). Unfortunately, well-validated and reliable methods to evaluate these outcomes are sparse (Ratanawongsa et al., Reference Ratanawongsa2008). It has also been noted that a commitment to change from learners may increase the likelihood of successful change. According to Mazmanian and associates, (Reference Mazmanian, Mazmanian and Waugh1997), the concept of “commitment to change” grounded in communication, law and philosophy could be used to promote change in others and has potential for evaluating behavioral change as well. Assessment of the impact on behavior of professionals in the work setting (and, eventually, on patient outcomes) is essential yet difficult to achieve. It involves follow-up assessments which are more time and resource intensive, and logistically more difficult than obtaining data at only one time point (Ratanawongsa et al., Reference Ratanawongsa2008).
Traditionally, outcomes of CEU and CME programs focus on satisfaction with the program and on self-reported gains in knowledge (Tian et al., Reference Tian, Atkinson, Portnoy and Gold2007). More recently, programs have focused on a broader range of outcomes such as improved skills and changes in attitudes towards treating patients with certain characteristics. In a large-scale educational program conducted in five states involving one to two day regional workshops for community-based clinical faculty (N = 282) analyzed teaching skills associated with effective teaching behaviors, and identified individual teaching goals and educational needs in their institutions. The post-training follow-up showed significant increase in teaching ability (p < 0.001) and sense of integration with affiliated institutions (p < 0.001). The majority of these physician faculty were in primary care settings where they taught and supervised medical students (Skeff et al., Reference Skeff, Stratos, Bergen, Sampson and Deutsch1999). Results suggested that better patient outcomes were associated with improved faculty teaching skills, which was the program's focus.
In general, it is agreed that educational programs that focus on specific patient populations or clinical diagnoses (such as heart disease, diabetes, or cancer) are more effective than broader scope programs (Miller et al., Reference Miller2008). There is strong interest in training health care professionals on the care of patients with dementia (PWD) and their families, due to the expected dramatic increase in incidence rates of dementia as the population ages. It is widely known that the prevalence of Alzheimer's disease and other forms of dementia are age-related, affecting an estimated 5% of those aged 71–79, 24% of those aged 80–89, and 37% of those aged 90 and above according to a large, recent U.S. study (Plassman et al., Reference Plassman2007). These are major challenges to health care professionals who typically have little or no training in the care of PWD and their families. The Center for Health Workforce Studies (2005) projection of workforce shortages for geriatric mental health care, including dementia, are partly related to the complex needs of older adults with dementia and multiple comorbidities.
A multidisciplinary educational program on dementia care was conducted with content delivered to urban and remote areas of a predominantly rural state, using both traditional classroom and telecommunication technology. The three-month, post-training follow-up showed that 77% of trainees (N = 325) reported having used the acquired knowledge in their work setting with dementia patients (Murphy-Southwick and McBride, Reference Murphy-Southwick and McBride2006). However, the specific changes associated with the training were not identified.
In an effort to further address the training needs of professionals working with PWD, the Stanford Geriatric Education Center (SGEC) collaborates with several local health care provider organizations (e.g. the Alzheimer's Association regional office serving northern California and northern Nevada) to provide (for more than a decade) an annual 7-hour conference entitled “Updates on Dementia.” The program informs clinicians and non-clinicians working with some aspect of dementia care about new research developments and clinical practices. Included in the evaluation component of the conferences is an “action plan” that participants are requested (but not required) to formulate. They are asked to specify changes they plan to make in their work, based on information learned at the conference. A three-month follow-up contact is made with each participant to determine whether or not they were able to implement their “action plan” and whether it proved to be useful in their work setting.
The specific research objectives addressed in this paper are to: (1) ascertain the usefulness of “action plans” as a monitoring tool to evaluate concrete change over time, (2) investigate factors that influence their successful implementation, and (3) investigate factors that present challenges and barriers to their successful implementation.
While this method does not directly measure impact of education on patient care outcomes, it is a significant advance over widely used traditional measures of learner's knowledge or satisfaction. In our project, generating an action plan based on newly acquired knowledge or skill set provides the learner with a tool to track an issue in the work setting or one's professional practice behavior that needs attention. This monitoring tool could also serve to foster a commitment to change among the learners and increase the likelihood of actual change as indicated by Mazmanian and associates (Reference Mazmanian, Mazmanian and Waugh1997). Finally, the chosen action may also give cues to the evaluator on need for resources and further training.
Training and assessment methods
Participants
A total of 1,947 participants (957 clinicians and 990 non-clinicians) attended five conferences on “Updates on Dementia” (UOD) from 2004 through 2008. On average, about 400 health care professionals and students attended each year. Multidisciplinary experts in dementia care and ethnogeriatrics (aging, health, and cultural issues) provided a wide variety of curriculum content on treatment and management issues in different service settings where patients with dementia and their families receive care. The SGEC and partner organizations coordinated the selection of speakers from academic institutions and service organizations. Faculty presenters came from such institutions as Mayo Clinic College of Medicine at the University of Rochester, Minnesota; University of California at San Francisco; University of North Carolina at Chapel Hill; San Francisco State University; and Stanford University. The co-sponsors of the series included: Alzheimer's Association, Northern CA/Northern NV chapter, Family Caregiver Alliance (San Francisco), El Camino Hospital (Mountain View, CA), Institute on Aging (San Francisco), Older Adult and Family Center, (Stanford, CA), the Veterans Administration Palo Alto Health Care System (VAPAHCS), including the Mental Illness Research, Education, Clinical Center (MIRECC) and the Geriatric Research, Education and Clinical Center (GRECC) of the VAPAHCS. A conference planning committee met on a regular basis and invited nationally and internationally known researchers and health care providers to present the latest findings in the fields of basic and clinically relevant research on dementia. The format was lecture-based with question-and-answer segments included, as well as extensive informal networking opportunities, and Resource Fairs. Conference sponsors invited health care providers to participate by mail, e-mail, and through their work sites. The registration fee ranging from $45 to $80 covered conference attendance, breakfast, lunch, parking, and exhibitor fair. Registration fees varied from year to year depending on venue and related expenses. On average, about two-thirds of the participants who completed action plans were from the Bay area, with 65% from California, and 35% from the rest of the U.S.A.
To evaluate the impact of the training, several measures were developed and administered at each conference, including: registration forms asking for detailed demographic information; a satisfaction survey; and short pre- and post-questionnaires assessing knowledge, skills and attitudes regarding practice with ethnically diverse elderly patients. Each conference had either a plenary presentation or a panel discussion focused on diversity issues and dementia care aspects of ethnogeriatrics. In addition, a more precise approach to impact assessment called the “Action Plan” (AP) was adapted from the SGEC summative evaluation strategy for the different training programs offered since the inception of SGEC in 1987 (Gamel and Yeo, Reference Gamel and Yeo1993). The AP asks participants to state one to two specific ways they would apply the new knowledge and skills learned at the conference to their current professional role. The document was completed in duplicate and the participant retained a copy of his/her declared commitment. In this paper, we focus only on the usefulness of formulating an individual AP following participation in dementia training conferences, as an evaluation tool to assess actual changes (i.e. implementation of the planned behavior) over a three-month period.
Action plans
At the end of the training session participants were asked to reflect on their experience and develop a plan of action that they would incorporate into their work and which included some of the new knowledge they acquired at the UOD conference. Each participant was instructed to be as specific as possible and list up to two action plans that they consider feasible in their work role.
Three months after the conference, participants who completed APs were contacted via email, using a short survey technique commonly available on the internet, and referred to as “survey monkey.” They could complete the online survey in 5 minutes on line. Project staff initiated telephone contact up to three calls to participants who did not respond to the online survey. AP data were collected systematically for five years from 2004 through 2008. Aggregated data from this 5-year period is reported in this paper.
A total of 388 participants completed APs, a figure which represents about 20% of the total number of participants who attended the UOD conferences over the five-year period. Given that some participants completed more than one AP, a total of 666 APs were collected at baseline, and a total of 366 APs (55%) were collected from participants who responded at the three-month follow-up. Thus, the sample used for the data analyses in this paper consists of 366 APs, since that corresponds to the number of participants who both completed an initial AP and responded to the follow up (n = 222, or 57%).
Table 1 presents demographic information on the population sample (n = 388 participants) who submitted either one or two APs, and a demographic comparison of participants who submitted initial action plan and also responded at the three months follow-up. The sample consisted of 56% white or Caucasian, 14% Asian, 3% African American, 2% Hispanic/Latino, and 28% did not report their ethnicity. About 67% were from California and 56% of the participants (n = 218) stated they were clinicians. Thirty nine percent (n = 151) were health care practitioners, 11% were administrators or managers, 8% academic or clinical faculty, and 32% gave no response. When compared with the total sample of training participants, the percentage of clinicians who returned the action plans versus those who did not was very similar (about 50% in each case); however, other demographic information for those who did not complete action plans was limited.
Demographic data, completion of Action Plan 1 and Action Plan 2
Measures and analytic plans
For analytical purposes the APs were coded and rated on level of difficulty. Coding was done by three trained raters. All raters were trained by one of the authors, Renee Marquett. Each rater coded each action plan separately using codes and code definitions which were developed and approved by an expert panel (SGEC Director, Co-Directors, and Evaluation Specialist). Table 2 lists the code definitions and the overall percent endorsement for each code, across the grand total of 666 APs collected from the five conferences.
Action Plan code definitions, (n = 666 total action plans submitted)
The raters also rated the APs separately on level of difficulty (based on considering the participant's main responsibility), the content of what they aspired to complete, and overall perceived difficulty of the AP. The raters then went over all of the codes and difficulty ratings and came to consensus on any that differed substantially between them. Overall, there was 91% agreement between raters before conferencing and 100% consensus was reached after contrasting the results and the rationale.
The coding method is similar to coding methods used in prior research studies in that we used more than one rater and we classified the qualitative data into reasonable subcategories (Hasnain et al., Reference Hasnain, Onishi and Elstein2004). We differ in that we used an expert panel to determine the code definitions, instead of using experts to code the data (Memoona et al., Reference Memoona, Onishi and Elstein2004; Tonnesen et al., Reference Tonnesen2007), although both raters were trained directly by these on-site experts.
We used both quantitative and qualitative techniques to address the three research questions of interest: (1) the usefulness of developing an “action plan” as a monitoring tool to evaluate concrete change over time, (2) the factors that influence the successful implementation of the plan, and (3) the factors that present challenges and preclude a successful implementation of the intended changes outlined in the “action plan” as developed at the end of the training sessions.
In our quantitative analysis we explored the data using descriptive statistical techniques, χ2 analysis, and logistic regression models. A logistic regression model was used to investigate predicting factors associated with the likelihood of completing Action Plans 1 and 2 elaborated by the participants during their training, as reported three months after their training. Predictive variables included as covariates in the model were gender, age, ethnicity, clinician status (i.e. practicing clinicians versus others), primary work role (i.e. health care practitioner, administrator, in-service/continuing education, coordinator or other), and the level of difficulty of the selected AP as coded by raters into four different groups (i.e. easy, somewhat easy, medium difficulty, and most difficult).
Our qualitative data analysis is based on the responses to open-ended questions about factors that prevented participants from implementing their action plan. Question responses were coded and grouped to generate a list of barriers and facilitators. In the next step, coded data were categorized as either reflecting barriers or facilitators to implementation. Finally, for each of the major categories (i.e. facilitators and barriers) subcategories were identified. Subcategories for barriers included time, support (financial and other), lack of knowledge, lack of training, experience/expertise, other priorities, personal life, staffing and facilities and “other” barriers. Subcategories for facilitators included available resources, education (including medical and cultural knowledge), institutional support, positive feedback and “other.” The qualitative data provided additional useful information about the process of implementing behavior change, which we plan to use to help improve training methods for physicians and other allied health professionals in the future.
Results
Descriptive analyses
As previously noted, about 20% of the total participants in the five-year period studied actually completed APs. Given that some participants completed more than one, a total of 666 APs were collected at baseline, and a total of 366 action plan outcomes were collected at three-month follow-up. At the three-month follow-up, the reported success in AP implementation was very high among those who turned in at least one AP: 73% of the respondents reported at least some level of success. It is noteworthy that 27% of planned activities were not implemented, and among those who reported success there was great variation in the types of activities implemented and in the degree of success reported. Most indicated “moderate” success, with only about 17% of the respondents reporting that they had been “very successful” in implementing their plans. Table 3 presents a description of the percentage of coded responses according to AP type, by level of success and level of difficulty. The majority of APs were initially coded as “somewhat easy.” These were the plans generally reported at follow-up as having “moderately successful” implementation. Some examples of this were: “used therapeutic techniques” and “incorporated research/information into trainings, meetings, and classes.”
Types of Action Plan 1 and 2 vs success (3A) and difficulty (3B)
Qualitative analysis
Results of an analysis of barriers to and facilitators of implementation are summarized in Table 4. A subsample of 145 participants from the 2008 UOD conference were queried about this, with coding done by authors LH and MMB. Barriers included lack of time or being too busy (indicated in 30% of cases). In addition, having other priorities, and forgetting, were reported by 15% of respondents, and lack of knowledge, training, experience or expertise by 11%. About 4% of respondents reported lack of support (financial, leadership or other staff), and 4% reported institutional barriers. Conversely, among the facilitators to implementation, the ones most frequently mentioned were having sufficient resources and systems of support, and having sufficient education and knowledge to be able to complete the plan.
Barriers and facilitators of implementation in a subsample of respondents
Quantitative analyses: χ2 approach
To assess possible differences in the ability to complete an AP between clinicians and non-clinicians who attended the conference, a series of χ2 analyses were used. Results indicate that clinicians were not statistically different from non-clinicians in their likelihood of accomplishing their stated goals (p > 0.10). Psychologists were not significantly different from other groups, nor were gender or ethnicity associated with the likelihood of implementation success. On the other hand, when we examined the level of difficulty of the APs elaborated by different types of professionals, we found that licensed social workers and marriage and family counselors were less likely than other clinicians to develop APs that were considered easy to implement. They were somewhat more likely to develop difficult plans (p = 0.10). Nurses were also more likely to develop difficult action plans in comparison to other clinicians (p < 0.05 for all comparisons between nurses and others).
Logistic regression analyses
A logistic regression model was used to investigate predictive factors associated with the likelihood of completing the first and second action plans. When controlling for the covariates stated earlier that were included in the model (i.e. gender, age, ethnicity, clinician status, primary work role, and the level of difficulty of the AP), the only variable that was found to be statistically significant in predicting the likelihood of completing the first action plan was the level of difficulty of the proposed plan. Compared to those who developed an AP that was coded as “easy”, those who developed an AP coded as “most difficult” were about six times less likely to complete it (see Table 5).
Likelihood of completing Action Plan 1 during “Updates on Dementia” Conferences in 2004, 2005, 2006, 2007 and 2008: results of logistic regression model
Note: OR = odds ratio.
When we added the “action plan type” to the logistic regression model, results remained the same as described in Table 5, and the type of plan was not significant in predicting the likelihood of successful implementation.
We also analyzed the 144 participants in the dementia training who developed a second action plan, which they intended to accomplish after the training. This plan was successfully completed by 100 of them. Most (99%) had also been successful in completing their first AP. In a logistic regression model controlling for the same variables described above, difficulty level was not significant, but age and ethnicity were significant in predicting the likelihood of completing a second AP. Participants over 60 years of age were six times more likely to complete a second AP than respondents aged less than 40 years. Participants of ethnicity other than White (mostly Asian) were about three times more likely to complete it. Adding the type of plan to the model did not change these findings, and the type of plan was again not a significant predictor of success.
In conclusion, the use of a technique such as an “action plan” (with follow-up to determine degree of compliance) appears to be a useful way to stimulate change in the behavior of clinicians towards patients with dementia and their family members. Specifically, our analysis indicates that (1) “action plans” are a useful monitoring tool to evaluate concrete change over time; (2) the main factor that influenced their successful implementation was level of difficulty of the plan specified, and (3) factors that present challenges and preclude a successful implementation of the intended changes include lack of time or being too busy, having other priorities, forgetting; and lack of knowledge, training, experience or expertise to carry out the planned behavior.
Discussion
Our results indicate that while a commitment to change from learners might increase the likelihood of successful change as hypothesized by Mazmanian et al. (Reference Mazmanian, Mazmanian and Waugh1997), a crucial element to understand the likelihood of success is how realistic or difficult it is to implement a specific plan of action. In other words, it is not only how committed someone might be to implement change, but how capable she/he is of envisioning a feasible path of action that predicts successful change. In addition, our, results support the view that developing and writing APs can be a useful tool in self-monitoring behavioral change among trainees over time.
The main limitation of this study was the low response rate of the participants – only about 20% of attendees completed the AP at the UOD conference itself and were later available for follow-up. As previously indicated, when compared with the total sample of training participants, the percentage of clinicians was about the same among those who completed the AP (about 50% in each case); however, other information on the demographics of those who did not complete an action plan was limited. Therefore, the results presented here are from a self-selected sample, and those completing the AP could have had a higher motivation or commitment level than the rest of the participants. In addition, behavior change is not observed directly, but inferred from the self-reported change described by attendees.
Yet, despite those limitations, our findings are consistent with those of Murphy-Southwick and McBride (Reference Murphy-Southwick and McBride2006). While they found that 77% of trainees reported having used the acquired knowledge in their work setting with dementia patients, we found that 73% of the respondents reported at least some level of success in implementing specific changes.
As a follow-up to this study, a second smaller study was designed to address some of the limitations in the action plan response rate. This study was designed in collaboration with supervisory social work staff at On Lok Lifeways in San Francisco, California. On Lok Lifeways is a major health care provider to frail elderly (particularly low-income minority elderly such as Chinese) in the greater San Francisco Bay Area. This organization's goal in 2009 was to increase the number of advance directives for end-of-life care actually on file in their medical records system. A review of their medical records found that less than 50% of the patients had an advance directive on file. Training was focused on one specific topic: how to increase the proportion of older adults who had advance health care directives posted in their medical records – both individuals with dementia and those without measurable cognitive impairment. Training was provided in a workshop format with 30 participants. The workshop included eight hours of intensive discussion, role-playing, and problem-solving led by experts in this area. Action plans were developed focusing on how to work more effectively with patients and families to achieve the agreed goal. All of the social workers who attended the workshop completed an action plan at the time of training and were available for follow-up six months later. Results indicated that 69% of the social workers were able to complete their AP successfully. In terms of patient care outcomes, initially, 48% of the patients in this system had advance directives on file; after six months, the proportion had risen to 54%. This 6% improvement translated into about 75 additional patients having filed the advance directives. This was a meaningful change for On Lok Lifeways and represented partial success in reaching their goal. Taken together with the AP results, this led to further discussion about what the barriers were, and what staff training was needed to address those barriers.
This experience taught us several lessons about how best to implement behavior change in staff who work directly with older adults in health care settings. We conclude that success in getting 100% compliance with the action plan both at the time of training and at time of follow-up was associated with three factors: (1) size of the group; (2) teaching and learning methods; and (3) clearly specified achievable outcomes stated during the training. Group size was small (in contrast to the average of 400 attendees at the UOD conferences); content was delivered in workshop format that was highly interactive, with brief didactic sessions (in contrast to the lecture style of the UOD conferences); and the curriculum was designed based on identified needs to improve specific patient care outcomes that were supported by management and well articulated in the learning objectives for the training program (in contrast to the UOD conferences where each individual set up their own plans – which might not have been consistent with the larger goals of their employer). In short, when the system is supportive of behavioral change, it is much more likely to occur, and staff are much more likely to comply with requests to document behavior. Thus, the degree of relevance to trainees’ performance goals in their respective roles is a strong motivation to achieve the expected outcomes of a continuing education program. In contrast, participants of the UOD conferences have a wide range of motivations for attending the training. As such, the challenge may be to identify common ground in the context of a multidisciplinary system of dementia care, which all attendees could support.
After carefully considering our experience with the studies reported in this paper, we are confident in recommending use of the AP strategy to other educational researchers and educators responsible for training health care providers and/or administrators working in the dementia field. However, we would like to make several recommendations at this time to improve this process. First, we suggest developing incentives to increase completion of APs at the conference, and to increase completion of the follow up survey. Participants in the UOD conferences were encouraged in various ways to comply; however, it was a voluntary activity. For the 2010 UOD conference, AP completion was required to receive CEU or CME credits. We also plan to offer prizes (such as gift cards), drawn randomly, to those who complete the follow-up surveys. Second, we recommend providing some concrete examples of APs (including those coded as relatively easy, compared to those likely to be more difficult) so that attendees have specific information to guide them in their thinking. Third, we suggest conducting follow-up surveys with all attendees so that everyone can be asked how participating in the educational experience impacted their behavior in the work setting. This can be done either instead of, or in addition to, collecting action plans at the time of the training. By doing this, a much larger pool of responses will be available for subsequent analyses. At the same time, it should be noted that staff time (and thus, the total cost) required to collect and code this type of follow-up data is significant and may not be available in all settings. On the other hand, behavioral methods of data collection, such as participant observation of behavior change, are even more expensive and time-consuming, and may be impractical in many settings. Fourth, we recommend close collaboration with staff when an agency or institution request specific educational training (such as the On Lok Lifeways example). It is clear that group size, educational methodology used, and specification in advance of a clear behavioral outcome to be achieved will all increase the success of implementing the knowledge and skills gained.
In conclusion, further research is needed to replicate these findings and to study other methods for promoting behavior change in persons who come for CEU and/ or CME training – particularly with regard to dementia care, as well as care of older adults with multiple, chronic health and mental health problems. Our experience with the action plan methodology described here highlights the improvement of an evaluation strategy initially developed in the early years of SGEC (Gamel and Yeo, Reference Gamel and Yeo1993). In future UOD programs and other types of training, it would be useful to incorporate an experimental design into the AP evaluation model wherein “mandated” outcomes are specified rather than outcomes that are self-generated. We hope that the recommendations provided in this paper will enable others to build on our experience, and ultimately to improve medically related outcomes for older adult patients and their family members.
Conflict of interest
None.
Description of authors’ roles
Eunice Rodriguez participated in designing the analyses and evaluation of the project and took the lead in writing the paper. Renee Marquett also participated in designing the analyses and evaluation of the project, conducted the statistical analysis, prepared the tables and contributed to writing the paper. Ladson Hinton conducted the qualitative analysis and contributed to writing the paper. Melen McBride helped with writing the paper. Dolores Gallagher-Thompson, as the principal investigator, led the project implementation and analysis, reviewed the data, and participated in writing the paper.
Acknowledgments
Dr. Gallagher-Thompson, Principal Investigator, received substantial support for this study through a grant from the Department of Health and Human Services, Health Resources and Services Administration (HRSA), Bureau of Health Professions (D31HP08825). Dr. Ladson Hinton received support from the National Institute on Aging (AG010129).
