Introduction
Recent focus on dissemination of cognitive behavioural therapy (CBT) treatments to routine clinical practice (RCP) has raised a fundamental question: ‘What are the most effective methods to train such skills?’ (Shafran et al. Reference Shafran, Clark, Fairburn, Arntz, Barlow, Ehlers and Wilson2009; Rakovshik & McManus, Reference Rakovshik and McManus2010; Becker & Stirman, Reference Becker and Stirman2011). The need to most efficiently allocate training resources requires a more thorough understanding of the same issues considered when providing empirically supported psychological treatments (EST) to patients: what works for whom, when and under what circumstances?
Decades of research have allowed CBT to develop time-limited treatment protocols, which show clear and significant effects for targeted patient populations (e.g. NICE, 2004a,b, 2005). Dismantling and process studies seek to further hone the contents of these protocols by evaluating which aspects are merely inert components and which are ‘active’; i.e. have clinically significant effects on the presenting problem (e.g. Longmore & Worrell, Reference Longmore and Worrell2007; McManus et al. Reference McManus, Peerbhoy, Larkin and Clark2010a). However, the methods by which these treatments are disseminated have limited empirical support, thus leaving CBT trainers with a choice either of using extensive (and expensive) training programmes, which have shown significant effects on therapist competence and patient outcome (Shaw et al. Reference Shaw, Elkin, Yamaguchi, Olmsted, Vallis, Dobson and Imber1999; Kuyken & Tsivrikos, Reference Kuyken and Tsivrikos2009), or choosing more brief and feasible methods, which have little empirical support in terms of effectiveness. A further block to implementing the evidence-based extensive training approaches is that they are impractical for many therapists and services, not only because of resource constraints but also because many populations have no access to formal CBT training due to absence of local resources. Thus, there is a need to develop scalable methods of disseminating CBT interventions (Fairburn & Cooper, Reference Fairburn and Cooper2011).
In the multi-disciplinary literature regarding transfer of training, training design and content are consistently included as potential determinants of outcome (Baldwin & Ford, Reference Baldwin and Ford1988; Blume et al. Reference Blume, Ford, Baldwin and Huang2009; Herschell et al. Reference Herschell, Kolko, Baumann and Davis2010; Beidas et al. Reference Beidas, Koerner, Weingardt and Kendall2011). Beidas & Kendall (Reference Beidas and Kendall2010) review training studies from a systems-contextual perspective and conclude that the quality of training (i.e. content and method) is critical to success.
The consensus on the importance of training quality, however, is not accompanied by clear evidence as to the specific design, method and content most likely to facilitate successful training outcomes. Beidas & Kendall (Reference Beidas and Kendall2010) suggest that the ‘gold standard’ for training would include a workshop, a manual, and clinical supervision, as was the case in Sholomskas et al. (Reference Sholomskas, Syracuse-Siewert, Rounsaville, Ball, Nuro and Carroll2005). Although this combination of interventions produced significant improvement in clinical skill, the assessment was of only a small subset of skills using standardized role-plays, an outcome that may not be equivalent to the ability to demonstrate competence with a broad range of real patients in RCP. It is therefore not known whether the ‘gold standard’ methods used by Sholomskas et al. (Reference Sholomskas, Syracuse-Siewert, Rounsaville, Ball, Nuro and Carroll2005) would result in successful transfer of EST protocols into RCP.
This description of a gold standard of quality training also does not address the ‘micro’ aspects of the training process (Rakovshik & McManus, Reference Rakovshik and McManus2010): what active elements were present in the workshop or supervision, which modality of training (i.e. workshop, supervision, or self-study) was the most efficacious, and what kind of interactions had the greatest or least impact within these modalities (e.g. reflection, direction instruction from trainers, discussions with peers)?
Aspects of CBT training can be examined from the perspective of smaller units of a learning experience or elements (e.g. a role-play within supervision, theoretical or clinical materials presented by a trainer), larger blocks of training or modalities (e.g. workshop vs. supervision), or sources of learning which are present across individual elements and larger modalities (e.g. from trainer, peers or personal reflection). Bennett-Levy et al. (Reference Bennett-Levy, McManus, Westling and Fennell2009) surveyed 120 therapists about their perceptions of learning during a 2-day workshop and found that the trainees perceived the workshop's varying elements (e.g. lectures/talks, role-plays and reflective practice) to be differentially effective for various kinds of learning (e.g. declarative vs. procedural, reflective and interpersonal skills). It may be helpful, therefore, to understand which aspects of an extensive CBT training course are perceived as most active by trainees on a course which has been shown to significantly increase CBT skills (McManus et al. Reference McManus, Peerbhoy, Larkin and Clark2010b).
As noted in two reviews of the supervision literature (Milne et al. Reference Milne, Aylott, Fitzpatrick and Ellis2008; Beidas & Kendall, Reference Beidas and Kendall2010), available models and theories of clinical supervision lack empirical support and conceptual-methodological rigour. Thus, while the current data and recent reviews (Beidas & Kendall, Reference Beidas and Kendall2010; Rakovshik & McManus, Reference Rakovshik and McManus2010) suggest that supervision promotes confidence and learning, the data are sparse concerning the effectiveness of specific supervision methods or models.
A strategy of investigating the role and nature of supervisory experiences as an aspect of the learning processes and active elements present within the overall training experience may be most fruitful at the current stage of supervision development. This open approach promotes flexibility in how these experiential interactions could be incorporated into diverse training protocols, and would reduce the risk of prematurely endorsing a specific model of supervision before empirical support is gained for that supervision approach per se.
If supervision is regarded as one method of promoting learning within a particular training programme, then its design and function can remain flexible, and can be considered as a focused interaction between trainer and therapist. A recent review of 24 studies by Milne and colleagues (Reference Milne, Aylott, Fitzpatrick and Ellis2008) would seem to support this approach. Their analyses provide an empirically generated inventory of 26 different supervision interventions. The most frequently cited of these was teaching and instruction (75% of studies), and was followed by the use of corrective feedback (reported in 63% of the studies), live or video-based observation of the trainee (42%), goal setting (38%) and information gathering/clarifying (38%). These interactions represent active elements, which may be present in a variety of approaches to supervision.
If we can understand which elements, methods and sources of learning are ‘active’ (i.e. providing the greatest effect on therapist competence) within extensive training, then empirically guided choices could be made when creating more scalable cost-effective training programmes (e.g. internet-based training programmes), thus retaining maximal impact on competence within more accessible and less resource-intensive training programmes. Similarly, in developing effective scalable CBT training protocols it is useful to know if the impact of training varies across different trainee subgroups (Rakovshik & McManus, Reference Rakovshik and McManus2010). For example, is the impact of supervision greater for those with less prior CBT experience? (Beck, Reference Beck1986; Mannix et al. Reference Mannix, Blackburn, Garland, Gracie, Moorey, Reid and J2006). This data will generate hypotheses that can then be tested examining the effects of discrete training elements on competence development (dismantling studies) and effectiveness trials of training protocols.
Aims
This study examines trainees’ perceptions of the impact of elements, modalities and sources of learning on the development of CBT competence, in order to assess which aspects of training are considered most ‘active’ (i.e. perceived as having greater impact on competence).
Predictions
Given the absence of existing data, this study is oriented towards discovery rather than testing specific hypotheses. The aim is that the ratings of the perceived impact of different course elements will be useful for planning different forms of training appropriated to varied circumstances and groups, as well as to establishing preliminary findings from which hypotheses could be drawn for future research. In keeping with the current emphasis on evaluating discrete aspects of learning within larger training experiences (e.g. Milne et al. Reference Milne, Aylott, Fitzpatrick and Ellis2008; Bennett-Levy et al. Reference Bennett-Levy, McManus, Westling and Fennell2009; Rakovshik & McManus, Reference Rakovshik and McManus2010), this study's focus of discovery concerns three approaches to conceptualizing learning within the course, i.e. elements, modalities and sources of learning:
(1) Elements. The perceived impact of different course elements (e.g. theoretical instruction, preparing for supervision, ratings of clinical recordings, informal discussions with peers during breaks) was compared.
(2) Modalities. The perceived impact of the two learning modalities found in this training course was compared [supervision vs. clinical instruction (i.e. workshop format)]. In addition, due to the demonstrated importance of supervision for maintenance of new skills (Mannix et al. Reference Mannix, Blackburn, Garland, Gracie, Moorey, Reid and J2006), comparisons between more and less experienced trainees’ (i.e. lower baseline CBT competence scores and fewer hours of prior CBT training, supervision and practice) endorsements of the perceived impact of supervision are made.
(3) Sources of learning. The perceived impact of interactions with trainers, peers or self-reflection is compared.
Method
Study design
This study has a mixed design, using a survey design for a detailed course evaluation completed by three cohorts of CBT trainees, and observational pre- and post-training CBT competence ratings collected as part of the regular course assessment. The competence ratings provide data regarding the change in trainees’ CBT skill during this course.
The course
The course is a collaboration between Oxford University and the Oxford Cognitive Therapy Centre (part of Oxford Health NHS Foundation Trust) and leads to the award of a master's-level postgraduate diploma in CBT from the University of Oxford. The course is also accredited by the British Association of Behavioural and Cognitive Psychotherapists, and the prospectus and details of the selection criteria and process, and course structure and content, can be found on the Oxford Cognitive Therapy Centre website (http://www.octc.co.uk/university-courses/pg-diploma-in-cognitive-behavioural-therapy). The course is aimed at qualified mental health professionals with most successful applicants being from the core mental health professions of clinical psychology, psychiatric nursing, psychiatry and counselling.
The course consists of 36 days of training over three 12-week terms. Trainees are usually employed in clinical settings and are released to attend the course 1 day a week. Each training day comprises a 90-minute group supervision session (one supervisor with three trainees) followed by a 5-hour workshop. The workshops aim to train clinical skills within the framework of relevant theory and research. The focus is primarily on the application of CBT in clinical practice and demonstrations and experiential role-plays are frequently used.
In addition to attending the 36 training days trainees are expected to set aside a further 6–8 hours per week, throughout the duration of the course, for the private study and preparation that is required for successful completion (preparing for supervision, listening to and reflecting on therapy recordings, completing academic assignments). During the course, they are expected to treat, under close supervision, at least six patients using CBT.
Procedure
Trainees completing a comprehensive and multi-modal CBT training course were surveyed regarding their perceptions of the impact that various elements of the training had had on their competence as CBT practitioners. The questionnaire was completed as part of a course evaluation during the final month of the course. To encourage non-biased responses, trainees were informed that individual responses would only be reviewed by the first author, who had no affiliation with the course. Surveys were distributed and collected by administrative staff, and were not seen by any course staff or those involved with marking trainees’ assignments. Course elements were analysed individually, as modalities, and for sources of learning.
Measures
Competence ratings
Trainees’ CBT skills were assessed via supervisors’ ratings on the Cognitive Therapy Scale (CTS; Young & Beck, Reference Young and Beck1980), based on audio-recordings of therapy sessions. While it is recognized that there are difficulties in defining CBT competence and what constitutes a valid measure of it (Sharpless & Barber, Reference Sharpless and Barber2009), supervisors’ ratings of clinical performance are thought to have the greatest validity because they bear the closest relationship with patient outcomes (Chevron & Rounsaville, Reference Chevron and Rounsaville1983; Kuyken & Tsivrikos, Reference Kuyken and Tsivrikos2009). A thorough review of the CTS and other CBT competence measures can be found in Muse & McManus (Reference Muse and McManus2013).
Course impact questionnaire
A course assessment questionnaire, completed during the final month of the course, asked the participants to rate the perceived impact of 26 different course elements on their clinical competence. The items were rated on a 6-point Likert scale with the following anchors: 0 = none, 1 = minimal, 2 = somewhat, 3 = moderate, 4 = sizable, and 5 = very large. Areas of assessment included theoretical instruction, informal interactions with peers and presenters, supervision, preparing and writing case reports, and self-/expert assessment of recordings of their clinical work on the CTS (a full list can be found in Table 2). Trainees were also asked about basic professional and demographic information, results of which may be found in Table 1.
Trainees’ baseline demographic and clinical characteristics
* Prior to beginning the course.
Mean rating of perceived impact and source of learning for all course elements (in descending order)
CTS, Cognitive Therapy Scale.
Analysis
Comparison of elements, methods and sources
Paired t tests were used to compare trainees’ ratings for the different elements of training. Although multiple comparisons were made between trainees’ ratings, a repeated-measures univariate analyses of variance (ANOVA), a common strategy for controlling the risk of Type I errors (false positive) in multiple comparisons, was not used. This decision was based on the nature of the data and the goals of the study. A repeated-measure ANOVA, often used to control ‘family-wise’ errors (the chance of making at least one Type I error in a set of comparisons), is appropriate for repeated comparisons of data with ordinal levels of measurement (i.e. time, dosage). It allows for an analysis of the overall pattern of data that shows uniform intervals in its pattern of variation, thus confirming a trend (e.g. linear). This kind of analysis is not appropriate for non-ordinal data (Howell, Reference Howell2013) such as comparisons of varied modalities of therapy, drugs (compared to drug dosage, which would have ordinal levels of measurement), or as in this case, elements of training. Instead, a post-hoc correction (i.e. Bonferroni) has been applied to multiple t tests to control the comparison error rate; the adjusted significance level is noted in the results for each multiple t test. To assess the impact of differential levels of CBT experience on endorsements of supervision's impact, non-parametric correlations were calculated between ratings for modalities and variables pertaining to trainees’ baseline (prior to beginning to course) CBT experience level.
Results
Sample characteristics
Trainees (n = 73, response rate 91.3%) had an initial (pre-training) CTS mean item score of 2.95 (s.d. = 0.64), and this increased to 3.83 (s.d. = 0.69) by the final assessment (post-training). There was a significant difference between pre- and post-training CTS mean scores (t 72 = 8.31, p < 0.001), indicating an improvement in trainees’ CBT competence by the end of the course. The majority of trainees (43/73, 58.9%) had mean CTS item scores above the cut-off of 4.0 (the recommended threshold of competence for therapists on clinical trials; Shaw & Dobson, Reference Shaw and Dobson1988; Shaw et al. Reference Shaw, Elkin, Yamaguchi, Olmsted, Vallis, Dobson and Imber1999) on the final assessment. A description of trainees’ baseline characteristics can be found in Table 1.
Course elements
Mean ratings and standard deviations for the perceived effectiveness of elements of learning are shown in Table 2. Notably, the top four elements were all associated with supervision, with ‘supervision in general’ receiving the highest rating. For the purposes of analyses by source of learning, elements were also categorized in terms of the source of learning: ‘trainer’, ‘peer’, ‘self’, or ‘combined’ (Table 2).
Sources of learning
Mean ratings and standard deviations for sources of learning were calculated from course elements. A paired t test yielded a significant difference (p < 0.016, in line with a Bonferroni correction for three statistical tests) between trainer-related learning (mean = 3.76, s.d. = 0.63), personal reflection (mean = 3.50, s.d. = 0.67), and peer-related learning (mean = 3.26, s.d. = 0.78). Trainer interactions were rated significantly higher than personal reflection [mean (s.d.): 3.76 (0.63) vs. 3.50 (0.67); t 67 = 4.60, p < 0.001] or learning from peers (3.26 (0.78); t 67 = 7.50, p < 0.001), and personal reflection received significantly higher ratings than learning from peers (t 67 = 3.25, p = 0.002).
Course modalities (clinical instruction vs. supervision)
Ratings of the perceived effectiveness of clinical instruction (workshop format) vs. supervision (small group format) were compared using a paired t test. Trainees rated supervision significantly more highly than clinical instruction [mean (s.d.): 4.49 (0.75) vs. 3.74 (0.69); t 72 = 8.34, p < 0.001].
Given trainees’ relatively varied exposure to CBT training before beginning the course (e.g. a range of 0 to 300 hours of prior CBT supervision), associations were examined to determine relationships between baseline levels of CBT experience and preferences for a particular modality of instruction. There were no significant correlations found between baseline therapist variables and ratings of instruction or supervision.
Within-supervision sources of learning
Paired t test comparison of the different sources of learning showed significant differences (p < 0.016) between all sources, with a pattern similar to ratings of sources of learning for the course overall. Trainer interactions were rated significantly higher than personal reflection (preparation for supervision and CTS self-ratings), [mean (s.d.): 4.47 (0.67) vs. 4.06 (0.79); t 72 = 5.66, p < 0.001] or learning from peers (observation and direct feedback) [3.51 (0.97); t 72 = 10.27, p < 0.001], and personal reflection received significantly higher ratings than learning from peers (t 72 = 5.52, p < 0.001).
Discussion
Findings
Course element and modalities
There were significant differences between trainees’ perceptions of the ‘activeness’ (impact on development of CBT skills) of different modalities within the course. Learning experiences associated with supervision were most highly rated by trainees, both when looking at discrete elements and when comparing supervision to clinical instruction as more general instruction modalities. This result is consistent with previous findings in which supervision was rated highly in course evaluations (e.g. Westbrook et al. Reference Westbrook, Sedgwick-Taylor, Bennett-Levy, Butler and McManus2008), and adds to these findings by providing a comparison of the perceived impact of clinical instruction vs. supervision. Perhaps more surprising is the finding that ratings for experiencing CBT from a patient's perspective (i.e. trainees’ experiences as patients in peer therapy and ‘using CBT on oneself’) were in the lower third for perceived impact on learning. A recent study by Bennett-Levy & Lee (Reference Bennett-Levy and Lee2012) reported positive endorsement for self-reflection/self-practice by CBT trainees. It may be that this approach is helpful, but is perceived as less effective than other learning experiences. The finding may also be a function of that particular sample of trainees (of 19 trainees, five had no previous experience in delivering therapeutic interventions and eight had no previous exposure to CBT) and may suggest that the perceived benefit of experiencing CBT interventions is stronger for novices.
Sources of learning
Interactions with trainers were rated as having greater impact than interactions with both peers and individual reflection for the courses overall, while individual reflection was rated significantly higher than peer-related learning. Additionally, within-supervision for sources of learning showed the same pattern of trainers > individual reflection > peers. Although trainer-related aspects of supervision were rated most highly (i.e. direct feedback during supervision and supervisors’ CTS ratings and comments), there was a notable finding regarding reflection for supervision: trainees rated preparing for supervision and self-rating therapy sessions to be the fifth and sixth most helpful component in increasing CBT competence, with ratings significantly higher than any peer-related interactions during supervision (i.e. direct peer feedback and vicarious learning through observing peers). This finding brings into question the relative benefits of shared learning in group supervision, and may also suggest that learning from personal reflection is a salient aspect of supervision's perceived impact on competence development.
Factors associated with course ratings
There were no associations between baseline CBT experience and the trainees’ perceptions of course modalities (i.e. workshop vs. supervision format). The lack of association between experience and ratings for supervision was surprising given the evidence for the importance of supervision for consolidation of new skills (Beck, Reference Beck1986; Mannix et al. Reference Mannix, Blackburn, Garland, Gracie, Moorey, Reid and J2006). However, these results may indicate that although more experienced trainees may have less of a need to consolidate novel CBT skills, supervision may still be a highly effective modality for their long-term maintenance (Waller, Reference Waller2009). Alternately, although the range of prior CBT experience for trainees was quite varied (e.g. range of prior training days = 0–118, range of prior supervision hours = 0–300) it may not have been varied enough to detect potential differences.
Relevance to training practices
There are differences in how trainees value course elements, sources of learning, and modality (workshop vs. supervision). These are consistent across levels of previous CBT experience, and may be useful in allocating limited resources (e.g. time, finances) when planning a course. In particular, the reflective aspects of supervision (i.e. rating one's own therapy sessions on a competence scale or preparing for supervision) may develop trainees’ competence without adding additional time or expense to a training programme.
These findings may also indicate which elements of training may or may not be excluded in more scalable forms of training. For example, internet-based training (IBT) may not risk loss of efficacy due to exclusion of peer interaction, but may benefit from the addition of some form of supervision. Although development of CBT competence may require more standard (and resource-intensive) trainer-led supervision, structured self-supervision (akin to preparing for supervision or CTS self-rating in this study) may also be ‘active’ enough to have a significant effect as well. Although the extent to which trainees’ perceptions represent course elements’ actual effect on competence is not established, these ratings can elucidate the potential social validity of non-traditional forms of training such as internet-based programmes.
Limitations
Limitations in the present study have been noted throughout the Discussion, and mainly relate to the current study's methodology. Most importantly, the data presented here represents trainees’ perceptions of their training experiences, rather than an objective measure of the actual impact of differential training approaches on trainee skill. For this reason, it is not known to what extent these rating represent the actual impact of elements, sources and modalities compared to expectations of which aspects of training should have the greatest effect. For example, while it may be true that interactions with trainers had the greatest effect on trainee learning, these perceptions may also be influenced by trainees’ beliefs that trainers have greater expertise and therefore should have a larger impact on changes in CBT competence. It is also possible that trainees were influenced by the timing of the evaluation. Endorsements may have been inflated or restrained depending on trainees’ perceptions of their own success as the course reached its conclusion. Despite the assurance that individual evaluations would not be seen by course markers and administrators, some trainees might have been hesitant to answer honestly before graduation.
Given the study's retrospective design, there are several issues limiting the conclusions that can be drawn both for the current sample and for the external validity of the results. This study was not designed to test specific hypotheses, but took a pragmatic approach in examining existing competence data and results of a course evaluation questionnaire in order to form tentative hypotheses for future research. The population sampled (therapists with the resources and motivation to undergo extensive training) most likely does not represent the broader range of practitioners who might attend briefer forms of CBT training, or those undertaking training in other contexts.
Future research
It is hoped that results from the current study will guide future research in providing more definitive conclusions regarding active elements of training through prospective hypothesis-testing studies, such as dismantling studies in which the efficacy of discrete elements are compared, or when the impact of particular training interventions is compared across different trainee subgroups (e.g. more or less prior CBT experience). In this way, future research can address the actual impact of differential training inventions, compared to the self-reported perceptions of trainees represented by the current study. Results of IBT training trials also may show whether training can remain effective when peer interactions are absent.
Acknowledgements
The authors thank Emily Lodge and Jaclyn Voskoboinikov for their help as research assistants on this project.
Declaration of Interest
None.
(1) To be aware of the current status of empirically based training.
(2) To understand trainees’ differential endorsement of effect on competence by discrete aspects of an extensive CBT training programme.
(3) To describe the relevance of these finding to non-traditional forms of CBT training.
Comments
No Comments have been published for this article.