Introduction
Despite a limited body of evidence for the empirical benefits of supervision and training (e.g. Ellis and Ladany, Reference Ellis, Ladany and Watkins1996; Milne and James, Reference Milne and James2000), good-quality randomized trials of psychological therapy ensure that therapists not only adhere to treatment protocols but also deliver the therapy in a competent manner. Without this, internal validity can be compromised, limiting clinicians’ and researchers’ ability to attribute any differences in client outcomes to the treatment they received. The impact of deviations in treatment fidelity is widely recognized, and well-conducted trials take steps to address this, for example, by recruiting experienced therapists, ensuring that training is offered before the study starts, offering ongoing supervision and monitoring the delivery of interventions (Roth and Fonagy, Reference Roth and Fonagy2005).
Elkin (Reference Elkin1999) has argued that trial protocols should clearly specify the training and supervision offered to therapists. Although prominent features of trial design (and hence the delivery of therapy), these elements can be neglected or even forgotten when evidence-based therapies are transported into routine clinical settings (where their efficacy may be attenuated relative to the research context). This might be less likely if services in routine settings could easily identify the extent of training and supervision found in research trials, and hence assess the similarities and differences between their own service characteristics and those operative in research contexts. In this way they could identify any shortfalls in training and supervision in their existing procedures. However, to do this they would need a “minimum dataset” giving information about:
• The background training of therapists participating in trials
• The training therapists receive in preparation for the trial
• The supervision received during the trial
• The ways in which therapist performance in clinical sessions is monitored, and the uses to which this monitoring is put
There are two broad aims to this study. First, to determine whether this dataset was readily available in research reports and, second, to identify the extent of training and supervision offered in clinical trials.
Method
Design
Our research question is not best addressed by a systematic review, since the inclusion criteria are unrestricted (in effect, we would need to include all trials in which a psychological therapy has been shown to be effective). Instead, we have modified the “best evidence synthesis” method (Slavin, Reference Slavin1995) by identifying a subsample of “landmark” trials that have established the evidence for the efficacy of a specific therapy, thereby increasing the likelihood that routine clinical services would import these therapies into their own practice. We have further delimited sampling in order to take advantage of the identification of just such a set of “exemplary” trials as part of a project that mapped the competences used to deliver high-quality CBT for people presenting with depression or anxiety disorders (Roth and Pilling, Reference Roth and Pilling2007, Reference Roth and Pilling2008). Our criteria for inclusion were that:
a) The trial methodology assured high levels of internal validity
b) The therapy was delivered to a high standard
c) The trial has had a major impact in the fields of research and clinical practice.
We initially identified a set of 32 key trials of CBT for depression and anxiety based on a re-examination of databases from two sets of high-quality systematic reviews:
a) The database of trials of CBT located and considered by the guideline developers for the NICE Guidelines on depression and anxiety (NICE, 2004a, 2004b, 2005, 2006).
b) Roth and Fonagy's (Reference Roth and Fonagy2005) review of research on the efficacy of psychological therapies.
The “long list” that emerged from this process was appraised by an Expert Reference Group (ERG) overseeing the competence project (group members were selected for their expertise both as researchers, trainers and treatment innovators in CBTFootnote 1). They were asked to focus on the scientific rigour and clinical impact of the trials, a process that explicitly relied on both expert knowledge and consensus; on this basis the ERG reduced the long list to a sample of 27 exemplar trials.
Included studies
Of the 27 trials (listed at the end of this paper) seven focused on depression, six on generalized anxiety disorder (GAD), five on social phobia, three on panic disorder and six on PTSD (hence covering a broad range of presentations and including most anxiety disorders).
Extraction of information about training and supervision
Initially information was extracted from the primary paper(s) reporting the trial. We attempted to extract the following information:
• The qualifications and the previous experience of the therapists, including any prior experience and training in CBT
• The training and ongoing supervision provided in the trial, including the content of training (for example, training could focus on broad therapeutic competences in CBT, or have the specific aim of ensuring treatment fidelity)
• Whether there were procedures for observing and monitoring treatment adherence and/or fidelity, and the uses to which this information was put.
Where papers did not give sufficient detail to extract this information, the primary authors were contacted (resulting in additional information for 17 of the 27 trials)Footnote 2. Initial analysis and tabulation were carried out by JT. Papers were re-examined by AR to confirm the accuracy of extraction, and a framework for coding adherence procedures was developed through discussion between JT, AR and SP (based on an iterative review of the papers). This led to the emergence of a set of categories that captured the range of procedures used to enhance adherence and fidelity.
Results
Background training and experience of therapists
Almost invariably, trials recruited therapists with prior experience of the therapies they were expected to deliver; and almost all were clinicians with a master's or doctoral level of training. Only three trials (Barlow, Rapee and Brown, Reference Barlow, Rapee and Brown1992; Borkovec and Costello, Reference Borkovec and Costello1993; Heimberg et al., Reference Heimberg, Dodge, Hope, Kennedy, Zollo and Becker1990) included therapists without an advanced clinical qualification, although in all cases these were identified as senior or advanced doctoral students. Fourteen of the 27 trials gave data on the length of time therapists had been practising, but this is not easily summarized because of variations in reporting. As can be seen in Table 1, five trials gave the range of experience (a mean of between 2.4 and 20.8 years), three declared the mean number of years of prior experience (a mean of 9.1 years), and four indicated the minimum amount of therapist experience (a mean of 3.0 years). Of the remaining two trials, one had two therapists (with 10.0 and 2.0 years’ experience; Durham et al., Reference Durham, Murphy, Allan, Richard, Trevling and Fenton1994) and one a single therapist (with 7.0 years’ experience; Sharp et al., Reference Sharp, Power, Simpson, Swanson, Moodie, Anstee and Ashford1996).
Extent of prior experience of therapists
*In studies where prior therapist experience was reported as a range, the average number of years of minimum experience was 2.4 years and of maximum experience 20.8 years.
Clearly most therapists have “generic” experience, and information from 19 of the 27 trials indicated that they were specifically experienced in CBT. Seven trials did not give any information on this point. Only one trial explicitly stated that not all therapists were experienced in CBT before the trial started (Barlow, Gorman, Shear and Woods, Reference Barlow, Gorman, Shear and Woods2000).
In a number of cases, the extent of previous training and experience was considerable. For example, Elkin et al. (Reference Elkin, Shea, Watkins, Imber, Sotsky, Collins, Glass, Pilkonis, Leber, Docherty, Fiester and Parloff1989) required therapists to meet specific competence criteria before enrolling in the trial (through interview and assessment of a previously video-taped session with clients). In some cases, trial therapists were also involved in developing the treatment protocol, or had been involved as supervisors in previous trials of the therapy being tested, making it reasonable to assume that they were highly competent in the procedures they were evaluating (for example, Ehlers et al., Reference Ehlers, Clark, Hackmann, McManus, Fennell, Herbert and Mayou2003; Ehlers, Clark, Hackmann, McManus and Fennell, Reference Ehlers, Clark, Hackmann, McManus and Fennell2005; Ladouceur et al., Reference Ladouceur, Dugas, Freeston, Leger, Gagnon and Thibodeau2000). Overall it is clear that most trials are recruiting clinicians with extensive and high-level background training, and with a good level of prior experience in CBT.
Training offered specifically for the trial
Twenty-one studies included detailed information on the training provided in order to equip therapists with relevant skills. In the remaining trials, therapists were already highly experienced and hence no further training was deemed to be required.
Only eight trials indicated the length of training, which varied widely from 1 day to 16 months. In many instances trials employed experienced therapists and offered them only brief training, but where they were inexperienced training could be extensive. For example, Hollon et al. (Reference Hollon, DeRubeis, Evans, Wiemer, Garvey, Grove and Tuason1992) and Jarrett et al. (Reference Jarrett, Schaffer, McIntire, Witt-Browder, Kraft and Risser1999, Reference Jarrett, Kraft, Doyle, Foster, Eaves and Silver2001) trained a cadre of inexperienced therapists over a period of 14 to 16 months, selecting for the trial only those who were able to demonstrate competence.
The initial training methods varied widely but usually included a mix of didactic teaching and video case examples, working from manuals and role playing. As might be expected, trials employing less experienced therapists focused on teaching basic therapeutic principles and familiarizing the therapists with the CBT model (for example, Dimidjian et al., Reference Dimidjian, Hollon, Dobson, Schmaling, Kohlenberg, Addis, Gallop, McGlinchey, Markley, Gollan, Atkins, Dunne and Jacobson2006). Conversely, trials that recruited experienced therapists focused on assuring treatment fidelity in each arm of the trial (for example, Butler, Fennell, Robson and Gelder, Reference Butler, Fennell, Robson and Gelder1991).
In 16 of the 21 studies that included a training component, therapists were required to demonstrate competence, based on formal evaluation by members of the research group, before participating in the trial. Thirteen trials also noted that therapists needed to treat “pilot cases” before the trial proper (a range of 2 and 10).
Supervision during the trial
Almost universally, trials reported that therapists received regular model-specific supervision throughout the trial; only Ost and Breitholtz (Reference Öst and Breitholtz2000) did not explicitly do so. However, not all studies detailed the format or frequency of supervision. Fourteen trials indicated that therapists received individual supervision; in four of these, therapists received additional peer or group supervision. Sixteen trials reported on supervision frequency; in 12 cases supervision was weekly, in two fortnightly, and in another two bi-weekly. In addition to supervision, some trials reported that there were also periodic meetings involving researchers, therapists and supervisors focused on adherence and fidelity (DeRubeis et al., Reference DeRubeis, Hollon, Amsterdam, Shelton, Young, Salomon, O'Reardon, Lovett, Gladis, Brown and Gallop2005; Dimidjian et al., Reference Dimidjian, Hollon, Dobson, Schmaling, Kohlenberg, Addis, Gallop, McGlinchey, Markley, Gollan, Atkins, Dunne and Jacobson2006).
Use of treatment manuals. In all cases, interventions were based on a specified intervention model. Nineteen trials cited a published treatment manual, and two referred to unpublished manuals prepared for the trial. Although the remaining six studies did not refer explicitly to a manual, they did cite published sources that outlined a clear model of treatment.
Observation and monitoring of therapist performance. Twenty-five of the 27 studies recorded therapy sessions and monitored and assessed therapist adherence (and sometimes competence). Seven trials used a standardized measure of CBT competence (either the Cognitive Therapy Rating Scale or the Collaborative Study Rating Scale); 10 trials based ratings on study-specific adherence checklists. Seven reported that supervisors observed recordings, but did not indicate whether they used any systematic procedure for doing so. In one trial (Cottraux et al., Reference Cottraux, Note, Albuisson, Yao, Note, Mollard, Bonasse, Jalenques, Guérin and Coudert2000) therapists completed a checklist based on the manual, indicating the techniques they used in sessions. Only two trials did not report any checks on adherence and competence (Durham et al., Reference Durham, Murphy, Allan, Richard, Trevling and Fenton1994; Ost et al., Reference Öst and Breitholtz2000). Few trials reported on how systematic these integrity checks were. Some indicated that tapes were “randomly selected”, although the number of sessions reviewed was not specified frequently enough for any clear pattern to emerge. In some instances the results of detailed integrity checks were reported (for example, Barlow et al., Reference Barlow, Rapee and Brown1992; Elkin et al., Reference Elkin, Shea, Watkins, Imber, Sotsky, Collins, Glass, Pilkonis, Leber, Docherty, Fiester and Parloff1989; Foa et al., Reference Foa, Dancu, Hembree, Jaycox, Meadows and Street1999; Jarrett et al.,Reference Jarrett, Schaffer, McIntire, Witt-Browder, Kraft and Risser1999; Ladouceur et al., Reference Ladouceur, Dugas, Freeston, Leger, Gagnon and Thibodeau2000; Monson et al., Reference Monson, Schnurr, Resick, Friedman, Young-Xu and Stevens2006; Resick, Nishith, Weaver, Astin and Feuer, Reference Resick, Nishith, Weaver, Astin and Feuer2002).
Discussion
What emerges from this review is a picture of the major investment in training and supervision made by researchers in high-quality trials. Therapists are almost invariably carefully selected, trained in a specific and well-specified set of interventions, supervised intensively, and monitored closely, usually on the basis of tape-recordings. The consistency with which this coherent “package” of activities appears across the trials we have reviewed makes clear that, for researchers, implementing treatment protocols entails much more than simply “following a treatment manual”.
This observation is not surprising – at least for researchers – but it has important implications for the way in which routine services transport therapies from a research setting, especially if they wish to implement an evidence-based therapy. Our point is a simple one: the manual or protocol is only one part of the package which, when implemented in a trial, resulted in good clinical outcomes for patients. What has actually been demonstrated is the impact of the therapeutic intervention in the context of dedicated training and supervision for trial therapists. This strongly suggests that services implementing evidence-based practice need to mirror (or at least closely approximate) the training and supervision that enabled the intervention to be delivered effectively in the research context. Supporting this contention, studies that have evaluated attempts to “transport” evidence-based CBT protocols into a routine service setting and that have found effect-sizes comparable to those obtained in studies of efficacy have invariably taken care to include comprehensive packages of training and supervision (for example, Brewin, d'Ardenne and Ehlers, Reference Brewin, d'Ardenne and Ehlers2008; Gillespie, Duffy, Hackmann and Clark, Reference Gillespie, Duffy, Hackmann and Clark2002; Lincoln et al., Reference Lincoln, Rief, Hahlweg, Frank, von Witzleben, Schroeder and Fiegenbaum2003; Persons, Roberts, Zalecki and Brechwald, Reference Persons, Roberts, Zalecki and Brechwald2006; Shultz, Resick, Huber and Griffin, Reference Shultz, Resick, Huber and Griffin2006; Wade, Treat and Stuart, Reference Wade, Treat and Stuart1998).
If we are correct in asserting that training and supervision cannot be ignored as a component of evidence-based treatments, how is it that, as Elkin (Reference Elkin1999) has observed, this simple point is easily overlooked? The challenge of conducting this review tells its own story: information about training and supervision is presented inconsistently from paper to paper, and sometimes only in outline form, even where it is clear that extensive training and supervision must have taken place.
A change in journal policies for reporting trials could improve this situation. For example, Bellg et al. (Reference Bellg, Borrelli, Resnick, Hecht, Minicucci, Ory, Ogedegbe, Orwig, Ernst and Czajkowski2004) have identified a framework for appraising treatment fidelity at each stage of the research process, and have suggested that trials should report on how they achieve this in a manner analogous to the CONSORT system for reporting participant flows through a trial (Altman et al., Reference Altman, Schulz, Moher, Egger, Davidoff, Elbourne, Gøtzsche and Lang2001). If this were to be adopted as standard practice, clearer statements about therapist selection, training and supervision would become a routine feature of trial reporting, and this in turn could lead to improved evaluation of RCTs of psychological interventions in systematic reviews and meta-analyses (Moher et al., Reference Moher, Cook, Eastwood, Olkin, Rennie and Stroup1999). (Such an approach could have value beyond psychological treatment trials and be applied to a wide range of complex healthcare interventions where practitioner competence has a key bearing on the outcome of the intervention.) Our own suggestions for a set of reporting standards are shown in Table 2, based on the coding scheme that emerged from our review.
Minimum standards for reporting training and supervision in trials
The emphasis on the potential benefits of training and supervision for a specific set of techniques could be seen as controversial, since there is an historical and ongoing debate arguing against privileging technique over the many other sources of outcome variance (for example, Norcross, Reference Norcross1995; Beutler, Reference Beutler2002). In anticipation of this objection, it is important to be clear that whatever the status of the “dodo-bird” verdict (that all therapies have equivalent outcomes), equivalence of outcome does not imply the irrelevance of technique, only whether one set of techniques can claim advantage over another. The expectation that clinicians practise in a competent manner is axiomatic, especially so if “technique” is not defined narrowly (and seen as synonymous with the rote delivery of a set of protocolized procedures). We have developed an evidence-based specification of the competences required to conduct psychological therapy in the domains of CBT, psychodynamic, systemic and (broadly configured) humanistic therapy (Roth and Pilling, Reference Roth and Pilling2007, Reference Roth and Pilling2008; forthcoming a, b, c), along with a competence-based framework for the supervision of these therapies (Roth and Pilling, forthcoming d). All these frameworks have a common base of generic therapeutic competences, many of which emphasize the crucial importance of attending to interpersonal elements of therapy (for example, through the therapeutic alliance). Further, all require therapists to implement therapy using procedural knowledge, which enables them to deploy clinical judgments about how therapy is best implemented in relation to the individual client. The important point is that therapists need to have a rationale for deciding on a particular course of therapeutic action and be confident that they are implementing therapeutic procedures in an appropriate and skilled manner. While training and ongoing supervision are key to achieving this, our contention is that this may not be routinely provided and could well account for the reports of attenuation of effect in routine practice (for example, Roth and Fonagy, Reference Roth and Fonagy2005).
Direct evidence of the benefit of training and supervision on client outcome is limited, perhaps in part because demonstrating this association is methodologically challenging (for example, Freitas, Reference Freitas2002), However, recent studies have indicated the benefits of training on skill level (for example, Milne, Westerman and Hanner, Reference Milne, Westerman and Hanner2002; Milne, Pilkington and James, Reference Milne, Pilkington and James2003; Mannix et al., Reference Mannix, Blackburn, Garland, Gracie, Moorey, Reid, Standart and Scott2006), improvements in outcome consequent on training and supervision (for example, Bradshaw, Butterworth and Mairs, Reference Bradshaw, Butterworth and Mairs2007; Callahan, Almstrom, Swift, Borja and Heath, Reference Callahan, Almstrom, Swift, Borja and Heath2009; Grey, Salkovskis, Quigley, Clark and Ehlers, Reference Grey, Salkovskis, Quigley, Clark and Ehlers2008) and also the impact of supervision on adherence and in turn outcome when therapies are transported into routine settings (for example, Schoenwald, Chapman, Sheidow and Carter Reference Schoenwald, Chapman, Sheidow and Carter2009; Schoenwald, Sheidow and Chapman, Reference Schoenwald, Sheidow and Chapman2009). Clearly more research is required, but if it is the case that client outcomes are enhanced by matching levels of supervision and training to those available in clinical trials then this needs to be recognized in the commissioning of routine clinical services – as is the case, uniquely in the NHS, in the Improving Access to Psychological Therapies programme (Department of Health, 2008). Providing the training and supervision embodied in demonstrations of the efficacy of psychological therapies should be routine and a matter of systemic, rather than ad hoc, planning; this clearly has resource implications, but neglecting to attend to these elements may have substantive consequences for the effectiveness and cost-effectiveness of services and, most importantly, a direct and negative impact on the welfare of clients.
Comments
No Comments have been published for this article.