Introduction
The assessment of treatment integrity plays a crucial role in psychotherapy research, as it is needed not only to improve the internal validity of intervention studies (Moncher and Prinz, Reference Moncher and Prinz1991), but also enables researchers to determine the amount of variance in treatment outcome that is attributable to specific factors (Webb, Rubeis and Barber, Reference Webb, De Rubeis and Barber2010) and to explore and improve the effectiveness of treatments, quality of training (Miller and Binder, Reference Miller and Binder2002; Stein and Lambert, Reference Stein and Lambert1995) and practical application of empirically supported treatments (Shafran et al., Reference Shafran, Clark, Fairburn, Arntz, Barlow and Ehlers2009).
Checking for treatment integrity should thus become a standard procedure in intervention studies and routine clinical practice (Bellg et al., Reference Bellg, Resnick, Minicucci, Ogedegbe, Ernst and Borelli2004). However, there have so far been few investigations that have systematically addressed treatment-integrity procedures (Perepletchikova, Treat and Kazdin, Reference Perepletchikova, Treat and Kazdin2007). This research gap is presumably due to the enormous effort and resources required to conceptualize, implement, measure, and evaluate treatment integrity (Waltz, Addis, Koerner and Jacobson, Reference Waltz, Addis, Koerner and Jacobson1993; Barber, Triffleman and Marmar, Reference Barber, Sharpless, Klostermann and McCarthy2007).
Treatment integrity comprises three components: treatment adherence, therapist competence, and treatment differentiation. The latter refers to whether the treatments under investigation differ from one another with regard to various critical aspects. Adherence scales can be used to measure the degree of differentiation between treatments, provided that they have been designed for this purpose (Perepletchikova and Kazdin, Reference Perepletchikova and Kazdin2005). Adherence refers to the extent to which therapist behaviour complies with the treatment standards specified in a particular manual, whereas competence refers to the level of therapist skill in delivering the prescribed treatment. Barber, Sharpless, Klostermann and McCarthy (Reference Barber, Sharpless, Klostermann and McCarthy2007) point out that empirical investigations of competence do not assess universal expressions of competence across treatments (global competency), but rather are confined to measuring the specific application of psychotherapeutic techniques in limited domains. Consequently, it is important to derive valid conceptions of such domain-limited intervention competence from treatment manuals and the corresponding theories of change (Waltz et al., Reference Waltz, Addis, Koerner and Jacobson1993).
The Cognitive Therapy Scale (CTS; Young and Beck, Reference Young and Beck1980) and its various revisions are the most frequently used tools for assessing competence in cognitive therapy (Barber, Sharpless et al., Reference Barber, Triffleman and Marmar2007). The scale was first developed to evaluate therapist competence in cognitive therapy for depression and was subsequently modified for other disorders (Haddock et al., Reference Haddock, Devane, Bradshaw, McGovern, Tarrier and Kinderman2001). In our study, we adapted the measurement of treatment integrity to recent developments in cognitive therapy for social phobia (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003, Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006), based on the cognitive model of Clark and Wells (Reference Clark, Wells, Heinberg, Liebowitz, Hope and Schneier1995). Interventions based on this model include deriving a cognitive model of social phobia, conducting role-play-based behavioural experiments to reduce self-focused attention and safety behaviours, video feedback and imagery for restructuring distorted self-imagery, and in-vivo behavioural experiments to test negative beliefs in anxiety-provoking social situations (Clark and Wells, Reference Clark, Wells, Heinberg, Liebowitz, Hope and Schneier1995; Stangier, Clark and Ehlers, Reference Stangier, Clark and Ehlers2006). To assess the competence with which therapists delivered this cognitive therapy programme, we used the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP; Clark, Consbruch, Hinrichs and Stangier; Reference Clark, Consbruch, Hinrichs and Stangier2007), an adapted version of the CTS. In the present study, we present data on the interrater and the retest reliability of the CTCS-SP, in order to determine whether it is sufficiently reliable for use in research studies, clinical practice and the evaluation of training.
Method
Treatment and therapists
The data for this paper were obtained from a major German clinical multicentre trial - the SOPHO-NET study - that compared supportive-expressive therapy (SET) and cognitive therapy (CT) in the treatment of social phobia (Leichsenring et al., Reference Leichsenring, Hoyer, Beutel, Herpetz, Hiller and Irle2009), with each set of treatments comprising 25 sessions. In the CT condition, therapists were trained in cognitive therapy for social phobia, based on a manual describing the specific interventions (Stangier et al., Reference Stangier, Clark and Ehlers2006). During the training period, each therapist treated at least one patient diagnosed with social phobia and received intensive group supervision. Therapists were requested to submit at least two videotapes of treatment sessions during the training phase. Only those therapists who adhered sufficiently to the manual and proved competent in delivering treatment - as evaluated by an external rater - were allowed to enter the study as therapists.
Measures
The Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP; Clark et al., Reference Clark, Consbruch, Hinrichs and Stangier2007) is an adaptation of a competence scale for assessing CT for panic disorder (Clark, Reference Clark2002), which itself was an adaptation of Young and Beck's (Reference Young and Beck1980) more depression-focused cognitive therapy scale (CTS). The two anxiety disorder scales include some common items that were considered important in the treatment of anxiety disorders in general, plus some changes in wording that are intended to more precisely capture the key features of CT for particular anxiety disorders (e.g. inducing physical symptoms in panic disorder, using video feedback in social phobia). For our study, a German version of the scale was used. Additionally, the item “Resource orientation” was included, so as to measure a common factor in psychotherapy. Stangier, Consbruch, Schramm and Heidenreich (Reference Stangier, Consbruch, Schramm and Heidenreich2010) found that in a study comparing CT and interpersonal psychotherapy (IPT) in the treatment of social anxiety disorder, CT therapists reported greater use of resource activation than IPT therapists, which furthermore predicted therapeutic outcome, even though no additional resource activating interventions were implemented. As found by Stangier et al. (Reference Stangier, Consbruch, Schramm and Heidenreich2010), we also expect sufficient levels of resource activation in our trial, i.e. we expect the therapist to enable the patient to be aware of the patient's positive characteristics and skills and focus on how these can be used to reach self-set goals in therapy.
The final scale consisted of 16 items: agenda; dealing with questions, objections, problems; clarity of communication; pacing and efficient use of time; interpersonal effectiveness; resource orientation; review of social-phobia diary, questionnaires and other measures; reviewing previously set homework; use of feedback and summaries; guided discovery; focus on social-phobia-related cognitions, self-focused attention, safety-behaviours, and biased imagery (cognitive model); rationale; selection of appropriate strategies for change in social phobia related cognition and maintaining factors; appropriate implementation of techniques for change in social phobia related cognition and maintaining factors; integration of discussion and experiential techniques; homework setting. In addition to these 16 items of the scale, observers were asked to rate two more items: one serving as an estimate of overall session-competence, which was included to examine whether the overall impression of competence can be measured with sufficient reliability, and one item measuring the degree of difficulty associated with working with the particular client, which was included to focus the rater on the needs of the particular client when assessing therapist competence. Both items were excluded, when the total CTCS score was calculated.
Each item was rated on a 7-point Likert-type scale, with ratings of 0 indicating poor therapist performance and ratings of 6 indicating excellent competence. In the pilot phase, therapists had to achieve at least Level 3 on the total CTCS-score on at least two tapes to be included in the study. Level 3 was considered a reasonable minimum target bearing in mind the prior experience of the trainees. It was recognized that this was slightly lower than the minimum of 4 recommended by Clark (Reference Clark2002). During the trial, therapists received feedback and additional supervision whenever the therapeutic quality fell below this level.
Raters
Seven psychologists (qualified to at least Masters level) served as raters for this study. Six of the seven were clinical psychologists in training and one was a licensed clinical psychotherapist. All of the raters had received additional training and supervision in cognitive therapy for social phobia by US. Manual-based training (Consbruch, Clark and Stangier, Reference Consbruch, Clark and Stangier2007) was conducted to ensure a satisfactory level of interrater reliability. During 2 days of training, each item of the scale was explained and demonstrated using videotaped therapy sessions, and the ratings were discussed in detail. Before rating the relevant video sessions, the raters were required to evaluate a further five pilot tapes and to discuss discrepant ratings. Throughout the training period, the raters met continuously to discuss tapes.
Materials
All videotaped therapy sessions were collected and catalogued. For the purposes of quality assurance, the treatment of every second patient from each therapist was evaluated by rating two different therapy sessions. These sessions were drawn randomly from three different therapy phases; from the initial sessions (1–3), the middle sessions (10–12), or the final sessions (20–22) of therapy. For the current analysis, a total of 161 videotaped sessions that included 51 therapists and 98 patients was selected randomly. From each of 63 treatments, two sessions were selected, and from each of 35 treatments, one session was selected (see Table 1). While most sessions took 50 minutes, those including in vivo behavioural experiments lasted 100 minutes. One of the raters (KvC, “standard rater”) assessed all the tapes, whereas the other raters assessed only some of the tapes. Interrater reliability was assessed by pairing the ratings of the first rater with the other raters.
Descriptive data on the video samples for reliability calculations
Statistical analysis
Interrater-reliability coefficients were determined by calculating ICC (2,n; Shrout and Fleiss, Reference Shrout and Fleiss1979), where the rater was considered to be a fixed effect and n was the number of raters. Absolute consistency between raters was required. In the current study, ICC (2,2) was thus an estimate of how two raters performed. Cronbach alpha coefficients were computed to determine the internal consistency of the full scale. Scores were summated across the different raters within each rater pair and across rating occasions for the standard rater.
In order to determine the retest reliability, 15 video sessions drawn randomly from the sample of treatments were re-assessed. To minimize memory effects, only those videos were analyzed that had been evaluated during the first 6 months of the study period. The time interval between test and retest assessment ranged between 18 and 24 months.
Results
High interrater reliability was observed for the total CTCS score (range .73–.88). While ICCs for individual items ranged from relatively low to high values (–.06 to .98), no item yielded poor interrater reliability across all rater pairs (see Table 2). To obtain information on the reliability of our ratings based on the whole sample, we calculated the ICC, thus capturing consistency between the standard rater and the other raters. While aggregation in this calculation still occurred across no more than two raters, ICCs in the complete sample were found to be high, even for individual items (ranging from .62 to .92.). The items with the highest reliabilities were “review of social phobia diary” and “review of previously set homework”; the lowest rates were observed for “feedback and summary”, dealing with problems”, and “rationale”.
Intraclass correlations* of six different pairs of raters, global reliability indices, test-retest reliability, and internal consistency of the CTCS-SP
* all correlations in bold type are significant
High retest reliability was observed for the total CTCS score (.92), while ICCs for individual items (.55–.96) range from moderate to substantial. As can be seen in Table 2, there was a wide fluctuation between pairs of raters. The internal consistencies proved to be very high, with Cronbach's alpha ranging from .82 to .92. Cronbach's alpha, if an item was deleted, was not significantly higher than the overall alpha for any item; consequently, no item was removed from the scale.
Discussion
The results of the present study indicate that the CTCS-SP has high interrater reliability and good internal consistency. Interrater reliability in the current study compares positively with that found for other measures of competence in cognitive therapy (Table 3). Vallis, Shaw and Dobson (Reference Vallis, Shaw and Dobson1986) have shown that the CTS can be used with moderate interrater reliability (ICC = .59), as replicated by James, Blackburn, Milne and Reichfelt (Reference James, Blackburn, Milne and Reichfelt2001). The unusually high ICCs of .83-.89 in Bryant, Simons and Thase (Reference Bryant, Simons and Thase1999) and of .94 from Haddock et al. (Reference Haddock, Devane, Bradshaw, McGovern, Tarrier and Kinderman2001) seem to be due to the high number of raters used to calculate reliability rates (aggregation). Furthermore, Haddock et al. (Reference Haddock, Devane, Bradshaw, McGovern, Tarrier and Kinderman2001) asked their raters to evaluate five videotapes, which were preselected, in order to ensure a wide range of competence scores. Thus artificially increased variance might have lead to elevated ICC.
Reliability of cognitive therapy competence measures
* CTACS = Cognitive Therapy Adherence and Competence Scale; CTS = Cognitive Therapy Scale; MACT = Manual assisted Cognitive Behavioural Therapy Rating Scale; CTS-Psy = Cognitive Therapy Scale for Psychosis; CTS-R = Cognitive Therapy Scale-Revised; STCRF = Short Term Anxiety Provoking Psychotherapy - Therapist Competence Rating Form **Based on the study by Blackburn et al. (Reference Blackburn, James, Milne, Baker, Standart and Garland2001), the scale was revised, and is currently used in the 12-item version.
While high interrater reliability was observed in all rater pairs for overall scores, the reliability of individual items varied from low (–.06) to high (.98). Discussions of diverging ratings revealed that raters often agreed on perceptions of less competent aspects of therapist behaviour, but attributed the observed behaviours to different dimensions of competence. A possible explanation of this phenomenon might be that, given the interdependence of the components of therapeutic competence, raters might differ in their implicit or explicit assumptions about the relationships between the components. For instance, some raters might attempt to evaluate “guided discovery” and “clarity of communication” independently, whereas others might base their ratings on the assumption that guided discovery may not be performed to a high standard when the clarity of communication is inappropriate. However, despite the low reliabilities for some of the items, the high internal consistencies suggest a homogenous construct, with Cronbach's alphas proving to be very high (.82-.92). Internal consistencies for the competence measures, however, have also been high in other studies (Barber, Liese and Abrams, Reference Barber, Liese and Abrams2003; James et al., Reference James, Blackburn, Milne and Reichfelt2001; Vallis et al., Reference Vallis, Shaw and Dobson1986). Nonetheless, beyond the definitional overlap of individual items, the high inter-item correlation may also be due to rater bias, such as the influence of perceptions of specific therapist behaviour by ratings that had previously been attributed to other behaviours.
Ratings did not prove reliable when those of only single raters were considered (results for single raters not reported). In a study by Vallis et al. (Reference Vallis, Shaw and Dobson1986), ICCs did not exceed .59 for the ratings of just one rater, but the authors did demonstrate that aggregation can increase reliability to ICC = .73 when two raters are used, or even to ICC = .84 when three are used. As aggregation can be used to increase reliability (Barber, Sharpless et al., Reference Barber, Sharpless, Klostermann and McCarthy2007), we decided, in accordance with current approaches (see Table 3), to aggregate the ratings of two judges.
While the reliability of a number of items failed to reach a satisfactory level when ICCs were calculated for individual rater-pairs, interrater reliability for the entire sample of raters was found to be high, even for individual items. These higher coefficients may be due to the larger sample size, which potentially minimizes deviations caused by outliers and/or to the greater variance in therapist performance in the larger sample (N = 161). In the small samples (N = 15–60), used for the individual calculations, the target behaviours yielded less variance, which may have resulted in lower coefficients. The results for the total sample indicate that not only total scores but also individual items may be reliable enough to warrant examination of their predictive validity. To the best of our knowledge, this has not previously been demonstrated in any other study. Items with relatively low reliabilities (e.g. use of feedback and summaries; dealing with questions, objections, problems; rationale) were not removed from the scale, in order to facilitate more accurate comparisons with the results of studies using the original CTS. In future research, the items in questions should be operationalized in greater detail.
Another drawback of our study is that, at times, raters had to assess the same therapist more than once and also assess two videos from the same patient. Therefore, as in similar studies (Bryant et al., Reference Bryant, Simons and Thase1999; James et al., Reference James, Blackburn, Milne and Reichfelt2001; Svartberg, Reference Svartberg1999), the data set is not independent and we cannot rule out rater confounds.
As recommended by Barber, Sharpless et al. (Reference Barber, Triffleman and Marmar2007), we devoted considerable effort to training our judges, doing so across sites to avoid systemic differences, and we aggregated multiple scores of judges, all of which seems to have increased reliability rates to a satisfactory level. Our results suggest that the multidimensional assessment of competence is necessary in order to obtain high reliability rates (ICC for total CTCS score = .81) and should not be substituted by simple estimations of therapist competence (ICC for estimation of overall competence = .70). Nevertheless, the high level of effort associated with the assessment of competence, such as the availability of trained raters, time considerations, and the expenses incurred by such research projects (Waltz et al., Reference Waltz, Addis, Koerner and Jacobson1993), continue to be an issue. Further examination of the CTCS-SP should include the question of whether reliable and valid ratings can be obtained by evaluating less than complete sessions (e.g. time-segments of 15 min), with the aim of reducing the time needed for evaluation (Castonguay, Goldfried, Wiser, Raue and Hayes, Reference Castonguay, Goldfried, Wiser, Raue and Hayes1996) and/or by using raters with less expertise (undergraduates or bachelors-level students), so as to reduce costs.
Wu, Whiteside and Neighbors (Reference Wu, Whiteside and Neighbors2007) point out that while high interrater reliability demonstrates consistency between raters, it does not prove that raters have used the scale correctly. They suggest assessing not only the consistency between raters, but also their accuracy; that is, the consistency of rater evaluations with a predetermined standard rating, in order to ensure accurate scale use. In the case of our competence ratings, no such “gold” standard was available. In this context, one important potential criticism of the study is that all correlations refer to the standard rater (KvC), with no information being available regarding the consistency of ratings between other rater constellations. If the standard rater used the scale incorrectly, then our high reliability rates would create false confidence in our data. We therefore considered it important to ensure that the standard rater herself rated in a homogenous manner across the entire 3-year rating period. While high test-retest reliability is not a substitute for accuracy measures, it does at least guarantee consistency over time. As reported above, retest reliability was high for the total CTCS score (ICC = .92), as well as for individual items (ICC = .55-.96). In a study by Svartberg (Reference Svartberg1999), test-retest reliability for the total score of the competence measure (Therapist Competence Rating Form; STCRF) was also high (r = .84). A favourable interpretation of our results is that the standard rater relied on similar aspects of therapist behaviour to make her ratings throughout the entire rating process. However, we cannot of course rule out the possibility of memory effects as an alternative source of high test-retest reliability.
Ultimately, the claim that accuracy measures are required boils down to the problem of validity; the CTCS-SP can be used reliably, but is the scale used correctly? Wu et al. (Reference Wu, Whiteside and Neighbors2007) suggest that expert ratings might serve as a gold standard. However, even experts will diverge in their judgment of whether a specific intervention during a therapy session represented the best strategy option at that point in time for that particular patient, since they lack important contextual information that could explain why a therapist has chosen a certain strategy or behaved in a certain way. In addition, it is unclear how many and which therapy sessions need to be evaluated in order to achieve a valid assessment of therapist competence. Currently, there is still little data available to suggest “which specific moment-to-moment interventions are associated with improvement for a specific patient within a particular therapeutic interaction” (Barber, Sharpless et al., Reference Barber, Triffleman and Marmar2007). As recommended by Waltz et al. (Reference Waltz, Addis, Koerner and Jacobson1993), we defined competence relative to a specific treatment manual and instructed our raters to consider aspects of the therapeutic context, such as the stage of therapy, the difficulty of working with the particular client and problems presented by clients. Given these specifications, we are aware that the CTCS-SP is not applicable to other treatment approaches and disorders.
Apart from these limitations, our study suggests that the CTCS-SP is indeed a reliable instrument for assessing therapeutic competence in the application of a treatment, which has proven to be effective in several trials. Further studies are needed to clarify its validity and to define empirically-based criteria for competence levels that are required for therapists to obtain sufficient outcomes.
The high reliability of the CTCS-SP allows for various applications of the scale, including the measurement of changes in skill acquisition, the correlation with self-rated competence for supervision purposes, and the impact of competence on outcome criteria. Furthermore, differences in effect sizes between trials of cognitive therapy for social phobia (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003, Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006; Stangier, Heidenreich, Peitz, Lauterbach and Clark, Reference Stangier, Heidenreich, Peitz, Lauterbach and Clark2003; Mörtberg, Clark, Sundin and Wistedt, Reference Mörtberg, Clark, Sundin and Wistedt2007; Borge et al., Reference Borge, Hoffart, Sexton, Clark, Markowitz and McManus2008), as well as the poor generalizability of research findings to routine settings (Shafran et al., Reference Shafran, Clark, Fairburn, Arntz, Barlow and Ehlers2009), might be explained by different levels of competence. Therefore, ensuring acceptable levels of competence might bridge the gap between results obtained from randomized controlled trials and clinical practice.
Acknowledgments
This research was supported by a grant from the Ministry of Education and Research in Germany. We would like to thank Anja Berg, Anna Dietrich, Natalie Fromme-Schwarzer, Viktoria Ritter and Simone Salzer for organizing the trial, and Wolfgang Hiller, Jürgen Hoyer, Eric Leibing and Ulrike Willutzki for supervising CT. We also gratefully acknowledge the assistance of Kathleen Engelhardt, Katrin Heinrich, Christiane Bohn, Denise Ginzburg, Sarah-Maria Werner and Florian Weck in conducting the competence ratings and the support of the NIHR Biomedical Research Centre for Mental Health at South London and Maudsley NHS Foundation Trust and Kings College London, UK. Finally, we are grateful to Brian Bloch for his comprehensive editing of the English in the final version of the manuscript.
Comments
No Comments have been published for this article.