Introduction
Supervision can be thought of as an educational process which aims to support and extend therapist competence (Watkins & Wang, Reference Watkins and Wang2014) and has normative, formative and restorative functions (Inskipp & Proctor, Reference Inskipp and Proctor1993, Reference Inskipp and Proctor1995). There are a variety of definitions of supervision (Milne, Reference Milne2007; Bernard & Goodyear, Reference Bernard and Goodyear2013), all of which include an explicit or implicit acknowledgement of the normative function; that is, for supervisors to evaluate the supervisee's competence and to ensure that it meets the required standard. Competence can be defined as the skill with which a particular intervention is delivered, and encompasses both general therapeutic skills and skills specific to the therapeutic modality in question (Muse & McManus, Reference Muse and McManus2013). Although there are inconsistent findings in the literature, existing evidence largely suggests that therapist competence is related to therapy outcome (Shaw et al. Reference Shaw, Elkin, Yamaguchi, Olmsted, Vallis, Dobson, Lowry, Sotsky, Watkins and Imber1999; Trepka et al. Reference Trepka, Rees, Shapiro, Hardy and Barkham2004; Kuyken & Tsivrikos, Reference Kuyken and Tsivrikos2009; Strunk et al. Reference Strunk, Brotman and DeRubeis2010; Ginzburg et al. Reference Ginzburg, Bohn, Hofling, Weck, Clark and Stangier2012; Brown et al. Reference Brown, Craske, Glenn, Stein, Sullivan, Sherbourne, Bystritsky, Welch, Campbell-Sills, Lang, Roy-Byrne and Rose2013). Consequently, determining how we can best reliably assess and feedback information about therapist competence as part of supervision is an important area for exploration. This paper seeks to add to this literature by (1) presenting the findings of a systematic literature review of the inter-rater reliability (and rater training) of a commonly used competence measure, the Cognitive Therapy Scale (CTS) and Cognitive Therapy Scale – Revised (CTS-R), and (2) presenting a study exploring the impact of supervisor training on the CTS-R on the supervisors’ confidence in using the measure and inter-rater reliability.
Fundamental to supervising therapists in a particular therapeutic modality such as Cognitive Behaviour Therapy (CBT) is assessing their competence in delivering that therapy. Frameworks which detail supervisory skills such as the Roth & Pilling (Reference Roth and Pilling2008) framework specify assessment and evaluation as a core supervisory skill. Competence assessment has a number of functions. The formative components serve to guide learning through the provision of feedback (Nichol & MacFarlane-Dick, Reference Nichol and MacFarlane-Dick2006). The summative (or normative) assessments provide evidence that the minimum required level of competence (a predefined standard) has been obtained by the end of training and is maintained once a therapist is qualified, thus ensuring the quality of treatment provision.
The aims of clinical supervision include ensuring that therapy is delivered competently (Milne, Reference Milne2010) and that therapists (particularly in the context of a training course) maintain and improve their competence (Rosenbaum & Ronen, Reference Rosenbaum and Ronen1998). To become a competent supervisor requires substantial formal instruction in the knowledge, skills and values required to supervise, as well as experiential supervision of supervision within which experiential learning can take place (Watkins & Wang, Reference Watkins and Wang2014). The literature regarding training supervisors is relatively sparse, with Culloty et al. (Reference Culloty, Milne and Sheikh2010) concluding that the generalizability of the 11 studies they reviewed was limited by methodological issues and a lack of established training methods. Increasingly, accreditation requirements (e.g. British Association of Cognitive and Behavioural Psychotherapists; BABCP) dictate that formal assessments of clinical competence should be used in supervision. Providing clear and direct feedback is a feature that enhances the relationship quality and alliance in supervision and supervisee satisfaction with supervision (O'Donovan & Kavanagh, Reference O'Donovan and Kavanagh2014). Therefore, it is prudent to address the question of how best to provide supervisor training to enable them to use such measures reliably (normative function) and in a way which enables supervisee learning and development through the provision of feedback (formative function).
There are a variety of methods for assessing clinical competence, and no consensus about which is best. Muse & McManus (Reference Muse and McManus2013) identified 64 studies relating to methods of assessing therapist competence when working with clients with adult mental health problems using CBT. They proposed a hierarchy of CBT therapist competence measures as follows: (i) knowledge-based measures, which assess whether the therapist has the abstract understanding of the therapy, such as multiple choice questionnaires, (ii) practical understanding measures, which assess whether the therapist can relate the abstract knowledge to practical problems such as clinical scenarios and case reports, (iii) measures which assess the therapist's practical application of the knowledge. In other words, whether the therapist can demonstrate the required skills, which would include, for example, standardized role plays, and (iv) assessment of clinical practice, which tests whether the therapist actually utilizes the skills in practice, for example by rating treatment sessions or by client surveys or outcomes. The most valid assessments of competence focus on what the therapist actually does with the client in clinical practice. The ultimate means of competence assessment is through the supervisor assessing the therapist delivering the required intervention in real-world clinical practice settings, either by live observation or video or audio recording. Ideally, competence in CBT delivery would be rated on an established rating scale, such as the CTS (Young & Beck, Reference Young and Beck1980) or the CTS-R (Blackburn et al. Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001).
The CTS is an 11-item scale which is scored on a 7-point Likert scale, resulting in a total possible maximum score of 66. Six items on the CTS assess general skills such as interpersonal and relationship factors, and five items assess technical skills in utilizing specific cognitive therapy techniques. A number of criticisms have been levelled at the CTS, including that several skills are subsumed within single items, that different levels of inference are required when making judgements about ratings, and the narrow focus on a CBT for depression protocol, all of which result in questionable reliability and validity (McManus et al. Reference McManus, Rakovshik, Kennerley, Fennell and Westbrook2012). The CTS-R is a 12-item scale which is a reconstruction of the CTS to incorporate Kolb's Experiential Learning Cycle (Kolb, Reference Kolb1984), linking each item to the modes of learning or change (i.e. doing, reflecting, formulating, planning). The scale assumes that the aim of therapy is to enable the client to move through the stages of the cycle in order to facilitate learning and change (Milne et al. Reference Milne, Claydon, Blackburn and James2001). It is not clear to what extent users of the scale bear the Kolbian structure in mind, or how strongly embedded the structure really is within the scale; for example, the authors do not discuss whether therapy should reflect the preferred learning styles of the client and/or the therapist. The CTS-R also incorporates a Dreyfus scale (Dreyfus & Dreyfus, Reference Dreyfus and Dreyfus1986) of competence, ranging from incompetent to expert in an attempt to improve clarity and the scoring procedure. The CTS-R is utilized widely as an assessment of clinical competence by postgraduate CBT courses including, for example, the University of Southampton postgraduate diploma in CBT (Gordon, Reference Gordon2006) and the Newcastle Cognitive Therapy Course (Keen & Freeston, Reference Keen and Freeston2008).
Thus, this study seeks to add to the literature around training supervisors, specifically in their use of the CTS or CTS-R as an assessment of clinical competence and as a tool for giving feedback within supervision, by presenting a systematic literature review and some exploratory outcome data on supervisor training on the CTS-R.
Systematic literature review
A systematic literature review was undertaken with the aim of establishing what evidence exists on how to train raters to use the CTS and CTS-R. This was operationalized in the form of two questions: (1) What is it reasonable to expect in terms of inter-rater reliability on the CTS and CTS-R?, and (2) What do we know about how to train raters to use the CTS and CTS-R?
EMBASE, ScienceDirect, PubMed and APA Psycnet were systematically searched, using strings of search terms designed to identify articles containing the words ‘CTS-R’ or ‘CTS’ or ‘cognitive therapy scale’ or ‘cognitive therapy scale revised’ or ‘cognitive therapy rating scale’ and ‘rater training’ or ‘inter-rater reliability’. The reference lists of included articles were hand searched, and a Google Scholar search was also undertaken, reviewing the first 50 results for each search string entered. Original research studies which utilized the CTS (or a closely linked derivative) or the CTS-R (or a closely linked derivative) were included. Review articles and essays were excluded. Studies were also excluded if the version of the measure was a translation into another language. Studies were only included if inter-rater reliability or information on training raters on the competence measure was given.
A total of 79 articles were identified. Of these, 17 were excluded as irrelevant at the screening stage, with 62 articles included for full-text review to determine if they met the inclusion and exclusion criteria. A further 40 articles were excluded after full-text review (see Fig. 1). In 27 of these 40 articles, the CTS or CTS-R was used in some form, either as an outcome measure or a fidelity measure, without any description of rater training or analysis of inter-rater reliability.
PRISMA flow chart of included studies.
Twenty-two articles, reporting on 20 studies, met the requirements of the inclusion and exclusion criteria.
Findings
In these 20 original studies, the CTS, CTS-R, or a closely related derivative, was utilized as a way of ensuring therapy fidelity, measuring outcome or checking the validity of another measure (see Table 1).
Summary of data from existing studies using the CTS-R
ICC, Intraclass correlation coefficient; CTS, Cognitive Therapy Scale; CTS-R, Cognitive Therapy Scale – Revised; CTCS-SP, Cognitive Therapy Competence Scale for Social Phobia (derivative of CTS); CTS-Psy, Cognitive Therapy Scale for Psychosis (derivative of CTS); CTC, Cognitive Therapy Checklist (derivative of CTS).
Question 1: What is it reasonable to expect in terms of inter-rater reliability on the CTS and CTS-R?
Dobson et al. (Reference Dobson, Shaw and Vallis1985) reported good inter-rater reliability for the four expert raters in their initial study of the CTS [intraclass correlation coefficient (ICC) = 0.96], and similarly, Blackburn et al. (Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001) reported good inter-rater reliability for the four expert raters who developed the CTS-R (ICC = 0.63). More widely, inter-rater reliability data is not always reported, and where it is, it demonstrates variability (see Table 1). Notably, Gordon (Reference Gordon2006) reports a mean discrepancy between raters of 7.48 points (s.d. = 6.33) on the CTS-R, and half of the rater pairs had a discrepancy of >5 points. This discrepancy would only be of significant concern in relation to the normative function of the scale if the discrepancy crossed Dreyfus boundaries and the minimum threshold criteria of 50% overall, with a minimum score of 2 per subscale.
There does appear to be variability in the inter-rater reliability at an item level, with some items on the CTS and CTS-R having better inter-rater reliability than others. For example, Maissi et al. (Reference Maissi, Ridge, Treasure, Chalder, Roche, Bartlett, Schmidt, Thomas and Ismail2011) utilized two raters who double-rated 10 patient sessions and found that some subscales of the CTS-R had good inter-rater reliability (e.g. agenda setting ICC = 0.84, conceptual integration ICC = 0.83, total ICC = 0.64) while other subscales demonstrated much poorer inter-rater reliability (e.g. eliciting and planning behaviours ICC = 0.29, eliciting key cognitions ICC = 0.33). Similarly, on the CTS, Williams et al. (Reference Williams, Moorey and Cobb1991) found relatively good inter-rater correlations on the understanding (r = 0.654–0.692) and agenda setting (r = 0.529–0.792) items, but poorer agreement on the feedback (r = 0.000–0.292) and pacing (r = −0.404 to 0.489) items. However, Stallard et al. (Reference Stallard, Myles and Branson2014) demonstrated consistently high ICCs of >0.90.
Question 2: What do we know about how to train raters to use the CTS-R and the CTS?
Of the included studies, nine (45%) did not mention rater training. A further six studies mentioned training, but only gave scant details; Branson et al. (Reference Branson, Shafran and Myles2015) state that 1 day of training was provided annually, Ginzburg et al. (Reference Ginzburg, Bohn, Hofling, Weck, Clark and Stangier2012) state that 2 days of training were provided, and Haddock et al. (Reference Haddock, Devane, Bradshaw, McGovern, Tarrier, Kinderman, Baguley, Lancashire and Harris2001) state that rater training was manualized.
Only five studies (25%) give further detail on rater training procedures. Reichelt et al. (Reference Reichelt, James and Blackburn2003) assessed inter-rater reliability before and after a 3.5-h rater training session (the objectives of which were to improve inter-rater agreement on CTS-R by increased familiarity, mutual understanding and clarification of rationale for items). Reichelt et al.’s (Reference Reichelt, James and Blackburn2003) sample consisted of 10 supervisors, all of whom were experienced in supervising for the same CBT course (Newcastle), and all of whom had experience of using the CTS-R previously. Each supervisor rated a tape of a CBT session pre- and post-training, with each tape being rated by two raters from the course. The findings were that training improved overall agreement on the CTS-R significantly (from 0.44 to 0.67), with post-training inter-rater reliability reaching ‘substantial agreement’ (Landis & Koch, Reference Landis and Koch1977) but with room for further improvement. Thus, there is some initial evidence that rater training does improve inter-rater agreement However, Reichelt et al. (Reference Reichelt, James and Blackburn2003) note that reliability achieved in a study ‘by highly trained experts cannot necessarily be expected by the average user in everyday practice’ (Reichelt et al. Reference Reichelt, James and Blackburn2003, p. 95). Gordon (Reference Gordon2006) found that in a subgroup of five markers who had both attended the same rater training (half-day workshop, no further details given), the inter-rater correlation was much higher (0.76) than for pairs of markers (n = 9) who had not both attended the training (0.38). Maissi et al. (Reference Maissi, Ridge, Treasure, Chalder, Roche, Bartlett, Schmidt, Thomas and Ismail2011) utilized two raters who attended training (unspecified amount) in the therapies they were rating (i.e. CBT and motivational interviewing). The raters scored two full patient sessions and compared their responses, and also scored sections of two further recordings. They found an inter-rater reliability of 0.64, which is considered to be substantial agreement. von Consbruch et al. (Reference von Consbruch, Clark and Stangier2012) report that rater training on the Cognitive Therapy Competence Scale for Social Phobia (a derivative of CTS) was manual-based, and took place over the course of 2 days, during which each item on the scale was explained and demonstrated, using video recordings from therapy sessions, with ratings discussed in detail. Following this, each rater undertook five pilot ratings, with discussion of discrepant ratings. No data are available to enable inter-rater reliability to be compared before and after rater training. Stallard et al. (Reference Stallard, Myles and Branson2014) present data on a therapy session scored on the CTS-R. In a pool of 12 raters, the variability in scores before attending CTS-R training was large, with scores ranging from 24% to 70%. The training involved the raters meeting as a group, having rated the session independently, to discuss their marks and to agree a consensus rating for each subscale. Further inter-rater exercises were undertaken and feedback provided to the raters on how their individual ratings compared to the group of raters. Subsequent to training, the average intraclass correlation for the 12-item scale was 0.95 (95% confidence interval 0.90–0.98).
Summary
Inter-rater reliability and rater training are often not described. Where it is documented, inter-rater reliability on the CTS and CTS-R is variable. On the CTS-R, for example, the ideal would be to achieve inter-rater reliability scores of ≥0.70 (Reichelt et al. Reference Reichelt, James and Blackburn2003). Some studies have achieved this (Gordon, Reference Gordon2006; Lysaker et al. Reference Lysaker, Davis, Bryson and Bell2009; Stallard et al. Reference Stallard, Myles and Branson2014; Branson et al. Reference Branson, Shafran and Myles2015) while others have not (Milne et al. Reference Milne, Baker, Blackburn, James and Reichelt1999; Reichelt et al. Reference Reichelt, James and Blackburn2003; Maissi et al. Reference Maissi, Ridge, Treasure, Chalder, Roche, Bartlett, Schmidt, Thomas and Ismail2011). It appears that rater training does improve inter-rater reliability. However, little information is available about the form, duration and frequency of such training. Therefore, the following exploratory study was undertaken.
Exploratory study
To explore the impact of training supervisors to use the CTS-R, an exploratory study was conducted to address (1) whether supervisors perceive themselves to be more familiar with and more confident in using the CTS-R at the end of a training workshop compared to at the beginning, and (2) how attending a training workshop on using the CTS-R impacts on inter-rater reliability on three subscales of the CTS-R. The hypotheses were that attending the training would lead to an increase in self-rated familiarity with and confidence in using the CTS-R, and attending the training would lead to an improvement in inter-rater reliability.
Method
Participants
The participants were 34 (8 male, 26 female) experienced mental health professionals who attended an advanced supervisor training workshop (1 day), hosted by the Clinical Psychology Doctorate programme at the University of Bath. Most participants were clinical psychologists (n = 32, 94%), with the remaining participants being CBT therapists. Most (79%) participants were within the age range 30–45 years, with the remainder (21%) aged ≥46 years. Number of years post-qualification experience ranged from 1 to 29 [mean (s.d.) = 9.94 (6.095) years]. Ninety-seven percent of the sample declared their BABCP accreditation status; the majority of participants were either accredited practitioners (29.4%), accredited supervisors/trainers (5.9%) or working towards accreditation (23.5%). The remaining participants (38.2%) were not accredited and not working towards accreditation. Ninety-four percent of the sample responded to a question asking them to rate their knowledge and skills in practising CBT on a 0–10 scale, where 0 = no experience and 10 = highly experienced; the mean rating was 6 (median rating 6, range 2–8, s.d. = 1.402).
Procedure
The participants were a convenience sample of experienced mental health professionals who opted to attend an advanced supervisor training workshop (1 day) on using the CTS-R as an assessment of clinical competence, delivered by the Clinical Psychology Doctorate programme at the University of Bath. Advertisements about the workshop were placed in the Clinical Psychology programme newsletter, which was circulated to all regional supervisors, and also, circulated via email to the contact list of regional supervisors held by the course team. Supervisors opted to sign up to attend the supervisor workshop.
On arrival, they were provided with information about the study, and completed a consent form and brief demographic information questionnaire. They then listened to a 10-min audio clip of a trainee doing therapy (CBT) with an adult client and were given an additional 10 min to rate the performance of the trainee on three CTS-R subscales (agenda setting, collaboration, conceptual integration). The rationale for restricting the material presented to 10 min was predominantly a pragmatic one, given the amount of time available for the workshop and the proportion of this dedicated to the research tasks. However, Weck et al. (Reference Weck, Grikscheit, Hofling and Stangier2014) have recently demonstrated that rating segments of therapy sessions is as reliable as rating entire sessions, and previous studies have also used subscales of the CTS rather than the entire scale (Ryum et al. Reference Ryum, Stiles, Svartberg and McCullough2010). The particular subscales of the CTS-R were selected as they spanned a range of generic and CBT-specific skills, and could reasonably be assessed on the basis of the material in the clips.
The workshop was then delivered by the second author (P.A.), and was entitled ‘Two aims in the use of CTS-R: consistency in its use as a competency measure; aiding the development of therapists’. In total, the workshop was of 6 h duration, including breaks. The content and form of the workshop is detailed in Table 2. The workshop aimed to equip supervisors with knowledge and skills related to both the normative and formative functions of the CTS-R.
Description of form and content of training workshop
CTS-R, Cognitive Therapy Scale – Revised.
At the end of the day, the participants all listened to another 10-min clip of another trainee doing therapy (CBT) with an adult client. Again, they were given 10 min to rate the performance of the trainee on the same three CTS-R subscales. Both audio clips were of trainees in their final year of training.
Measures/data collection
Background information questionnaire
This was specifically designed for this study to gather demographic information and information about relevant experience, including gender, age, number of years since qualification, accreditation status with the BABCP, self-rated experience in CBT, self-rated familiarity with the CTS-R and self-rated confidence in using the CTS-R.
CTS-R (Blackburn et al. Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001)
The CTS-R is a 12-item scale. Each item is rated on a Likert scale ranging from 0 to 6, where 0 indicates that the skill was not demonstrated at all (incompetent), and 6 indicates that the skill was proficiently demonstrated, even when faced with challenges to its demonstration (expert). Four scales relate to general therapeutic skills (feedback, collaboration, pacing/use of time, and interpersonal effectiveness) and seven scales relate to CBT-specific skills (facilitation of emotional expression, identifying key cognitions, application of behavioural techniques, guided discovery, conceptualization, application of change methods, and homework setting), with one further scale (agenda setting) which is considered to be both generic and CBT-specific. A minimum score of 36 is generally accepted as the competence threshold (Blackburn et al. Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001). The CTS-R demonstrates high internal consistency (α range 0.75–0.97; Reichelt et al. Reference Reichelt, James and Blackburn2003).
At the beginning and end of the workshop, the three CTS-R subscales were provided. The participants were asked to listen to a 10-min clip of a trainee doing CBT with a working age adult client. Participants were then given 10 min to rate the trainee's performance on the three subscales from the CTS-R, with instructions to rate each item on the Likert scale.
At the end of the workshop, as part of a workshop feedback form, participants were asked to re-rate their familiarity with the CTS-R and their confidence in using the CTS-R.
Results
Research Question 1: Do supervisors perceive themselves to be more familiar with and more confident in using the CTS-R at the end of a training workshop compared to at the beginning?
Participants rated their familiarity with the CTS-R on a 0–10 scale (where a higher score indicates more familiar) at the beginning and the end of the workshop. The mean rating at the beginning of the workshop was 4.71 (range 0–9, s.d. = 3.158). The mean rating at the end of the workshop was 6.26 (range 1–10, s.d. = 2.079). A paired-samples t test demonstrated that this increase was significant (t = -3.517, df = 33, p = 0.001). Therefore, supervisors appear to have perceived themselves to be more familiar with the CTS-R by the end of the training workshop.
Participants also rated their confidence in using the CTS-R on a 0–10 scale (where higher scores indicate increased confidence) at the beginning and the end of the workshop. The mean rating at the beginning of the workshop was 4.06 (range 0–8, s.d. = 2.645). The mean rating at the end of the workshop was 6.34 (range 2–10, s.d. = 2.025). A paired-samples t test demonstrated that this increase was significant (t = -6.313, df = 32, p = 0.000). Therefore supervisors appear to have felt more confident with using the CTS-R by the end of the training workshop.
Research Question 2: How does attending a training workshop on using the CTS-R impact on inter-rater reliability on three subscales of the CTS-R?
To establish the inter-rater reliability of the raters, Krippendorff's alpha (Hayes & Krippendorff, Reference Hayes and Krippendorff2007) was calculated for each of the three CTS-R subscales pre-training and post-training, and for the summed total score pre- and post-training. An increase of 0.2 was sought to evidence improved inter-rater reliability as a result of training, and a Krippendorrf's alpha of at least 0.8 assumed to indicate good agreement. Both pre- and post-training, across all three subscales and the total score, Krippendorff's alpha was 0.0000. Therefore, support was not found for the hypothesis that the training would lead to an improvement in inter-rater reliability and very poor inter-rater reliability was found pre- and post-training.
Discussion
A systematic review found that inter-rater reliability data on the CTS and CTS-R is not consistently available, and where it is presented, is found to be variable, ranging from poor to good consistency. Information about rater training is presented even less frequently, although there is some limited evidence that rater training does improve inter-rater reliability. Where information about the form and content of training is available, the training appears to take the form of rating recordings and agreeing consensual ratings through exploration of discrepancies as a group. Rating a therapist's performance on the CTS or CTS-R requires the supervisor or rater to make qualitative judgements, which are open to biases (Tversky & Kahneman, Reference Tversky and Kahneman1974). Therefore, such ratings may be affected by a vast number of factors, including mood, dynamic issues, experience of ratings, knowledge of CBT and the environment in which the ratings are made. Training supervisors to use the rating scale consistently and objectively may be central to mitigating against such biases.
The current study added to the relatively sparse literature about training supervisors by presenting exploratory data on a 1-day training workshop on the CTS-R for a large group of placement supervisors from a Clinical Psychology Doctorate programme. Workshop attendees’ self-rated familiarity with the CTS-R and self-rated confidence in using the CTS-R increased from pre-training to post-training; it is not known whether this finding was the result of having received didactic instruction on using the tool or as a result of the experiential component of rating the audio clips and hearing other supervisors’ reasoning as to how they arrived at a particular score, or both, or attributable to other uncontrolled factors. Supervisors’ inter-rater reliability did not improve as a group as a result of the training and was very poor both pre- and post-training.
The finding that inter-rater reliability failed to improve following training was in contrast to the findings of Gordon (Reference Gordon2006) and Reichelt et al. (Reference Reichelt, James and Blackburn2003). One possible reason for this contradictory finding is that the current study engaged a much larger group of raters than the previous studies. Engaging a larger group of supervisors may have made training less effective at improving inter-rater consistency. For example, perhaps smaller groups allow for better engagement in the teaching (e.g. more time to ask individual questions) and thus more effective use of the tool. In such a large group, it is much more difficult to achieve absolute consensus amongst the entire group during discussions. Using a number of subscales of the CTS-R but not the entire scale is an approach that has been used previously (Ryum et al. Reference Ryum, Stiles, Svartberg and McCullough2010). However, it may be that, in this study, using only three specific subscales rather than the entire 12-item scale highlighted inconsistencies in ratings at a subscale specific level, which may have been present in the aforementioned studies, but masked by total score inter-rater consistency. Indeed, Reichelt et al. (Reference Reichelt, James and Blackburn2003) presented subscale inter-rater data which illustrates that only seven of the 12 items reached significant levels of inter-rater agreement. This raises questions about the raters’ reasons for and means by which they arrive at a score; it is possible that some raters arrive at an overall score by considering each item as an independent scale, and then sum the subscale scores, whilst other may begin with a global impression and fit individual subscale scores to that impression. The group of raters in the current study was also drawn from a much more varied pool in terms of background and experience in working in training settings, and therefore, might need more training than a 1-day workshop in order to achieve inter-rater consensus. It is promising that the attendees’ self-rated confidence in using the tool and familiarity with it improved, and if this were attributable to the training, it suggests that the training was of benefit, even though improvements in inter-rater reliability were not evidenced. However, it could be argued that this is a potentially dangerous outcome if supervisors become more confident in their ability to use the tool without actually getting better at it. Despite this, satisfaction (assessed via self-ratings of increased confidence and familiarity in this instance) as a subjective impression of training quality is important as it emphasizes the validity of the supervisors’ views and their right to receive high-quality training (O'Donovan & Kavanagh, Reference O'Donovan and Kavanagh2014).
The degree to which the CTS-R recognizes therapist improvement and development (formative function) remains unclear. A number of items on the CTS-R do not increase significantly following therapist training (Milne et al. Reference Milne, Baker, Blackburn, James and Reichelt1999; Blackburn et al. Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001). This may indicate variable item sensitivity (Muse & McManus, Reference Muse and McManus2013), and may also confuse the picture in terms of using the scale to measure therapist development. Therapist training may be expected to, at least temporarily, have negative effects on certain subscales. For instance, interpersonal effectiveness was shown to significantly increase in both Blackburn and Milne's studies, but often, trainees’ interpersonal skills may suffer early in training, when their focus on improving their performance on CBT techniques may be detrimental to their ability to be empathic, warm and genuine within the therapeutic encounter. Similarly, training supervisors on using the CTS-R may initially have a negative impact on inter-rater consensus as supervisors begin to question the processes by which they arrive at their ratings and the rationale for their ratings. In the current study, it may be that part of the development of the supervisors’ ability to rate fairly and consistently on the CTS-R is increasing discrepancy between them, and with time to integrate their learning through reflection and supervision of supervision (Milne & Reiser, Reference Milne and Reiser2016), consensus may be improved, at least partly as a result of formal instruction in a workshop context.
The main strength of this exploratory study was that a much larger sample of raters was utilized than in previous studies, with much more varied experience of using the CTS-R pre-training. Furthermore, all raters scored the same audio clips pre- and post-training, enabling direct comparisons to be drawn. A multi-method approach was taken to assessing training outcome which did not solely rely on self-report data, but also included inter-rater reliability, which overcomes some of the limitations of previous supervisor training studies noted by Watkins & Wang (Reference Watkins and Wang2014) which have solely relied on self-reported outcomes.
However, there are a number of methodological limitations which future studies should aim to address. Only one trainer and one group of supervisors were utilized. Thus, it is not known whether these effects would be replicable and generalizable. Without a control group, it is not possible to conclude that the effects were the result of attending this specific training, or may have been a result of attending any training day. Only three CTS-R subscales and a 10-min segment of a therapy session were used, not the entire instrument or the entire session. While this design made it possible to undertake this study (and having 34 raters rate an entire session pre- and post-training on the 12-item instrument would have been prohibitive), it does limit the extent to which conclusions can be drawn about the findings in relation to the 12 item CTS-R. The recordings were audio and not video; the latter might have provided useful information (e.g. about the therapist's non-verbal behaviours and the therapy context) to enable the raters to make more reliable judgements. The raters did not have any information about the context of the clinical cases and the stage of the casework, which may have further enabled more reliable judgements and decreased the impact of individual assumptions. That said, the issue of contextual information remains controversial, with James (verbal communication to second author at a BABCP round table discussion on the CTS and CTS-R) maintaining that contextual information compromises the scale's metric qualities, while others (at the same round table discussion) took the opposing view. In practice, supervisors are likely to have at least some contextual information, and certainly more than they had in this study. Furthermore, it cannot be assumed that both recordings were equally easy to rate, thus a false negative may have resulted from the post-training recording being more difficult to rate than the pre-training recording, masking an improvement in rater consistency. The post-training ratings were undertaken at the end of a training day when fatigue may have been a factor. Finally, the number of supervisors with post-qualification specialist CBT training is not known, and therefore, conclusions cannot be drawn about the impact of such training on the ability to reliably rate competence on the CTS-R.
Conclusions: implications for future research and practice
To improve the dissemination of CBT in clinical practice, Shafran et al. (Reference Shafran, Clark, Fairburn, Arntz, Barlow, Ehlers, Freeston, Garety, Hollon, Ost, Salkovskis, Williams and Wilson2009) and Simons et al. (Reference Simons, Rozek and Serrano2013) highlight the need for reliable assessment tools of CBT competence. Therapist competence is an ethical issue and also a prerequisite of any determination of a therapy's effectiveness (Sharpless & Barber, Reference Sharpless and Barber2009), and therefore, we must continue to seek to refine and improve the tools available for undertaking competence assessments. The current study has highlighted the challenges of training a large group of raters to use the CTS-R consistently as a normative measure for assessing competence. The CTS-R is a widely used instrument, and the findings of this systematic review and the exploratory study illustrate that with the right training conditions, acceptable inter-rater reliability can be achieved, but cannot necessarily be assumed.
A priority for future research will be to further explore the normative aspects of the CTS-R and ways in which reliability can be improved through rater training. It may be that 1 day of training is insufficient, and a more lengthily training period is required, particularly with a group of raters from a varied background in terms of experience and prior knowledge of the instrument. A study including a control group who rate a therapy session on the full 12-item scale pre- and post-training but attend training on a subject other than the CTS-R or ratings of CBT competence would build upon the current study. Didactic instruction on using the CTS-R could also be compared to an experiential workshop format in which attendees rate the same session on the CTS-R and then discuss their ratings to achieve consensus. A follow-up evaluation after some time has elapsed would provide data on the extent to which the benefits of training are maintained, and possibly enhanced by specific tasks such as post-workshop reflection (Bennett-Levy & Padesky, Reference Bennett-Levy and Padesky2014). The form of training could also be varied, to include an online training condition (Dimeff et al. Reference Dimeff, Koerner, Woodcock, Beadnell, Brown, Skutch, Paves, Bazinet and Harned2009) which may be most cost-efficient when scaling up training to a large group of raters. The current study assumes that training supervisors to use the CTS-R more consistently will enhance the experience of their supervisee's, whose competence they will rate and to whom they will give feedback. Future research could seek to examine the impact of supervisor training on supervisee experience of competence assessment by the supervisor, and thus, contribute to the debate on whether supervisors should act as normative assessors. Certain factors such as formal post-qualification training and core profession have been found to be associated with competence in cognitive therapists (Brosan et al. Reference Brosan, Reynolds and Moore2006), and it may be that certain factors are correlated with competence in rating competence. This could be explored in order to enable training to be optimally tailored to specific groups of supervisors.
With regard to the formative use of competence tools like the CTS-R, future research could evaluate the outcome of various forms of training, and a variety of training content, on the feedback supervisors give to supervisees, and the impact of this on the supervisee and their perceived and actual learning. Future research could also explore the relationship between the formative and normative assessment aspects of the CTS-R; it is possible that supervisors might give appropriate formative feedback, but then seeing the volume of room for improvement, mistakenly derive too much of their normative rating (score on the CTS-R) from this, rating overly harshly. Alternatively, it may be that supervisors who are focused on the formative aspect of developing the supervisee's confidence and skills may be reluctant to rate the supervisee objectively on the CTS-R for fear of impacting negatively on their learning experience by undermining their confidence. In future studies, the aims and content of the training could focus on either the formative function of the CTS-R or on the normative function, or both, and evaluate outcomes on both dimensions for both trainings, in order to establish what to prioritize in training, and whether a combination (as was used in this study) is preferential to focusing solely on one or the other.
In practice, whatever formal competence tool is being used, supervisors should be aware of the reliability issues that persist with such tools; it is advisable for supervisors to undertake regular inter-rater reliability checks, followed up with discussions aimed at achieving consensus (as is the case in a number of CBT training programmes). Although total objectivity is likely to be an unattainable goal, such conversations are likely to enable raters to become more familiar with the tool and also more aware of their biases, and therefore, to improve the objectivity of their ratings, allowing for more balanced and realistic feedback to those they are assessing. Rating competence on formal instruments such as the CTS-R, which is a highly complex and demanding task, should also be discussed within supervision of supervision (Milne & Reiser, Reference Milne and Reiser2016) to enable the supervisor (rater) to become more aware of their biases and mitigate against these. When the roles of supervisor and normative assessor are combined, the supervisor-assessor should be mindful of the influence that the range of their roles as supervisor may have on the specific role of assessor/CTS-R rater. Following Inskipp & Proctor's (Reference Inskipp and Proctor1993, Reference Inskipp and Proctor1995) account of supervisory functions as normative, formative and restorative, the use of a scale such as CTS-R to provide normative feedback (scores) can be further complicated by the need to provide formative feedback (commentary on performance and guidance for development) and, especially where supervisee performance has been poor or confidence low, restorative support.
When instruments such as the CTS-R are being utilized for research purposes, data on treatment fidelity including inter-rater reliability checks and details of rater training procedures should be routinely provided.
It appears that training raters in the use of tools designed to directly assess clinical competence within the therapeutic encounter may hold promise, although the form, content and duration of such training is variable and evaluating the outcome of training presents a number of methodological challenges. Achieving inter-rater consensus continues to be the holy grail.
Summary of main points
-
(1) Supervisors play a key role in evaluating and enhancing supervisee competence.
-
(2) The CTS and CTS-R can be used as a direct assessment of clinical competence in supervision, with consideration of normative (i.e. is the supervisee meeting a required minimum standard score), formative (i.e. giving feedback to aid learning and development) and restorative (i.e. supporting the supervisee with the impact of the formative and normative) aspects.
-
(3) Inter-rater reliability on the CTS-R has been shown to be variable, and there is limited existing evidence about how best to train raters to use the CTS-R consistently (normative aspect). Inter-rater reliability on three subscales of the CTS-R was found to be poor at the beginning and end of a 1-day workshop. Further research is needed to establish the optimum methods for training raters on such measures.
-
(4) It is advisable for supervisors who are utilizing the CTS-R as a competence rating scale to seek formal instruction in using the scale and also to seek supervision of supervision to enable reflection on the task, including noting their own biases which may be impacting on their objectivity and considering the impact of the normative and formative functions on the supervisee and the supervisory relationship.
Ethical standards
Ethical permission was sought from and granted by the Department of Psychology Ethics Committee at the University of Bath.
Acknowledgements
The authors thank the supervisors who attended the training and consented to their data being used for research purposes. This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Declaration of Interest
None.
This paper seeks to:
-
(1) Introduce the CTS and CTS-R as direct assessments of clinical competence which can be used within supervision.
-
(2) Summarize the existing data on inter-rater reliability on the CTS and CTS-R and what we know about training raters to use these measures of CBT competence reliably.
-
(3) Provide an evaluation of the outcomes of a 1-day training workshop which aimed to better enable supervisors to use the CTS-R normatively and formatively in supervision.
-
(4) Discuss the current status of the literature regarding the CTS and CTS-R and make recommendations for research and consider the implications for supervisory practice.
Comments
No Comments have been published for this article.