Introduction
The large-scale implementation of evidence-based psychotherapies such as cognitive behaviour therapy (CBT) relies on accurate measurement of therapy competence. Effective CBT delivery is widely regarded to depend on the related constructs of therapist adherence: implementation of the correct therapeutic procedures – and therapist competence – and the skilful delivery of these interventions (Barber et al., Reference Barber, Liese and Abrams2003; Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001; Fairburn and Cooper, Reference Fairburn and Cooper2011; Sharpless and Barber, Reference Sharpless and Barber2009). Adherence is widely incorporated into assessments of competence as it is regarded as a necessary element of competent therapy delivery (Fairburn and Cooper, Reference Fairburn and Cooper2011) and the two constructs are strongly correlated (Barber et al., Reference Barber, Liese and Abrams2003). Gold-standard assessments of therapist competence for training and evaluation involve an assessor observing a therapist delivering CBT and rating their performance on a competence assessment scale (Clark, Reference Clark2018; McHugh and Barlow, Reference McHugh and Barlow2010; Rosen et al., Reference Rosen, Matthieu, Wiltsey Stirman, Cook, Landes, Bernardy and Watts2016). Evaluating the psychometric properties of these competence rating scales is consequently important given their key role in CBT training and delivery, and as their widespread use in training and clinical practice may influence therapists’ conceptualisation of competence.
The Cognitive Therapy Scale – Revised (CTS-R) has become an established measure of competence in CBT training (Branson et al., Reference Branson, Shafran and Myles2015; Keen and Freeston, Reference Keen and Freeston2008; Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2018; Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019), routine practice and clinical trials (e.g. Freeman et al., Reference Freeman, Dunn, Startup, Pugh, Cordwell, Mander and Kingdon2015; Wilkinson et al., Reference Wilkinson, Alder, Juszczak, Matthews, Merritt, Montgomery and Jacoby2009). Adapted from the related but distinct original Cognitive Therapy Scale (CTS; Young and Beck, Reference Young and Beck1980), the CTS-R consists of 12 items that assess various elements of skilful therapy delivery, including interpersonal and time management skills, adherence to CBT protocols, and effective use of CBT-specific session features. Competence is scored on a scale between 0 and 6 for each item (0 = non-competent, 3 = competent, 6 = expert).
The measure has demonstrated promising psychometric properties, namely high internal consistency (α range = .75–.97; Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001; Kazantzis et al., Reference Kazantzis, Clayton, Cronin, Farchione, Limburg and Dobson2018; Reichelt et al., Reference Reichelt, James and Blackburn2003), responsiveness to increased competence with training (Branson et al., Reference Branson, Shafran and Myles2015; Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019), and some evidence of clinical predictive validity in depression (Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019). While inter-rater reliability estimates vary (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001; James et al., Reference James, Blackburn and Reichelt2001; Kazantzis et al., Reference Kazantzis, Clayton, Cronin, Farchione, Limburg and Dobson2018), agreement improves with rater training (ICC = .38 to .76, Gordon, Reference Gordon2007; r = .44 to .67, Reichelt et al., Reference Reichelt, James and Blackburn2003) and recent estimates are high (Finn’s r = .88, Kazantzis et al., Reference Kazantzis, Clayton, Cronin, Farchione, Limburg and Dobson2018). The CTS-R is consequently a measure of successful CBT training and an important tool to guide supervision and clinical practice. Both the CTS-R and the original CTS are often divided into two subscales that are believed to reflect two hypothetical factors underlying competence: general interpersonal competence (CTS-R items 1–5) and skill at delivering CBT-specific interventions (CTS-R items 6–12) (James et al., Reference James, Blackburn and Reichelt2001). The factor structure of the CTS-R, however, has never been empirically investigated.
Factor analytic investigation into the original CTS has generated surprising findings. A two-factor structure (‘quality of therapy’ and ‘session structure’) was indicated based on depression trial data (Vallis et al., Reference Vallis, Shaw and Dobson1986). Notably, this did not correspond with the widely used hypothetical two-factor structure previously discussed (general versus CBT-specific competencies), nor any of the similar but less common hypothetical three subscale divisions previously proposed for the CTS [McManus et al. (Reference McManus, Westbrook, Vazquez-Montes, Fennell and Kennerley2010) and Westbrook et al. (Reference Westbrook, Sedgwick-Taylor, Bennett-Levy, Butler and McManus2008): interpersonal effectiveness, general therapeutic skills and cognitive behavioural skills; Simons et al. (Reference Simons, Padesky, Montemarano, Lewis, Murakami, Lamb and Beck2010): general therapy skills, CBT skills and structure]. A recent factor analysis of the CTS in community clinicians treating a range of disorders (Affrunti and Creed, Reference Affrunti and Creed2019) supported neither Vallis et al.’s (Reference Vallis, Shaw and Dobson1986) two-factor solution nor the hypothetical two-factor solution, and instead endorsed distinct structures in CBT for adults (two-factor solution) and children (one-factor solution). The CTS-R is also divided into general versus CBT-specific competency subscales (James et al., Reference James, Blackburn and Reichelt2001) despite factor analyses of the CTS demonstrating that the underlying structure of competence measures does not correspond to this theoretical understanding and can vary across populations. Consequently, investigation into the factor structure of the CTS-R is warranted.
The variability in CTS factor structures suggested across studies may reflect systematic differences in how assessors conceptualise competence and score the measure in different contexts and with different patient presentations. Vallis et al. (Reference Vallis, Shaw and Dobson1986) was based on a highly monitored clinical trial of depression – the original context for which the CTS was developed – while Affrunti and Creed (Reference Affrunti and Creed2019) was based on community treatment of adults and children with a variety of clinical presentations. Measurement invariance analysis is consequently indicated for factor analysis studies of competence measures, including the CTS-R, where different therapist and/or patient populations are assessed. This is particularly relevant for populations that are commonly collapsed together or compared with one another in studies involving competence assessment (Van de Schoot et al., Reference Van De Schoot, Schmidt, De Beuckelaer, Lek and Zondervan-Zwijnenburg2015), e.g. patient diagnostic groups which are often combined in studies of therapist competence.
This paper aimed to conduct the first investigation into the factor structure of the CTS-R – including exploratory factor analysis and confirmatory factor analysis of the suggested solution and of the hypothetical two-factor model. Data were drawn from postgraduate CBT training for adult depression and anxiety disorders. Given that structure may vary across populations (Affrunti and Creed, Reference Affrunti and Creed2019), we also assessed measurement invariance across diagnostic groups (depression and anxiety cases).
Method
Participants
Participants were 382 trainees from the High-Intensity IAPT Postgraduate Diploma at King’s College London. All available data from ten training cohorts (2008–2018) were included in the present study. Reported ethnicity was 81% (n = 311) White and 19% (n = 71) Black, Asian or minority ethnic. Gender distribution was 76% (n = 292) female and 24% (n = 90) male. Median age was 32 years (IQR = 7). Trainees’ professions were psychological wellbeing practitioner (38%, n = 145), clinical psychologist (26%, n = 101), counselling psychologist (12%, n = 46), mental health nurse (9%, n = 36), counsellor/psychotherapist (8%, n = 31), occupational therapist (3%, n = 12), social worker (1%, n = 4) and other (2%, n = 7).
Measures
Therapist competence was assessed using the Cognitive Therapy Scale – Revised (CTS-R; Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001). The CTS-R consists of 12 items (detailed in Table 1) which assess general therapeutic skills (items 1–5) and CBT-specific competencies (items 6–12). Items are rated between 0 (non-competent) and 6 (expert), producing a total score out of 72. The recommended competence threshold is item mean score ≥3 or total score ≥36 (James et al., Reference James, Blackburn and Reichelt2001). Raters in the present study were course staff with training and experience in delivering and assessing CBT (see Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019). Estimates of inter-rater reliability based on randomly selected tapes from the current sample were examined separately as part of a course assessment study (Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019) and were excellent between internal course raters [one-way random single measures ICC with absolute agreement (114, 113) = .95] and good between internal and external expert raters [two-way random single measures ICC with absolute agreement (42, 41) = .68]. CTS-R internal consistency and factor structure are reported in the Results.
Factor structure and internal consistency of the Cognitive Therapy Scale-Revised
h 2 = communalities.
Procedure
Data were collected as part of course assessments and a larger programme of CBT training research. The one-year course is delivered as part of the UK IAPT initiative and focuses on treatment of adult anxiety disorders and depression in primary care. The content of the training course has been described in detail elsewhere (see Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019). Trainees submit five audio-recordings of mid-treatment therapy sessions. One baseline tape within the first month of training (any depression/anxiety presentation) and two mid-module tapes (one depression and one anxiety presentation) provide informal feedback for skills development. Two end-of-module tapes (one depression and one anxiety presentation) provide formal examination of therapist competence. While course staff mark assessments due to resource limitations, several steps are taken to blind markers to trainee identity. Tapes are submitted without identifying information. Course staff also have varying degrees of trainee contact, and tapes are allocated to markers with limited familiarity with the given trainee. To assess the CTS-R factor structure at the end of training (i.e. factor structure of the CTS-R for therapists entering the workforce), we included available end-of-training anxiety and depression tapes in the present study. All available recordings were included in the present analysis, including those available for trainees who did not complete the course. A total of 746 recordings were included, consisting of 373 depression cases and 373 anxiety disorder cases. Trainees provided informed consent to use data from their course submissions for research purposes. Patient recordings were not directly handled as part of the current study, as only anonymised CTS-R result data obtained from course records were analysed.
Statistical analysis
As trainees submitted both an end-of-training anxiety tape and an end-of-training depression tape, we ensured that the same trainees were not included multiple times in a given analysis by splitting the data by diagnosis for exploratory and confirmatory factor analysis (i.e. a given trainee’s anxiety tape would be included in the exploratory factor analysis and their depression tape in the confirmatory factor analysis or vice versa). This also ensured counterbalance of diagnostic groups. Four trainees provided only one submission (due to drop-outs, two anxiety and two depression) and these were divided between the two datasets. The remaining missing tapes (n = 14) occurred due to withdrawals/fails and human error. The full exploratory and confirmatory datasets each consisted of 373 tapes counterbalanced by diagnosis, with 186 depression tapes and 187 anxiety tapes in the exploratory factor analysis diagnostic subsample, and 187 depression tapes and 186 anxiety tapes in the confirmatory factor analysis diagnostic subsamples.
We performed exploratory factor analysis using principal axis factoring due to skew observed in some variables. As we expected factors underlying competence to be correlated, we decided a priori to apply oblique factor rotation (Promax) where required. Exploratory factor analysis was conducted on the full sample and on diagnostic subsamples, as the CTS-R is widely used to measure competence regardless of the presentation being treated, but we were interested in uncovering any potential differences in the factor structure by diagnosis.
We planned to assess goodness of fit via confirmatory factor analysis for a univariate model, the model generated by exploratory factor analysis, and for the hypothetical two-factor model commonly used in the literature (Generic Subscale items 1–5, and Specific Subscale items 6–12). Only two models – the univariate and two-factor models – were tested as exploratory factor analysis indicated a univariate model. We assessed the following measures of absolute and relative fit: the relative χ2 (close fit: values from close to 2 to no larger than 5; Schumacker and Lomax, Reference Schumacker and Lomax2004; Ullman, Reference Ullman, Tabachnick and Fidell2001), the root mean square error of approximation (RMSEA; close fit: <.05, adequate fit <0.8; Browne and Cudeck, Reference Browne, Cudeck, Bollen and Long1993), the Taylor–Lewis index (TLI; close fit: >.9; Bentler and Bonett, Reference Bentler and Bonett1980), and the comparative fit index (CFI; close fit: .9; Bentler, Reference Bentler1990). We compared the fit between models based on their Akaike’s information criterion (AIC; Akaike, Reference Akaike1974) and Bayesian information criterion (BIC; Schwarz, Reference Schwarz1978) values, with better model fit indicated with lower values. Measurement invariance of CTS-R items across diagnostic groups (depression and anxiety) was assessed using the multiple independent groups confirmatory factor analysis model.
Results
Exploratory factor analysis
High Kaiser–Meyer–Olkin values (.92 full and anxiety subsample, .90 depression subsample) and significant Bartlett’s tests (all p < .001) suggested that the items were suitably factorable for the full sample, and for the anxiety and depression subsamples.
Exploratory factor analysis (Table 1) suggested a one-factor solution for the full sample and subsamples based on Kaiser’s criterion and scree plots, respectively, explaining 98.68% (full sample), 92.81% (depression) and 95.16% (anxiety) of the variance for the set of 12 variables. Rotation was not performed due to the one-factor structure. Factor loadings (.39–.82) indicated that all items clustered well onto the single factor and were similar for the full sample and subsamples. Communalities ranged from .15 to .60, and were relatively low (≤30% common variance) across the full sample and both subsamples for items 5 and 8, and additionally for item 12 on the full and anxiety subsamples and items 1 and 4 in the depression subsample only.
Confirmatory factor analysis
High Kaiser–Meyer–Olkin values (.93 full scale) and significant Bartlett’s test (p<.001) suggested that the items were again suitably factorable. Item means and factor loadings for the one- and two-factor models are reported in Table 2.
Confirmatory factor analysis and internal consistency of the Cognitive Therapy Scale-Revised
All factor loadings significant, p < .0001; EFA, exploratory factor analysis; CFA, confirmatory factor analysis.
Confirmatory factor analysis yielded similar indices of relative and absolute fit to the data (Table 3) for both the one- and two-factor models. The one-factor model demonstrated lower AIC and BIC values, supporting the univariate structure suggested by the exploratory factor analysis. Relative χ2 values indicate close fit for either model (one-factor = 3.01, two-factor = 3.04). RMSEA indicated adequate fit (.07 for both models) and TLI values exceeded .9, also indicating adequate fit.
Goodness of fit for one-factor versus theoretical two-factor model of the Cognitive Therapy Scale -Revised
**p < .0001. CFI, comparative fit index; RMSEA, root mean square error of approximation; TLI, Taylor–Lewis index; AIC, Akaike’s information criterion; BIC, Bayesian information criterion.
Measurement invariance
We assessed measurement invariance for the CTS-R items across diagnostic groups using multiple groups confirmatory factor analysis (Table 4), which applies a series of confirmatory factor analysis models with increasingly strict invariance assumptions at each step and compares the fit to the less constrained model using a likelihood ratio test. The CTS-R demonstrated full metric invariance (equal corresponding factor loadings), and partial scalar invariance (equal corresponding thresholds), with only item 10 (Conceptual Integration) demonstrating scalar non-invariance. Raters appeared to endorse higher scores on item 10 in anxiety than depression (unstandardised thresholds: 3.24 for anxiety and 3.07 for depression). However, as the invariance occurs in one item only, we conclude that the CTS-R is not biased with respect to diagnosis
Multiple groups confirmatory factor analysis by diagnosis for the Cognitive Therapy Scale-Revised
Reliability and item-level descriptive statistics
Descriptive statistics and internal consistency are reported for the exploratory factor analysis data set in Table 1 and for the confirmatory factor analysis data set in Table 2. Based on both full samples, the scale demonstrated high Cronbach’s alpha (α = .88–90) and strong item-total correlations (.51–.83) supporting retention of all items. There was no evidence that removing any item would increase Cronbach’s alpha values for either data set. Item means ranged from 2.84 (SD = .55) to 3.72 (SD = .45) for the exploratory factor analysis data set and 2.92 (SD = .59) to 3.72 (SD = .47) for the confirmatory factor analysis data set, with overall means of 3.15 (SD = .35) and 3.18 (SD = .37), respectively.
Discussion
This study is an important first investigation into the factor structure of the CTS-R in a large sample of IAPT CBT trainees treating depression and anxiety disorders. Competence – the capable delivery of appropriate therapeutic procedures (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001; Sharpless and Barber, Reference Sharpless and Barber2009) – is proposed to underlie effective CBT provision. Regular assessment using validated rating scales, such as the CTS-R (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001), is recommended to evaluate and guide trainee and qualified therapists (McHugh and Barlow, Reference McHugh and Barlow2010; Rosen et al., Reference Rosen, Matthieu, Wiltsey Stirman, Cook, Landes, Bernardy and Watts2016), and to ensure the availability of quality therapy for patient benefit (Clark, Reference Clark2018; Strunk et al., Reference Strunk, Brotman, DeRubeis and Hollon2010). Exploratory factor analysis suggested a one-factor solution including all 12 items, which was supported by confirmatory factor analysis and was largely invariant across diagnostic categories. As in previous studies (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001; Kazantzis et al., Reference Kazantzis, Clayton, Cronin, Farchione, Limburg and Dobson2018; Reichelt et al., Reference Reichelt, James and Blackburn2003), internal consistency was high. Strong item-total correlations and no gain in reliability with the removal of any item supported retention of all 12 CTS-R items.
The CTS-R has been hypothetically divided into two constructs underlying CBT competence, i.e. generic interpersonal/therapeutic competence (items 1–5) and specific competence at delivering CBT interventions (items 6–12) (James et al., Reference James, Blackburn and Reichelt2001). Consistent with investigations into the factor structure of the original CTS (Affrunti and Creed, Reference Affrunti and Creed2019; Vallis et al., Reference Vallis, Shaw and Dobson1986), findings from the present study did not support division of the scale into subscales based on this hypothetical two-factor model. This departure from the hypothetical two-factor model is notable, considering this theoretical division has been used to conceptualise competence as scored on the CTS-R in both therapist and supervisor training (Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2018; Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2019; Loades and Armstrong, Reference Loades and Armstrong2016; Loades and Myles, Reference Loades and Myles2016). The current findings support conceptualisation of trainee competence scored on the CTS-R as a single construct containing all items of the measure, which appear closely related rather than clustering into separate dimensions of general versus CBT-specific skill. Consequently, CBT training and supervisor/rater training should focus on assessment across all items of the CTS-R, including both generic and CBT-specific elements.
This unidimensional CTS-R structure was also distinct to that of the CTS, which has previously demonstrated varying two-factor structures for adult CBT (Affrunti and Creed, Reference Affrunti and Creed2019; Vallis et al., Reference Vallis, Shaw and Dobson1986). The current populations of therapists and patients varied from CTS factor analysis studies, however, with Vallis et al. (Reference Vallis, Shaw and Dobson1986) based on a clinical trial for depression involving highly experienced and monitored therapists, and Affrunti and Creed (Reference Affrunti and Creed2019) based on a sample of community clinicians treating a range of disorders with varying severity. The present study was based on a sample of CBT trainees at the end of training, who were treating patients with depression and anxiety disorders with limited psychosocial complexity or co-morbidity outside of these diagnostic categories. How competence is conceptualised and scored on the CTS-R, and therefore its factor structure, may vary systematically by therapists’ level of experience (i.e. trainee versus experienced therapist), the context in which they practice, and the population being treated. For example, the factor structure of the CTS appeared non-invariant for therapists treating children (one-factor structure) versus adults (two-factor structure) (Affrunti and Creed, Reference Affrunti and Creed2019). The one-factor CTS-R structure identified in the current study was suggested and confirmed on a large sample of CBT trainees at the end of long-term training. Corresponding investigation into the CTS-R factor structure in other populations and contexts is consequently recommended.
Notably, the scale was not fully invariant across diagnostic categories. Item 10 – ‘Conceptual Integration’ – demonstrated diagnosis-related scalar non-invariance. This item assesses competence at CBT conceptualisation, including whether a presentation-relevant conceptualisation was used and whether it was incorporated into treatment in an appropriate manner. Findings indicated that raters endorsed higher scores for this item in anxiety than depression (unstandardised thresholds respectively 3.24 and 3.07), although these scores were within the boundaries of the same CTS-R competence category (3 = ‘competent’). Systematically higher ratings on item 10 for anxiety presentations may reflect the strong role of disorder-specific conceptualisations in CBT protocols for specific anxiety presentations. However, strong invariance held for the majority (11/12) of CTS-R items, indicating that diagnostic category was not a substantial source of bias in competence ratings.
Additionally, means and standard deviations across items in both the exploratory factor analysis and confirmatory factor analysis samples were similar, tending towards mean values around 3 (‘competent’) with relatively limited variation. Trainees received highly structured and intensive training over one year (please see Liness et al., Reference Liness, Beale, Lea, Byrne, Hirsch and Clark2018 for details), probably contributing to relatively homogenous post-training scores. While this indicates an appropriate level of post-training competence across the sample, the relatively homogenous CTS-R scores may have attenuated the correlations between items. However, overall findings are unlikely to be substantially affected as a single-factor structure is likely to remain consistent in a more heterogenous sample due to higher variance. While selection of training cases for end-of-training evaluation occurred prior to recording the sessions submitted in the present study, self-selection of tapes may have influenced overall scores and score variability if trainees selected their strongest session recordings.
Data in the present study were drawn from a single training course and generalisability of this factor structure should be assessed both through confirmation with other training courses and investigation in other populations and contexts, such as assessment of experienced therapists practising in various settings common to CBT practice. Furthermore, while the CTS-R appeared to be largely invariant across the depression and anxiety diagnostic categories in the present study, these findings may not be consistent when the scale is used to rate therapists treating other diagnoses or at higher levels of patient severity or complexity.
While tapes were submitted without identifying information and scored by markers with limited familiarity with individual trainees, we could not ensure that markers were always blind to trainee identity. Recognition of a trainees’ identity and recollection of personal factors, e.g. prior training, interpersonal factors, etc., could affect CTS-R ratings. However, this was unlikely to systematically affect ratings as course markers were allocated tapes from less familiar or unfamiliar trainees and varied in their contact with trainees overall.
Conclusion
This study adds to the CBT competence literature with a focus on an important missing element in the psychometric evaluation of the CTS-R. Measures such as the CTS-R influence how competence is conceptualised in training and practice. Results based on a large sample of CBT trainees demonstrate the reliability of CTS-R and its component items, and support its continued use to evaluate competence. The one-factor structure identified in this study indicates that conceptualising and supporting competence as a holistic construct consisting of both general therapeutic skills and CBT-specific techniques when using the CTS-R may be beneficial in CBT training. Further investigation is warranted to confirm the structure underlying the CTS-R and its stability across the various contexts and populations in which CBT is applied.
Data availability
The datasets generated for this study are not currently publicly available as they comprise records of a clinical training course.
Acknowledgements
With many thanks to the course staff, trainees and patients who contributed to this research.
Funding
This work received no financial support for research, authorship, and/or publication. Silia Vitoratou is funded by the Biomedical Research Centre for Mental Health at South London and Maudsley NHS Foundation Trust and King’s College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.
Conflicts of interest
The authors report no conflict of interest.
Ethics statement
This study received ethical clearance from the King’s College London Psychiatry, Nursing, and Midwifery Research Ethics Committee: reference number PNM/12/13-50. Authors abided by the Ethical Principles of Psychologists and Code of Conduct as set out by the BABCP and BPS.
Comments
No Comments have been published for this article.