Introduction
In October 2011 the Improving Access to Psychological Therapies (IAPT) programme for working age adults was extended to children and young people (CYP-IAPT; Department of Health, 2011). The primary aim of CYP-IAPT is to increase the availability of evidence-based psychological interventions for children and young people supported with ongoing clinical supervision. A notable difference from the adult IAPT programme is that training is focused upon clinicians working within existing child and adolescent mental health services (CAMHS) rather than creating separate IAPT services. By so doing, CYP-IAPT aims to transform current services by embedding the CYP-IAPT philosophy of evidence-based practice, child/young person centred services and outcome informed practice throughout CAMHS.
The development of evidence-based practice requires the implementation of the knowledge, skills and attitudes (KSAs) from successful randomized controlled trials into everyday practice. For the adult IAPT programme, Roth and Pilling extracted these KSAs and developed a competency framework that subsequently underpinned IAPT training (Roth and Pilling, Reference Roth and Pilling2008). This framework has been extended to CAMHS, and was further developed to define the competences required to deliver CBT for children and young people suffering with anxiety and depressive disorders, and provide the backbone for the national CYP-IAPT training curriculum (Roth, Calder and Pilling, Reference Roth, Calder and Pilling2011). However, in order to determine if a sufficient level of competence in evidence-based practice has been acquired, an appropriate method is required to determine whether or not competence has been achieved.
The most widely used tool for measuring CBT competence with adults is the Cognitive Therapy Scale-Revised (CTS-R) by Blackburn et al. (Reference Blackburn, James, Milne, Baker, Standart and Garland2001), which was a revised version of the original Cognitive Therapy Scale developed by Young and Beck (Reference Young and Beck1980, Reference Young and Beck1988). The CTS-R consists of 12 items that assess important generic CBT skills. These include four general skills (feedback; collaboration; pacing and efficient use of time; and interpersonal effectiveness) and seven specific CBT skills (eliciting appropriate emotional expression; eliciting key cognitions; eliciting behaviours; guided discovery; conceptual integration; application of change methods and homework setting). The remaining item, agenda setting, overlaps both sets of items and is included in the general and specific CBT skills sub scales. Whole video-tapes are rated on a 7-point scale reflecting different levels of competence (0 = incompetent through to 6 = expert), with a score of less than 2 on any individual item or a total score of less than 50% (a score of 36) reflecting unsatisfactory performance.
The applicability of the CTS-R to assess competence in the use of CBT with children and young people has been questioned (Fuggle, Dunsmuir and Curry, Reference Fuggle, Dunsmuir and Curry2012). In particular, the discriminative ability of the total score to determine competence and the absence of developmentally appropriate criteria were identified as limitations. This led Fuggle and colleagues to develop the CBT Session Competency Framework (CBTSCF) to assess competence in the use of CBT with children and young people (Fuggle et al., Reference Fuggle, Dunsmuir and Curry2012). The CBTSCF is a 24-item scale that includes all 12 items from the CTS-R, but includes additional items more specific to working with children, such as child-centred practice and creativity. The authors propose that 15 domains of the scale should be evident in every session (e.g. therapeutic alliance, collaborative practice) with the remaining 9 being demonstrated in some sessions depending upon the stage of therapy (e.g. discovering cognitions, recognizing emotions, developing a shared formulation). Each item is rated as “competent”, “partially competent” or “failed”; there is no total score, rather an overall recommendation of pass or fail. No psychometric evaluations of the scale have been reported and the robustness and sensitivity of the 3-point scoring system is unknown. Similarly, the inclusion of optional items to assess competence is problematic. It is unclear whether their absence appropriately indicates that the items were not required or that the trainee inappropriately failed to recognize their importance.
Aim
The absence of a psychometrically robust scale developed specifically to assess CBT competence with children and young people led to the development of the Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP). The scale was modelled on the widely used CTS-R; all skills were expected to be demonstrated in each session and were rated on a 7-point scale of competence. The purpose of this paper is to describe the development and preliminary psychometric analysis of the Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP). The development of the scale items will be described, psychometric properties evaluated, and comparison with the CTS-R examined.
Method
Cognitive Behaviour Therapy Scale for Children and Young People (CBTS- CYP)- Scale development
Guiding principles
Although the reliability, validity, item overlap and thresholds of the CTS-R have been questioned, it is nonetheless widely used and considered to provide a comprehensive overview of the generic skills required to competently practise CBT with adults (Kazantzis, Reference Kazantzis2003; Fairburn and Cooper, Reference Fairburn and Cooper2011; Keen and Freeston, Reference Keen and Freeston2008). It was therefore decided that, in as far as possible, any measure developed for use with children and young people should build upon and be consistent with the CTS-R. First, in terms of content, the CTS-R assesses the specific use of CBT methods as well as general skills that facilitate their effective delivery. It was therefore decided that the CBTS-CYP would contain items that assessed both competence in the application of specific methods and in the process of using CBT with children and young people. Second, it was decided that all 12 items of the CTS-R should be covered by the CBTS-CYP, although these might need to be modified and others added to reflect the use of CBT with children and young people. Third, the framework for defining the development of competence proposed by Dreyfus (Reference Dreyfus and Burke1986) and adapted into a 7-point Likert scale on the CTS-R would form the basis of the CBTS-CYP assessment of competence. Fourth, by using the same rating scale the thresholds for assessing competence would be the same as those on the CTS-R (i.e. a score of 2 or more on each item and a total score of 50% or more). Finally, it was decided that the scale would be developed to assess both verbal and non-verbal behaviours and so could be used like the CTS-R to assess both audio and video recordings of whole clinical sessions.
It was anticipated that specific items would not necessarily be mutually exclusive. For example, a formulation requires the development of a shared conceptualization in which important cognitions, emotions and behaviours are bound together within the CBT model. The elicitation and identification of key cognitions and processes would therefore be expected to be associated with the formulation. Similarly, CBT typically involves developing an understanding of the links between cognitions, emotions and behaviours and as such there will inevitably be overlap between these different aspects of the cognitive behavioural model.
CBTS-CYP development
Process items
A modest but consistent relationship has been found in child psychotherapy between the therapeutic alliance and treatment outcomes (Shirk and Karver, Reference Shirk and Karver2003; Karver, Handelsman, Fields and Bickman, Reference Karver, Handelsman, Fields and Bickman2006; McLeod, Reference McLeod2011). Research into which specific aspects of the alliance are particularly important is generally lacking and hindered by the absence of consistent terminology and frameworks (Elvins and Green, Reference Elvins and Green2008; Fjermestad, Haugland, Heiervang and Ost, Reference Fjermestad, Haugland, Heiervang and Ost2009). Despite these limitations Creed and Kendall (Reference Creed and Kendall2005) identified collaboration as an important predictor of alliance in CBT with children and young people. This was defined by behaviours suggesting a partnership with the child and therapist working together as a team, agreeing shared goals with the therapist actively inviting the young person's participation and involvement. Excessive formality resulting in the therapist talking to the child in an aloof or patronizing way and pushing the young person to speak about uncomfortable emotions had a negative effect on the relationship, whilst Russell, Shirk and Jungbluth (Reference Russell, Shirk and Jungbluth2008) found that therapists’ responsiveness characterized by warm, positive and empathic behaviours had a positive effect. Similarly, flexibility or creativity whereby the therapist makes sessions more active, uses different methods, such as games and role-plays, to explain ideas, involves others and attempts to match the concepts to the child's interests was positively related to child engagement (Chu and Kendall, Reference Chu and Kendall2009). Engagement was subsequently associated with positive outcomes leading the authors to suggest that particular attention may need to be paid to securing and maintaining engagement with children by making sessions interesting and enjoyable.
Children are typically referred to psychological services because of concerns identified by others and may not themselves recognize or acknowledge any problems or the need to do anything different (McLeod and Weisz, Reference McLeod and Weisz2005). It is therefore unsurprising that the development of self-efficacy through motivational techniques highlighting positive treatment expectancies, challenging pessimism and providing praise have been identified as important (Russell et al., Reference Russell, Shirk and Jungbluth2008). A further aspect of the therapeutic process that requires specific attention is that of collaborative inquiry, which requires people to “become scientific investigators of their own thinking” (Beck and Dozios, Reference Beck and Dozois2011, p.400). The promotion of a reflective and investigative approach requires careful attention when working with children who are typically used to being provided with information and answers. Finally, the need to adapt CBT to the developmental level of the child or young person has been emphasized by many writers (Stallard, Reference Stallard2002; Friedberg and McClure, Reference Friedberg and McClure2002) and has been reflected in different versions of CBT programmes for children and adolescents (Barrett, Lowry-Webster and Turner, Reference Barrett, Lowry-Webster and Turner1999). This requires the therapist to ensure that CBT is pitched at the right level and is consistent with the child's cognitive, emotional, verbal and reasoning ability.
These key aspects of the process of CBT with children and young people have been encapsulated by the acronym PRECISE (Stallard, Reference Stallard2005). The therapeutic process involves the child and their family working in a partnership (P) with the therapist. The partnership is based upon collaborative empiricism and highlights the active role of the child and their parents/carers in securing change. The therapist needs to pitch the intervention at the right developmental level (R) to ensure that it is consistent with the child's cognitive, linguistic, memory and perspective taking ability. The therapist adopts a warm, caring, respectful and empathic (E) relationship and creatively (C) and flexibly conveys the concepts of CBT in a way that matches the child's interests and understanding. A key concept of CBT is that of guided discovery and investigation (I), which is developed through the adoption of a curious and reflective stance. Self-efficacy (S) is promoted as the therapist helps the child or young person to discover and build upon their strengths, skills and ideas. Finally, therapy sessions should be enjoyable (E) and engaging in order to maintain the child's motivation and commitment to change. These seven items form the basis of the process sub-scale of the CBTS-CYP. Table 1 demonstrates how these are assumed to relate to items from the CTS-R.
CBTS-CYP Process and CTS-R equivalent items
CBT method items
The CTS-R includes items that assess the eliciting of emotions, cognitions and behaviours as well as an item assessing application of change methods. These four items are subsumed within three items on the CBTS-CYP relating to the awareness, identification and management of emotions (E), cognitions (C) and behaviours (B). Formulation (F) relates to the conceptual integration dimension of the CTS-R. Unlike the CTS-R, there is no specific homework setting dimension on the CBTS-CYP. Instead, homework is subsumed within the discovery (D) dimension, which also includes the development of self-discovery and understanding. Similarly, there is no separate item assessing agenda-setting, which is incorporated within a wider item assessing general session planning and organizational skills (G). Finally, a core characteristic of CYP-IAPT is the use of routine outcome and goal planning measures resulting in an additional item of assessment (A) being included. The CBTS-CYP and CTS-R equivalent method items are summarized in Table 2.
CBTS-CYP method and CTS-R equivalent items
CBTS-CYP framework
The 14 items that constitute the CBTS-CYP are summarized in Figure 1. The assessment (A) informs the formulation (F) which encapsulates key cognitions (C), emotions (E) and behaviours (B). These are evaluated and more functional ones developed through the process of discovery (D), with general organizational skills (G) facilitating the effective use of CBT methods. This occurs within a therapeutic process based on partnerships (P) working with methods being pitched at the right (R) developmental level. The relationship is empathic (E), with the therapist being flexible and creative (C) in the way CBT is adapted to reflect the child's interests. A curious stance encourages self-discovery through investigation (I) and reflection with a positive and hopeful approach promoting self-efficacy (S). Finally, engagement and interest is maximized by ensuring that sessions are enjoyable (E).
CBTS-CYP framework
The 14 items are each rated using the same 7-point Dreyfus Likert Scale (Dreyfus, Reference Dreyfus and Burke1986) as used in the CTS-R (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart and Garland2001). By using the 0–6 measure of competence across the 14 items, the total score can range between 0–84 and is converted to an overall percentage by dividing the total score by 84 and multiplying by 100.
The internal reliability, convergent validity and discriminative ability were assessed in two studies. In the first, 12 markers for the University of Reading CYP-IAPT programme independently rated a single video recording of a clinical CBT session with a young male adolescent (aged 12 years) with an anxiety disorder using both the CBTS-CYP and CTS-R. In the second, 48 audio recordings submitted to the University of Reading by 18 trainees in fulfilment of the CYP-IAPT CBT training requirements were rated by university assessors on both the CBTS-CYP and CTS-R. Assessments and recordings were submitted in relation to the three, national curriculum led modules covering core CYP-IAPT principles, anxiety and depression (see www.iapt.nhs.uk/cyp-iapt). The primary problems in the submissions were depression (n = 20); separation anxiety (n = 9); social anxiety (n = 6); OCD (n = 5); panic (n = 3); generalized anxiety (n = 3), specific phobia (n = 1) and PTSD (n = 1). Young people in the recordings were predominantly female (n = 31, 64.6%) and ranged in age from 9–17 years (M = 14.44, SD = 2.31).
Results
Face validity
An overarching description of each item was developed by the author (PS) and some examples of specific behaviours that evidence the competence were generated. The conceptual framework, headline description and detailed descriptors were then discussed with a cross-section of child CBT therapists (n = 61) ranging in expertise and experience.
First, training in using the CBTS-CYP was delivered to seven experienced child CBT supervisors (applying for accreditation as CBT therapists by the British Association for Behavioural and Cognitive Psychotherapies - BABCP) and four experienced child mental health specialists who were markers on the CYP-IAPT course at the University of Reading. Following feedback, a second iteration was developed which included greater specification of each competence level. A second group of four experienced CBT supervisors (applying for BABCP accreditation) and 12 markers were provided training in how to use the CBTS-CYP and gave feedback on the framework, which was considered to comprehensively capture both the process and skills involved in using CBT with children and young people. In total 27 people were involved in the initial discussions and training. The group were predominantly clinical psychologists (89%), aged 25–44 (63%) with over 5 years of experience working with children and young people (81%). Finally, the framework was discussed with 18 CBT trainees on the Reading CYP-IAPT course and with a group of 16 clinical psychologists undertaking their clinical training. Feedback was unanimously positive, with the CBTS-CYP being viewed as comprehensive and more accurately reflecting the process of CBT with children and young people than the CTS-R.
Internal reliability
Intraclass correlations (ICC) were calculated for the 12 ratings of the single video. A two-way mixed effects approach ICC with absolute agreement is appropriate where several raters assess the same target (Shrout and Fleiss, Reference Shrout and Fleiss1979). For the seven process items ICC was 0.95 (95%CI: 0.90–0.98), for the seven method items, 0.90 (95%CI: 0.79–0.97), and the ICC for the total 14 items (0.96) of the CBTS-CYP (95%CI: 0.91–0.99) were substantial demonstrating good reliability.
These compared well with the CTS-R where the equivalents were 0.93 (95%CI: 0.84–0.98) for the 5 general items, 0.91 (95%CI: 0.80–0.97) for the 8 specific items, and 0.95 (95%CI: 0.90–0.98) for the total 12 items. Although the total score sums all 12 items, the agenda setting item is included in both the general and specific item sub-scales.
Convergent validity
Correlations between CBTS-CYP and CTS-R sub-scale and total scores
Convergent validity was determined by comparing the CBTS-CYP against the CTS-R. In the first study (n = 12 raters) Pearson correlation coefficients between the CBTS-CYP process and CTS-R general scores (r = .98, p < .001), CBTS-CYP method and CTS-R specific scores (r = .91, p < .0001) and total scores (r = .98, p < .0001) were very high. Similarly, in the second study (n = 48 session recordings) correlations between process and general scores (r = .79, p < .0001), method and specific skills (r = .91, p < .0001) and total scores (r = .93, p < .0001) were again very high, suggesting that the two measures are highly correlated.
Correlations between individual items on the CTS-R and CBTS-CYP
To investigate the a priori assumed relationship between individual items on the CBTS-CYP and CTS-R, correlational analysis (Pearsons R) was undertaken of the 48 session recordings.
Results, summarized in Table 3, indicate that the predicted correlations were all significant, ranging from r = .93 between formulation and conceptual integration to r = .31 between emotional techniques and application of change methods.
Correlations (Pearson R) between CBTS-CYP and equivalent CTS-R items (n = 48)
Discriminative ability
Results from the 12 raters using the two pass and fail criteria on the CBTS-CYP and CTS-R (i.e. no single item is scored less than 2 and that the total score exceeds 42, i.e. 50%) are shown in Table 4. In study one there was agreement between the CBTS-CYP and CTS-R on whether the session recording was passed or failed by 11/12 (91.7%) raters using the 50% criteria and by 11/12 using the single item criteria. In terms of the total percentage criteria, 8/12 failed the tape on both the CBTS-CYP and the CTS-R. There was one instance (rater 6) where the case failed on the CTS-R but passed on the CBTS-CYP. In terms of single items, 9/12 raters failed the tape on both the CBTS-CYP and CTS-R with one (rater 4) failing the tape on the CBTS-CYP only. Using both criteria, 10/12 raters failed the tape on both the CBTS-CYP and the CTS-R.
Comparison of raters’ scores (total percentages and number of items scoring <2) on the CBTS-CYP and CTS-R (n = 12 raters)
All 12 raters met to discuss individual rater scores, and a group rating agreed for each item. The agreed total CBTS-CYP percentage indicated that the session should be failed; eight raters consequently lowered their scores and four increased them. Using the individual item criteria, the consensus was that three items on the CBTS-CYP were failed (creative, assessment and discovery).
The discriminative ability of the CBTS-CYP and CTS-R was also explored by examining ratings of the 48 session recordings (Study 2). Agreement between the CBTS-CYP and CTS-R on whether a case had passed or failed (on either or both of the individual items or total score criteria) was found in 37 (77%) cases. Of the remainder, eight recordings were failed only on the CTS-R and three only on the CBTS-CYP. An analysis of the individual items that were failed on either measure are summarized in Table 5. Session recordings were failed on all 12 individual items of the CTS-R, with homework setting and conceptual integration being more frequently failed. For the CBTS-CYP, all items were failed except for assessing the right developmental level. The most frequently failed items were formulation, discovery experiments and behavioural techniques.
Summary of individual failed items on the CBTS-CYP and CTS-R
Finally, changes in competence of the 18 trainees as they progressed through training were examined by comparing total scores on the CTS-R and CBTS-CYP for each of the three assessed modules. The results highlight that competence increased by approximately eight points on both measures from the start to the end of training. Average total scores on the CTS-R and CBTS-CYP were respectively module one (m = 47.17, SD = 7.11; m = 50.83, SD = 8.18), module two (m = 52.22, SD = 9, 37; m = 56.06, SD = 10.36) and module three (m = 55.42, SD = 8.31; m = 58.42, SD = 8.45).
Conclusion
This study reports on the development and initial evaluation of the Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP) specifically developed to assess competence in undertaking CBT with children and young people. The results indicate that the CBTS-CYP has good reliability and face validity and compares well, using the total score and single item criteria, with the CTS-R in discriminative ability. This is encouraging and suggests that the CBTS-CYP is an appropriate measure to determine competence with children. However, in reaching this conclusion it is recognized that there are a number of issues that need to be highlighted.
First, whilst the internal consistency of the CBTS-CYP is high, this also suggests that there is considerable overlap between individual items. This interconnectedness was also found with the CTS-R in this study and in previous studies (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart and Garland2001). From a theoretical perspective this is not surprising since it would not be expected that items would be mutually exclusive. Whilst the emphasis of individual sessions might vary in their primary focus, CBT is an integrated therapy that will typically draw upon many aspects of the CBT framework in each session. This point was highlighted by Blackburn et al. (Reference Blackburn, James, Milne, Baker, Standart and Garland2001) who noted that CBT training does not typically focus on one specific skill/aspect at a time but involves training in the global model. The high internal consistency of the CBTS-CYP and CTS-R may therefore indicate considerable overlap between items but may also reflect the way that CBT skills are acquired.
Second, whilst the CTS-R has been widely used to assess CBT competence, our use of this measure to validate the CBTS-CYP raises issues that need to be acknowledged. For example, the CBTS-CYP includes all the items (except homework) from the CTS-R and as such the strong correlation between these measures may reflect the considerable overlap in content between them. Furthermore, despite specific scoring guidelines there continues to be wide variations on the CTS-R between individual raters, a finding that has been noted in this and previous studies (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart and Garland2001; Keen and Freeston, Reference Keen and Freeston2008). Equally, the criteria for determining competence have not been demonstrated and so it is unclear whether someone who scores 48% is actually “incompetent” compared to another who scores 55%, or whether this perhaps reflects differences in client co-operation or engagement. Issues such as these raise questions about whether the CTS-R is an appropriate comparator against which to assess the CBTS-CYP and whether the pass criteria are valid. These limitations are acknowledged, but given that the CTS-R is the most widely used scale for assessing competence in CBT with adults, this comparison using the same pass criteria does seem appropriate.
Third, an examination of the specific item failure criteria revealed that in the 48 session recordings assessed, the right “developmental level” item on the CBTS-CYP was never failed. Similarly, other CBTS-CYP process items such as “partnership working”, “creative and enjoyable”, and “engaging” were seldom endorsed, thereby questioning whether they are important determinants of competence. However, this may reflect the experience of the trainees and the videos assessed. In terms of experience, the selection process for the trainees was rigorous and required a minimum of 2 years of relevant experience of working with children and young people. The average number of relevant years of experience of the 18 trainees was just over 10 years (range 2–31 years). Our cohort was therefore already very experienced in working with children and young people and would be expected to have already acquired the process skills to work therapeutically with this age group. The low frequency of failure on process items may therefore reflect established competence compared to the higher rate of failure on the CBT method items where the trainees are inexperienced and acquiring specific skills. Further studies using the CBTS-CYP to assess therapists with more limited experience of working with children should be undertaken to clarify this issue. The second possibility that needs to be considered relates to the age of the young people in the recordings assessed since there was a tendency towards submitting sessions involving adolescents as opposed to children. The average age of the young people in the recordings was 14.44 years, with only four (8.3%) submissions involving children under the age of 11. Undertaking CBT with younger children will require greater attention to the process issues defined by the CBTS-CYP. For example, creativity is more important with younger children in order to maintain their interest. Similarly, pitching the intervention at the right developmental level is more challenging when working with younger children who are still acquiring skills. Process issues are therefore possibly more important with this younger age group and it is probable that trainees will be more likely to fail to demonstrate competence.
This study has highlighted considerable pre-training variability between raters, with total scores of the 12 raters assessing the same video ranging from 19%-68% on the CBTS-CYP and 24%-70% on the CTS-R. Although the CBTS-CYP and CTS-R include detailed descriptions of individual items and specify differing competence levels, simply reading the scoring instructions does not appear sufficient to ensure good consistency between raters. In this study, after completing the CBTS-CYP and CTS-R independently, the 12 markers were convened as a group to discuss their marks and to agree a consensual rating for each item. Eight markers lowered their initial ratings on the CBTS-CYP following the discussion by up to 37 points with the lowest four raters increasing their scores by up to 14 points. A similar pattern emerged with the CTS-R where nine raters lowered their scores by up to 41 points and three increased their scores by up to 5 points. During the consensual discussion it emerged that whilst the highest scorers had noted evidence of each item, the lower raters had also identified missed opportunities and had considered the extent to which the competence was displayed across the whole of the recording. Future studies should explore the effect of consensual group discussions on reducing variations between raters.
Despite these limitations, this study has demonstrated that the CBTS-CYP, developed specifically for assessing CBT with children and adolescents, has good face and convergent validity and discriminative ability. These results are promising and suggest that the CBTS-CYP provides a useful way of assessing CBT competence for therapists working with this younger age group.
Acknowledgements
The authors are grateful to everyone who has contributed to the development and evaluation of the CBTS-CYP. Particular thanks go to the University of Reading CYP-IAPT supervisors who provided many helpful suggestions during early drafts of the questionnaire. The authors would also like to thank the first year CYP-IAPT trainees from the University of Reading for trialling the CBTS-CYP.
Comments
No Comments have been published for this article.