Introduction
The impact of psychological therapy depends on it being delivered competently. There are generally limited opportunities for others to directly observe therapy and therefore, for a therapist to receive direct feedback on their competence. Although accrediting bodies such as the British Association for Cognitive and Behavioural Psychotherapists (BABCP) stipulate that live observation should occur regularly as part of supervision, this does not appear to take place in routine clinical practice (Reiser & Milne, Reference Reiser and Milne2011). Hence, the governance of therapy via the recognition of training the need to seek supervision or training to improve competence depends on a therapist accurately self-assessing their own competence and appropriately addressing their limitations (Brosan et al. Reference Brosan, Reynolds and Moore2008). Little is known about what facilitates accurate self-assessment of competence, or of how self-assessment is affected by training.
Development of competence over the course of training
Therapist competence in delivering cognitive behaviour therapy (CBT) can be defined as the skill with which the intervention is delivered, including general therapeutic skills and CBT-specific skills (Muse & McManus, Reference Muse and McManus2013). To assess therapist competence in CBT, the following need to be evaluated (James et al. Reference James, Blackburn, Milne and Reichfelt2001): (1) the therapist's adherence to the cognitive model, (2) the degree of skill demonstrated in applying the various cognitive techniques; and (3) the therapist's interpersonal effectiveness.
There is mixed evidence about whether training improves competence, and it is possible that specific training, such as CBT training, may target some areas of therapist competence and not others. Brosan and colleagues found that while number of years of experience, frequency of supervision and accreditation were not related to ratings of competence made by an objective rater (n =24 qualified CBT practitioners), formal post-qualification CBT training was significantly related to competence (Brosan et al. Reference Brosan, Reynolds and Moore2006). CBT trainees (n = 278) on a postgraduate diploma course demonstrated higher levels of CBT skills as measured by the Cognitive Therapy Scale (CTS; Young & Beck, Reference Young and Beck1980) at the end of training than at the outset, with most trainees achieving the pre-determined criteria for competence (McManus et al. Reference McManus, Westbrook, Vazquez-Montes, Fennell and Kennerley2010). The trainees in this study showed greater improvement on the CBT subscales of the CTS than on the General Skills or Interpersonal Effectiveness subscales. Therapists' gender was not found to have any effect on outcome, although older trainees did not do as well as younger trainees and trainees from a clinical psychology background performed better than those from nursing and psychiatry backgrounds. James et al. (Reference James, Blackburn, Milne and Reichfelt2001) found that certain therapist characteristics (n = 20 postgraduate CBT trainees) were correlated with a greater improvement in competence over the course of training including previous CBT experience (although not general post-qualification experience), and gender of therapist (with male trainees improving more than female trainees). These two studies demonstrate inconsistency with regard to therapist factors related to development of competence over the course of training.
Reflective ability
Reflection is generally defined as thinking back on action, reviewing it, and trying to make sense of it (Schon, Reference Schon1983). The Declarative-Procedural-Reflective model for conceptualizing therapist skill development formulates reflection as the driving force for therapist skill development (Bennett-Levy, Reference Bennett-Levy2006). Reflective practice leads to increased self-awareness, professional expertise, and critical thinking (Knight et al. Reference Knight, Sperlinger and Maltby2010), and may therefore be related to accurate self-assessment of competence.
Various factors may influence how effective a therapist is at reflection. For example, reflection can result in self-doubt, which may be difficult to manage depending on individual and circumstantial factors, thus limiting openness to reflection (Farrell, Reference Farrell2001). Reflective practice can also result in feelings of distress (Knight et al. Reference Knight, Sperlinger and Maltby2010). Laitila & Oranen (Reference Laitila and Oranen2013) noted that trainees' first reflections highlight ‘uncertainty, insecurity . . . harsh self-criticism’ (p. 609) but that trainees gradually feel secure enough to become more reflective about their practice.
The nature of the relationship between reflective ability and therapists' accuracy at self-rating their CBT clinical competence has not been previously explored.
Why is self-assessment of competence important?
Like reflection, self-assessment can be summarized as ‘looking back on action' (Driessen et al. Reference Driessen, van Tartwijk and Dornan2008, p. 829). Self-assessment of competence can be further defined (and differentiated from reflection) as ‘a form of appraisal that makes a comparison between one's behavioural outcomes and an internal or external standard’ (Boekaerts, Reference Boekaerts1991, p. 11).
Self-assessment is a skill which enables therapists to become more ‘questioning, analytical, self-motivated, self-challenging and curious’ (Mathieson et al. Reference Mathieson, Barnfield and Beaumont2008, p. 44). Without reviewing and assessing their own practice, therapists risk becoming increasingly incompetent without being aware of it. Thus, accurate self-assessment is critical in maintaining and improving competence, in conjunction with actual scrutiny of practice.
The existing evidence points towards the need for practitioners to become better at improving and maintaining competence; for example, Kraus et al. (Reference Kraus, Castonguay, Boswell, Nordberg and Hayes2011) found that a large number of therapists' average clients (3–16%) ended treatment significantly worse off than when they started, which illustrates the limitations of therapy and the potential for incompetent practice (although this also might reflect worsening psychopathology). Walfish et al. (Reference Walfish, McAlister, O'Donnell and Lambert2012) undertook an investigation of self-assessment bias in mental healthcare providers, and found that 25% of therapists surveyed perceived their skills to be above the 90th percentile compared to their colleagues, and none perceived themselves to be below average. Thus, therapists in routine practice may be systemically biased in their self-assessment of their skills. Brosan et al. (Reference Brosan, Reynolds and Moore2006) demonstrated that a number of BABCP-accredited CBT practitioners (13/24) working in routine clinical practice did not meet widely accepted minimum thresholds of 50% competence on the CTS. Thus, it is vital that training programmes equip trainee therapists with the ability to accurately self-assess their clinical practice in order to ensure safe, competent practice and ongoing therapist development and learning throughout their careers.
Accuracy of self-assessment of competence in CBT has been previously investigated, although not extensively. Brosan et al. (Reference Brosan, Reynolds and Moore2008) sampled 22 practising CBT therapists. A mid-therapy session was rated on the CTS by both the therapist and an independent observer. It was found that therapists tended to significantly over-rate their own competence, particularly less competent therapists. The authors highlighted the need for further research to examine the effects of training on self-evaluation of competence. Brosan et al.'s study was limited by a very low response rate and the use of only one objective rater. Mathieson et al. (Reference Mathieson, Barnfield and Beaumont2008) compared self-assessment to other forms of assessment of CBT competence in the context of a postgraduate CBT training course. The study sample included 34 multidisciplinary professionals. The authors investigated the relationships between self-report (student self-rating form), supervisor assessment by indirect observation (using a supervisor rating form designed to capture supervisor ratings of observations across a number of supervision sessions and observed therapy sessions), and assessments by objective raters via direct observation (CTS-R). They found that self-assessment had a non-significant relationship with supervisor-rated competence and with observer-rated competence. A significant limitation of this study is the lack of reliability and validity data for two of the measures and therefore it is questionable as to whether the tools measured what they purported to measure. In addition, observers rated a single session, while supervisors and therapists rated themselves across a number of sessions and clients. McManus et al. (Reference McManus, Rakovshik, Kennerley, Fennell and Westbrook2012) compared CBT therapists' self-assessments on the CTS to those of their supervisors (n = 64). Contrary to the findings of Brosan et al. (Reference Brosan, Reynolds and Moore2008), they found moderate correlations, with an underestimation of skills compared to supervisors' ratings. Ratings by less competent trainees were not significantly different from their supervisors, and ratings by more competent trainees were significantly lower than their supervisors.
In summary, competence does appear to increase over the course of training in CBT, although there does not appear to be a linear relationship between competence and training input, with various therapist factors potentially influencing the acquisition of competence. Accurate self-assessment of competence enables therapists to recognize the need for further training and supervision, and thus to continue to develop their CBT skills and maintain safe and effective practice beyond their initial formal training in CBT. Therefore, it is important to establish what factors are related to accurate self-assessment of competence to enable training providers to develop self-assessment skills as part of the training experience. One factor that may be related to accurate self-assessment of competence is reflective ability; if a therapist is more able to critically evaluate their performance, they may be more accurate at rating their competence. Reflective practice is thought to enhance self-assessment, although this relationship has not been extensively examined.
Research questions
This exploratory study aims to investigate the following questions:
-
(1) How accurate are trainees' self-evaluations of their competence according to ratings by an independent assessor?
Hypothesis. Trainees will tend to underestimate their competence compared to ratings by an independent assessor.
-
(2) Does the accuracy of self-evaluation of competence change over the course of postgraduate CBT training?
Hypothesis. Level of agreement between self-evaluation of competence and evaluation by an independent assessor improves over the course of training.
-
(3) Does reflective ability change over the course of postgraduate CBT training?
Hypothesis. Reflective ability improves over the course of training.
-
(4) Is reflective ability related to accuracy of self-evaluation of competence? Is reflective ability related to accuracy of self-evaluation of competence on generic skills? Is reflective ability related to accuracy of competence on CBT-specific skills?
Hypothesis. Reflective ability correlates with higher agreement between self-evaluation of competence and the competence ratings made by an independent assessor. This applies to evaluation of both generic and CBT specific skills.
Method
Participants
The participants were experienced mental health professionals who attended postgraduate training in CBT for children and young people within the Improving Access to Psychological Therapies (IAPT) initiative, delivered by the University of Reading, 11.5 h (2 days) per week for 30 weeks. The course was available to applicants from seven regional Child and Adolescent Mental Health Services (CAMHS), and the selection process included an interview and reviewing references.
Forty-three eligible participants were approached and asked to retrospectively consent to their coursework assignments being used for this study. Eighteen participants (42%) consented, with complete data available for 13 participants (30%). As the study took place subsequent to the completion of the training course, it is possible that the relatively low participation rate was due to trainees having disengaged from the training programme.
The final sample consisted of two males and 11 females. The mean age of the participants was 43 years (range 21–59, s.d. = 12.27). Four participants were from cohort 1 of the training course, and nine participants from cohort 2. Participants were from a range of professional backgrounds, including six mental health nurses, two social workers, two occupational therapists, two clinical psychologists and one participant who had evidenced equivalence of a core professional training by producing a portfolio demonstrating relevant knowledge, skills and attitudes. The highest academic qualification obtained prior to this training was a Bachelor's degree for seven participants, a Master's-level qualification for four participants, and a Doctorate-level qualification for two participants. In comparison to the overall sample of 43 potential participants, this subsample was fairly similar with regard to gender, core profession and highest academic qualification attained prior to this training. They were also comparable to the overall sample in terms of mean, standard deviation and range when considering self-assessed competence, assessor-rated competence, and reflective ability scores. The exception to this was that at time 2; in those who consented to participate, while comparable in terms of mean reflective ability, the range of reflective ability scores was not as broad as in the overall sample.
Procedure
Participants attended the training course for an academic year, during which they were required to deliver CBT to children and young people under close supervision within their workplaces. They submitted a number of summative assignments related to their clinical practice of CBT; at three points during training (early-, mid-, late- training), they submitted a batch of assignments related to a particular clinical case (anonymized) which included a case report, a reflective analysis, a video recording of a therapy session, self-rated using the CTS-R. Coursework was marked by a team of experienced BABCP-accredited practitioners who had undergone extensive rater training and inter-rater reliability checks (for example, one a single video, the intraclass correlation coefficient between raters was 0.95, 95% confidence interval 0.90–0.98). Coursework was distributed across raters so that a different rater marked a particular trainee's work at each time point.
Measures
Cognitive Therapy Scale – Revised (CTS-R; Blackburn et al. Reference Milne, Claydon, Blackburn and James2001). This was self-rated and rated by the coursework markers including moderation (objective review of marking by a BABCP-accredited therapist, trainer and supervisor). The CTS-R is a 12-item scale. Each item is rated on a Likert scale ranging from 0 to 6, where 0 indicates that the skill was not demonstrated at all (incompetent), and 6 indicates that the skill was proficiently demonstrated, even when faced with challenges to its demonstration (expert). Five scales relate to general therapeutic skills (agenda setting, feedback, collaboration, pacing/use of time, and interpersonal effectiveness) and seven scales relate to CBT-specific skills (facilitation of emotional expression, identifying key cognitions, application of behavioural techniques, guided discovery, conceptualization, application of change methods, and homework setting). Agenda setting can also be a CBT-specific skill, straddling both generic and CBT-specific competencies. The pass mark is 36 (50%) with a minimum score of 2 on every item, which is the generally accepted competence threshold (James et al. Reference James, Blackburn, Milne and Reichfelt2001). The CTS-R demonstrates high internal consistency [range α = 0.75–0.97 (James et al. Reference James, Blackburn, Milne and Reichfelt2001; Milne et al. Reference Milne, Claydon, Blackburn and James2001; Reichelt et al. Reference Reichelt, James and Blackburn2003)]. Inter-rater reliability has been shown to be variable (Reichelt et al. Reference Reichelt, James and Blackburn2003; Gordon, Reference Gordon2006; Keen & Freeston, Reference Keen and Freeston2008).
Reflective ability. As part of their coursework assignments, the participants submitted a reflective analysis of up to 1000 words which focused on the assessment and application of the therapeutic skills demonstrated in their submitted recording of a therapy session. Participants were expected to critically evaluate the therapy skills demonstrated within the session, judge the effectiveness of the strategies and interventions employed, drawing upon relevant CBT theory, research and literature, and identify areas for improvement of practice. One section of the mark scheme for this assignment was the ‘critical analysis’, on which participants were summatively assessed on their evaluation of the session. They were expected to demonstrate an understanding of the strengths and limitations of their own practice with an evaluation of major and minor issues of relevance. All sections were marked on a 100-point scale, ranging from 0 to 10 (using one decimal point), with 5 as a minimum threshold score to pass. For the purposes of this study, this critical analysis section score was utilized as an index of reflective ability.
Results
Question 1: To what extent do trainees' self-evaluations of their competence correlate with ratings undertaken by a more experienced CBT assessor?
Inter-rater reliability on the CTS-R has been shown to be variable (Reichelt et al. Reference Reichelt, James and Blackburn2003; Gordon, Reference Gordon2006; Keen & Freeston, Reference Keen and Freeston2008). In approximately 50% of cases, the discrepancy between pairs of raters on the CTS-R is >5 points (Gordon, Reference Gordon2006). Therefore, a discrepancy of ≤5 points in either direction between the self-rating and the assessor's rating of competence was assumed to be an acceptable level of agreement (see Table 1).
Patterns of agreement between self-rating and assessor rating across the course of training
CTS-R, Cognitive Therapy Scale – Revised.
Visual inspection of the data revealed that participants tended to switch from overestimating their competence to accurately or underestimating their competence, or vice versa, from assessments 1 to 2 to 3 (Fig. 1). Thus, rather than the same participants consistently under- or overestimating their competence, the trend was for participants to deviate from the assessors' ratings in a different direction on the subsequent occasion of assessment.
Graph showing patterns of self-evaluation of competence compared to competence ratings made by an experienced assessor at time points 1, 2 and 3.
A Pearson χ2 test of association was conducted for each of assessments 1, 2 and 3. The association between self-rated CTS-R scores and assessor-rated CTS-R scores was not found to be significant at any of the three assessment points (see Table 1).
In summary, trainees' self-ratings of competence on the CTS-R were not significantly associated with assessors' ratings. The trainees' level of agreement with assessors on the CTS-R did appear to improve over the course of training. Trainees appeared to overestimate their competence at the beginning of training, with an increase in underestimation of competence midway through training, and a relative increase in overestimation of competence at the end of training.
Question 2: Does the level of agreement between self-evaluation of competence and evaluation undertaken by a more experienced CBT assessor change over the course of postgraduate CBT training?
The level of agreement between self-rated and assessor-rated CTS-Rs improved over the course of training, with a trend towards a decrease in mean discrepancy score (see Table 2). A proportion of participants overestimated their competence, and a proportion underestimated their competence across the course of training, relative to the ratings of a more experienced CBT assessor. However, an ANOVA with repeated measures with a Greenhouse-Geisser correction showed that the within subjects CTS-R discrepancy scores were not statistically significantly different over time [F(1.864, 1239.359) = 1.139, p = 0.336]. This finding should be interpreted with caution given the small sample size and resultant high potential for error.
Agreement in scores between self-rating and assessor rating across the course of training
CTS-R, Cognitive Therapy Scale – Revised.
Question 3: Does reflective ability change over the course of postgraduate CBT training?
To explore the hypothesis that reflective ability improves over the course of training, visual inspection of the data (see Table 3) and an ANOVA with repeated measures with a Greenhouse-Geisser correction was conducted. This showed that the within subjects reflective ability scores were not statistically significantly different over time [F(1.826, 23.443) = 1.288, p = 0.293]. This finding should be interpreted with caution given the small sample size and resultant high potential for error.
Reflective ability scores across the course of training
Thus, the results do not support the hypothesis that reflective ability improves over the course of training.
Question 4: Is reflective ability related to accuracy of self-evaluation of competence? Is reflective ability related to accuracy of self-evaluation of competence on generic skills? Is reflective ability related to accuracy of competence on CBT-specific skills?
Total reflective ability score was calculated by summing reflective ability at the three assessment points, and comparing this to a sum of average discrepancy between the self-rated CTS-R and the assessor-rated CTS-R. A related samples Wilcoxon signed rank test found a significant relationship (Z = −3.180, p = 0.001) between total reflective ability score and the level of agreement between self-evaluation of competence on the CTS-R and assessor-rated competence on the CTS-R in the predicted direction (i.e. as reflective ability score increases, the discrepancy between the self-rated and assessor-rated CTS-R decreases, see Fig. 2).
Graph showing total reflective ability scores and average discrepancy between self-rated and expert-rated competence scores. CTS-R, Cognitive Therapy Scale – Revised.
The items on the CTS-R were divided into ‘generic’ skills (agenda setting and adherence, feedback, collaboration, pacing and efficient use of time, interpersonal effectiveness) and ‘CBT-specific’ skills (eliciting appropriate emotional expression, eliciting key cognitions, eliciting and planning behaviours, guided discovery, conceptual integration, application of change methods and homework setting) and related samples Wilcoxon signed rank tests were applied. The level of agreement between self-ratings and assessor ratings on the generic CTS-R items was found to be significantly related to reflective ability at the same time point (time point 1: Z = −2.481, p = 0.013; point 2: Z = −3.112, p = 0.02, point 3: Z = −2.727, p= 0.006). Total level of agreement between self-ratings and assessor ratings on the generic CTS-R items, summed across the three time points, was significantly related to total reflective ability across the three time points (Z = −3.181, p = 0.001). The same pattern held true for the CBT-specific skill items, with the level of agreement of self-ratings and assessor ratings on these items found to be significantly related to reflective ability at the same time point (time point 1: Z = −2.900, p = 0.004; point 2: Z = −2.970, p = 0.003; point 3: Z = −2.204, p = 0.028). Total level of agreement between self-ratings and assessor ratings on the CBT-specific CTS-R items, summed across the three time points, was significantly related to total reflective ability across the three time points (Z = −2.936, p = 0.003).
Thus, reflective ability appears to be positively related to the level of agreement between self-evaluation and assessor evaluation on the CTS-R, for both the generic items and the CBT-specific items.
Discussion
The level of agreement between trainees' self-ratings of their competence in delivering CBT to children and young people and ratings of more experienced CBT assessors, rated on the CTS-R, did improve over the course of training, although was not statistically significantly associated at any of the three time points. This result suggests that trainees became more objective in their self-evaluations over the course of training, and adds to previous research (Brosan et al. Reference Brosan, Reynolds and Moore2008; Mathieson et al. Reference Mathieson, Barnfield and Beaumont2008; McManus et al. Reference McManus, Rakovshik, Kennerley, Fennell and Westbrook2012), none of which explored the development of self-evaluative ability over time. However, it appears that discrepancies persist between self-rated competence and assessor-rated competence, even at the end of training.
Previous studies have presented a mixed picture in terms of therapists' trend towards underestimation or overestimation when self-evaluating clinical competence. Dunning et al. (Reference Dunning, Heath and Suls2004) synthesized studies from health, education and workplace settings which illustrated the tendency for people to overrate themselves. The authors postulate that this systematic bias is a result of information deficits such as the double curse of incompetence, unknown errors of omission, the difficulty with defining what competence is, and also information neglect, such as the tendency to become more egocentric when evaluating skill in comparison to others. More specifically, within therapist samples, Brosan and colleagues (Reference Brosan, Reynolds and Moore2008) found that qualified CBT therapists tended to overestimate their competence as did Walfish et al. (Reference Walfish, McAlister, O'Donnell and Lambert2012), while McManus et al. (Reference McManus, Rakovshik, Kennerley, Fennell and Westbrook2012) found that trainee CBT therapists tended to underestimate their competence, and Mathieson and colleagues (Reference Mathieson, Barnfield and Beaumont2008) found no significant relationship between self-assessment and observer-rated competence (trainee sample). The current study found that some trainee therapists tended to overestimate their competence, while others tended to underestimate their competence. In this sample, rather than particular participants continuing to over- or underestimate their competence across time, the trend was for participants to become more accurate or to deviate in a different direction from the objective rating at the subsequent assessment point. This result may reflect a response to the feedback trainees received at the previous point of assessment, with participants over-compensating for their previous flawed self-evaluation, consequently resulting in flawed self-evaluation in the opposite direction. This may also reflect participants' journeys through learning new skills, and progressing at differing rates through stages of unconscious incompetence, conscious incompetence, unconscious competence and conscious competence, described by Adams (Reference Adams2011). Qualified therapists may be more likely to overestimate their competence due to the absence of ongoing direct feedback and scrutiny of their skills, while the picture for trainee therapists is more variable, and the accuracy of their self-evaluation of their competence may be related to a number of different factors, including feedback previously received, nature of ongoing supervision and teaching, learning style and their stage in the journey through skill acquisition. Future research could explore these possibilities.
The current study found that reflective ability did not significantly increase over the course of training. This finding may be attributable to limitations in the design of this study, given that reflective ability was assessed via a marked, written coursework assignment, which may not be a true index of reflective ability, but may rather reflect factors like academic ability and previous experience of postgraduate education (and thus expertise in satisfying the marking criteria). Further research should explore reflective ability over the course of training within a more robust design.
In addition, this study found that reflective ability did appear to be significantly correlated with level of agreement between self-evaluation and assessor-evaluation on the CTS-R, both on the generic items, and on the CBT-specific items. This finding adds to previous research which has explored whether factors such as age, gender, professional background, training and experience (Brosan et al. Reference Brosan, Reynolds and Moore2008) influence the accuracy of self-evaluation. It could be that reflective ability leads to a more accurate assessment of one's competence, and if this were found to be true, it may be indicated that training programmes place more of an emphasis on activities that enhance reflective ability. This conclusion is consistent with the arguments about the centrality of reflection to therapist skill development, presented in the adult education literature (Kolb, Reference Kolb1984) and in the therapist training literature in CBT (Bennett-Levy et al. Reference Bennett-Levy, Turner, Beaty, Smith, Paterson and Farmer2001, Reference Bennett-Levy, Thwaites, Chaddock, Davis, Stedmon and Dallos2009 a,b). An example of an approach being integrated into CBT training courses that aims to improve therapist skill through reflective activities is the Self-Practice–Self-Reflection (SP-SR) approach (Bennett-Levy et al. Reference Bennett-Levy, Thwaites, Chaddock, Davis, Stedmon and Dallos2009b ). Future research should be designed to enable the causality of the relationship between reflective ability and accuracy of self-assessment of competence to be established, and could, for example, explore whether enhancing reflective ability during training impacts on accuracy of self-assessment of competence once in qualified practice.
The current study, although exploratory in nature, addressed some of the methodological issues of previous studies, for example, Brosan et al. (Reference Brosan, Reynolds and Moore2008), who utilized only one objective rater (with a small percentage marked by a second rater). In the current study, a number of raters were utilized, all of whom had undergone extensive marker training. The design of the current study also allowed for comparison over time, given that measures were available at multiple time points. Most of the literature regarding therapist competence and reflective ability relates to therapists working with adult populations, and this study adds to this literature by its focus on therapists working with children and young people.
Professionals who are actively pursuing further training in CBT may not represent larger populations of therapists delivering CBT in clinical practice (McManus et al. Reference McManus, Westbrook, Vazquez-Montes, Fennell and Kennerley2010), which limits the extent to which the study's participants and therefore, the findings are representative of the wider community of CBT professionals working in routine practice. As participation was optional and recruitment was retrospective to the training course, those who agreed to participate may be a skewed sample (for example, more reflective individuals may have been more open to participating in the study), and thus there is the potential of selection bias. Furthermore, the therapists selected the recordings which they had submitted to be evaluated, most likely leading to their filtering out poorer performances, and therefore, it is difficult to extrapolate from the current findings conclusions about therapists' self-assessments of their poorer performances.
The sample size is a key limitation to generalizability, and it would be helpful to replicate this study on a larger scale, potentially across different training institutions to enable more robust conclusions to be drawn.
It is not known what changes, if any, to self-assessment and/or reflective ability would have occurred in this group in the absence of training, given that there was no control group. Furthermore, the assessors were not blind to stage of training, which may have influenced their ratings in either direction (McManus et al. Reference McManus, Westbrook, Vazquez-Montes, Fennell and Kennerley2010), although when they undertook their ratings, they were not aware of this study and were merely completing coursework marking procedures.
In relation to the measures, given that the scores were based on assessed coursework, it could be that practice and familiarity with the course requirements, rather than increased ability, was attributable to the reflective ability scores. Furthermore, the CTS-R, although frequently utilized as an assessment tool, does have acknowledged limitations with regard to inter-rater reliability (Reichelt et al. Reference Reichelt, James and Blackburn2003; Gordon, Reference Gordon2006; Keen & Freeston, Reference Keen and Freeston2008); the current study assumed that the coursework markers' rating on this measure were an objective and reliable index of actual assessment. All coursework markers had undergone inter-rater training and regular inter-rater checks, and all coursework assignments were moderated by a member of the course team. Despite this, the task of rating remains a judgement, and is therefore an imperfect science, which can be influenced by a multitude of factors. Given the limitations of the current measurement tools available as direct assessments of CBT (Simons et al. Reference Simons, Rozek and Serrano2013), it is possible that overestimations or underestimations of competence on the part of the trainees may represent issues related to achieving a ‘true’ measurement of competence on this (and other) measures of clinical competence.
Trainee therapists do not necessarily demonstrate the bias for overestimation of self-rated competence found in qualified therapists. Over the course of training, the tendency of a trainee to over- or underestimate their competence may not remain static, but in general, tends to become more accurate. Those trainees who demonstrated better reflective ability tended to be more consistent with more experienced CBT assessors in their self-assessments of competence; hence, enhancing reflective ability during training may equip therapists with the skills required to accurately self-evaluate their competence beyond training and thus enable them to maintain and improve their practice, although further research is required to fully test this hypothesis.
Ethical standards
Ethical approval was granted by the School of Psychology Ethics Committee, University of Reading. All therapists gave their written consent for the authors to utilize the coursework marks they obtained while studying for a postgraduate qualification in CBT for submission to a journal, in anonymous form. Therefore, all identifying information has been changed or altered to maintain anonymity.
Declaration of Interest
None.
Acknowledgements
The authors thank the trainees who agreed for their data to be utilized for the purposes of this study, and also thank Dr Craig Steel for his input at the early stages of the project. No specific funding was sought for the purposes of carrying out this study.
-
• To summarize what is known about the development of therapist competence through training.
-
• To define and explore reflective ability in the context of therapist development.
-
• To understand the importance of self-assessment of competence.
-
• To consider how self-assessment of competence and reflective ability may be linked in a small sample of CBT diploma postgraduate students.
Comments
No Comments have been published for this article.