Introduction
How good are we as therapists at assessing our own competence? This exploratory study investigated the reliability of self-assessment of therapeutic competence through exploring the relationships between self-assessment and direct/indirect assessment by others.
Verifying competence is an important responsibility of training programmes. It is a complex task, and it is important that we find rigorous methods of assessing competence (James et al. Reference James, Blackburn, Milne and Reichfelt2001).
A number of reviews have emphasized the need for multiple methods of assessment of competence particularly in relation to determining the outcomes of psychotherapy training programmes (Scofield & Yoxtheimer, Reference Scofield and Yoxtheimer1983; Alberts & Edelstein, Reference Alberts and Edelstein1990; Milne et al. Reference Milne, Baker, Blackburn, James and Reichelt1999; Barnfield et al. Reference Barnfield, Beaumont and Mathieson2007). However, little is known about the relationships between the various methods of assessment of competence.
This study investigated the relationships between three different measures of therapist competence: self-report, supervisors' assessment via indirect observation (therapist self-report which includes written formulations and case reports) and assessment by raters via direct (videotape) observation using a rating scale and asked: What are the relationships? Do they add value? Are they all necessary?
A case for self-assessment
Self-assessment of competence can be defined as ‘a form of appraisal that makes a comparison between one's behavioural outcomes and an internal or external standard’ (Boekaerts, Reference Boekaerts1991, p. 11). Elks & Kirkhart (Reference Elks and Kirkhart1993) argue that clinicians continually evaluate their own practice, albeit privately: we form ongoing global appraisals of our work based on intuition and experience, through monitoring our own reactions and make judgements in comparison to guidelines for good practice. Watts (Reference Watts1990) argues that self-assessment is valuable because through it we become more questioning, analytical, self-motivated, self-challenging and curious, while clinicians who do not may be sliding towards incompetent practice, may easily become careless or out of date without realizing it and may waste valuable efforts to learn.
Moreover, Bjork (Reference Bjork, Gopher and Koriat1999) suggests that self-assessment is arguably as important as our actual competence in that it affects time allocation, whether we seek further study or practice, volunteer for tasks and instil confidence in others. Self-assessment of a clinician's skills and abilities is therefore, arguably, a crucial element in developing and maintaining competence.
A case against self-assessment
In his book on ‘self-insight’, Dunning (Reference Dunning2005) suggests that it is a complicated and subtle matter to know one's self. He argues that it is surprisingly difficult and that we come to flawed and sometimes downright incorrect conclusions about ourselves.
For example, a number of studies have found a weak to non-existent connection between people's self-estimation and actual ability, with a tendency to overestimation. Overestimation of ability by the trainee has been found to be particularly evident with less able trainees (Bjork, Reference Bjork, Gopher and Koriat1999). Ames & Krammath (Reference Ames and Kammrath2004) found that most people overestimate their social judgement and mind-reading skills and they suggest narcissism, not actual competence may predict self-estimated ability. This may fit with a broader tendency for people to overestimate their competence across a range of areas, dubbed the ‘above average effect’ (Alicke et al. Reference Alicke, Mark, Klotz, Breitenbecher, Yurak and Vredenberg1995).
Kruger & Dunning (Reference Kruger and Dunning1999), in studies of university undergraduates, found that people who overestimate their competence make poor choices; a further manifestation of their incompetence is their lack of meta-cognitive ability to identify their unfortunate choices. Kruger & Dunning argue that the skills that engender competence in a domain are often the same skills needed to evaluate competence in that domain.
Miller (Reference Miller1998) argues that although self-assessment of one's own therapy is clearly open to bias, in that it is subjective and potentially self-serving. However, LaSala (Reference LaSala1997) did not find support for a self-serving bias: clinicians underrated therapeutic gains compared to client ratings.
Limitations of other forms of assessment
The reliability and validity of assessment of students by supervisors was explored in a review of the literature by Ellis & Ladany (Reference Ellis, Ladany and Watkins Jr1997). They noted that supervisees may be evaluated primarily qualitatively via their indirect report of therapy and suggest that perceptions of the supervisee by the supervisor may influence evaluation. They conclude that using this process to assess supervisee competence may be unreliable, bringing into question its usefulness.
The unreliability of indirect supervisee report was evident in a study by Ladany et al. (Reference Ladany, Hill, Corbett and Nutt1996) who found that supervisees frequently did not disclose key material from therapy during supervision. Trainees' reasons for not disclosing included a perception that the material was too personal; that they had a poor alliance with the supervisor, and that they felt uncomfortable about disclosing the material (embarrassed by a lack of empathy).
The limitations of self-assessment and indirect assessment by supervisors underscore the importance of incorporating external, direct observation techniques (video-/audiotapes) in obtaining valid and reliable judgements of therapeutic competence, as direct observation may provide a more objective and accurate measure of competence. This method would generally involve observation of therapy sessions via a two-way mirror, videotape or audiotape and the therapist's skills being rated using a structured rating scale (Ellis & Ladany, Reference Ellis, Ladany and Watkins Jr1997). This is what the Cognitive Therapy Scale – Revised (CTS-R) (Blackburn et al. Reference Blackburn, Milne and James1997, Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001; Milne et al. Reference Milne, Blackburn and James1997), used in the current study, is designed for. The limitation with this approach, however, as noted by Miller (Reference Miller1998) is that it only draws on limited data from a single session.
Relationships between the measures
Within the psychotherapy field there is little research that examines the relationships between subjective self-assessment of competence, indirect (supervisor) assessment measures and the more objective assessment via direct observation.
Acosta (Reference Acosta1995) investigated psychologists' self-assessment of multicultural competence, in comparison to an objective measure (a multicultural awareness-knowledge-skills survey). Results revealed the subjective self-assessment to be positively related to the objective scores.
The relationship between measures of self-assessment of competence and those involving objective assessment is unclear. This lack of clarity supports the argument that self-assessment of competence, as a sole measure of competence, is not a good stand-alone measure if we want valid, reliable judgements of therapeutic competence.
The purpose of this exploratory study was to examine the relationships between self-assessed competence, supervisor-rated competence via indirect observation and other-rated competence via direct observation. Two exploratory measures were devised, the Student Self-Rating Form and the Supervisor Rating Form (T. V. Barnfield, unpublished course materials; both available as Supplementary material on the Journal's website). It was expected that measurement of competence based on direct observation would be superior to measurement of competence based on indirect observation or self-assessment.
Hypotheses
(1) There would be a weak but positive relationship between self-assessment of competence and other forms of assessment of competence.
(2) There would be a positive relationship between ‘other’ measures, i.e. between direct and indirect (supervisor) external measures
Method
Participants
Participants were 34 experienced mental health professionals enrolled in a half-time (20 hours per week on average) 30-week postgraduate-level CBT course. There were 24 females and 10 males. Thirty participants were identified as European (Pakeha) New Zealanders, two as Maori New Zealanders and two from other unspecified ethnicities. The age range was from 24 to 61 years (mean age 39 years). Fourteen were qualified social workers, 12 were registered nurses, five were occupational therapists and the remaining three were from other mental health disciplines. Fifteen participants had more than 6 years' clinical experience, nine had between 4 and 6 years, and ten had 3 or fewer years of experience. Participants were selected for the course on the basis of years of mental health experience, interpersonal effectiveness in a selection interview, referee endorsement and curriculum vitae. None of the course participants had undergone prior formal training in CBT.
Procedure
CBT training course
The training consisted of a half-time postgraduate 30-week course. It was designed to provide workforce development for experienced mental health professionals and has been running for 5 years.
The course consisted of two components, a 300-hour academic component consisting of lectures, tutorials, reading and assignments, and a 300-hour practicum. There were three block-taught (40-hour) weeks at intervals throughout the course. A mixture of teaching modalities were used during block weeks, including didactic lectures, practical exercises, role-plays, videotapes of experts demonstrating specific skills and small group discussions. The approach taught was based on the work of Dr Aaron Beck. A special aspect of the course was the inclusion of a component on working with Maori, the indigenous people of New Zealand.
For the practicum component, each trainee worked full time in a mental health setting and developed CBT skills while working with patients as part of their regular client load. Participants received individual supervision during their training from clinical psychologists with expertise in CBT. Each participant received 50 hours of supervision. Supervisors also received supervision and further training from the CBT course lecturers.
Measures
Cognitive Therapy Scale – Revised (CTS-R)
The CTS-R (Blackburn et al. Reference Blackburn, Milne and James1997, Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001) is a 14-item scale designed to measure competence across a range of therapeutic skill areas. Items are rated on a 7-point likert scale ranging from incompetent (0) to expert (6). The CTS-R has been shown to have high internal consistency and adequate inter-rater reliability (Blackburn et al. Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001).
Supervisor Rating Form (SRF)
This measure was developed for the present study, due to the absence of similar supervisor measures of competence in CBT. It is based on the CTS-R (Blackburn et al. Reference Blackburn, Milne and James1997), but is based on supervisor observations from supervision sessions and several observed therapy sessions, rather than rating a single session, as occurs with the CTS-R. The form requested the supervisor to rate the student's skills, knowledge and behaviours across a range of areas pertinent to CBT. The twenty-four items measuring skill areas sampled on the CTS-R and additional skills and behaviours were rated on the same 6-point likert scale (0, poor performance; 5, excellent performance). This measure also allowed supervisors to comment on the strengths and weaknesses of the student, and asked whether supervisors would refer selected patients to the student. For the purposes of the present study, only the 24 skill development items were used. Participants could obtain a score between 0 and 120, with higher scores indicating greater levels of skill.
Student Self-Rating Form (SSRF)
This measure was also developed for the present study, again, due to the absence of a suitable available measure. It is the student self-rating version of the SRF. The items were identical to the SRF but were self-rated by the students. The students were also asked to comment on their perceived strengths and weaknesses and state whether they were comfortable using CBT with selected patients. Again, only the 24 skill development items were used in the present data analysis, and were scored out of a maximum of 120. Higher scores indicating greater levels of skill.
Design/data collection
A videotape of each trainee's therapy session was assessed using the CTS-R after 2 and 6 months of training. Three clinical psychologists, experienced in CBT training, therapy, supervision, and using the CTS-R, independently rated the videotapes. The supervisors and course participants completed the SRF and SSRF, respectively, after both 2 and 6 months of training.
Results
The inter-relationships between CTS-R, SRF, and SSRF were examined. Pearson product-moment correlations were calculated and are presented in Table 1.
Pearson product–moment correlations between CTS-R, SRF and SSRF at time 1 and time 2
CTS-R, Cognitive Therapy Scale – Revised; SRF, Supervisor Rating Form; SSRF, Student Self-Rating Form.
Although none of the correlations presented in Table 1 reached significance, they are low and in the predicted direction. CTS-R ratings have low and positive relationships with the SRF and SSRF ratings at both time 1 and time 2. It is of interest to note that the CTS-R has a greater correlation with the SRF than it does with the SSRF. This relationship is consistent at both time 1 and time 2. The SRF and SSRF have moderate and positive relationships with each other at time 1 and time 2, although these correlations do not reach significance.
Discussion
The results of this study were that self-assessment had a non-significant relationship with supervisor-rated competence and with other-rated competence assessed via the CTS-R but there was a trend in the hypothesized direction. In a recent article by Bennett-Levy & Beedie (Reference Bennett-Levy and Beedie2007) it is suggested that it is likely that self-perception of competence is related to actual therapeutic performance. The results of the present study suggest that this relationship may be weaker than was anticipated.
The lack of a significant correlation between supervisor-rated competence and competence assessed via the CTS-R is of interest. It is possible that supervisor ratings were affected by the fact that they often shared their evaluations with their supervisees, so supervisors may have been kinder as a result. It may also have been that supervisors' views of their students were influenced by their experiences of the student's competence in other areas, such as in a social work or nursing role.
Even if student self-assessment, on further investigation, proves unreliable as a measure of competence, it may be important to pay attention to it because it is likely to reflect students' confidence in their competence as CBT therapists, which is likely to affect whether they take on cases, what cases they choose to take on, etc. Student self-assessments may also prove useful for course lecturers, through making it possible to identify students who are overestimating their competence in a way that may lead them to work in a harmful way or get out of their depth.
Although the results of this study suggest that self-assessment may not have a strong relationship to more (supposedly) objective measures of competence, such as the CTS-R, it must be emphasized that reliability/validity data for the two measures developed for this study are not available. It may be that these tools did not measure what they were intended to measure. A future investigation into the reliability and validity of the SSRF and SRF would clarify this. The measures of competence used in this study differed between the supervisors, the students and the independent raters. A future study where students, supervisors and independent raters rate the same video sessions with the same measure, e.g. the CTS-R may also assist in further clarifying the relationship between self-assessment and competence.
Summary of main points
• It is important that we find ways to verify psychotherapeutic competence as a result of training.
• Current evidence suggests trainee self-assessment is not necessarily reliable.
• Other measures such as supervisor ratings and direct assessment of videotaped sessions also have their limitations.
• There is little evidence of relationships between these measures.
• This exploratory study compared these three forms of assessment of competence and found no significant relationship between the measures.
• Trainee self-assessment may have an important role in professional development, through the development of confidence and the identification of strengths and weaknesses.
• Follow-up reading suggestion: Bennett-Levy & Beedie (Reference Bennett-Levy and Beedie2007).
Note
Supplementary material accompanies this paper on the Journal's website (http://journals.cambridge.org).
Declaration of Interest
None.
Learning objectives
• Knowledge of the current state of the evidence regarding self-assessment of therapeutic competence.
• Knowledge of different measures of assessing competence and their strengths and limitations.
• Awareness that despite little evidence of correlation with other measures of competence, it may be useful to include self-assessment measures in training programmes.
Comments
No Comments have been published for this article.