Introduction
An important yet often neglected area of investigation within the field of training and development has been the exploration of factors associated with training and academic performance. Whilst attention has been given to the role of the therapist on treatment outcomes, little is known about factors associated with performance in therapy training. Cognitive ability and academic attainment are consistently found to predict training and job performance (Richardson, Abraham and Bond, Reference Richardson, Abraham and Bond2012), and personality may add meaningfully to the variance explained (Barrick and Mount, Reference Barrick and Mount1991). A few studies have explored the effect of demographic variables (e.g. age, gender, experience) on performance on postgraduate therapist training programmes; however, courses are not all cognitive behavioural in nature and results are mixed (Siqueland et al., Reference Siqueland, Crits-Christoph, Barber, Butler, Thase and Najavits2000).
The Improving Access to Psychological Therapies (IAPT) initiative affords a valuable opportunity to investigate the influence of therapist characteristics on training outcomes. The English Government has made a commitment to train 6000 therapists to deliver evidence-based Low-Intensity interventions and CBT for depression and anxiety disorders. Within the IAPT framework, Low-Intensity interventions are provided by Psychological Wellbeing Practitioners (PWPs) and CBT is provided by High-Intensity therapists. Training is provided at a postgraduate levelFootnote 1 in partnership between National Health Service (NHS) IAPT Services and Universities, following national course curricula (www.iapt.nhs.uk).
The aim of this study was to investigate the impact of individual characteristics on the training outcomes of trainee PWPs and High-Intensity therapists. It was predicted that: 1) cognitive-ability and academic attainment (degree classification) would be positively associated with clinical and academic performance; 2) the personality dimensions of conscientiousness (positively), openness (positively) and neuroticism (negatively) would be associated to academic work and agreeableness (positively) to clinical work; 3) age and experience would be related (positively) to academic and clinical performance and that gender would be unrelated to either aspect of performance.
Method
Sample
Participants were trainees enrolled on the Postgraduate Certificate (PWP) and Diploma (High-Intensity) courses at Reading University between 2008 and 2012. Of the 239 trainees enrolled on these courses, 140 (59%) consented to be involved (n = 81 (65%) and n = 59 (51%)Footnote 2 ). Trainees were predominantly female (78% and 73%), PWPs were younger (M = 29.5, SD = 7.27 vs. M = 37.7, SD = 8.53) and had fewer years of experience (M = 1.84, SD = 1.86 vs. M = 9.49, SD = 17.01) than High-Intensity therapists. Of those who consented to be involved, 114 trainees completed the measure of personality [NEO PI-R, n = 60 (74.1%) and n = 52 (88.1%)] and 122 completed the measure of cognitive ability [Ravens Progressive Matrices, n = 68 (84%) and n = 54 (91.5%)]. No significant differences emerged across cohorts in terms of age, gender, years of experience or degree grade (see extended report).
Assessments and measures
Training (grades) and demographic data were collected from student records. Clinical and academic work is rated by experienced CBT therapists; in-house inter-rater reliability is good, interclass correlations for High-Intensity and PWP clinical and academic assessments range from .50 −.84.
Clinical assessments
Observed Standardized Clinical Examinations (OSCEs) were used to assess the competence of PWPs. Trainees undertook three OSCEs, which were rated using standardized scales detailed in the PWP curriculum (Reach Out; Richards and Whyte, Reference Richards and Whyte2009).
Cognitive Therapy Scale Revised (CTS-R; Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart and Garland2001) was used to assess High-Intensity clinical competence. Trainees submitted three audio-recordings of therapy sessions which were rated against 12 items on a 7-point Likert scale (0 = incompetent to 6 = expert). The threshold for competence on the CTS-R is 36.
Academic assessments
Written assessments examine trainees’ knowledge and understanding of clinical theory and practice. Three reflective-analyses of 1000 words (PWP and High-Intensity), three case reports of 2500 (High-Intensity), and four written examinations (PWP) were undertaken. Academic assessments were graded 0–100 using internal rating-scales; the pass mark was 50%.
Personality
The Big-Five personality dimensions of neuroticism, extraversion, openness to experience, conscientiousness, and agreeableness were measured using the 240-item NEO-PI-R (Costa and McCrae, Reference Costa and McCrae2006).
Cognitive ability
Ravens Progressive Matrices were implemented. Ravens is a measure of non-verbal reasoning comprising five sets of 12 problems, resulting in a raw score between 0 and 60 (Raven, Raven and Court, Reference Raven, Raven and Court1998).
Demographic variables
Age (in years), gender, and years of relevant clinical experience (not specifically CBT) were collected from student records.
Results
Descriptive statistics
Distribution of assessment grades and mean NEO PI-R and Ravens scores, and comparison with published UK norms (Costa and McCrae, Reference Costa and McCrae2006) can be found in the extended version of this report.
Associations between variables
With alpha set at .01 to control for type 1 error, significant correlations were observed between all assessment types in PWP and High-Intensity trainees (rs = .5 to .77, p < .001 for all) suggesting overlap in the knowledge and skills under evaluation.
Patterns of association between therapist characteristics and performance were generally weak and inconsistent. Hypothesis 1 was partly supported: Ravens was unrelated to clinical or academic performance (rs = .031 to rs = .256, p > .01). However, degree classification was significantly and positively associated with academic work in both groups (rs = −.201 to rs = −.457, p < .01 for all), and with clinical work in High-Intensity trainees (rs = −376, p < .01). Higher degree classifications were associated with higher assessment grades. Results did not support hypothesis 2: agreeableness was the only dimension of personality to be significantly related to performance, but only with the PWP reflective-analyses (rs = .33, p = .01).
Limited support was also found for Hypothesis 3. Age was negatively associated with clinical and academic work across groups but associations only reached significance for the PWP exam (rs = −.35, p = .001), and High-Intensity reflective-analyses (rs = −.34, p = .009). Experience was not significantly associated with performance.
Significant Point-Bisarial correlations between gender and OSCE (rp b = .35, p = .001), reflective-analyses (rpb = −.29, p = .008) and exams (rpb = −.30, p = .008) indicated that female PWP therapists outperformed males, but this was not observed in High-Intensity therapists.
Effect of prior academic attainment
A series of 1-Way ANOVAs were conducted to further explore the relationship between degree classification and performance (see Figure 1). With alpha set at .05, ANOVA revealed a significant effect of degree classification on mean OSCE grade, (F(2,75) = 3.51, MSE = 66.3, p = .04); and CTS-R, (F(2,56) = 5.37, MSE = 54.5, p = .007).A similar pattern emerged for the PWP exam (F(2,75) = 10.12, MSE = 43.81, p <.00), and reflective-analyses, (F(2,75) = 6.93, MSE = 55.51 p = .002). Degree classification also significantly affected high-intensity therapists’ reflective-analysis grades (H(2) = 12.29, p = .002). Results were less robust for the high-intensity case report; although data followed a similar pattern, ANOVA was non-significant, (F(2,55) = 1.80, MSE = 50.5, p = .17).
Mean assessment grades by degree classification for PWP and High-Intensity (HI) therapists
Discussion
This study investigated relationships between therapist characteristics and performance on clinical and academic assessments in trainee Psychological-Wellbeing-Practitioners and High-Intensity CBT therapists. Results provided limited support for the majority of hypotheses.
A surprising finding was the weak association between cognitive-ability and all assessment types. As the range of intelligence narrows from primary to tertiary education, correlations between cognitive-ability and academic success decrease in magnitude (Chamorro-Premuzic and Furnham, Reference Chamorro-Premuzic and Furnham2005); this restriction of range may have attenuated the strength of observed associations.
Degree classification was the only variable associated with clinical (High-Intensity only) and academic assessments in PWP and High-Intensity training programmes. Trainees achieving higher classifications in their undergraduate degree do better clinically and academically than those with lower classifications. This finding mirrors the extensive body of research observing a positive relationship between academic attainment and subsequent training across educational spheres, and extends the limited literature focusing on postgraduate populations. The evidence to date suggests that the best predictor of academic and clinical performance is likely to be past academic performance.
Limited support was found for an association between training outcomes and therapist personality, and the pattern of association was inconsistent across assessments and training groups. This supports existing evidence for the lack of generalizability of personality measures across settings (Barrick and Mount, Reference Barrick and Mount1991).
Older trainees performed less well than their younger counterparts, replicating previous findings (McManus, Westbrook, Vazquez-Montes, Fennell and Kennerley, Reference McManus, Westbrook, Vazquez-Montes, Fennell and Kennerley2010). One possible explanation for this finding is proactive interference; a phenomenon whereby previously acquired knowledge, skills and attitudes attenuate the acquisition of new information. A second explanation is role conflict; older trainees may be juggling multiple roles resulting in reduced quantity and/or quality of study time. Experience was unrelated to clinical or academic performance. It may therefore be the type of experience, rather than experience per se that is important in the prediction of performance.
Contrary to prediction, female trainees performed significantly better, both clinically and academically, than males, but this was true only for PWP trainees. As the association was not consistent across training groups this finding may reflect random variation in ability in male PWP trainees in the current sample. Future research should investigate the male-female performance gap across other IAPT courses to see whether findings are replicated. A gender gap in performance has important implications for course developers and trainers in terms of establishing how to optimize training for both genders.
Strengths, limitations and future lines of enquiry
A strength of the current study is the range of therapist factors that were investigated and the inclusion of a large sample of trainees spanning three academic years on two different courses. Nevertheless, the uncontrolled retrospective nature of the study means that results cannot be conclusive. Since the study was conducted on a subset of PWP and high-intensity therapists trained at one university, the data may not be representative, and the possibility of this is highlighted through the presence of cohort effects in clinical performance and PWP examinations (see extended report). Before conclusions can be drawn, researchers will need to incorporate representative samples and afford greater attention to therapist characteristics in order to more fully understand how they impact upon training performance.
If future research confirms degree classification to be the most important correlate of training success, there are potentially important implications for the selection of candidates for postgraduate training. However, as degrees are not a requirement for PWPs, it may be that a broader perspective on past performance is more useful, including assessments such as appraisal scores as well as GCSE results. A potentially concerning finding was the negative association between age and performance. Correlational analyses provide confirmation of a relationship, but conclusions cannot be drawn on causal link. A qualitative approach may offer a richer, more comprehensive understanding of the barriers faced by older trainees, and will help course developers to provide adequate support to reduce the inequality of performance across the age range.
Conclusions
The current study extends the limited body of research focusing on the effect of individual characteristics on postgraduate performance. With the exception of degree classification, no variables were consistently associated with performance. If individual characteristics are to be used in the selection of therapists for postgraduate programmes a range of variables should be considered, particularly past performance. Furthermore, whilst experience is a prerequisite of entry onto the High-Intensity programme, this study indicates that it would be unwise for course developers to make assumptions that experience necessarily equates to excellence.
Acknowledgements
This study was supported by the 2010 Research Grant from the British Association for Behavioural and Cognitive Psychotherapies (BABCP).
Supplementary materials
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S1352465814000642
Comments
No Comments have been published for this article.