Introduction
It is important to evaluate the effectiveness of specific instances of training (Beidas & Kendall, Reference Beidas and Kendall2010). In addition to large-scale studies of general efficacy, developing an evidence base for training should incorporate an examination of training in real-world contexts (i.e. practice-based evidence; Barkham & Mellor-Clark, Reference Barkham and Mellor-Clark2003) to test whether general findings hold for specific situations. In terms of the broader evidence base, there is a dearth of literature examining whether immediate training-related gains in knowledge and confidence are transferred into subsequent clinical practice (Herschell et al. Reference Herschell, Kolko, Baumann and Davis2010). This study evaluates a workshop-style training programme, facilitated by an accredited practitioner and supervisor in Schema Therapy (ST). To the authors’ knowledge, this is the first study to evaluate the effectiveness of ST training. The training programme was designed to address the extra-curricular learning needs of a regional cohort of trainee clinical psychologists. The trainees had requested an opportunity to develop their knowledge and competence in using ST, as an approach to working with clients who have been diagnosed with personality disorders.
A need for training: working with personality disorders
There is a need to improve service provision for individuals diagnosed with personality disorder (Fanaian et al. Reference Fanaian, Lewis and Grenyer2013). Broadly definable as ‘an enduring pattern of inner experiences and behaviours that deviates markedly from cultural expectations’ (National Co-ordinating Centre for NHS Service Delivery and Organisation; NCCNSDO, 2007), personality disorderFootnote † is highly prevalent in mental health services. It has been estimated that up to 40% of those who are referred to mental health services in the UK meet diagnostic criteria for personality disorders (NICE, 2009). Those presenting to mental health services are often associated with high risk to self (e.g. of self-harm and suicidal behaviours) and consequently require high levels of support (NICE, 2009). In spite of these needs, individuals with a diagnosis of personality disorder have historically been considered untreatable within mental health services (NCCNSDO, 2007). More recently, practice guidelines (NICE, 2009) have acknowledged the mental health needs of those with a diagnosis of personality disorder and recommend the use of psychologically informed interventions by all practitioners working with this client group.
Despite NICE recommendations, there are still concerns about the quality of services for those who have a diagnosis of personality disorder (NCCNSDO, 2007). Mental health professionals often feel they are unable to support such individuals because they lack the knowledge and confidence to work with this population (NCCNSDO, 2007). Service user experiences would seem to reflect this professional unease, as reports indicate that upon receiving a personality disorder diagnosis their treatment and involvement with mental health services deteriorates (NCCNSDO, 2007).
Considering the above, it is important that mental health professionals are trained in approaches that have demonstrated efficacy in working with personality disorder presentations. Evidence-based approaches have the potential to improve service provision, and thereby address the difficulties reported by both providers and service-users with a diagnosis of a personality disorder. An approach that has demonstrated particular efficacy with personality disorder presentations is ST (Nordahl & Nysaeter, Reference Nordahl and Nysaeter2005; Giesen-Bloo et al. Reference Giesen-Bloo, van Dyck, Spinhoven, van Tilburg, Dirksen, van Asselt, Kremers, Nadort and Arntz2006; Farrell et al. Reference Farrell, Shaw and Webber2009).
Schema Therapy
ST is an evidence-based psychological therapy, which integrates principles from a range of therapeutic approaches (e.g. cognitive, psychodynamic, attachment, interpersonal psychotherapy, gestalt) to provide an understanding of, and approach to intervening with, the difficulties associated with personality disorders (Young, Reference Young1990). Although developed for personality disorder, there is some inchoate evidence suggesting that ST may be useful with other complex clinical presentations (see Bamelis et al. Reference Bamelis, Bloo, Bernstein, Arntz, van Vreeswijk, Broersen and Nardort2012 for a review of the empirical evidence).
The central premise of ST is the concept of psychopathology resulting from Early Maladaptive Schemas (EMS; Young et al. Reference Young, Klosko and Weishaar2003). EMS are defined as self-defeating emotional and cognitive patterns that develop early in childhood and are strengthened and elaborated throughout life. Young et al. (Reference Young, Klosko and Weishaar2003) propose that when EMS are ‘triggered’ by events within the environment they result in unhelpful behavioural responses. A range of therapeutic techniques are utilized to enable individuals understand their EMS, the coping strategies they have developed in response to adverse childhood experiences, and more effective ways of meeting their core emotional needs in adulthood.
ST may be useful for improving service provision for those with personality disorders and as such is taught on some clinical training courses in the UK where clinicians with the necessary expertise are available. Where adequate training opportunities and supervision are available, ST could be the ‘one other’ therapy that trainees gain a level of competence in as course requirements dictate. An important question is how can learning and practice of ST achieve this?
Evaluating the effectiveness of training
There is a growing body of literature that supports the benefits of workshop-style training, which integrates expert consultation and supervision (Herschell et al. Reference Herschell, Kolko, Baumann and Davis2010). Herschell et al. (Reference Herschell, Kolko, Baumann and Davis2010) report three methodological flaws: (1) most studies to date have only used pre- and post-intervention data and have not attempted to gather follow-up data to explore whether training was used in practice; (2) researchers have not made the type of teaching style or learning objectives explicit in the method sections of their reports, which makes it difficult to make comparisons across the training evidence base; and (3) there is no conclusive evidence to suggest whether training can be generalized to clinical practice.
The model of training that primarily informed the approach under investigation in this study was one of experiential learning and reflection. In terms of underpinning theory, Kolb's (Reference Kolb1984) model was central to the applied training approach and identifies four phases. These phases are conceptualized in a sequential learning cycle: (1) concrete experience, (2) observation and experience, (3) forming abstract concepts, and (4) testing in new situations. Smith (Reference Smith2010) highlighted two phases that are particularly noteworthy: the use of concrete, real-life experience to test ideas; and the use of feedback (e.g. emotional, cognitive and behavioural) to change practices. Schön's (1983) model of learning from reflective practice (i.e. ‘reflecting-in-action’ and ‘reflecting-on-action’) also influenced the delivery of the training.
Training delivery was further augmented through processes such as didactic teaching and expert modelling. The value of multicomponent approaches to training has been supported empirically (Herschell et al. Reference Herschell, Kolko, Baumann and Davis2010) with evidence to suggest that different components may have differential effectiveness (Bennett-Levy et al. Reference Bennett-Levy, McManus, Westling and Fennell2009) – for example, experiential learning and reflective practice are perceived to be particularly effective for development of relational/interpersonal skills.
The local context and impetus for evaluation
Trainees in the region had expressed an interest in developing their competence in ST and approached a local clinician who agreed to provide training. The facilitator met trainees’ demands by creating a specifically tailored 3-day ST workshop.
The facilitator, working in local specialist services, was keen to have the training formally evaluated and to receive feedback so that future training could be adapted. The facilitator was further interested in whether any increase in knowledge, confidence and willingness led participants to use ST in clinical practice and, if not, identifying barriers to using ST in practice.
Aims
This study aimed to evaluate an ST training programme by:
(1) Reviewing whether ST training increased knowledge, confidence and willingness to use ST and whether any gains were sustained at follow-up.
(2) Examining whether an increase in knowledge, confidence, and willingness were associated with subsequent use of ST in practice.
(3) Identifying facilitators/barriers to using ST in clinical practice.
Achievement of these aims would allow the facilitators to monitor the effectiveness of this ST training programme and any recommendations would offer ways of adapting ST training in general.
Method
Participants
Fifty-seven trainees from the local DClinPsy training course were invited to attend a 3-day ST training event. Nineteen participants (33%) enrolled on the training programme. Participants ranged in age from 24 to 42 years (mean = 29.4, s.d. = 4.95 years); the majority (17/19, 89%) were female. Of the 19 participants, nine (47%) were in the first year of DClinPsy training, nine (47%) were in the second year, and one (5%) was in the third year.
Materials
The first and second authors developed the pre-, post- and follow-up questionnaires, which were approved by the facilitator (third author). All three questionnaires captured the same information on knowledge, confidence and willingness. Participants rated their knowledge, confidence and willingness on an 11-point Likert scale (0 = no knowledge/confidence/willingness to 10 = extremely knowledgeable/confident/willing). The follow-up questionnaire additionally sought to capture whether ST was used in clinical practice. Free text boxes enabled participants to identify barriers/facilitators of using ST in clinical practice. Questionnaires are available from the first author on request.
Procedures
Training was provided by a local facilitator and was self-funded by trainees (or training budget monies were used). Participants were briefed about the study and informed that completion of the questionnaires was optional. Participants were encouraged to use a personal identifier and could withdraw their data, within 3 months of submitting it, by emailing the first author.
Questionnaires were distributed: (1) at the start of training; (2) at the end of training; and (3) 3 months after completion of training. To ensure confidentially, any questionnaire returned by email was deleted once it had been printed. Workshops were delivered over a 2-month period, allowing participants to take time to incorporate ST into clinical practice. A co-facilitatorFootnote † delivered training during the second workshop describing how they adapted ST for an adolescent population.
Analysis
Quantitative analysis was performed using IBM SPSS for Windows v. 19 (www.ibm.com/software/uk/analytics/spss/). The data met the assumptions for parametric tests and were subsequently analysed using repeated-measures analysis of variance (ANOVA) and correlational analysis: examining group-level changes in outcomes of interest (knowledge, confidence, willingness) in addition to their inter-relationship and association with variables gauging reported application in practice. This study further used Reliable Change Index (RCI) calculations to conduct individual-level analysis of changes in the outcomes of interest (Jacobson & Truax, Reference Jacobson and Truax1991). Quantitative content analysis was used to analyse free-text responses (Graneheim & Lundman, Reference Graneheim and Lundman2004) and purposively identify barriers/facilitators to application in practice.
ST training programme
The programme content (Table 1) has been categorized by the authors to address the criticisms levelled at previous research, which typically omitted descriptions of the training content/approach used. The programme used an experiential approach to training, which was underpinned by Kolb's (Reference Kolb1984) learning model and Schön's (1983) reflecting model. These approaches are considered suitable for all learning styles (Smith, Reference Smith2010). The facilitator advised that this style of training mirrors the delivery of ST in clinical practice in that participants would experience both client and therapist roles. Such techniques are core to the ST approach, their aim being to both activate the emotional content of schemas and then to provide corrective emotional experiences using experiential techniques such as imagery re-scripting or chair work in the context of meeting childhood unmet needs. These techniques are considered more powerful than cognitive and behavioural techniques as they capitalize on our capacity to process information more effectively in the presence of affect (Young et al. Reference Young, Klosko and Weishaar2003).
Overview of the 3-day Schema Therapy (ST) training programme
Results
Of 19 participants in the training workshop, 17 completed pre- and post-training questionnaires (89% response rate) and 16 (84%) additionally completed the follow-up questionnaire. One trainee discontinued their training during the 3-month follow-up period, which may have explained the attrition from post-training.
A series of one-way repeated-measures ANOVAs was conducted comparing aggregated scores in knowledge, confidence, and willingness across three time points: (1) pre-intervention, (2) post-intervention, and (3) 3-month follow-upFootnote †. Means and standard deviations are presented along with descriptive statistics in Table 2Footnote ‡. We further examined disaggregated changes by applying RCI computations (Jacobson & Truax, Reference Jacobson and Truax1991). These analyses identified individuals showing statistically reliable changes from pre- to post-training (i.e. changes exceeding what would be expected due to measurement error and test–retest artefacts alone) – and whether individual changes were maintained at the 3-month follow-up.
Descriptive statistics, pairwise comparisons, and reliable change criteria for dependent variables
ST, Schema Therapy.
Dependent variables were scored from 0 to 10, with higher scores indicating greater knowledge, confidence, or willingness (as appropriate).
a Calculated according to Jacobson & Truax (Reference Jacobson and Truax1991), based on the test–retest reliability and pre-training s.d. for each variable; individual-level changes of greater magnitude than this criterion value were considered to be statistically reliable.
Knowledge
There was a significant effect for knowledge over time (F 2,30 = 37.27, p < 0.001). Pairwise comparisons indicated a statistically significant difference between pre- and post-training (F 1,15 = 47.87, p < 0.001). There was no significant difference between post-training and follow-up (p = 0.70).
In terms of individual-level analysis, RCI computation indicated that 9/17 group participants (53%) reported reliable improvements in knowledge from pre- to post-training (i.e. positive change >3.8 points). No individuals showed reliable change between post-intervention and follow-up.
Confidence
Confidence was measured in two domains: (1) confidence in use of ST concepts (theoretical knowledge) and (2) confidence in use of ST techniques (procedural knowledge).
There was a significant effect for confidence in using ST concepts with clients over time (F 2,30 = 71.92, p < 0.001). Pairwise comparison indicated that there was a statistically significant difference between pre- and post-training (F 1,15 = 75.96, p < 0.001). There was no significant difference between post-training and follow-up (p = 1.00)
There was a significant effect for ‘confidence in using techniques with clients’ over time (F 2,30 = 69.48). Pairwise comparison indicated that there was a statistically significant difference between pre- and post-training (F 1,15 = 92.64, p < 0.001). There was no significant difference between post-training and follow-up (p = 0.51).
RCI computation indicated that 11/17 participants (65%) reliably improved their confidence in ST concepts from pre- to post-training (positive change >3.5 points); no change was reported from post-intervention to follow-up. Similarly, 11/17 group participants (65%) reliably improved their confidence in ST techniques (positive change >3.4 points). However, one participant reported deterioration in confidence to use ST techniques from post-training to follow-up (i.e. negative change >3.4 points), which was attributed to a lack of supervision with a qualified Schema Therapist (in qualitative response data). No other participants reported reliable changes from post-intervention to follow-up, suggesting that gains were sustained for 10/11 participants for whom confidence in using ST techniques was reliably improved after training.
Willingness
There was a significant effect for willingness over time (Greenhouse-Geisser: F 2,30 = 5.91, p < 0.001). Pairwise comparisons found that there were no significant differences between pre- and post-training or between post-training and follow-up. However, the difference between pre-training and follow-up reached significance, indicating that willingness to use ST increased from pre-training to follow-up.
RCI computation indicated that only 2/17 participants (12%) reported reliable change from pre- to post-training, and no participants reported reliable change from post-training to follow-up. Limited change in this domain may be attributed to ceiling effects, as pre-training willingness was relatively high (in comparison with confidence and knowledge) – as might be expected in a sample of individuals seeking training. Given that mean willingness at pre-training was 7 (out of 10) the ‘average’ respondent could not have demonstrated reliable improvement (i.e. positive change > 4.9 points).
Partial correlations
Partial correlations (Pearson's r; see Table 3) were applied to examine associations between use of ST with clients (assessed at follow-up) and post-training scores on dependent variables, while controlling for pre-training scores on dependent variables. These analyses thus examined relationships between training-related changes (in ST knowledge, confidence, and willingness) and subsequent application of ST in practice. Preliminary analyses were performed to ensure that data were appropriate for these analyses (i.e. all statistical assumptions were met)
Partial correlations between post-training gains and subsequent integration of Schema Therapy (ST) training into clinical practice
Partial correlations controlled for pre-intervention measures.
*p ≤ 0.05, **p ≤ 0.01.
Post-training knowledge, confidence and willingness showed considerable inter-relationship (r’s = 0.43–0.82); in particular, confidence in using ST techniques was highly (and significantly) correlated with both ST knowledge (r = 0.80) and confidence in applying ST concepts (r = 0.82).
In terms of actual practice of ST after training (as reported at follow-up), the number of clients introduced to ST was significantly associated with training-related gains in ST knowledge (r = 0.58) and confidence to use practical ST techniques with clients (r = 0.67). Conversely, consideration of ST concepts when thinking about clients was significantly associated with training-related gains in confidence to apply theoretical ST concepts to clients (r = 0.62).
Content analysis
Participants were asked to identify barriers and facilitators to transfer of training into clinical practice (free-text responses). The data were subjected to content analysis and four modulating factors were identified: (1) placements, (2) supervision, (3) personal factors and (4) the training process (see Table 4).
Content analysis of the barriers/facilitators to integration of Schema Therapy (ST) training into clinical practice
Numbers in parentheses give the number of participants who commented on this theme within the free text in the follow-up questionnaires.
Placements
A majority of the participants reported that the type of placement (e.g. in terms of population and presenting problems) acted as either a facilitator or barrier to using ST. The feedback indicated that participants were reluctant to use ST with clients who did not exhibit personality disorder characteristics, which is surprising given that an expert facilitator discussed how ST could be implemented for other clinical presentations and client groups (including children and adolescents).
Supervision
Most of the participants reported that they had positive experiences of sharing the ST training with their supervisors and many supervisors offered encouragement for ST to be used within their service. However, supervisors could act as both barriers and facilitators for using ST in clinical practice. Consistent with the evidence base on training therapists (Herschell et al. Reference Herschell, Kolko, Baumann and Davis2010), participants commented that they required on-going support or supervision (including group or peer supervision) to enable them to practice ST in their clinical practice.
Personal factors
Participants who perceived that ST was compatible with their therapeutic orientation, and those who were able to integrate ST within academic assignments, were more likely to use ST in their clinical practice. Some indicated that the training was poorly timed with respect to their personal readiness for change (in terms of available capacity, concordance with other developments, or foundational knowledge)
Training process
Encouragingly, 11 participants reported that training gave them the opportunity to ‘try out’ ST techniques which they claimed enhanced their confidence. In part this seemed to fit with the applied experiential model and intentions of the facilitator. However, it appeared that some participants wanted more practice in applying techniques in a range of situations and with different presentations.
Discussion
Quantitative measures indicated that participants’ knowledge and confidence around the use of ST increased from pre- to post-training, with increases largely sustained at follow-up. Explanatory content analysis of qualitative responses indicated that maintenance of gains was afforded by the opportunity to implement ST in clinical practice. Reported willingness to use ST also showed an increase (from pre-training to follow-up).
There appeared to be particularly strong relationships (r’s = 0.51–0.67) between confidence acquired through training and subsequent introduction of clients to ST (i.e. reported application of learning in practice). However, confidence was not independent of perceived ST knowledge or willingness to use ST, and it would seem logical to expect that successful transfer of learning into practice would require all these attributes.
Individual-level analyses indicated that the majority of participants experienced statistically reliable improvements, but identified an important minority who seemed to benefit less from the training. Content analysis of qualitative responses pertaining to the ‘training process’ helped to elucidate particular ways in which the content of training might be improved for future implementations, drawing attention to individual differences in learning needs and preferences (discussed further below).
Moreover, content analysis identified a number of facilitators and barriers to using ST in clinical practice (i.e. applying learning from training).
Barriers and facilitators to use of training in practice
The three most salient areas that appeared to impact on the use of ST in clinical practice were type of placement, supervision, and training process. Participants indicated a reluctance to use ST on placements where the client group did not typically exhibit personality disorder characteristics: this was in in spite of orientation towards the potential usefulness of ST with other complex clients, who may not respond to first-line interventions (such as CBT). It would seem that participants did not fully embrace the potential for applicability of training across multiple clinical presentationsFootnote †. Where training in evidence-based practices is considered to have broad and transferrable utility, it would seem important to emphasize this – future instances of training may benefit from attention to e.g. breadth of case examples.
Participants further reported that supervisors had a critical role in moderating use of ST in clinical practice. While supervisors were generally perceived to be encouraging, the majority were unable to provide expert supervision, as they were personally unfamiliar with ST. This highlights the need for ongoing expert supervision, which is consistent with previous reports (e.g. Herschell et al. Reference Herschell, Kolko, Baumann and Davis2010). Inviting local supervisors to training sessions could help them to support trainees in transferring learning to practice. Additionally, linking participants to external networks and sources of supervision (including peer supervision) may enable practice which is less dependent on the knowledge, skills, and interests of local supervisors.
Participants appeared to endorse the experiential model of training, indicating that practice of skills within the workshop enabled them to gain confidence. However, it appeared that some participants wanted more practice in applying techniques in a range of situations and with different clinical presentations.
Evaluating the training approach
In evaluating the training approach, it appeared that participants wanted to know more about what to do and when to do it. Correlational analysis suggested that there were two strong associations: (1) between confidence in theoretical concepts (declarative knowledge) and thinking about ST concepts with clients; and (2) between confidence in techniques with clients (procedural knowledge) and introducing ST with clients. These distinct relationships can only be tentatively posited, as there was significant overlap/inter-correlation between conceptual and practical variables which makes it hard to identify specific associations. Training workshops aim to enhance both declarative and procedural knowledge; consequently, to ensure that trainees incorporate both aspects of knowledge into their practice, application of an alternative model of learning may have been useful.
One model that may support learning of declarative and procedural knowledge is a cognitive model of learning (Bennett-Levy, Reference Bennett-Levy2006). Bennett-Levy (Reference Bennett-Levy2006) suggested that therapists in training need to: (1) reflect on declarative knowledge (knowing the theory; e.g. principles and concepts); (2) reflect on procedural knowledge (knowing what to do; e.g. two-chair work, imagery, behavioural pattern breaking); and (3) reflect on both declarative and procedural aspects (knowing what to do and when to do it). Using this approach may overcome above-identified barriers pertaining to the training process.
Limitations and recommendations
In addition to meeting the local requirements for evaluation, this study also contributes to a wider evidence base, supporting the effectiveness of workshop-style training and its potential transferability into clinical practice. There were, however, a number of limitations to this evaluation. Principally, findings were subject to self-report biases. For example, participants may have been motivated to under-report pre-training knowledge and confidence, and subsequently over-report training-related gains (e.g. in response to perceived demand characteristics). Similarly, reports of application in practice were not independently verified, and may be subject to motivated responding or recall inaccuracies. The anonymity of responding was designed to mitigate potential biases, and there was some evidence of more critical commentary and individual differences in reporting of training gains (which may counter suggestions of systematic bias). Nonetheless, future evaluations of this kind would benefit from objective measurement, such as behavioural observation or testing of knowledge and skills (e.g. through case studies, simulated clients, or written assessments).
Acknowledgements
The study was carried out and funded as part of a doctoral course of study (in clinical psychology), undertaken by the first author. It was conducted in accordance with BPS ethical standards and received approval as a service evaluation by the local Trust R&D.
Declaration of Interest
None.
After reading this paper the reader should be able to:
(1) Understand how instances of training can be evaluated using a mixed-methods approach, including analyses of change at both group and individual levels.
(2) Understand a number of potential barriers and facilitators to application of training in practice, with implications for designing future training and post-training support.
(3) Design evaluations of training activities that address limitations of previous work (including those affecting the current study) and thereby strengthen practice-based evidence for approaches to disseminating cognitive-behavioural therapies.
Comments
No Comments have been published for this article.