Introduction
The efficacy of cognitive behaviour therapy (CBT) has been demonstrated in several psychiatric conditions (Hofmann et al., Reference Hofmann, Asnaani, Vonk, Sawyer and Fang2012) and its efficacy has prompted calls for its increased dissemination in routine clinical settings. To achieve effects similar to the original efficacy trials, effective dissemination of CBT is needed and requires good training and supervision of therapists. Unfortunately, systematic descriptions and empirical evaluation of supervision of psychotherapy – including CBT supervision – have been sparse. Milne and James (Reference Milne and James2000) provided a systematic review of CBT supervision and concluded that CBT supervision seems to improve skills and therefore potentially improves the quality of treatment and impact of psychological interventions. Alfonsson and colleagues (Reference Alfonsson, Parling, Spännargård, Andersson and Lundgren2018) were more cautious in their review of the documented benefits of supervision, stating that the empirical support for the effects of supervision on therapist competence and patient outcomes is virtually non-existent (Alfonsson et al., Reference Alfonsson, Parling, Spännargård, Andersson and Lundgren2018). As Milne et al. (Reference Milne, Reiser, Cliffe and Raine2011a) have also noted, there are relatively few valid and reliable instruments to evaluate supervision. Milne and Reiser (Reference Milne and Reiser2011) recommended measuring competency of clinical supervision by direct observation and to focus on the evaluation of tools to observe and quantify the quality of supervision on design issues (psychometric properties), implementation issues (practical use) and clinical relevance (utility). Reviewing the available methods, they concluded that there is a need for observational instruments to measure the quality of CBT supervision. Furthermore, these instruments should have acceptable psychometric properties as well as being practical to use (not time-consuming or demanding extended training).
Existing tools such as ‘Teachers PETS’ (Milne et al., Reference Milne, James, Keegan and Dudley2002), SAGE (Milne et al., Reference Milne, Reiser, Cliffe and Raine2011a) and the Supervisory Competency Scale (SCS) (Kennerley and Clohessy, Reference Kennerley, Clohessy, Mueller, Kennerley, McManus and Westbrook2010) have attempted to address this need for a valid and reliable tool for evaluating supervision. Whilst these instruments are noteworthy they still contain many items (23–33 items), which may prohibit their regular use in a busy clinical setting. Thus, it could be argued that there is still a need for a brief and valid tool to evaluate CBT supervision in everyday clinical settings.
Supervision as a clinical act implies a process of communication between supervisor and supervisee, consisting of intertwined components. These components can be separated to analyse the style of a supervisor. Based on a review of the current supervision literature, with a particular focus on CBT supervision (Corrie and Lane, Reference Corrie and Lane2015; Pilling and Roth, Reference Pilling, Roth, Watkins and Milne2014), we identified six competency domains covering the core aspects of CBT supervision, the supervisor’s: Theory, Focus, Learning strategy, Techniques, Structure, and Interpersonal style (Møller and Rosenberg, Reference Møller and Rosenberg2016) (see also Table 1).
Moeller, Moerch, Rosenberg Supervision Scale (MMRSS)
The Moeller, Moerch, Rosenberg Supervision Scale (MMRSS) was developed to evaluate a supervisor’s performance within each of these six domains after direct observation of supervision. The scale enables the assessment of the supervisor’s current supervision skills, and it can guide skill development within one or more of the six domains. The MMRSS has an explicit focus on learning strategies and cognitive behavioural techniques applied by the supervisor, which is unique compared with other supervision tools, which as a rule have primarily focused on the supervision alliance (Milne et al., Reference Milne, Reiser, Cliffe, Breese, Boon, Raine and Scarratt2011b). Furthermore, the MMRSS was designed to be easy to implement and not too time consuming to promote clinical utility. As this tool has been recently developed, the psychometric properties and utility of MMRSS needs to be tested.
The aim of the present study was to examine the feasibility of MMRSS in the evaluation of CBT supervision. We focused on reliability, construct validity, and clinical utility. Reliability was assessed with the inter-rater agreement of two independent raters. Construct validity was examined by comparing some of the domains in the MMRSS with another CBT supervision measure, the SCS (Kennerley and Clohessy, Reference Kennerley, Clohessy, Mueller, Kennerley, McManus and Westbrook2010; Section I). We used the SCS as it was clinically easy to apply and seemed to include similar theoretical constructs as the MMRSS. The clinical utility from a supervisor and supervisee perspective was examined by a questionnaire on user satisfaction.
Method
Participants
A total of eight (two male, six female) CBT supervisors were recruited for the study from the Mental Health Centre Capital Region, Northern Zealand (n = 7 psychologists) and from Mental Health Centre North Jutland (n = 1 medical physician). These supervisors provided videos of their group supervision, resulting in 21 videos in total (2–6 videos per supervisor). The supervisors all had more than 4 years of experience in the treatment and provision of supervision according to CBT principles. The age range for supervisors was 26 to 68 years (mean 44.3 years).
Training of raters
The two independent raters were qualified CBT therapists and supervisors with >15 years of experience. They participated in a 1-day training workshop in the use of the MMRSS and the SCS (Kennerley and Clohessy, Reference Kennerley, Clohessy, Mueller, Kennerley, McManus and Westbrook2010; Section I). This training included rating supervision videos to establish inter-rater reliability. Raters participated in further calibration ratings after rating 4 and 8 videos to ensure that good inter-rater reliability was maintained.
Procedure
To evaluate the inter-rater reliability and construct validity, the participating supervisors were asked to submit digital recordings of their group supervision for the feasibility study. Those digital recordings were collected by the study coordinator (S.B.M.). To examine inter-rater reliability and construct validity, the two independent evaluators rated all the video-recordings of supervision separately. Each rating was based on watching a video of the supervision session. The two raters completed both the MMRSS (Møller and Rosenberg, Reference Møller and Rosenberg2016) and the SCS (Kennerley and Clohessy, Reference Kennerley, Clohessy, Mueller, Kennerley, McManus and Westbrook2010; Section I).
To evaluate the clinical utility and satisfaction of the MMRSS to evaluate CBT supervision, this measure was used by the supervisors and supervisees in supervision training. Participating supervisors and their supervisees viewed and rated a video of their own supervision as part of the group supervision training session. Afterwards the supervisors and supervisees completed a questionnaire about their experience of using MMRSS in supervision. The questionnaire was completed by the supervisors and the supervisees in their group supervision right after using the MMRSS as part of the group supervision training session. In this evaluation in total 25 supervisors and supervisees completed the satisfaction questionnaire.
Measures
Moeller, Moerch, Rosenberg Supervision Scale (MMRSS; Møller and Rosenberg, Reference Møller and Rosenberg2016)
The MMRSS is an observer-rated supervision rating scale, consisting of six domains of a supervisor’s performance: Theory, Focus, Learning strategy, Techniques, Structure, and Interpersonal style. Each of the domains are conceptualized from a cognitive behavioural perspective. Theory refers to the use of CBT theory by the supervisor and how the supervisor responds to theoretical input from the supervisees when these are not relevant or congruent with CBT. Focus refers to the degree to which the supervision question(s) are clear and transparent, and to which degree the focus of the process is related to them. Strategy refers to the incorporation of relevant learning strategies (e.g. dialogue, feedback, questioning, role play, brainstorm, reflection, teaching, reflective team, experiential practices) based on the supervision question, with higher scores allocated for combining reflection and experience. Techniques (e.g. thought record, functional analysis, case formulation, imaginary rescripting, exposure) refers to the specific CBT methods discussed or used in the supervision, including the degree to which the supervisor uses them appropriately throughout the supervision session. Structure refers to how successful the supervisor is in establishing a plan for the session and is aware of providing a clear structure in the supervision process, including the degree to which other supervisees are involved to participate in the process. Finally, Interpersonal style refers to the type of contact and learning climate the supervisor promotes. The supervisor needs to demonstrate a relevant balance in being active and directive, supportive and helpful, collaborating and sharing responsibility, and show a relevant expectation of performance from the supervisee(s).
The MMRSS was developed to evaluate a supervisor’s performance within these domains after direct observation of one or more sessions (live or recorded). The evaluation is presented as a simple profile for the six domains (six items in total), each rated on a 5-point Likert scale summing to a total of 30 points. This scale covered the following ratings: 4, clearly present and very well applied; 3, clearly present and adequately applied; 2, present but not adequately applied; 1, not clearly present or not relevantly applied; 0, not at all present or inappropriately applied.
Supervisor Competency Scale (SCS; Kennerley and Clohessy, Reference Kennerley, Clohessy, Mueller, Kennerley, McManus and Westbrook2010; Section I)
This was developed to evaluate supervision. The scale is based on recognized competency domains within psychotherapy supervision practice and informed by the Pilling and Roth (Roth and Pilling, Reference Roth and Pilling2008; Pilling and Roth, Reference Pilling, Roth, Watkins and Milne2014) competencies with the intention to facilitate the development of good supervisor skills following the guidelines for supervisory practice. A supervision session (live or recorded) is viewed and rated on six domains concerning supervisor’s abilities: structuring the session, enhancing learning, supervisory relationship, other process issues, professional and ethical practice, and reflective practice (this last item concerns the supervisor’s development and was not included). Each domain has 3–9 items, which are rated on a 5-point scale for competency. The scale contains 33 items and is currently being investigated in a larger validation study; preliminary data supporting the psychometric properties of SCS was presented at the British Association for Behavioural and Cognitive Psychotherapies (BABCP) conference (Kennerley, Reference Kennerley2014; conference proceedings).
Satisfaction questionnaire
A questionnaire regarding experience and satisfaction using the MMRSS was developed for the study. This questionnaire included seven questions covering ease of use, discrimination between different supervision domains, clinical utility, experienced skill development, and the experience of time resources used to complete questionnaire. Each question was rated on a 4-point Likert scale ranging from completely disagree, to totally agree. In addition, participants were given the opportunity to provide qualitative feedback on using the MMRSS.
Statistical considerations
Inter-rater agreement of the MMRSS was assessed using the kappa coefficient. The MMRSS internal consistency was estimated using Cronbach’s alpha. To examine the construct validity of the MMRSS, correlations between the MMRSS parameters and the corresponding SCS domains were undertaken. All analyses were conducted using IBM SPSS 25.
Results
The MMRSS internal consistency was estimated using Cronbach’s alpha (αc = 0.75), which indicates acceptable internal consistency. Descriptive results showed variability in the scoring of the 21 videos for the MMRSS. The item’s scores ranged from 0 to 4, with a summed mean of MMRSS of 22.3, range 16–29. The inter-rater agreement for the domains is given in Table 2. The kappa values for the six domains ranged between 0.71 to 0.84, which indicates moderate to good agreement between raters and an acceptable level of inter-rater reliability.
Inter-rater reliability on separate domains for the MMRSS
Examining the validity of the constructs in the MMRSS was difficult as the SCS which we used in the study as a validated measure for CBT supervision has some subscales that do not directly correspond to the MMRSS constructs. Four domains from the MMRSS were compared with three corresponding domains from SCS. The SCS domain Structuring the session was compared with a mean composite score in Focus and Structure domains in MMRSS. The SCS Enhancing learning domain was compared with the MMRSS domain of Learning strategy, and the SCS domain of Supervisory relationship was compared with the MMRSS domain of Interpersonal style. The correlations between MMRSS and SCS domains ranged between 0.60 to 0.86. A full summary of the correlations is shown in Table 3.
Correlations between the Moeller, Moerch, Rosenberg Supervision Scale (MMRSS) domains and Supervisor Competency Scale (SCS) domains
To determine the supervisor and supervisee experience of using MMRSS, the questionnaire responses were summarized. Most participants rated the MMRSS as easy to rate across the six domains, although roughly 20% of participants found the Focus, Techniques and Interpersonal style domains slightly difficult to rate. The majority of participants found it relatively easy to distinguish between the six domains; they felt the MMRSS was a useful tool to develop supervision skills and indicated that they would use it in their role as a supervisor. Over 85% of participants indicated that using MMRSS was not too time consuming. A complete summary of the results is provided in Table 4.
Satisfaction using the Moeller, Moerch, Rosenberg Supervision Scale (MMRSS) as part of the supervision training session; n = 25
Discussion
The following study examined the feasibility of the MMRSS as a tool to evaluate CBT supervision. Results indicated that MMRSS had acceptable internal consistency and construct validity as several of the domains in MMRSS with a validated questionnaire (SCS) for cognitive supervision. Regarding the investigation of the construct validity, the correlation between SCS domain of structure and MMRSS domains focus was high, indicating that these two domains are measuring the similar constructs. There was moderate correlation between the SCS domains Enhancing learning and Supervisory relationship compared with the MMRSS domains of Strategy and Interpersonal style, respectively. Thus, while the SCS and MMRSS domains share some variance, the results also indicate that there are some differences between the constructs the SCS and MMRSS are measuring. As such, the MMRSS seems to provide a picture of the supervisor’s competency with emphasis on the professional methodology.
The inter-rater agreement was overall satisfying, with the Theory, Focus and Structure domains showing good inter-rater reliability and the Learning strategy, Method and Interpersonal domains acceptable inter-rater reliability. It is important to place these ratings in the context that they are based on only a relatively small number of videos and further psychometric testing is needed with a larger number of supervision sessions and raters. Whilst acknowledging the limitations of these results, it appeared that MMRSS could be used to reliably evaluate supervision competencies, and based on these preliminary results the MMRSS displayed adequate psychometric properties as an instrument.
Results from the satisfaction questionnaire (80% of the participants reported the MMRSS easy to use and 100% reported it clinically helpful) supported the satisfaction and acceptability of MMRSS as an evaluation tool for supervision. These results also indicated that the MMRSS has the potential to assist both a supervisor and a supervisor–supervisee in the training and development of supervision competencies. The MMRSS is applicable to everyday clinical practice because it can be completed in a short amount of time and can be integrated into training by being used as a structure to provide clinical feedback on actual supervision. As such, it demonstrates clinical utility and could be implemented in a busy clinical setting to support the development of competent CBT clinicians.
The MMRSS pays equal attention to crucial supervisor competencies as CBT methods involved in the supervision, the applied learning strategy, and the focus of the supervision process. This should mirror a modern evidence-based view on supervision conceptualizing supervision as a learning opportunity focusing on the development of trainees’ CBT competencies in supervision. Furthermore, the MMRSS is a simpler instrument than the SCS in the sense that the SCS has many items that might not be relevant in a given session, while all items of the MMRSS are considered relevant for every session. This makes it suitable for evaluation of all supervision sessions and all you need for an evaluation is one supervision session.
Limitations
The study is a small, preliminary study including only a small assessment battery and inter-rater reliability data derived from only two trained raters. The specificity of the domains of the MMRSS has not been substantiated in the present study. As such, the findings of this study need further empirical investigation with a larger number of supervision sessions and raters before leading to clinical decision-making. Additionally, using the scale requires training to secure reliable ratings. Moreover, the increased priority of focusing on the learning perspective of supervision may have the disadvantageous consequence of paying less attention to the therapeutic alliance and process issues. Another potential limitation that the MMRSS does not highlight ethical issues and diversity that are included in some other supervision assessments. Regarding the MMRSS as a supervision tool, it places more focus on summative evaluation but can also be used as part of a formative supervision process if ratings across the five domains are used to highlight which supervisory skills need to be developed and a plan how these skills can be attained. Finally, the preliminary psychometric properties of the SCS presented at the BABCP conference (2014) were unable to be retrieved.
Summary
This study examined the feasibility of the MMRSS as a valid and easy to use supervision tool, and preliminary results indicated this tool is psychometrically sound and clinically useful. Future studies are needed to evaluate this tool in a range of supervisory settings including testing the psychometric properties in comparison with other newly developed tools such as short-SAGE (Reiser et al., Reference Reiser, Cliffe and Milne2018) and to examine the ability of the MMRSS to detect changes in skills throughout supervision.
Acknowledgements
We are grateful to the supervisors and their supervisees for participating in the study.
Financial support
This research did not receive grants from any funding agency, commercial or not-for-profit sectors.
Conflicts of interest
Stine Moeller, Stephen Austin, Morten Hvenegaard, Krista Straarup, and Nicole Rosenberg have no conflicts of interest with respect to this publication.
Ethical statements
The study was conducted according to the Ethical Principles of Psychologists and Code of Conduct as set out by the APA. The participating supervisors received verbal informed consent from their supervisees regarding the recording of the supervision session. All supervisors and supervisees received written information about the study. Participating supervisors signed a consent form. The study was approved by the National Committee on Health Research Ethics in Denmark (case number RHP-2017-051, med I-Suite no. 05921).
Key practice points
(1) There is a need for observational instruments with which to measure CBT supervision that have acceptable psychometric properties and display clinical utility (not time consuming to complete or demanding lengthy complicated training).
(2) The Moeller, Moerch, Rosenberg Supervision Scale (MMRSS) (Møller and Rosenberg, Reference Møller and Rosenberg2016) covering six domains (Theory, Focus, Strategy, Method, Structure, and Interpersonal style) was developed to provide a tool to evaluate clinical supervision of CBT.
(3) The MMRSS has an explicit focus on learning strategies and cognitive behavioural methods applied by the supervisor, which is unique compared with other supervision tools that have typically focused more on the supervision alliance.
(4) The MMRSS could be used to reliably evaluate supervision competencies, and based on these preliminary results the MMRSS displayed adequate psychometric properties as an instrument.
Comments
No Comments have been published for this article.