Introduction
The biopsychosocial model of illness has widened the psychiatrist's set of clinical goals. Psychiatrists are asked to reconcile biopsychosocial data gathering with the development of a trusting relationship and a shared decision-making process and to learn the patient-centred interviewing skills that facilitate achieving these goals. Open-ended inquiry, active listening in terms of reflections, checking, summarizing, clarifying and patient-orienting expressions, together with relationship-building skills such as emotion-seeking questions and emotion-focusing statements improve the information-gathering process and contribute to patient satisfaction and participation and thus to better treatment adherence and health status in different medical settings (Stewart et al. Reference Stewart, Brown, Donner, McWhinney, Oates, Weston and Jordan2000; Bodenheimer et al. Reference Bodenheimer, Lorig, Holman and Grumbach2002).
Biopsychosocial information gathering and building rapport are particularly important for first consultations in psychiatry for which high treatment drop-out rates (Percudani et al. Reference Percudani, Belloni, Contini and Barbui2002; Rossi et al. Reference Rossi, Amaddeo, Ruggeri, Thornicroft and Tansella2002; Young et al. Reference Young, Grusky, Jordan and Belin2002), poor treatment adherence (Nosé et al. Reference Nosé, Barbui and Tansella2003), low patient satisfaction (Morgan, Reference Morgan1999) and lack of patient involvement (Goss et al. Reference Goss, Del Piccolo, Mazzi, Moretti, Rimondini and Zimmermann2008) are reported. Patient-centred communication skills would be helpful resources to deal with such qualitative shortcomings.
The importance of enhancing psychiatrists' communication skills is widely recognized. Healthcare organizations such as the Academy of Royal Colleges, the World Psychiatric Association and the American Board of Psychiatry and Neurology have introduced communication skills as ‘core skills’ in the curriculum of psychiatry (Walton & Gelder, Reference Walton and Gelder1999; Swick et al. Reference Swick, Hall and Beresin2006). Programmes on doctor–patient communication are being extended to the psychiatric residency years, and a wide range of assessment tools has been developed (Swick et al. Reference Swick, Hall and Beresin2006).
However, studies that assess the efficacy of time-limited interviewing skill training courses have remained rare in psychiatry, compared with other medical specialities. Three early studies, based on quantitative analyses of word-by-word doctor talk during the consultation, evaluated the effects of a training course on psychiatrists' use of specific patient-centred interviewing techniques (Maguire et al. Reference Maguire, Goldberg, Hobson, Margison, Moss and O'Dowd1984; Lieberman et al. Reference Lieberman, Cobb and Jackson1989; Harrison & Goldberg, Reference Harrison and Goldberg1993). Findings suggested that training in interview skills increased desirable interviewing behaviours, although some skills were more amenable to learning than others. For example, checking and summarizing increased while acknowledging and responding to patients' emotions did not. Undesirable behaviours reducing patients' communicative flow such as asking multiple or closed-ended questions proved resistant to change. Such observations present a challenge to communication research in psychiatry, which only recently has been taken up again (Nuzzarello & Birndorf, Reference Nuzzarello and Birndorf2004; Rimondini et al. Reference Rimondini, Goss, Del Piccolo, Mazzi, Paccaloni and Zimmermann2006).
The present study takes this stagnant area of research on the evaluation of interview skills in psychiatry a step further. The study aims to add methodologically sound and updated evidence by applying a design and advanced methods of evaluation fitting the complexity of factors involved. To assess the effectiveness of time-limited communication training for second-year residents in psychiatry in producing measurable changes in interviewing behaviour we used an interrupted time-series (ITS) design (Campbell & Stanley, Reference Campbell and Stanley1966) with a segmented regression and multilevel analysis to take into account the individual training effect of each participant. ITS is an alternative to randomized clinical trials and is considered the best design to evaluate longitudinal effects of time-limited interventions (Ramsey et al. Reference Ramsey, Matowe, Grilli, Grimshaw and Thomas2003).
The hypothesis was that training would increase patient-centred interviewing skills and/or decrease the use of doctor-centred expressions which reduce patient participation and involvement.
Method
The Verona communication skills training in psychiatry
The course aims to improve residents' interviewing performance during a first psychiatric consultation in out-patient community-based care. Residents learn how to collect reliable information and to establish a relationship with the patient within the time constraints that characterize first psychiatric out-patient visits at most mental health services by using patient-centred interviewing skills.
The course consists of four consecutive weekly small group sessions of 4 h each, with two facilitators (M.R. and C.G.) experienced in learner-centred teaching (Kurtz et al. Reference Kurtz, Silverman and Draper1998; Joyner & Young, Reference Joyner and Young2006). Teaching methods used are feedback on the videotaped consultations of each participating resident, analyses and exercises regarding the transcripts of these consultations, critical incident reports, and role-play with video feedback. Psychiatrists are taught a reasoned and systematic use of open inquiry, patient-orienting and patient-supporting statements, and active listening skills. They are expected to reduce behaviours that inhibit patients' comunicative flow, such as closed-ended questions, instructions and suggestions and disapproving comments.
Subjects
Psychiatric residents
All 10 psychiatric residents (four females, six males), aged between 28 and 44 years, in their second year of the 4-year postgraduate course in psychiatry of the Medical School of the University of Verona took part. This school accepts residents after a careful selection process among applicants from all over Italy. The selection is based on the academic curriculum and a written admission test on basic knowledge in medicine and psychiatry. In the first 2 years residents' clinical activities in the psychiatric ward, the mental health centre and the out-patient clinic are carried out under close supervision. During the second year they may start to see out-patients but have to report back to their senior colleagues.
Simulated patients
Female (n=10) and male (n=10) actors, aged between 20 and 65 years, from a School of Theatre acted as unannounced psychiatric patients at their first visit at the out-patient clinic of the Community Based Mental Health Service of South Verona. Half of each gender had to present either an anxiety or mood disorder, disorders which characterize the great majority of the psychiatric out-patient population attending the clinic. The actors were paid.
Role scripts
For each of the 20 standardized patients (SPs) a different personal role script was created. The scripts were standardized and contained the same set of information items (although of different content) to collect by the interviewer: five sociodemographic characteristics (name, age, education, current profession, marital status), four symptoms of either anxiety or mood disorder [defined by DSM-IV diagnostic criteria], with information on onset and course of symptoms and their impact on functioning and quality of life, the reason for consultation, patient's theory regarding symptoms, treatment expectation, one stressful life event, one severe social problem and information on the patient's social support network. All SPs had been coached individually and had played their role at least twice with one of the authors as interviewer and with feedback by another author after each role-play.
Procedure
The study was approved by the local ethics committee. Residents and SPs gave written, informed consent to be videotaped but were blind to the design of the study. Residents ignored that patients were SPs and SPs that there was a training intervention. The setting of data collection was made as realistic as possible. In September and October 2002, at pre-training (T1), each of the residents interviewed in the out-patient clinic four different female and four different male SPs in random order, half of each gender either presenting an anxiety or mood disorder. Residents had been told by the trainers that these patients had accepted to be videotaped when fixing their out-patient appointment. The training took place in November. In December 2002, at post-training (T2), residents interviewed two new male and female SPs, one of each gender either presenting an anxiety or mood disorder (see Fig. 1). The task of the residents at pre- and post-training was to spend 30 min collecting all the information necessary to understand the SP's current problems and to establish a collaborative relationship. They were not asked to express a diagnosis or suggest a treatment.
Flow chart of the study design and data collection procedure. SP, Standardized patient.
Over the study period residents continued to carry out their clinical duties as usual but were encouraged, as part of the training course, to apply the taught skills in their daily clinical practice. Only after the conclusion of the study were the residents informed that the interviewed patients were actors.
The videotaped interviews were fully transcribed. The talk of the residents was divided into units of speech according to criteria proposed by Butow et al. (Reference Butow, Dunn, Tattersall and Jones1995). The interviews were equally and randomly attributed to two trained raters who coded the speech units independently using the Verona Psychiatric Interview Classification System (VR-PICS). The raters were blinded to the purposes of the study and to whether the interviews had been recorded during the pre- or post-training sessions.
Measures
VR-PICS
The VR-PICS is a modified version of the Verona Medical Interview Classification System (VR-MICS) (Del Piccolo et al. Reference Del Piccolo, Putnam, Mazzi and Zimmermann2004, Reference Del Piccolo, Mead, Gask, Mazzi, Goss, Rimondini and Zimmermann2005), adapted to the psychiatric setting (Rimondini, Reference Rimondini2004) by considering psychiatric in addition to medical content categories. It possesses satisfactory inter-rater reliability, with percentage agreements between 80% and 86% and Cohen's κ between 0.88 and 0.87 (Rimondini et al. Reference Rimondini, Goss, Del Piccolo, Mazzi, Paccaloni and Zimmermann2006).
The VR-PICS consists of 39 mutually exclusive categories, 23 for psychiatrist and 16 for patient speech. For the present study only psychiatrist talk was considered, and the coded speech units were divided into three groups: patient-centred expressions; doctor-centred expressions; and neutral expressions.
Patient-centred expressions
This group contains the selected patient-centred target expressions expected to increase as a result of the training. They are expressions of active listening such as reflections and open-ended and patient agenda completing questions (‘… something else?’), orienting transitions (‘I need more information on this topic before going on …’), sign-posting expressions of clarifying (‘what do you mean by occasional drinker?’), checking and summarizing, expressions of seeking and handling emotions (‘How do you feel about this? You must feel quite upset for this news …’) and asking patients' point of view.
Doctor-centred expressions
This group comprises the set of doctor-centred interventions expected to decrease after training: brief answers (‘yes’/'no'), instructions and suggestions (e.g. ‘you should be more self-confident’), disapproval (e.g. ‘this was not a good thing to do’) and closed-ended questions.
Neutral expressions
The remaining non-target expressions are included in the category ‘neutral’. They include information giving, simple reassurance (‘Don't worry!’), and undemanding behaviour such as simple facilitations (‘Hmm, Ah …’) and transitions (‘Now another question …’), conversation, asking for repetition (‘Sorry?’) or for understanding (‘Okay?’) and agreements (‘You're right!’).
Measure of interview performance
For each consultation a performance index was derived, defined by the ratio of the selected patient-centred to doctor-centred expressions (PtC:DcC) used by the resident. This index was used as an outcome variable to assess training effectiveness. The index increases with increasing use of patient-centred skills and/or with decreasing doctor-centred expressions.
Design
ITS design explores the performance trend within a series of observations and checks if the trend observed is modified by an intervention. Data are collected at multiple instances over time before and after the intervention (here training) to detect whether the intervention has an effect significantly greater than the underlying initially observed trend (Ramsey et al. Reference Ramsey, Matowe, Grilli, Grimshaw and Thomas2003). To increase the goodness of fit of the baseline trend, and consequently the validity of the training effect estimation, a greater number of pre-training than post-training interviews were collected.
Statistical analysis
Unpaired t tests were performed to check whether there was a difference between the mean frequency of the coded speech units per interview before and after the training.
Kolmogorov–Smirnov two-sample tests (D) assessed the effect of the confounding variables [patient gender and diagnosis (anxiety versus depression)] on the performance index PtC:DcC at T1.
The individual pre-training performance indices allowed the determination of the best applicable baseline regression model. First, visual inspection of each resident's time series confirmed a linear form of trends. Second, to explore the presence of heterogeneity among residents' series, within- and between-resident variances were calculated and a linear robust regression estimate (specifying residents' dummy variables and order of interviews as explanatory variables) was determined to obtain the specific level of each resident (Greene, Reference Greene2003). As a third step the presence of first-order autocorrelation was tested using the Wooldridge test for panel data (Drukker, Reference Drukker2003). Finally, a cross-sectional time-series linear model with first-order autoregressive errors was fitted to check the presence of ‘natural’ trends of improvement (here practice effect).
After that, a multilevel segmented regression analysis of ITS on panel data was performed to estimate the following equation:
where y ij is the performance index in the ith interview of the jth resident, t ij is the order of interviews (1 to 12) of the jth resident, a 0+b 0t ij is the linear baseline component estimated before the training (in detail: a 0 is the intercept, which expresses the constant level; b 0 is the slope, which expresses the ‘natural’ trend of outcome), a 1+b 1(t ij – c) is the linear variation component due to training [like above: a 1 is the intercept, which expresses the level change due to training; b 1 is the variation of slope attributable to training; c is the number of interviews at pre-training (8)], d is a dummy variable representing the phase of study (pre-post training), and e ij+u j is the error term at time i for the resident j (representing the random variability not explained by the model, and can be divided in two components: one due to the ith interview within the jth resident, the other to the jth resident).
This equation assesses how much the training changed the interview performance, immediately and over time, producing different kinds of change, gradual (b 1) or abrupt (a 1) (Wagner et al. Reference Wagner, Soumerai, Zhang and Ross-Degnan2002). All 10 performance profiles were assessed simultaneously by a cross-sectional time-series analysis with a first-order autocorrelation. Fixed-effect estimators were chosen to take into account the different characteristics of the residents rather than their representativeness.
To assess the variability among and within residents, analysis of variance (ANOVA) and the intra-class correlation coefficient (ICC) were calculated for both pre- and post-training performance indices. The analyses were performed using Stata version 9.2 (StataCorp LP, USA).Footnote 1Footnote †
Results
The analyses were based on 118 consultations (two pre-training videotapes of resident no. 9 were damaged), comprising a total of 12204 coded verbal speech units of residents' talk. There was a mean of 103 speech units per interview (s.d.=33.2). No difference emerged in the mean speech units per interview at T1 and T2 [106.6 (s.d.=31.4) and 97.3 (s.d.=36.2), respectively, t=1.39, p=0.17]. The performance index PtC:DcC derived from the pre-training interviews was unaffected by the potentially confounding variables of SP gender (1.8 and 1.5 for female and male gender, respectively; D=0.19, p=0.38) and psychopathology (1.8 and 1.5 for anxiety and mood disorder, respectively; D=0.21, p=0.28).
Table 1 shows the overall distribution of patient-centred and doctor-centred expressions (numerator and denominator, respectively, of the performance index) at pre- and post-training. The observed increase of patient-centred expressions, particularly emotion handling skills, and decrease of doctor-centred expressions, particularly closed questions, as well as the overall increase of the performance index from 1.6 (s.d.=1.1) to 3 (s.d.=1.9) at post-training (Table 2) suggests the sensitivity of the target index to hypothesis-expected changes. Table 2 shows how residents vary in their performance and performance change. The heterogeneity assumption is confirmed by ANOVA (pre-training: F=5.72, p<0.001; post-training: F=4.44, p<0.001) and ICC values of 0.34 [confidence interval (CI) 0.13–0.62] for pre- and 0.43 (CI 0.15–0.75) for post-training. Robust regression on pre-training performance indices showed relevant differences among residents (residents no. 1, 3, 6, 7, 8, 9 versus residents no. 2, 4, 5, 10) and the absence of a practice effect (estimated slope=−0.01, p=0.80). The Wooldridge test on pre-training series confirmed the assumption of autocorrelation [F(1, 9)=85.4, p>0.01].
Distribution of patient- and doctor-centred expressions per interview (expressed by VR-PICS categories) comprising the performance index
VR-PICS, Verona Psychiatric Interview Classification System.
Values are given as mean (standard deviation).
Mean frequencies of patient-centred skills (information gathering and emotion handling) and doctor-centred expressions per interview and mean performance index at pre- and post-training, by resident
The regression model evidenced a horizontal slope at T1 and T2 [b 0=−0.02 (s.d.=0.08), p=0.78; b 1=−0.12 (s.d.=0.19), p=0.53], suggesting the absence of a significant effect due to the course of time. The significant level change indicates a positive training effect [a 1=1.75 (s.d. 0.55), p<0.01]. The between- and within-resident variability increased after the training (s.d. from 0.74 to 1.47 and from 0.83 to 1.22, respectively).
Fig. 2 shows for each resident the individual performance profile based on his/her 12 consultations, with the estimated pre- and post-training regression slopes. Two residents (no. 5 and no. 10) showed an irregular performance over all pre-training interviews that was maintained at T2. Two residents (no. 2 and no. 7) showed a particularly irregular profile at post-training.
Line plot of performance index (PI; ratio of the patient-centred to doctor-centred expressions) per interview by residents (1–10) before (sessions 1–8) and after training (sessions 9–12). - - -, Regression slope.
Discussion
This study evaluated the effects of a communication skills training for psychiatric residents, in producing measurable changes in interviewing behaviour, defined by a performance index based on observed frequencies of a selected set of desirable and undesirable behaviours in a series of interviews with unannounced SPs. Our analyses demonstrated a significant performance change in the four interview sessions after the training and allowed us to exclude a practice effect on residents' interview performance in the eight pre-training sessions. Training tended to increase desirable skills, particularly emotion handling, and decreased undesirable verbal behaviour, particularly closed-ended questions.
The uneven performance of residents over their four post-training interviews, compared with that at pre-training, and the corresponding sudden rise of the variability of the performance index was unexpected and suggested difficulties to put the learned skills into practice.
Strengths
The evaluation of the effect of training courses on interview performance in psychiatric settings does not easily lend itself to the collection of data. Differences in consultation length, patients' sociodemographic and clinical characteristics, physicians' attitude and personal style have been shown to affect the interviewing performance of physicians during consultations (Hall & Roter, Reference Hall and Roter1995; Levinson & Roter, Reference Levinson and Roter1995; Del Piccolo et al. Reference Del Piccolo, Saltini, Zimmermann and Dunn2000; Freeman et al. Reference Freeman, Horder, Howie, Hungin, Hill, Shah and Wilson2002) and are likely to interact with training effects also in psychiatry. To account for this variability in a randomized clinical trial design would have demanded sample sizes of residents, patients and consultations beyond the available economic and human resources. The ITS design offered an acceptable alternative. To our knowledge this is the first time that the assessment of interview performance of health professionals was based on repeated interview sessions before as well as after the training and that an ITS design was used in evaluating the interview performance over time. Moreover, the individual training effect of each participant was taken into account in the application of multilevel analysis.
The study met the Cochrane Group criteria of an ITS design (Bero et al. Reference Bero, Grilli, Grimshaw and Oxman2002): at least three time points before and after the intervention; the intervention occurring at a clearly defined point in time; and health provider performance measured objectively.
The balanced use of 20 simulated patients, each portraying a different, although standardized script, assured greater homogeneity in terms of psychosocial data and number and type of symptoms and guaranteed that the interview tasks were similar between and within residents and not affected by SP gender and the presented psychopathology. Residents' blindness was maintained until the end of the study, facilitated by the realistic setting in which SPs were interviewed. It could be argued that some residents may have suspected that patients were not real and that this could have altered their performance. However, Maguire et al. (Reference Maguire, Fairbairn and Fletcher1986) demonstrated that although doctors were told that some of their patients were simulators they could not tell who these were.
Limits
The 10 participants represented the whole population of second-year residents in psychiatry at the University of Verona in 2002, and their interview performance, here expressed as an index, may not be representative of psychiatric residents in general. However, this measure proved to be sensitive to changes determined by the training course and as such could be utilized in other population samples of health providers independently of their performance level at pre-training.
The changes in the performance index after training do not inform us if they were due to an increase in patient-centred techniques, a decrease in doctor-centred interventions, or to both. Due to the small sample size the observed pre-/post-training differences could not be tested for statistical significance, but trends suggested that changes were particularly pronounced for emotion handling and closed enquiry. The high instability of our outcome measure over the four post-training observations would have required a greater number of observations and increased the robustness of our results. Given the new frontier nature of our study we could not anticipate the destabilizing effect of the training and the difficulty of residents to put their learning into practice. Without any preceding evidence on this phenomenon in the literature we predicted the pre-training performance estimates of each resident. Moreover, given the lack in the literature on repeated observation of communication skills over time, power calculations could not be performed.
Main findings
No gold standard can exist defining what would be the appropriate frequencies of single doctor- or patient-centred interventions in psychiatric consultations. However, inspection of the mean use of some patient-centred and doctor-centred expressions suggests that pre-training interviews were characterized by many closed-compared with open-ended questions, a modest use of orienting transitions and by rare emotion handling expressions (an interviewing approach examined in detail in a first analysis of this dataset; Rimondini et al. Reference Rimondini, Goss, Del Piccolo, Mazzi, Paccaloni and Zimmermann2006). Similar consultation characteristics have been previously observed in psychiatry (Maguire, Reference Maguire1982; Goldberg et al. Reference Goldberg, Hobson, Maguire, Margison, O'Dowd, Osborn and Moss1984) and in other medical specialties, such as general medicine and gynaecology (Suchman et al. Reference Suchman, Markakis, Beckman and Frankel1997; Van Dulmen et al. Reference Van Dulmen, Nuebling and Langewitz2003; Del Piccolo et al. Reference Del Piccolo, Mead, Gask, Mazzi, Goss, Rimondini and Zimmermann2005). Psychiatrists were reported to be more empathic than other physicians (Hojat et al. Reference Hojat, Gonnella, Nasca, Mangione, Vergare and Magee2002) and to have better relational skills (Sierles et al. Reference Sierles, Vergare, Hojat and Gonnella2004). These observations contrast with the infrequent use of emotion-handling skills of our residents at pre-training, despite the longer duration of the psychiatric compared with other medical consultations, and the greater prevalence of psychosocial and emotional problems, competently simulated by our SPs.
The increase of emotion-handling interventions after the training represents an affirmative answer to one debated issue in clinical education: whether empathy and acknowledging of patient emotions can be taught. The finding, however, refers to verbal expressions. Non-verbal indicators, crucial to modulate the verbalized emphatic message, were not considered here.
The reduction in the frequency of closed-ended questions was another desirable course effect, but was not substituted, as expected, by a parallel increase in the use of open questions. This last learning target was not achieved and should be considered in future training courses, although the observed increase in reflections, transitions and other patient-centred interventions at post-training could have diminished the need of directive open-ended questioning. An alternative hypothesis is that training improved the quality of open-ended questions so that fewer questions were needed to obtain the required information. Additional qualitative analyses that would complete the picture and allow a deeper understanding of residents' improvements were beyond the scope of this paper.
The interview performance index varied between and within residents before and particularly so after training. These findings have implications for communication research and teaching. The performance variation of residents, expressed by the ICC at pre-training, will help future studies in this field perform the recommended power calculations for the determination of sample size and expected size of improvement in performance (Bland, Reference Bland2000). The observed differences between residents before training parallel previous findings that course participants rarely start from the same level of expertise and have a distinctive interviewing style (Hulsman et al. Reference Hulsman, Visser and Makoul2005). Teaching should be tailored to individual learning needs, to be assessed beforehand. Different approaches and skill levels may initially generate conflict in trainees, but become powerful teaching resources by offering the chance to share opinions and to learn from peers (Lieberman et al. Reference Lieberman, Hafner and Crisp1978). The residents' performance profile as well as the pre-/post-training comparison in the use of patient-centred expressions and closed inquiry confirmed that individuals have different ways of putting into practice what has been learnt (Lieberman et al. Reference Lieberman, Cobb and Jackson1989; Harrison & Goldberg, Reference Harrison and Goldberg1993; Nuzzarello & Birndorf, Reference Nuzzarello and Birndorf2004). Moreover, the increase in the performance variability within residents from pre- to post-training implies that assessment of interviewing skills based on a single interview would not present a realistic estimate of the learner's performance. Post-training exercise and supervised practice are needed in order to establish a more regular performance at a higher skill level which in order to become observable would require a prolonged follow-up period for evaluation.
In conclusion, our findings on training effects are encouraging concerning the acquisition of skills, but their full integration into practice has still to be demonstrated, suggesting that special thought has to be dedicated to the organizational and content-related aspects of the follow-up period. The ITS design described here appears to be a useful method for the evaluation of time-limited interventions in psychiatry. Future research may take advantage of the parameters and observations offered in order to confirm and extend the results of the present study.
Acknowledgements
This paper received no external funding. The authors thank Professor G. Dunn of the University of Manchester (UK) for statistical advice.
Declaration of Interest
None.
