Introduction
The influence of idiosyncratic therapist factors on therapy outcome has been widely recognized in the psychotherapy literature (Beutler et al., Reference Beutler, Mailk, Harwood, Talebi, Noble, Wong and Lambert2004), and is sometimes put forward as the most important determinant of patient outcome (Wampold, Reference Wampold2001). For example, Beutler et al. (Reference Beutler, Mailk, Harwood, Talebi, Noble, Wong and Lambert2004) reported data from 15 studies indicating that therapist style, as measured by positive or friendly behaviours directed towards the client, positively impacted on treatment outcome and reduced dropout rates. More recently, Wampold and Brown (Reference Wampold and Brown2005) analyzed data from 6146 treatments of individual patients who were treated by 581 therapists in a naturalistic setting. They estimated that about 5% of the outcome variance was attributable to the therapists.
With the advent of computerized interventions, guided internet-delivered cognitive behaviour therapy (CBT) has emerged as a promising treatment for anxiety and depressive disorders (Andersson, Reference Andersson2009; Spek, Cuijpers et al., Reference Spek, Cuijpers, Nyklicek, Riper, Keyzer and Pop2007). Effects have at times paralleled the results found in face to face CBT (Bergström et al., Reference Bergström, Andersson, Ljótsson, Rück, Andréewitch and Karlsson2010; Carlbring et al., Reference Carlbring, Nilsson-Ihrfelt, Waara, Kollenstam, Buhrman and Kaldo2005; Kiropoulos et al., Reference Kiropoulos, Klein, Austin, Gilson, Pier and Mitchell2008; Spek, Nyklicek et al., Reference Spek, Cuijpers, Nyklicek, Riper, Keyzer and Pop2007). Although there is an emerging literature on predictors of internet-delivered CBT (e.g. Andersson, Carlbring and Grimlund, Reference Andersson, Carlbring and Grimlund2008), less is known about the role of the patient-therapist relationship in internet-delivered treatments. This is an important question as systematic reviews suggest that totally automated programs without any therapist input are less effective than internet interventions in which at least minimal therapist guidance is provided (Palmqvist, Carlbring and Andersson, Reference Palmqvist, Carlbring and Andersson2007; Spek, Cuijpers et al., Reference Spek, Cuijpers, Nyklicek, Riper, Keyzer and Pop2007). For instance, Spek, Cuijpers et al. (Reference Spek, Nyklicek, Smits, Cuijpers, Riper and Keyzer2007) found a small treatment versus control between group effect size in six internet interventions without therapist support (Cohen's d = .26) and, conversely, a large between group effect size in five internet interventions with therapist support (Cohen's d = 1.00). Palmqvist et al. (2007) found a correlation of rho = .75 (p < .005) between the amount of therapist contact in minutes and the between group effect size in 15 trials dealing with psychiatric conditions. In a paper on the role of therapist factors in internet treatment for depression Almlöv and colleagues included data from two trials involving 10 therapists treating a total of 103 patients (Almlöv, Carlbring, Berger, Cuijpers and Andersson, Reference Andersson2009). The results of a nested one-way model showed that the outcome on a measure of Quality of Life yielded a significant intraclass correlation coefficient for therapists (r = .24, p = .001). There were no signs of therapist effects on depression measures. Gellatly et al. (Reference Gellatly, Bower, Hennessy, Richards, Gilbody and Lovell2007) did a meta-analysis on predictors of outcome of self-help for depression and did not find that therapist background was associated with effects in guided self-help. A series of controlled studies by Titov and colleagues showed that guidance can be equally effective when provided by a clinician or a technician (Robinson et al., Reference Robinson, Titov, Andrews, McIntyre, Schwencke and Solley2010; Titov et al., Reference Titov, Andrews, Schwencke, Solley, Johnston and Robinson2009, Reference Titov, Andrews, Davies, McIntyre, Robinson and Solley2010). However, therapist background is not the same as the role of the individual therapist as two therapist with similar backgrounds theoretically can differ in how effective they are when working with their patients and two therapists with different backgrounds may be equally effective, as showed by Titov et al. (Reference Titov, Andrews, Davies, McIntyre, Robinson and Solley2010).
The role of the therapist in guided internet treatment is not obvious as most of the treatment material is not presented by the therapist. However, even in online programs with prepared materials, the therapist usually responds with e-mails (preferably in encrypted password protected systems) to task assignments, and provides support and encouragement (Andersson, Bergström et al., Reference Andersson, Bergström, Buhrman, Carlbring, Holländare and Kaldo2008). In some programs such as Interapy, the therapist also offers much support almost equalling face to face treatment (Lange, van den Ven, Schrieken, and Emmelkamp, Reference Lange, Van Den Ven, Schrieken and Emmelkamp2001). Therapists in internet treatments may differ in many respects, such as how much empathy they show in their texts, their writing skills, how well they respond to questions from the patient and how they manage to encourage the patient to move forward in the treatment. In addition, a few studies have focused on the therapeutic relationship in internet-delivered treatments. Klein, Richards and Austin (Reference Klein, Richards and Austin2006) found high patient ratings of therapeutic alliance in their online CBT treatment for panic disorder. Knaevelsrud and Maercker (Reference Knaevelsrud and Maercker2007) suggested that the therapeutic alliance can be important in guided internet treatments, but while they found high patient ratings of alliance, associations with treatment outcome were modest. Furthermore, there are some suggestions that therapist factors are important in online therapy (defined as purely e-mail delivered and primarily not containing prepared texts in the form of bibliotherapy). For example, Cook and Doyle (Reference Cook and Doyle2002) used an established a measure of working alliance (Horvath and Greenberg, Reference Horvath and Greenberg1989) and found high ratings of working alliance in online therapy. D'Arcy, Reynolds, Stiles and Grohol (Reference D'Arcy, Reynolds, Stiles and Grohol2006) found that clients in online e-mail therapy provided similar ratings of session impact and therapeutic alliance when comparing their data with previous results from face to face therapies. In sum, previous research indicates that therapist behaviour can have an influence on treatment outcome, even when it is delivered via the internet. However, the role of the individual therapist providing the treatment has not been studied systematically in relation to anxiety disorders.
Different statistical approaches have been used in the literature to estimate therapist effects (McGraw and Wong, Reference McGraw and Wong1996). Researchers increasingly use intraclass correlation coefficients (ICCs; Fleiss, Levin and Paik, Reference Fleiss, Levin and Paik2003) as a means of describing how variance within outcome measures can be attributed to the therapist factor. When investigating therapist factors, ICCs can reveal how participant scores correlate with each other within a specified factor (for example, a certain therapist, a group or a family). One way to obtain this information is to compute nested one-way intra-class correlation coefficients using the “Reliability” command in PSAW (formerly known as SPSS) 17.0.
Researchers often choose to set negative ICCs to zero, since negative proportions conceptually cannot exist if an ICC is interpreted as the proportion of variance in outcome associated with the therapist factor (e.g. Crits-Cristoph et al., Reference Crits-Cristoph and Mintz1991; Maxwell and Delaney, Reference Maxwell and Delaney2004). An alternative definition of a negative ICC would be to view the negative ICC as describing a factor that increases the variance within a group (for example, some patients experience symptom reduction and some experience symptom exacerbation within therapist X), instead of a measure of explained variance in outcome. When ICCs are negative, a diversifying factor is found when a unifying factor is sought. Since this can be confusing, we decided to set the negative ICCs we found to zero, in line with previous research.
Given previous research findings that suggest that therapist input enhances the effects of guided internet delivered CBT, the aim of this exploratory study was to investigate whether the individual therapist could be a contributing factor to the outcome of internet-delivered CBT. With the possibility of therapeutic alliance being formed in internet treatments, and that idiosyncratic therapist characteristics (such as showing empathy, directiveness) can be conveyed in written messages, we investigated if the therapist factor would emerge as a significant factor explaining a proportion of the variance in treatment outcome. It was hypothesized that therapists would significantly differ from each other, but in light of the mixed findings in previous research, with just one significant finding in our previous study on therapist factors in depression treatment (Almlöv et al., Reference Almlöv, Carlbring, Berger, Cuijpers and Andersson2009), we did not expect to find large therapist effects.
Method
Participants and therapists
Data were derived from three completed studies of internet treatment. One study examined generalized anxiety disorder (GAD; Paxling et al., unpublished manuscript), one investigated panic disorder (PD; Carlbring et al., Reference Carlbring, Bohman, Brunt, Buhrman, Westling and Ekselius2006), and in the third study data from a social phobia trial were analyzed (SP; Carlbring, Gunnarsdóttir et al., Reference Carlbring, Brunt, Bohman, Austin, Richards and Öst2007). Outcome measures from the treatment groups, provided by a total of 119 participants from the studies (N = 36 in GAD study, N = 26 in the PD study, and N = 57 in the SP study), were used in the analyses. We included data from patients with pre and post measures who had been treated by therapists with similar experience of CBT and internet treatment. Since the objective was to investigate variance within therapists, therapists had to treat a minimum of 12 patients in a study in order to be included. All included patients were diagnosed with their respective disorders according to DSM-IV criteria (American Psychiatric Association, 2000). More details on inclusion and exclusion criteria are provided in the original trials. In this report we included eight therapists who had treated a minimum of 12 patients (range 12–24). All were clinical psychology students approaching the end of their 5-year education and had completed their basic psychotherapy training, which included clinical work under supervision. The therapists (four female and four male) had received a similar amount of training prior to the studies and were all given clinical supervision during the administration of the internet treatment. The mean age of the therapists was 28.0 years (SD = 1.8) and the mean age of the included participants was 36.0 years (SD = 10.7). Across the three trials 66% of the participants were female. Each therapist treated an average of 13.7 patients (range 4–24). When ICCs were calculated, an even number of patients per therapist (N = 12) was randomly chosen from this sample to function as nested raters of effect within each therapist, yielding a total number of 96 nested patients of therapy outcome.
Treatments
The treatment materials consisted of written book length texts that were divided into modules pertaining to the condition treated in each study. Weekly e-mail contact between therapist and patient was included in all trials. In the PD study, short telephone calls (i.e. 10 minutes) from the study therapists were also included each week as an adjunct to the e-mail correspondence. On a weekly basis therapists had to read messages from the patients and reports on home-work assignments. Feedback had to be given via e-mail (Andersson, Bergström et al., Reference Andersson, Bergström, Buhrman, Carlbring, Holländare and Kaldo2008), often within 24 hours. There was no additional support in terms of participation in online discussion groups. The programs spanned 8 (GAD), 9 (SP) and 10 (PD) weeks respectively. Favourable outcomes were hypothesized to be achieved via the mixture of written texts, home-work assignments and online therapist support (Andersson, Reference Andersson2009).
Measures
Different measures were used in the three studies because different diagnoses were targeted. Three measures were however common across the three conditions studies and were thus available for direct comparisons. The Beck Anxiety Inventory (BAI; Beck, Epstein, Brown, and Steer, Reference Beck, Epstein, Brown and Steer1988), which measures anxiety symptoms over a one-week period via 21 items with a score range of 0–63 points (where higher scores indicate more distress) was used in all studies. Moreover, all studies included the Montgomery Asberg Depression Rating Scale – Self Rated (MADRS-S; Svanborg and Åsberg, Reference Svanborg and Åsberg2001), which measures depressive symptoms via 9 items with a score range of 0–54. Higher scores represent more distress and signs of major depression. The third and last common measure was the Quality of Life Inventory (QOLI: Frisch, Cornell, Villanueva and Retzlaff, Reference Frisch, Cornell, Villanueva and Retzlaff1992), which is a measure designed to assess life satisfaction over a range of circumstances that are believed to be important to most people. The measure includes 16 items that are rated in terms of importance and satisfaction and yields a score that ranges between -6 to +6. Positive scores indicate life satisfaction and negative scores indicate life dissatisfaction in the areas covered (e.g. work, family, health).
An index of the effects obtained from the disorder specific measures was obtained by calculation of a Z-score composite (in accordance with Rosenthal and Rosnow, Reference Rosenthal and Rosnow1991). This composite allowed the main outcome measures from each study to be included in this report. Three measures were used in the Panic Disorder specific composite score. First, the Body Sensation Questionnaire (BSQ: Chambless, Caputo, Bright and Gallagher, Reference Chambless, Caputo, Bright and Gallagher1984), which measures fear of bodily symptoms on a 17-item scale where the total result is the average item score between 1 and 5. Higher scores indicate severity. Moreover, the Agoraphobic Cognitions Questionnaire (ACQ: Chambless et al., Reference Chambless, Caputo, Bright and Gallagher1984) was used to measure catastrophic thoughts. This is a 9-item scale that yields an average score between 1 and 5 on the same premises as the BSQ. Finally, the Mobility Inventory for Agoraphobia (MI; Chambless, Caputo, Jasin, Gracely and Williams, Reference Chambless, Caputo, Jasin, Gracely and Williams1985) was used as a measure of agoraphobic avoidance. This scale is based on 26 situations in which avoidance is measured. Scores on this measure also range from 1–5 as the score is averaged from all items. Due to technical error and a subsequent loss of data, a composite could not be calculated for one of the therapists in the PD study.
Three measures were also used to calculate the composite score for the social phobia study. We used the self-report version of the Liebowitz Social Anxiety Scale (LSAS-SR; Oakman, Van Ameringen, Mancini and Farvolden, Reference Van Oakman, Ameringen, Mancini and Farvolden2003), measuring extent of fear and avoidance behaviours in socially contingent situations, via 24 items pertaining to fear and another 24 items associated with avoidance. The score range for this measure is 0–144 points with higher scores indicating elevated distress. The Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS; Mattick and Clarke, Reference Mattick and Clarke1998) were also used as measures of anxiety pertaining to social interactions. The SPS contains 20 items answered on a scale from 0–4 and summed up for a total score. The SIAS entails 19 items scored in similar fashion as the SPS.
For the GAD specific composite score we used two measures: first, the Penn State Worry Questionnaire (PSWQ; Meyer, Miller, Metzger and Borkovec, Reference Meyer, Miller, Metzger and Borkovec1990), which measures excessive worry. This instrument consists of 16 items, which the participant can give a value of 1–5, thus overall results can range from 16–80, where higher scores indicate more symptom distress. Second, the Generalized Anxiety Disorder Questionnaire-IV (GAD-Q-IV; Newman et al., Reference Newman, Zuellig, Kachin, Constantino, Przeworski and Erickson2002) was used. This scale entails 9 items that are question forms of the DSM criteria. Scores can range from 0–13 points since 4 items yield a maximum of 2 points and five items can yield 1 point each. Higher scores indicate presence of GAD symptoms and 5.7 is considered as a cut-off for the GAD diagnoses. All questionnaires were administered online. The outcome measures have good psychometric properties when administered via the internet (Carlbring, Brunt et al., 2007), which is in line with previous research on internet administration of questionnaires (Buchanan, Reference Buchanan2003).
Procedure and statistical analyses
The randomized trials had comparable designs with randomization to treatment or waiting list group. All three studies showed statistically significant reductions in anxiety and associated symptoms, and all yielded large between group effect sizes across the included measures, which coincidently were all the exact same size (d = 1.0), favouring treatment group outcomes over the waiting list control group conditions. More details about the effects of the interventions can be found in the earlier reports of the studies (Carlbring et al., Reference Carlbring, Bohman, Brunt, Buhrman, Westling and Ekselius2006; Carlbring, Gunnarsdóttir et al., Reference Carlbring, Gunnarsdóttir, Hedensjo, Andersson, Ekselius and Furmark2007; Paxling et al., unpublished manuscript). In the present report we have completed a two-fold re-examination of the change score data with the individual internet therapists introduced as nested variables in a mixed model design alongside calculations of the intra-class correlation coefficient within therapists. The z-score composites and the change scores of the outcome measures that were shared by all three studies were compared between patients grouped within therapists, which in turn acted in different studies.
The data were thus analyzed with two types of analyses. The main outcomes were first subjected to a mixed models design [therapist(study)] in order to reveal potential fixed effects of therapists that might have influenced the original study effects. Second, intraclass correlation coefficients (ICCs) were computed in search of correlation of outcome within patients treated by the same therapists. These coefficients would render significant results if data from patients correlated more strongly with other patients treated by the same therapist than they would with all other participants. The fixed effects analyses thus concern mean level differences between therapists, while the ICCs are correlations between outcomes within therapists (i.e. patients treated by the same therapist). The intraclass correlation coefficients for therapist effects within each data subset were computed in accordance with PSAW syntax derived from the procedures recommended by Kenny, Mannetti, Pierro, Livi and Kashy (Reference Kenny, Mannetti, Pierro, Livi and Kashy2002) and McGraw and Wong (Reference McGraw and Wong1996).
Results
Main outcomes
Descriptive statistics of mean change scores between therapists and studies are presented in Table 1. On the BAI, average change scores ranged between 4.4 to 13.3 points for the different therapists. The MADRS-S change scores range between the respective therapists was 3.2 to 11.8 points. On the QOLI this range spanned between 0.3 and 1.6 points and on the z-score composite the change scores ranged between 0.9 to 1.2 z-score points. The composite scores thus offered the least amount of variation.
Change score means and standard deviations associated with original studies, therapists and overall
PD = Panic Disorder; SP = Social Phobia; GAD = Generalized Anxiety Disorder
Fixed therapist effects
Table 2 describes the outcome of a “Mixed Model” analysis, with all eight therapists nested in the original trials as fixed effects. Statistical probabilities indicative of therapist impact ranged from F(5,111) = .73, p = .605 to F(5,111) = 0.30, p = .912. With an alpha level set at 0.05, no significant differences between therapists nested in the included studies were detected in the outcome measures used in this study.
Output for Mixed Model analysis of pre and post measurement data for BAI, MADRS-S, QOLI and the Z-scored composite
MS = Mean square; p = exact probability; η2 = eta squared (effect size); BAI = Beck Anxiety Inventory; MADRS-S = Montgomery Asberg Depression Rating Scale – Self Rated; QOLI = Quality of Life Inventory
Intra class correlation coefficients of therapist impact on post measure change scores
BAI = Beck Anxiety Inventory; MADRS-S = Montgomery Åsberg Depression Rating Scale – Self-Rated; QOLI = Quality of Life Inventory
Intraclass correlation coefficients of within therapist correlations between patients
Intraclass correlations from the original studies resulted in four intra-class correlation coefficients for the studied outcome measures. One statistically significant intra-class correlation coefficient was found for the Beck Anxiety Inventory (r = .09, p < .05), while no such correlations were found on for the MADRS-S, the QOLI or the composite index.
Discussion
The aim of this exploratory study was to investigate the role of the therapist in internet-delivered CBT for anxiety disorders. The study was motivated by the finding in previous studies that the amount of guidance is associated with better outcome. When therapist factors in internet-delivered treatment were investigated, this factor did not explain statistically significant amounts of variance. While some patient factors have been found to predict outcome of internet treatment, for example severity of initial levels of depression (Andersson, Bergström, Holländare, Ekselius and Carlbring, Reference Andersson, Bergström, Holländare, Ekselius and Carlbring2004; Spek, Nyklicek, Cuijpers and Pop, Reference Spek, Nyklicek, Cuijpers and Pop2008), the therapist factor in the examined internet treatments explained variance to a lesser extent. This is in contrast to the general psychotherapy literature as summarized by Wampold (Reference Wampold2001), who argued that therapist factors are major determinants of outcome.
In contrast, one significant intra-class correlation effect was found for the BAI. This finding can be interpreted in different ways and should be cautiously handled. Even though statistically significant therapist mean level effects were not obtained, this is in part to be expected since very powerful effects would have been needed due the small sample of therapists investigated. It is therefore possible that the analysis of correlations within therapists might be better indicators of therapist impact than the mean level analyses in this study. Another caveat also needs to be mentioned when interpreting the results. Critz-Christoph and Mintz (Reference Crits-Christoph, Baranackie, Kurcias, Beck, Carroll and Perry1991) estimated that studies with a 15 to 1 ratio of patients per therapist, combined with a 5% therapist attributed variance, is liable to render a 15% probability of a false treatment effect. Since the examined studies fall into this category, it is possible that the significant effects of the study protocol might have been driven by a therapist effect that does not come across as differences between therapists, but as between group differences when compared to the control groups. Moreover, the same writers proposed that therapist effects with p-values as high as .2 or .3 might be sufficient to influence treatment outcome. In other words, they suggested that p-values should be adjusted to avoid type II error. By such reasoning, it is possible the therapist impact in this study might have influenced the treatment outcome.
Some other limitations should be mentioned. One is the choice of statistical methods and outcome measures. It is possible that therapist effects are less observable in specific outcome measures, such as the ones included in the composites, and more likely to be found in global measures, such as the Symptom Check List-90 (Derogatis, Lippman and Coti, Reference Derogatis, Lippman and Covi1973). In other words, the role of the therapist could become more noticeable when it comes to less specific effects that are not directly related to the treatment components. Also, the way intraclass correlation coefficients were calculated in this study is more conservative than when therapist variance is contrasted with error variance only. Third, the eight therapists included in this study were of similar experience and provided support and guidance under the controlled conditions of randomized efficacy trials. Hence, therapist effects in more clinically representative settings (e.g. effectiveness studies), and with a larger variation in therapist characteristics such as level of experience and therapeutic orientation, might reveal stronger therapist effects.
In this study we found no significant therapist effects using a mixed model, while some indications of a therapist effect were evident using intra-class correlations. The intra-class correlations and amount of variance explained are not much different from previous face to face therapy studies (for example, Wampold and Brown, Reference Wampold and Brown2005), but with the major difference that we had more limited statistical power to detect effects. One obvious difference from previous studies lies in the treatment format, with markedly less therapist contact in internet treatments. Since treatment effects tend to replicate between administration formats (i.e. face to face vs. internet), a plausible conclusion from this study is therefore that therapist factors could be less influential in internet treatments. This should come as no surprise as the text material included in internet treatments forms a major part of the treatment (often more than 100 pages of text, written mostly in collaboration between authors), and the interaction with the individual patient is substantially reduced compared to regular face to face treatments. Unfortunately, we could not locate directly comparable studies in which therapist effects in manualized face to face CBT for anxiety disorders had been studied, and thus we cannot infer from this study that therapist effects exist to a greater extent in face to face settings in comparison to internet treatments. Therefore, comparative studies of therapist effects across different administration formats (for example, individual therapy, group therapy, guided self-help) are needed. Researchers also need to pay attention to the role of the text in self-help materials as the text may systematically include and pay attention to so called common factors in psychotherapy such as therapist responsiveness and alliance-rupture repair (Richardson, Richards and Barkham, Reference Richardson, Richards and Barkham2010).
In conclusion, in the present study we did not find a readily observable fixed level therapist effect in three manualized treatments for anxiety disorders provided online. However, the ICC-analyses, which can be considered more appropriate with the relatively low power of the study taken into consideration, indicated a presence of a therapist effect. Our analyses thus leaves questions unanswered, as the relevance of the therapist in internet delivered treatments is still not understood satisfactorily. A possible next step would be to study whether the hypothesis stated in this report is better supported by other datasets of internet-delivered CBT.
Acknowledgements
The present study was supported from grants by the Swedish Science Foundation and the Swedish Council for Working Life and Social Research.
Comments
No Comments have been published for this article.