Studies omitted

Perhaps most concerning is that three studies with favorable results for MCT were omitted. According to van Oosterhout et al. (Reference van Oosterhout, Smit, Krabbendam, Castelein, Staring and van der Gaag2015), the studies by Aghotor et al. (Reference Aghotor, Pfueller, Moritz, Weisbrod and Roesch-Ely2010) (d = 0.43 for positive symptoms) and Erawati et al. (Reference Erawati, Keliat, Helena and Hamid2014) [η ² partial = 0.20 for Psychotic Symptom Rating Scales (PSYRATS) delusions] were excluded because they did not report both pre- and post-test measures. Similarly, another paper of ours (Moritz et al. Reference Moritz, Kerstan, Veckenstedt, Randjbar, Vitzthum, Schmidt, Heise and Woodward2011a ) was not included because only subscale scores were reported. However, even if such data are not reported in the original papers, it is recommended practice to approach the authors (Higgins & Altman, Reference Higgins, Altman, Higgins and Green2008), particularly when the first author of this commentary is named in the manuscript as a consultant. The authors of the meta-analysis state that they have followed PRISMA guidelines and cite Liberati et al. (Reference Liberati, Altman, Tetzlaff, Mulrow, Gøtzsche, Ioannidis, Clarke, Devereaux, Kleijnen and Moher2009). These guidelines, however, suggest that ‘authors should report if they attempted to acquire any missing information … from investigators and sponsors’ (p. 8). Another recent randomized controlled trial (RCT) by So et al. (Reference So, Chan, Chong, Wong, Lo, Chung and Chan2015) reporting at least medium effect sizes in favor of MCT on delusions and positive symptoms was also omitted from the analysis, despite the fact that this study was brought to the attention of the authors (by us) before the meta-analysis was published.

Further, two novel studies that were not considered for the meta-analysis, presumably because they were recently published, also demonstrate the efficacy of MCT to improve positive symptoms (Gawęda et al. Reference Gawęda, Krężołek, Olbryś, Turska and Kokoszka2015) and insight (Lam et al. Reference Lam, Ho, Wa, Chan, Yam, Yeung, Wong and Balzan2015). The excluded studies suggest that MCT exerts a beneficial effect on symptomatology, which in our view appears to represent a reporting bias in the meta-analysis.

Heterogeneity and moderator analyses

Van Oosterhout et al. (Reference van Oosterhout, Smit, Krabbendam, Castelein, Staring and van der Gaag2015) state that there was ‘a moderate to high level of heterogeneity, which makes it difficult to interpret the findings’ (p. 6). It appears, however, that the purported heterogeneity is mainly due to one study in each analysis. A reanalysis of the data (performed by D.W.) reveals that without the study by Rocha & Queirós (Reference Rocha and Queirós2013), the estimated percentage of observed variation in effect sizes for positive symptoms that is due to heterogeneity drops considerably (from I ² = 36% to I ² = 0%, Q = 6.67, p = 0.464). Without this study, the mean effect size of the remaining eight studies is significant at g = 0.31 (s.e. = 0.103, p = 0.003). A similar situation arises for delusions. Without the original study conducted by van Oosterhout et al. (Reference van Oosterhout, Krabbendam, de Boer, Ferwerda, van der Helm, Stant and van der Gaag2014), the heterogeneity drops from I ² = 46.9% to I ² = 0% (Q = 2.47, p = 0.781). Excluding this study, the mean effect size of the remaining six studies is g = 0.33 (s.e. = 0.109, p = 0.003).

Van Oosterhout et al. (Reference van Oosterhout, Smit, Krabbendam, Castelein, Staring and van der Gaag2015) failed to explain the observed heterogeneity. The heterogeneity stayed the same or increased in almost all of their subanalyses. For example, the meta-analysis examined a Clinical Trial Assessment Measure (CTAM) score as a moderator. A formal test of this moderator would have shown that there is no significant relationship between CTAM score and effect size for positive symptoms (p = 0.880) and delusions (p = 0.244).

Furthermore, careful inspection of the Rocha & Queirós (Reference Rocha and Queirós2013) study, which is mainly responsible for the heterogeneity in the data for positive symptoms, reveals that the reported effect size of g = –0.546 in favor of the control group at post-test is completely due to pretest differences (pretest effect size: g = –0.549). The difference between the MCT and control groups did not change during the intervention [likewise, the study by Balzan et al. (Reference Balzan, Delfabbro, Galletly and Woodward2014) seemingly favors the control condition according to Figure 2 but actually found that MCT is significantly superior to the control condition because differences were very large at baseline]. The key issue here is that the Rocha and Queirós study is a non-randomized trial. There is a large literature on the topic whether and when to combine randomized and non-randomized trials in a meta-analysis (e.g. Reeves et al. Reference Reeves, Deeks, Higgins, Wells, Higgins and Green2008; Shadish, Reference Shadish2011). Indeed, it appears that study design is a significant moderator (p = 0.045) with a difference of about half a standard deviation in mean effect sizes between RCTs and non-RCTs.

A meta-analysis of the six RCTs in the van Oosterhout dataset indicate a significant mean effect size of g = 0.36 (s.e. = 0.122, p = 0.003) in favor of MCT on positive symptoms – similar to the effectiveness of CBT (Wykes et al. Reference Wykes, Steel, Everitt and Tarrier2008) – with a low amount of heterogeneity (I ² = 11.3%, Q = 5.64, p = 0.343).

MCT+ is not CBT

Van Oosterhout et al. (Reference van Oosterhout, Smit, Krabbendam, Castelein, Staring and van der Gaag2015) claim that successful findings in one study of ours (Moritz et al. Reference Moritz, Veckenstedt, Randjbar, Vitzthum and Woodward2011b ) might mirror the effects of concurrent CBT and not MCT. This suggests a fundamental misunderstanding of the individualized application of MCT. Our trial compared a combination of group and individualized MCT (MCT+) with an active control. We have always maintained that MCT+ is a hybrid of the group MCT, CBT and cognitive remediation. However, MCT+ is grounded in the spirit of group MCT and there are many features that distinguish MCT+ from CBT (e.g. focus on cognitive biases). Following the authors’ suggestion, it could be argued that any ‘third-wave’ CBT approach is effective only due to CBT-based common factors.

Evolution of MCT

Following the failed trial by van Oosterhout et al. (Reference van Oosterhout, Krabbendam, de Boer, Ferwerda, van der Helm, Stant and van der Gaag2014), which was the driving influence of the non-significant effect for delusions in their meta-analysis, we have attempted to address the reasons for these negative results. In their study, patients with moderate-to-severe delusions were recruited, which seems appropriate as group MCT aims to improve delusional symptoms. However, at that time we had not taken into account that a homogeneous group of highly deluded patients poses tremendous challenges, particularly for clinicians with less experience, as this subgroup is easily distracted, often sedated by medication, and not very engaged in therapy due to a lack of insight. In keeping with this, secondary analyses of their trial data (the authors generously provided us with the original data) indicate that the site with most expertise in conducting MCT, and where MCT was first translated and implemented, had the biggest effect in favor of MCT (medium to strong effect size). Accordingly, we have updated the inclusion criteria for group MCT and now recommend that clinicians should avoid recruiting patients with severe delusions, and instead consider individualized treatment for these individuals. In addition, an older version of MCT was utilized in the van Oosterhout et al. trial. In subsequent editions of MCT we emphasize the importance of attenuating confidence, which is thought to represent a major mechanism of change. We have also recently added two new modules addressing self-esteem and stigma in light of recent findings suggesting that some patients identify low mood and emotional difficulties as high treatment priorities.

While reviews from our group (narrative review and the meta-analysis by Eichner, Reference Eichner2015) suggest that MCT leads to a significant improvement, much can still be done, and we are currently working on improvements. We also agree with Jiang et al. (Reference Jiang, Zhang, Zhu, Li and Li2015) that further work will be needed to determine the best intensity and duration of the program (e.g. number of sessions). We welcome clinicians to write to us and suggest modifications of the MCT program. Like many other developers of novel therapeutic approaches, we are data driven, and are prepared to undertake revisions in the face of new evidence. However, we believe that the meta-analysis did not include some important studies with positive findings, and in its blanket negative pronouncement might prematurely condemn an approach which could be beneficial in the lives of many patients.