At the beginning of 2018, Cuijpers et al. published a paper on the role of bias in influencing the estimation of the effects of psychotherapy for depression (Cuijpers et al., Reference Cuijpers, Karyotaki, Reijnders and Ebert2018a). Prior to undertaking the analyses presented in that paper, the authors made a series of decisions and established criteria regarding the parameters of their analyses including: (1) excluding studies of waiting list controls and non-Western studies; (2) including only studies with low risk of bias and (3) using a methodology to estimate the number of unpublished studies as a measure of publication bias and the effect on effect size when data from missing studies were imputed. Cuijpers and colleagues found that the mean effect size for psychotherapies for depression was 0.31. This is below previous estimates, which included all the studies comparing psychotherapy with any control group, and corresponds to a small effect according to Cohen (Cohen, Reference Cohen1992; Higgins, Reference Higgins and Green2011). As a follow-up to this paper, and given their interest in meta-analyses for psychological interventions (Fluckiger et al., Reference Fluckiger, Del Re, Barth, Hoyt, Levitt, Munder, Spielmans, Swift, Visla and Wampold2018; Munder and Barth, Reference Munder and Barth2018), Munder and colleagues re-analysed Cuijpers’ data (Munder et al., Reference Munder, Fluckiger, Leichsenring, Abbass, Hilsenroth, Luyten, Rabung, Steinert and Wampold2018). Starting from different theoretical assumptions – mainly related to the choice of waiting list condition as the most appropriate control, the decision to not consider the study's risk of bias, and the inclusion of non-Western studies – this new analysis produced an estimate close to 0.70. This estimate was consistent with the moderate effect size identified in the Cuijpers’ analysis that included all the studies.

These papers by Munder and Cuijpers show how widely findings of meta-analyses may differ when performed based on different assumptions, with important implications in terms of results and interpretation. In this commentary we describe some methodological issues when considering the work of Munder and Cuijpers from a global mental health perspective.

First, the choice of control group is particularly relevant to research in low- and middle-income countries. In the Cuijpers’ paper, the choice of excluding the waiting list as a control condition was motivated by the issue of limiting participants seeking care for their mental condition elsewhere because they are waiting for the therapy (Mohr et al., Reference Mohr, Ho, Hart, Baron, Berendsen, Beckner, Cai, Cuijpers, Spring, Kinsinger, Schroder and Duffecy2014; Cuijpers and Cristea, Reference Cuijpers and Cristea2016). This is not a concern in many low- and middle-income countries where alternative treatments are not available. One way to mitigate concern over this problem in the future might be to limit data to people with long-lasting conditions. In many low- and middle-income countries, participants suffer from long-lasting and even chronic conditions because they lack the possibility of receiving evidence-based treatments. Although Cuijpers favours the choice of treatment as usual (TAU) as a control condition, he also recognises that TAU may vary according to populations and contexts, to the point that being in the TAU condition sometimes corresponds to not getting treatments at all (Cuijpers et al., Reference Cuijpers, Karyotaki, Reijnders, Purgato and Barbui2018b), and differentiating TAU from no treatment or from waiting list control might become difficult. This is true in many low- and middle-income countries where there are no available treatment options outside the study context.

Second, we stress the importance of including studies in which psychotherapy is provided by para-professionals. In many parts of the world there are limited resources to deliver standard psychotherapeutic interventions defined in Munder's paper (Munder et al., Reference Munder, Fluckiger, Leichsenring, Abbass, Hilsenroth, Luyten, Rabung, Steinert and Wampold2018). Evidence is mounting for the effectiveness of psychotherapeutic interventions delivered by para-professionals in low- and middle-income countries without extensive training or experience in mental health, but trained for the delivery of a specific intervention (Morina et al., Reference Morina, Malek, Nickerson and Bryant2017; Patel et al., Reference Patel, Saxena, Lund, Thornicroft, Baingana, Bolton, Chisholm, Collins, Cooper, Eaton, Herrman, Herzallah, Huang, Jordans, Kleinman, Medina-Mora, Morgan, Niaz, Omigbodun, Prince, Rahman, Saraceno, Sarkar, De Silva, Singh, Stein, Sunkel and Unutzer2018; Purgato et al., Reference Purgato, Gastaldon, Papola, Van Ommeren, Barbui and Tol2018). While Cuijpers et al. included in his analysis several types of psychotherapy, like self-help or web-based interventions, Munder et al. applied stricter criteria regarding what constitutes psychotherapy: face-to-face meetings and having special experience or training in the handling of human relationships. For example, Munder criticised the choice of Cuijpers of including a study in which a short cognitive behavioural therapy (CBT)-based intervention was delivered by nurses with no specific mental health expertise except a 4-day training on how to detect depression and how to deliver techniques based upon self-management and CBT (i.e. reattribution of negative cognitions and problem solving) (Lamers et al., Reference Lamers, Jonkers, Bosma, Kempen, Meijer, Penninx, Knottnerus and Van Eijk2010). While including interventions provided by para-professionals may increase heterogeneity of the interventions (Fletcher, Reference Fletcher2007; Purgato and Adams, Reference Purgato and Adams2012), however it may allow for inclusion of more studies conducted in countries with limited financial and human resources (Singla et al., Reference Singla, Kohrt, Murray, Anand, Chorpita and Patel2017, Reference Singla, Raviola and Patel2018; Cuijpers et al., Reference Cuijpers, Karyotaki, Reijnders, Purgato and Barbui2018b; Purgato et al., Reference Purgato, Gastaldon, Papola, Van Ommeren, Barbui and Tol2018). If heterogeneity is a concern, an option for including these studies in systematic reviews could consist of conducting separate meta-analyses within the same review, or planning subgroup analyses, for example accounting for the type of region in which studies were conducted, the type of interventionist, or the presence/type of humanitarian crisis. Exclusion of these studies would result in missing an important group of rigorously tested treatments that actually share many characteristics with standard psychotherapies (Barbui et al., Reference Barbui, Purgato, Churchill, Adams, Amato, Macdonald, Mccleery, Minozzi and Syed Sheriff2017).

Moreover, a similar reasoning on heterogeneity applies to other characteristics of psychotherapeutic interventions, as these interventions are complex and many variables contribute to increased heterogeneity. For example pooling interventions with different number of sessions (i.e. one session v. 60 sessions), or targeting different populations (i.e. young v. elderly) in the same meta-analysis is problematic. At present, psychotherapy literature has been mainly focused on the comparison between (different types of) psychotherapies together against ‘inactive’ controls as on waiting lists, no treatment, or TAU. However, the papers by Cuijpers and Munder highlighted that it is difficult to draw firm clinical conclusions from meta-analyses. First because each ‘inactive’ control presents specific characteristics that may contribute to influence the effect size. Differences across control groups have been detected also in a subgroup analysis accounting for the type of control group (waiting list, no treatment, ETAU) on the outcome depressive symptoms at endpoint (p  =  0.02) and at 1–4 month follow-up (p  =  0.0001) in a recent systematic review on psychotherapies in low- and middle-income countries (Purgato et al., Reference Purgato, Gastaldon, Papola, Van Ommeren, Barbui and Tol2018). The second reason is that even when we follow strict criteria in defining psychotherapy, meta-analysing different interventions even in the same category often results in clinically meaningful statistical heterogeneity (i.e. I ²≥75%). In this case the risk of merging ‘apples and oranges’ (Eysenck, Reference Eysenck1978) applies within interventions belonging to the same category.

Finally, many types of psychological and psychosocial interventions have been introduced and tested in low- and middle-income countries. Research in global mental health might benefit from head-to-head comparisons considering separately each type of psychotherapeutic interventions v. each other, v. ‘inactive’ controls, but also v. other intervention option including less resource-intensive psychosocial interventions in homogeneous population groups. Moreover, given that efficacy has been proven for many types of psychological interventions (Tol et al., Reference Tol, Barbui, Galappatti, Silove, Betancourt, Souza, Golaz and Van Ommeren2011; Morina et al., Reference Morina, Malek, Nickerson and Bryant2017), it would be worth investigating the mechanism of action of these interventions (i.e. mediators and moderators). This could help inform the best matches between population groups and evidence-based interventions. This would enable the generation of more informative research for clinicians and for policy makers involved in the management of health systems that incorporate mental health interventions across contexts.