Setkowski et al. (Reference Setkowski, Palantza, van Ballegooijen, Gilissen, Oud, Cristea and Cuijpers2023) published an ambitious paper asking, ‘Which psychotherapy is most effective…in the treatment of … borderline personality disorder?’ They employ network meta-analysis (NMA), a statistical procedure that estimates the comparative effectiveness of interventions never trialed against each other, but against the same comparator, most often ‘treatment as usual’ (TAU), but also, other interventions that are better matched in dose, focus, and expertise with the treatments of interest.

The authors enumerate limitations of their study responsibly, which include first and foremost the heterogeneity of control conditions in studies. This is a more methodologically concerning limitation than is discussed. Beyond this, we find other meaningful problems. Most importantly, the classification of the treatments analyzed is dubious, particularly around the distinctions between TAU, community treatment by experts, generic treatments (GT), psychodynamic psychotherapy (PDP), and ‘mixed’. Third, using borderline personality disorder (BPD) severity as one of the main outcomes is questionable – even though it is relevant from a clinical perspective, because the instruments assessing BPD severity is exceedingly diverse across the studies, rendering the synthesis of effect sizes less reliable, likely leading to biased results.

The choice of NMA has theoretical advantages but practical limitations. High-quality meta-analyses using pairwise comparisons, mostly with TAU as its comparison anchor (Cristea et al., Reference Cristea, Gentili, Cotet, Palomba, Barbui and Cuijpers2017; Storebo et al., Reference Storebo, Stoffers-Winterling, Vollm, Kongerslev, Mattivi, Jorgensen and Simonsen2020), report and replicate similar findings. Highly intensive, specialized, lengthy therapies targeting both symptoms and core mechanisms of BPD have better outcomes than the general treatments available in the community, historically known to elicit negative therapeutic reactions from patients with BPD. What NMA does beyond these studies is include studies using better matched comparisons than TAU. These studies find smaller differences between treatments. Matching comparators as much as possible to the treatment of interest facilitates assessment of whether the distinct mechanisms provide incremental value over the essential generic attributes including placebo effects, expectancy, and institutional prestige. Research demonstrates that specific elements of trial design do influence outcome, including for example use of active comparator or placebo (Rutherford, Sneed, & Roose, Reference Rutherford, Sneed and Roose2009). In addition, placebo responses in medication trials are not static, and appear to increase over different decades (Rutherford et al., Reference Rutherford, Pott, Tandler, Wall, Roose and Lieberman2014).

Mills, Thorlund, and Ioannidis (Reference Mills, Thorlund and Ioannidis2013) published a brief guide to ‘Desmystifying trial networks and meta-analysis’, which explains how NMA works. They use the example of trials for different anti-coagulants in the treatment of atrial fibrillation to prevent strokes. Examples such as this involve relatively standardized interventions such as warfarin or aspirin, whose efficacy is unlikely to be altered by the setting or agent who administers it. Furthermore, assessment of outcome, that is in this case stroke, is relatively easy to code and compare. This becomes much more complex and subjective in trials of therapies for BPD.

The primary problem in the BPD treatment literature is the classification of non-brand name comparison interventions. Setkowski and colleagues face a difficult task. The first substantial methodological error is that general psychiatric management (GPM) is classified as a GT (Table 1) but is analyzed in the NMA as a PDP (Table 2). GPM, which reduced BPD symptoms and suicidality/self-harm comparably to dialectical behavioral therapy (McMain et al., Reference McMain, Links, Gnam, Guimond, Cardish, Korman and Streiner2009), is not pooled into the GT effects and may therefore contribute to improving the effects for PDPs. The treatments classified as GTs include a diversity of approaches: supervised team management (STM; Amianto et al., Reference Amianto, Ferrero, Piero, Cairo, Rocca, Simonelli and Fassino2011); structured clinical management (SCM); client-centered therapy (CCT); and Rogerian supportive therapy. GPM is much like STM and SCM in that it incorporates APA or national guidelines for the care of BPD. But it is also like CCT which similarly focuses on patient's problems of aloneness. GPM was also led by Paul Links with his team of community experts when trialed so is it also a CBTE? GPM incorporates psychodynamic concepts so can certainly be considered a PDP but eschews any focus on childhood experiences, which Setkowski et al. identify as a main element of PDPs. This problem that GPM poses for this analysis is only one example of how controversial the classification of comparator treatments is.

We also question the ‘mixed category’. Manual assisted cognitive therapy (MACT) (Weinberg, Gunderson, Hennen, & Cutter, Reference Weinberg, Gunderson, Hennen and Cutter2006) consists of six individual sessions that are ‘structured around a chapter of a booklet, covering: functional analysis of episodes of parasuicide… emotion regulation strategies, problem-solving strategies, management of negative thinking, management of substance use, and relapse prevention strategies’. MACT organizes these traditional CBT techniques in a booklet format, making it a broadly defined CBT treatment. MACT can hardly be called an integration of dialectical behavioral therapy (DBT), CBT, problem solving therapy, and relapse prevention. Not only does the brevity of MACT preclude that, but also the very philosophy of these treatments has not been preserved in MACT. The introduction of the ‘mixed’ category obfuscates an important issue that most treatment approaches require adaptation of existing therapies to make them effective with BPD patients.

Our third objection regards the use of BPD severity as a main outcome, despite the fact many trials do not properly assess BPD severity. Among the trials with GTs, none use accepted measures of BPD severity, but are still included in the NMA for this outcome. Ideally, a pre-processing step should be conducted to first unify these different instruments, so that their marginal distributions are of similar shape and modality and the effect sizes across different trials, as measured by for example standardized mean differences, are not infused with artificial variability introduced by varying sensitivity of different instruments in detecting changes in BPD severity. This pre-processing is difficult in the context of meta-analysis, because access to the original trial data at the individual level is limited.

When specialized treatments become widely available, identifying what works best of many effective BPD treatments might be reasonable. However, it is unlikely gold standard therapies for BPD will ever be available in supply to meet the public health need for them worldwide (Iliakis, Sonley, Ilagan, & Choi-Kain, Reference Iliakis, Sonley, Ilagan and Choi-Kain2019). If BPD treatments become as available for general administration as aspirin or warfarin, then parsing the best between effective options will make sense.