Summary of evidence

We examined the dissemination of a highly influential paper (Leucht et al. Reference Leucht, Hierl, Kissling, Dold and Davis2012) in the scientific medical literature by means of descriptive citation analysis and exploratory regression analysis. This particular paper provided an interesting case study, because, apart from being highly cited, it combined a thought-provoking main finding (i.e., psychiatric drugs are no less effective that drugs in general medicine) with an exhaustive and judicious description of caveats qualifying it. The authors went to great length to underscore their interpretation can only be qualitative and label their analysis as solely intended to put the effectiveness of psychiatric drugs into perspective. It was thus relevant to examine whether the reception on the paper has taken on the whole ‘package’, including the qualitative essence of its findings and the limitations attached, or rather focused on selecting the fragments that could be then molded into support for a priori agendas or post hoc hypothesising.

Over half of the citing articles used the reference to Leucht and colleagues in a context where it served to justify small or modest effects. Our regression analysis suggests this might especially be the case when the point made was in line with the main claim of Leucht's paper (the idea that treatment effectiveness in psychiatry closely resembles that in general medicine). Interestingly, only one paper reversed this idea by citing the target paper to argue that treatment effectiveness in general medicine closely resembles that in psychiatry. In addition, while three quarters of the quotes commented on drugs, some were made to comment on other interventions, such as psychological interventions. It is possible that the contingent of quotes about psychological interventions are a particular subgroup, dealing with therapies with particularly trivial effects (Sayer et al. Reference Sayer, Noorbaloochi, Frazier, Pennebaker, Orazem, Schnurr, Murdoch, Carlson, Gravely and Litz2015). It is worth noting that Leucht and colleagues did not include any meta-analyses of psychological interventions, a fact which excluded the possibility of citing the paper to justify a certain effect. Conversely, for psychotropic drugs, such as for instance the effectiveness of methylphenidate for ADHD, some authors referenced Leucht et al. in lieu of the original meta-analysis included in the overview. Moreover, the paper was cited uncritically. Only seven of the 120 analysed papers mentioned any caveats. Out of these, one was a comment on the target paper (Seemuller et al. Reference Seemuller, Moller, Dittmann and Musil2012), and another was co-authored by its lead author, Stefan Leucht (Leucht et al. Reference Leucht, Helfer, Gartlehner and Davis2015). In spite of the original authors’ insistence on caveats, the subsequent citing literature seems to have overwhelmingly ignored them.

Interestingly, ten of the papers benefited from industry support, with the same pharmaceutical corporation being involved in six of these. Though these numbers are small, they do point to the possibility of the industry becoming sensitive to a potentially reassuring scientific justification of small effects and we would not be surprised to find Leucht et al. referenced as an argument in drug approval documentation. This would represent a dangerous path to follow, as evidence is already accruing that the bar is too low for drug approvals by regulatory agencies (Downing et al. Reference Downing, Aminawung, Shah, Krumholz and Ross2014; Naci et al. Reference Naci, Smalley and Kesselheim2017), as it was the case for the very recent approval of the paliperidone 3-month injection by the EMA (Ostuzzi et al. Reference Ostuzzi, Papola, Gastaldon and Barbui2017).

These findings further supports our thesis that the reception of Leucht et al. in the subsequent scientific research – and indirectly what elevated this paper to highly cited status – was a case of citation bias, by systematically ignoring content that conflicted with the narrative one aimed to advance (Greenberg, Reference Greenberg2009). Of note, such biases are not surprising since previous citation content analyses revealed a plethora of citation distortions (Greenberg, Reference Greenberg2009), and also highlighted worrisome issues with the way highly cited papers are subsequently used in the literature, the same data being interpreted antithetically by different investigators, depending on their prior support for one or the other side of the evidence (Tatsioni et al. Reference Tatsioni, Bonitsis and Ioannidis2007). Undoubtedly, scientific journals may establish clearer rules on their requirements for correct citation and more effort can be invested in the peer review phase to aid accurately checking references and detecting possible misleading citations. However, there are other qualities that cannot be forced upon scientists by means of policy and reporting guidelines, such as being reflective. Readers should always remain vigilant and critical.

Limitations

In the first draft of this paper, we suggested that the specific distortions we identified might reveal something about the psychiatric profession's desire to be integrated with general medicine as described by Joanna Moncrieff (Moncrieff, Reference Moncrieff2008). However, a reviewer suggested that our citation of Moncrieff was not quite accurate, pointing to the fact that she examined psychiatrists’ sense of identity from a historical perspective, referring in particular to the period in the 1950s and 1960s when psychiatrists were considered ‘second-class’ doctors, working in asylums rather than in hospitals. While it is possible that some of that spirit might have survived in modern psychiatrists, we may very well have been distorting the initial idea of the paper, so as to fit it into our own narrative. Therefore, we were adding, albeit unintentionally, a potential citation distortion. We decided to discuss this example here as an illustration of how easy it is to be swayed by one's own ‘story-telling’ and to speculatively decontextualise and interpret previously published literature as supportive.

Because the data we collected were very subjective, we resorted to two independent extractions. Nevertheless, to make our judgement, we relied on both the wording used by the authors quoting the paper and Cohen's classification (Cohen, Reference Cohen1988) defining small, modest and large effect sizes. Finding a more objective method of quantification would have been impossible given that the authors of the citing papers operate with different notions or small or modest. Moreover, results of our regressions analyses should be interpreted cautiously, as such analyses are only exploratory and prone to residual confounding. In addition, our sample of citing papers was small and estimates of regression analyses may be 1/underpowered to uncover subtler associations as for example with COIs and 2/overly imprecise for identified associations (large confidence intervals).

Perspective

It is interesting to note that, with the exception of ADHD, the citations making use of Leucht et al. were about psychiatric conditions and treatments for which effectiveness was shown to be small or modest to moderate at best. For instance, antidepressants in the treatment of affective disorders were consistently shown to have small magnitude benefits (Jakobsen et al. Reference Jakobsen, Katakam, Schou, Hellmuth, Stallknecht, Leth-Moller, Iversen, Banke, Petersen, Klingenberg, Krogh, Ebert, Timm, Lindschou and Gluud2017), particularly when corrected for publication bias (Turner et al. Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008). For antipsychotics for psychotic disorders, a very similar picture emerged with moderate overall effects, further reduced to small effects when corrected for small sample bias and publication bias (Turner et al. Reference Turner, Knoepflmacher and Shapley2012; Leucht et al. Reference Leucht, Leucht, Huhn, Chaimani, Mavridis, Helfer, Samara, Rabaioli, Bacher, Cipriani, Geddes, Salanti and Davis2017). The influential paper of Leucht and colleague provided on many occasions a narrative normalizing such low efficacy: it was particularly exploited to feed into ‘a story’ in which modestly effective psychiatric treatments were normalised by purporting a seeming equivalence of psychiatry with general medicine. The paper was summoned as proof of this equivalence and its quantitative findings were regularly referenced. Effect estimations obtained or discussed in a citing paper were compared to ESs in the Leucht et al. overview of meta-analyses, seen as some sort of benchmark against which the intervention in question can be weighted. The absence of any mentioning of caveats is likely to mislead the reader of the citing paper into forgetting that while numbers might seem similar, that impression is illusory. An effect size is not reducible to just numbers and the very idea of a small, medium or large effect (Cohen, Reference Cohen1988) is intricately interwoven with the type of outcome. A small ES for a psychiatric scale can simply mean a 1 or 2 points intensity change for a symptom rated on a Likert-type scale. However, it could also literally mean lives saved or, conversely, almost nothing at all, depending on whether it was observed on an outcome like a life-threatening event, or on a surrogate outcome with disputed clinical relevance, such as glycated haemoglobin for diabetes (Boussageon, Reference Boussageon2014). In addition, effect sizes depend on variability and may appear higher in small trials of highly selected population than in large-scale pragmatic studies. The fact that the citing literature overwhelmingly obliterated this part is of concern. This entire crucial conversation is by-passed when one authoritatively cites Leucht et al.’s paper. If psychiatry is like general medicine, the storyline seems to go, one need not worry about finding only modest, and often trivial, effects for psychiatric drugs or other treatments, as this is exactly what should be expected.

Post scriptum

To a reader interested in citing our work, please contact us to choose a correct wording. We will not endorse claims such as ‘general medicine is as bad as psychiatry’ or ‘psychiatry is as bad as evidence-based medicine’, but would be delighted to see citations of our work suggesting that some clinical sense and critical thinking are welcome to avoid spin, decontextualisation and oversimplification of important issues in the field of therapeutics.