In the target article, Hooks et al. aimed to summarise the most representative methodologies and results from the field of microbiota-gut-brain research, a controversial topic that has recently gained attention within the psychiatric community. To select the 25 most cited non-review publications, the authors performed two PubMed searches and retrieved respective citations from Google Scholar. The aim was to identify those studies that exerted a broader influence on subsequent research and that received extensive media attention. The limitation of excluding potentially relevant recent studies is acknowledged.

We recognize three major drawbacks to this literature search. First, limiting the search to only one database could be detrimental when trying to widely depict the panorama of a scientific field. PubMed relies primarily on MEDLINE, a database, which – if used alone – has been considered inadequate (Lefebvre et al. Reference Lefebvre, Manheimer, Glanville, Higgins and Green2011). This is because of several issues that could affect search results of randomised and non-randomised studies or systematic reviews (Bramer et al. Reference Bramer, Giustini and Kramer2016; Dickersin et al. Reference Dickersin, Scherer and Lefebvre1994; Wieland & Dickersin Reference Wieland and Dickersin2005), the impact of which on database coverage can float considerably between different topics (Rathbone et al. Reference Rathbone, Carter, Hoffmann and Glasziou2016).

From this perspective, we argue that an extension of the search to more than one database could improve the comprehensiveness of the work, potentially resulting in additional references (e.g., Berer et al. [Reference Berer, Mues, Koutrolos, Rasbi, Boziki, Johner, Wekerle and Krishnamoorthy2011], cited by more than 600 papers according to Google Scholar). This highlights the need to take into account several databases of both references and citation metrics and explore the differences (Kulkarni et al. Reference Kulkarni, Aziz, Shams and Busse2009). We acknowledge that this methodology can be very time consuming, so that an optimal balance with available resources should usually be reached.

Second, descriptive analysis of top-cited articles allows for inferential thoughts on the potential relationship between style and content of a manuscript, as well as the probability of achieving citations over the years (Hafeez et al. Reference Hafeez, Jalal and Khosa2019). Citation trends are potentially exposed to several biases, such as the “hot stuff” bias, the one-sided reference bias, and the positive results bias (Catalogue of Bias Collaboration Reference Aronson, Bankhead and Nunan2017a; Reference Plüddemann, Banerjee and O'Sullivan2017b; Reference Spencer, Brassey and Heneghan2017c; Greenberg Reference Greenberg2009; Ioannidis Reference Ioannidis2005b). Hence, distortions within the citation network can generate unfounded authority of claims through highly biased, non-evidence-based information cascades and persist despite strong contradicting evidence by following publications of higher reliability (Greenberg Reference Greenberg2009; Tatsioni et al. Reference Tatsioni, Bonitsis and Ioannidis2007).

A survey among more than 100 highly cited researchers assessing their top 10 cited papers hinted that top-cited publications may be generally perceived as evolutionary or revolutionary (Ioannidis et al. Reference Ioannidis, Boyack, Small, Sorensen and Klavans2014). Also, strong claims might deceive readers into misinterpreting the manuscript as being an innovative and a pivotal step forward within that research field. Even though an association has yet to be shown, it appears reasonable to suggest that strong claiming leads to more media coverage and, perhaps, to more citations over time. Whether this is proof of a significant impact of the research itself over time remains open to debate (Ioannidis & Panagiotou Reference Ioannidis and Panagiotou2011; Mackinnon et al. Reference Mackinnon, Drozdowska, Hamilton, Noel-Storr, McShane and Quinn2018): “Citation rates are determined by so many technical factors that it is doubtful whether pure scientific quality has any detectable effect at all” (Seglen Reference Seglen1998, p. 226). Several ways that cannot be depicted by bibliometric analyses or metrics indexes by which a research work could be significant have been described (Cheek et al. Reference Cheek, Garnham and Quan2006).

Several known factors and possibly unknown ones can provide a paper a large number of citations. To account for some of them, an analysis on citation sources could yield interesting results. Distinguishing between citations from different publication types (e.g., reviews, editorials, original contributions), sources of citations (e.g., oneself, co-authors, and others), and related research fields (e.g., human studies, animal studies, and microbiological studies) at a study level and extending appraisal of scientific accomplishment beyond citation-based metrics (Ioannidis & Khoury Reference Ioannidis and Khoury2014; Ioannidis et al. Reference Ioannidis, Boyack, Small, Sorensen and Klavans2014) might have added significant depth to understanding the citation network of included studies. Hypothetically, this could influence the conclusions drawn by Hooks et al.

A distinction should, however, be made between the quality of a scientific study and its probability of being cited. Highly cited clinical research is not immune to being contradicted by subsequent studies, especially in the case of non-randomised studies and small-sample-size studies reporting large effects (Ioannidis Reference Ioannidis2005a; Tajika et al. Reference Tajika, Ogawa, Takeshima, Hayasaka and Furukawa2015). Despite supporting the authors’ view that highly cited manuscripts could exert an influence on relevant fields, a warning should be issued: They should not be considered as reliably representative of a scientific area's production or its scientific advances.

Finally, when focusing on the role that a study could play through its citations, omitting its indirect influence through reviews could bias the final interpretation. The authors’ decision is understandable, because the attribution of the impact derived from a study through a review cannot be easily estimated. However, the number of published systematic reviews has increased dramatically over the past decade. Systematic reviews can both achieve a large number of citations and be regarded as increasingly pivotal in influencing the research community, both directly and indirectly through policy making. Hence, omitting their role in shaping both the core literature and the scientific panorama of the field could be considered an excessive limitation when drawing conclusions.

When arbitrary criteria are employed, unavoidable limitations are cast: The quest for comprehensiveness might not completely overcome them, but it reduces the cherry-picking bias. A comprehensive, systematic bibliometric review is therefore suggested to identify, describe, and assess influential methodologies within an emerging research field.