Introduction
Scientific research publications may be retracted when the results are no longer felt to be reliable. In 2023, an all-time high of 10,000 research publications were retracted, driven by the retraction of more than 8,000 articles by the publisher Hindawi.Reference Van Noorden1 Multiple studies have documented a rise in the rate of retraction over the past three decades. One determined that the retraction rate reached 0.2% in 2022,2 and a review of European biomedical science publications found that the retraction rate quadrupled between 2000 and 2021.Reference Freijedo-Farinas3 An analysis based on data from Scimago and the Retraction Watch database reported an increase from 3.5 retractions per 10,000 articles in 2014 to 11.2 per 10,000 articles in 2022.Reference Petrou4 Collectively, these analyses indicate that the increase in retractions is not merely attributable to an increase in the volume of the scientific literature.Reference Fortunato5
Retractions can be viewed as a positive sign that the scientific community is able to identify and remove flawed publications, demonstrating that science can correct itself. However, retracted publications also pose a threat to scientific integrity. Retracted content may be cited as fact, whether deliberately or in error,Reference Bolland, Grey and Avenell6 and may continue to promulgate misinformation for years after retraction.Reference Suelzer7 A retraction for any reason is usually an indication that a project has failed, and it would be preferable if retracted articles were not published.
Analyzing retracted publications can provide insights into ongoing problems in the state of the scientific enterprise. In 2012, Fang et al. conducted a comprehensive analysis of biomedical research articles retracted between the 1970s and early 2010s.Reference Fang, Steen and Casadevall8 Their analysis found that misconduct, which included fraud or suspected fraud, duplicate publication, and plagiarism, accounted for most retractions. Since then, several developments have changed the nature of scientific publishing, which might be reflected in the nature of retractions. Although a “publish-or-perish” culture and the use of bibliometric data to evaluate scientists continue to encourage research quantity over quality, at least in some countries and at some institutions,Reference Aliukonis, Poškutė and Gefenas9 the rise of open access models has shifted the financial burden of publication from readers and institutional libraries to authors themselves.Reference Linacre10 Open access and the consolidation of for-profit publishersReference LaRiviére, Haustein and Mongeon11 carry incentives for journals to publish as many articles as possible. In tandem, the rise of electronic publishing has increased the global reach of journals while decreasing the overhead costs of publication, allowing new journals to become established more quickly than in the past.Reference Stigbrand12 At the same time, the increasing use of social media for academic discourse has made it easier to identify and publicize flawed publications, in what is referred to as post-publication peer review (PPPR), using platforms such as PubPeerReference Yeo-Teh and Tang13 and X (formerly Twitter).Reference Knoepfler14
In view of these changes, the present study was undertaken to reassess the major causes of retraction during the COVID-19 pandemic of the early 2020s, which reflect changes in the nature of publishing along with the unique context of a public health emergency. While COVID-related retractions may not be generalizable to the rest of the scientific literature, the pandemic posed a major challenge to science, and its effects have significantly impacted scientific publishing. One obvious effect has been a rapid increase in research output directly relating to the pandemic. Elsevier journals recorded a 63% increase in health-related manuscript submissions from 2019 to 2020,Reference Squazzoni15 and the number of PubMed-indexed publications increased by 10%.Reference Sloane and Zimmerman16 In the midst of the rapidly evolving public health crisis, many journals expedited their peer review processes; in a sample of 11 journals during the first six months of 2020, COVID-related articles posted on the preprint server MedRxiv were published after peer review a mean of just 19 days later, compared with 91 days for non-COVID-related articles.Reference Aviv-Reuven and Rosenfeld17
It is also noteworthy that the COVID-19 pandemic was one of the first scientific crises to occur in the social media age. In parallel with the increasing politicization of science,Reference Stroebe18 scientific misinformation about COVID-19 was able to spread rapidly through platforms such as Facebook, Instagram, and X. Although enhanced visibility may have exposed flawed research to greater scrutiny, many problematic publications were widely read and influenced individual or community health decisions before (and sometimes long after) they were formally reviewed and retracted.Reference Tuccori19 The rate of retractions was already increasing before the pandemic,Reference Furuse20 and the COVID-19 pandemic may have further accelerated this trend.Reference Yeo-Teh and Tang21
The present study analyzes 244 COVID-19-related and PubMed-indexed articles that were retracted between 2020 and early 2024, with a focus on the reasons for retraction and the retraction of articles relating to highly politicized or controversial topics.
Methods
Search Criteria
The authors performed a search of the National Library of Medicine PubMed database, with a final search as follows:
(SARS-CoV-2) OR (COVID*) Filters: Retracted Publication
The search was limited to articles published on or before March 28, 2024. In total, 244 publications were returned. Between April 2 and April 18, 2024, each paper was assessed for the following variables: days from online publication to retraction, length of peer review process (calculated as date of acceptance minus date of submission), reason for retraction, author agreement (whether the retraction statement notes if authors agree/disagree), COVID-19 subtopic, PubPeer comments, Altmetric Attention Score (a weighted count of online attention for a given paper across policy documents, Wikipedia, mass media, and social media), country of origin (of the corresponding author), citations, open access status, and Conflict of Interest statement. A subset of articles was abstracted twice by different individuals to ensure accuracy.
Categorization of Retracted Articles
Articles were classified in the following categories of study: Computational Study (analysis of large database), Clinical Trial, Observational Study, Survey, Review, Editorial, Case Report or Miscellaneous other categories (see Table S3).
The reason for retraction was chosen from the following categories based on the published Statement of Retraction and other publicly available information: Misconduct/Suspected Misconduct: data alteration, falsification, figure modification, or misrepresentation of cited references; Plagiarism: deliberate copying of a previous work and presentation as one’s own, original work; Error: honest mistakes in analysis, interpretation, or data sources, including instances in which data use violations or a general lack of rigor led to retraction; Peer Review Manipulation: subversion of the peer review process by using deception to produce favorable reviews; Failure to Obtain Ethics Approval; Administrative Error: inadvertent publication due to journal error; Duplicate Publication; or Miscellaneous Ethical Issues (e.g., failure to disclose conflicts of interest, simultaneous submissions, sale of authorship, the omission of co-authors, or the addition of authors without their knowledge).
Statistical Analysis
Data analysis was carried out in Excel (Microsoft, Redmond, WA). Pairwise comparisons were performed in Graphpad using Student’s t-test, with p-values ≤ 0.05 considered significant.
Results
Summary Data
A total of 244 indexed publications were identified in the PubMed search. These publications came from 116 different journals and 34 different publishers, with 85 journals retracting only a single article related to COVID-19 (Tables 1 & S1; see Table S2 for papers not published by the Hindawi publishing group). Forty-seven publications (19%) noted in their retraction statements that the authors agreed with the suggested retraction. Other retraction statements noted that the authors disagreed, partially disagreed, could not be contacted, or failed to specify author concurrence. One hundred seventy-three articles (71%) did not declare a conflict of interest.
Key Demographics of Retracted COVID-related Papers
Note: Due to rounding, some sections may tally above 100%
* categories with ≤5 articles classified as other (see supplemental data for additional details)
** co-corresponding authors from different countries were listed in both country categories, making n=253 for this analysis
Retracted articles were most likely to be published by research groups based in China (20%), the United States (12%), India (9%), or Saudi Arabia (9%) (Tables 1 & S2). These data align with observations that the United States and China were two of the top three most productive countries in the performance of COVID-19 randomized controlled trials between 2020 and 2022.Reference Zyoud22 In addition, Saudi Arabia, India, and China are three of the eight countries with the highest retraction rates.23 Retracted articles were also most often computational studies (26%), observational studies (17%), or reviews (14%) (Tables 1 & S3).
Nearly all (95%) retracted articles were open access. Approximately two-thirds (68%) had entries in PubPeer, a major online platform for post-publication peer review.
Duration of Peer Review and Time to Retraction
Of the 244 publications analyzed, the mean duration of peer review, defined as date of acceptance minus date of submission, was 71 d (Table 2), with a median duration of 50 d. Forty-seven publications failed to provide information about the duration of peer review, while 30 publications reported a review time of less than 7 d. The average length of time between publication and retraction was 403 d (13.2 months).
Characteristics of Retracted COVID-related Papers
Reasons for Retraction
The most common reasons for retraction were error (51/244, 21%) and peer review manipulation (45/244, 18%) (Figure 1). Approximately 24% (58/244) of retracted publications included retraction statements that did not provide a clear reason for retraction.
Reasons for Retraction. The percentage of retracted COVID-related papers are shown for the full dataset (n=244) or publications excluding those published by Hindawi or Wiley (n=161). See text for details. Fraud includes both confirmed and expected misconduct. For comparison, the percentage of retracted papers in a 2012 analysis (n=2047) are also shown.68 *Other includes journal error, authorship disputes, and miscellaneous ethical issues.
The present dataset includes 83 publications that were retracted by two publishers, Wiley and its subsidiary Hindawi. In 2023, Hindawi retracted more than 8,000 articles due to concerns that the peer review process was compromised, particularly those contained in special issues. As a result, Wiley announced that it would absorb existing Hindawi titles back into the Wiley brand. A separate review of COVID-19-related Hindawi retractions found that in general, their findings had limited impact on COVID-19 management decisions.Reference Peterson24 Because these retractions were related to problems associated with a specific publisher and comprised a substantial subset of the retracted articles, a separate analysis was conducted that excluded publications published by Hindawi, Wiley, or a collaboration between these publishers (Figure 1), and Table S4 summarizes key demographics of non-Hindawi COVID-19 retracted papers. Among the non-Hindawi non-Wiley publications, error accounted for the largest proportion of retractions (~30%), followed by various ethical issues (19%) and misconduct/suspected misconduct (14%) (Figure 1). This represents a lower proportion of retractions for misconduct compared with the 2012 analysis25 (Figure 1).
Controversial Topics in Retracted Articles
Of the 244 publications in the present analysis, 56 focused on one of the following topics: unproven therapies for COVID-19 (46%), COVID vaccines (31%), lockdowns (14%), and face masks (9%). Retracted publications that focused on one of these subtopics had a higher average number of PubPeer comments (mean 3.9 vs. 0.9 comments for non-controversial topics, p<0.00001) and a higher Altmetric Attention score (mean 2238 vs. 70, p<0.00001) compared to retracted publications that did not relate to these topics. In addition, retracted publications that focused on these topics had a shorter mean time to retraction (mean 10.8 vs. 13.8 mos, p=0.02).
Discussion
The present study analyzed trends in the scientific retraction landscape during the COVID-19 public health emergency, which significantly impacted the scientific literature. Most retracted COVID-19-related articles were retracted due to peer review manipulation or error, rather than fraud, suspected fraud, or plagiarism, representing a substantial difference from patterns observed in a 2012 study, which found that misconduct was responsible for the majority of retracted articles in the biomedical literature.26 Only a small proportion of COVID-related articles were retracted for fraud, but a substantial proportion were retracted due to the newly emergent problem of peer review manipulation. In addition, we found that articles relating to politicized or controversial topics, such as vaccines or face masks, tended to receive greater attention on social media and faster retraction times.
The observed time to retraction was lower on average than in the 2012 study,27 although this finding is subject to bias. Not all COVID-related publications destined for retraction may yet have been retracted. Nevertheless, the mean 13.2 mos. time-to-retraction is markedly lower compared to the 2012 analysis, which measured a mean time-to-retraction of 32.9 mos. across different causes.28 This may in part be due to the rise of post-publication peer review and heightened public scrutiny of COVID-19 research during the pandemic.
A rapid increase in research output,29 expedited review times,30 and the rise of preprints suggest that the pandemic may have lowered standards for scientific publication. One review of 686 COVID-19 articles found that methodological quality scores were lower in COVID-19 clinical studies than for control studies in the same journals.Reference Jung31 In addition, the urgency of disseminating COVID-19 research may have led authors of COVID-19-related publications to cite fewer sources and write shorter articles overall, leading to the publication of findings with less scientific support.Reference Gai32 It is important to note that although retracted publications represent only a small subset of the scientific literature, their occurrence can impede research progress and trust in the scientific community.
Many of the publications in this study might not have been retracted if not for the rise of online post-publication peer review. This is supported by the existence of PubPeer entries for two-thirds of the retracted articles. One example is the retraction of two early 2020 articles from Mehra, et al. that claimed to have performed a multi-national observational analysis based on data from the Surgical Outcomes Collaborative (Surgisphere). The first study, published in The New England Journal of Medicine, suggested that the use of ACE inhibitors or statins was associated with better COVID-19 outcomes based on an analysis of more than 8,000 patients.Reference Mehra33 The second, published in The Lancet, claimed to analyze data from more than 96,000 patients and concluded that hydroxychloroquine was associated with poorer outcomes.Reference Mehra34 Soon after their publication, concerns were raised about the veracity of these datasets. A retrospective analysis of tweets in response to the second article by Mehra, et al. found that 720 tweets mentioned “Surgisphere” and 1,819 mentioned “data” in reference to the paper, with several users raising concerns before any retraction was formally announced.Reference Haunschild and Bornmann35 Following these concerns, Surgisphere declined to allow third-party auditors to conduct a private audit of the dataset. As a result, both articles were retracted due to suspicion of data fabrication. Although the swift action of post-publication reviewers is laudable, it was not before the World Health Organization had temporarily halted a study of hydroxychloroquine in response to the Mehra Lancet article.36 Before retraction could occur, the fraudulent research was able to mislead the scientific community and influence international health policy decisions.
COVID-19 research became highly politicized as government officials considered decisions on stay-at-home orders, masking, and vaccine mandates. We found that COVID-19 publications related to controversial or politicized topics had a lower average time to retraction and also a higher degree of media and PubPeer attention. The rapid and widespread dissemination of controversial research might be viewed as a double-edged sword that allows post-publication peer review and swift retraction if a problem is identified but also spreads misinformation among the public who are unable to evaluate the claims critically. A separate review of COVID-19 research found that retracted COVID-19 articles were cited at higher rates than other articles in the same journal, and the addition of “retracted” or “withdrawn” to the article title failed to affect citation rates.Reference Taros37
In mid-2021, after COVID-19 vaccines became more widely available, an article by Walach, et al. claimed that the COVID-19 vaccination caused two deaths for every three that it prevented.Reference Walach, Klement and Aukema38 None of the article’s authors were trained in vaccinology,Reference Wadman39 and their calculations inappropriately conflated a database of adverse events following vaccination with attributable causation.40 Six scientists resigned as editors of the journal Vaccines in protest of the article’s publication, and within three days, an expression of concern had been published.41 In total, only eight days elapsed between the paper’s publication and its retraction. However, in this brief interval, the article was widely shared among the antivaccine community, and an analysis of the article’s Altmetric page confirms that it continues to draw attention from antivaccination activists online. Following retraction, the article was republished online in Science, Public Health Policy, and the Law,Reference Walach42 a website run by the Institute for Pure and Applied Knowledge, which was founded by anti-vaccine activist James Lyons-Weiler.Reference Marcus43 Another article by the same first author claimed that face masks cause children to inhale unsafe levels of carbon dioxide.Reference Walach44 This article was retracted within months, after concerns were raised about its methodology and interpretations.Reference Christakis and Fontanarosa45 However, the same article was soon republished by the same authors in another journal, and there is no mention of a prior retraction.Reference Walach46 This particular article was not included in our analysis because the title did not mention the phrase “COVID-19” or “SARS-CoV-2”.
Articles can also be retracted because they relied on data that were either erroneous or fraudulent. For example, a meta-analysis of ivermectin trials was retractedReference Hill47 and revised after learning that the analysis was based on a study containing fraudulent data. The study in question was a preprint that failed to pass peer review and was withdrawn from Research Square due to concerns about fabrication.Reference Elgazzar48 Nevertheless, the false data contained in this preprint appeared in a published article that was cited by politicians to support false claims of ivermectin’s utility against COVID-19.Reference Oransky49
Although examples of fraud were certainly observed in COVID-19-related articles, our analysis uncovered new trends not seen in the 2012 analysis.50 The compromised peer review process at Hindawi accounted for the majority of all publications retracted in 2023, not just affecting the COVID-19-related articles in this analysis.51 After excluding this mass retraction event from our analysis, error was the most common attributed cause for retractions, rather than fraud (Figure 1). This aligns with an early 2020 analysis of retracted COVID-19 publications, which found that 16 out of 26 article retractions were attributable to errors in analysis, data, or results.Reference Soltani and Patini52 Another review of 27 COVID-19-related retractions performed in 2022 reported that most retractions were due to misconduct, although the inclusion criteria of this small dataset differed from our study.Reference Capodici53 Other reviews using different date ranges or data sources found that the majority of articles analyzed were retracted for unknown reasons.Reference Anderson, Nugent and Peterson54 This suggests that retraction cause analyses may yield variable results depending on their inclusion criteria, data source, and date of analysis.
One trend that has continued into the present decade is the publication of incomplete or uninformative retraction notices, many of which fail to specify the precise reason that a paper has been retracted. For example, the retraction statement for Zhang, et al., 2022 states that the publisher “uncovered evidence of one or more of the following [six] indicators of systematic manipulation of the publication process” (emphasis added).55 This retraction statement is repeated word-for-word in many different articles retracted by this publisher. In addition, culpability for a retraction may differ based on whether a retraction announcement is written by the authors themselves or by the publisher. This can make it difficult for readers to assess which aspects of an article, if any, are trustworthy.Reference Frampton, Woods and Scott56 The Committee on Publication Ethics (COPE) suggests that retraction announcements should “state the reason(s) for retraction” while maintaining objectivity. However, in another analysis of COVID-19 related retractions, authors found that one-third of identified publications did not specify a reason for retraction, while over half of retracted articles remained available as original electronic documents,57 leaving problematic publications available to the public without an indication that they had been retracted. This illustrates a lack of consistent adherence to COPE guidelines, even in PubMed-indexed journals.58
COVID-19 further accelerated the push for open access publishing that has occurred in the past decade,Reference Arrizabalaga59 and 95% of the publications that we analyzed were open access. Open access began with an intention of making research widely accessible, and although it has benefited scientific communication,Reference Huang60 it has upended the economic model of scientific publishing to shift the cost of publishing from libraries or institutions to authors. Whereas librarians formerly served a gatekeeping role in the evaluation of journal quality when they purchased subscriptions for their institutions, the onus of evaluating quality has now fallen on the reader or researcher.Reference Clark and Smith61 One unintended consequence of this shift has been the rise of predatory publishers, whose annual publication output increased from 53,000 articles in 2010 to 420,000 in 2014.62 Because predatory journals have lower average Article Processing Charges (APCs), they may appear more attractive to new researchers or students looking to enhance the ease of publication at the expense of rigor. However, since many predatory journals are not indexed in PubMed, our analysis was unable to assess problems specific to predatory journals.
Nevertheless, we note that some PubMed-indexed journals, including some Hindawi journals, have at times been classified as predatory, for instance in Beall’s listReference Butler63 and the Norwegian Scientific Index.64 In 2023, Wiley retired the Hindawi name, given its association with problematic journals.Reference Kincaid65 Our finding that Hindawi publications comprised 60 out of 244 COVID-19-related retractions suggest that predatory open access journals are contributing to the decreased reliability of the scientific literature. The association between COVID-related retractions, for-profit publishers, and open access suggests that the changing economics of scientific publishing has contributed to the increase in retractions.
This study has important limitations. The Retraction Watch blog reports more than 400 retracted COVID-19 publications, but we analyzed only 244 because we limited our search to PubMed-indexed articles. This meant that we did not analyze retracted or problematic publications in many predatory journals or in preprints that are not listed in PubMed. We also were not able to analyze articles that did not directly mention COVID-19 or SARS-CoV-2, such as some articles on masking. In addition, our analysis of time to retraction warrants revisiting at a later date. It is likely that some problematic publications are yet to be retracted, especially given that investigations of fraud may take years, and all the analyzed publications are less than five years old. Consequently, our calculation of mean time to retraction, especially in comparison to the decades-long date ranges of previous analyses, may underestimate the actual time interval.
Protecting the integrity of the research record requires the cooperative actions of researchers, institutions, agencies and publishers.66 Our analysis found that most COVID-19-related retractions resulted from problems that could have been detected prior to publication. More than half of COVID-related retractions resulted from compromise of the peer review process, plagiarism, authorship issues, lack of ethics approvals, or journal errors, which are largely preventable with appropriate journal procedures. Many journals are also implementing measures to detect problematic images prior to publication. For example, the Journal of Clinical Investigation has implemented image and statistical checks prior to publication and found that nearly a quarter of all accepted manuscripts had problems, some of which were sufficiently serious to result in the rescinding of acceptance decisions.Reference Williams, Casadevall and Jackson67 More careful screening and peer review by journals could reduce the number of retractions by preventing the publication of problematic articles. COPE recommends that publishers implement workflows to check for plagiarism, authorship, misconduct, and ethics approvals prior to publication. In addition, better peer review processes and vetting of guest editors could have averted many of the retractions. Unfortunately, while we can call on journals to adopt more rigorous publication practices, it is likely that low-quality journals with poor practices will continue to proliferate as long as present economic incentives persist. This means that post-publication peer review and swift, transparent retraction practices will continue to play an important role in maintaining the quality of the scientific literature. Readers must remain vigilant even when it comes to peer-reviewed science, as publication is not and has never been a guarantee of truth.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/jme.2025.33.
Acknowledgements
The authors are grateful to the reviewers for their critical feedback and constructive suggestions for improving our manuscript.
Competing interest
E.R.M. is a former intern of Cell Press, which is owned by Elsevier.
