1. Introduction
Scientific retraction is not always effective. Across many fields retracted papers are cited approvingly after retraction, sometimes at surprisingly high rates.Footnote 1 This poses a problem for scientific progress. New studies draw on existing findings but cannot do so effectively if those findings are not reliable. In the wake of the metascience movement, and under the influence of the Retraction Watch blog, there is increasing attention to this issue.
One relevant question is: Which findings are easier or harder to retract, and why? In previous work using models of social networks LaCroix et al. (Reference LaCroix, Geil and O’Connor2021) predict that findings with wide reach should be easier to retract. Humans tend to share information that is relevant. While new scientific findings are broadly relevant, information about retraction is most relevant to those already misinformed about a finding. For this reason, information about the retraction of widely known findings spreads effectively in their models.
We test this modeling prediction using citation data and Altmetric data (tracking media attention). In the first case, we predict a more significant drop in citations for highly cited papers after retraction. We find support for this prediction. More highly cited papers see significantly greater drops in citation count upon retraction. They also see a larger relative drop in citation rate. In the second case, we predict, and find, more online attention to highly cited papers during their retractions.
Our findings provide new insight into the process of retraction, and to the role of social aspects of knowledge in retraction. They also demonstrate the usefulness of combining methods of inquiry—in this case simple social models and data analysis—to the study of philosophy of science and metascience.
In section 2 we describe the model from LaCroix et al. (Reference LaCroix, Geil and O’Connor2021), motivating it with some relevant empirical findings. We also describe our main predictions. In section 3 we describe the two data sets we make use of, as well as our methods of analysis. In section 4 we present our main results, and section 5 provides relevant discussion.
2. Model and predictions
There are myriad candidate explanations for failures of retraction—journals do not actively communicate about retraction, some scientific search engines fail to clearly identify retracted papers, and individual scientists are often hesitant to communicate about their retractions for reputational reasons. Beyond these issues, features of human information-sharing systems may prompt such failures as well.
LaCroix et al. (Reference LaCroix, Geil and O’Connor2021) study the social dynamics of retraction on the empirically well-supported assumption that information spreads via social contact in human groups. They draw on SIR (susceptible, infected, recovered) models from epidemiology, which track the spread of infectious diseases, but are also used to study the spread of rumors or information in human groups (Rogers Reference Rogers2012; Hayhoe et al. Reference Hayhoe, Alajaji and Gharesifard2017).
Their basic model is a connected network of N agents who can be in one of three belief states. Neutral (susceptible) agents have never encountered a new, false scientific claim, false (infected) agents have encountered this claim and now hold a false belief, retracted (recovered) agents have encountered information about retraction and correctly believe the original claim to be false. In each round of simulation, an agent is randomly paired with a network neighbor, and their belief states may influence each other. Importantly, information spread is asymmetric. Neutral agents may learn false beliefs, and false agents may learn about a retraction. Thus agent states only flow from neutral → false → retracted.
A substantive assumption is that agents do not learn about retractions before learning false beliefs. As noted, the authors justify this via appeal to the role of relevance in human communication—people tend to avoid sharing irrelevant information (Grice Reference Grice, Cole and Jerry1975; Sperber and Wilson Reference Sperber and Wilson1986; Wilson and Sperber Reference Wilson, Sperber, Laurence and Ward2006). Since retractions are typically only relevant to those who learned the original finding, information about them will not spread well absent this context.
A central result is that when agents do not share new information indefinitely, false beliefs persist in these models as an accident of history. Some agents learn a false finding, and never happen to encounter its retraction. The authors explore conditions under which larger or smaller proportions of the population maintain permanent false beliefs.
Most important for us is the role of delay in retraction. One might think that a retraction should be issued as soon as possible to prevent eventual false beliefs. In their models, though, delayed retractions are issued in a network where many agents have already encountered the false belief. The retraction can thus spread more readily, and is more successful. We present this effect in Figure 1, which shows average final belief states in this model for different lengths of delay in retraction.Footnote 2
Delayed retractions are more effective. Results for a fully connected network of size N = 100 where agents share newly learned beliefs for 200 time steps of simulation.
The models just described are highly simplified and do not fully track the real dynamics of information sharing.Footnote 3 As such, they should not be taken to directly inform real scientific communities. But we can use them to generate testable predictions about causal processes that might occur in the real world.
In the models, delay improves retraction because delay leads false findings to spread more widely. But in the real world the importance of the research, publication venue, status of the authors, their social sway, and chance events all determine the degree to which a finding is widely shared in a community. Thus, the relevant prediction is that widely known findings should be easier to retract, and that this retraction should generate more attention. Specifically we predict that:
(P1) Highly cited papers will see a more significant drop in citations after retraction.
(P2) Highly cited papers will receive relatively more Altmetric mentions during retraction.
In the case of the first prediction, we assume citation rate correlates with reach of a finding in a scientific community. We also assume citations signal belief or trust in a piece of research.Footnote 4 Of course, a popular finding may still be widely cited even after a relatively successful retraction compared to a finding that received no attention in the first place. For this reason we look at two measures of drop in citation rate meant to track both the absolute drop and the drop as a percentage of expected future citations.Footnote 5
As a secondary test, we also look at Altmetrics. During retraction, papers tend to see a bump in attention, presumably corresponding to the spread of information about the retraction (Memon et al. Reference Memon, Makovi and AlShebli2025). We expect highly cited papers to see a more significant bump—corresponding to the model prediction that more relevant retractions will spread more readily in social networks.Footnote 6
3. Data, methods, and analysis
To test prediction P1, we relied on two primary datasets. The first is Retraction Watch (RW), the most comprehensive publicly available repository of retracted research, comprising over 26,000 papers (as of the time of data acquisition) across approximately 5,800 journals and conferences (Oransky and Marcus Reference Oransky and Marcus2012). The second is the Microsoft Academic Graph (MAG), which provides metadata and citation networks for over 263 million scientific publications and captures the publishing trajectories of more than 271 million authors. For our analysis, we removed all records that were retracted in bulk (e.g., more than 1,200 abstracts retracted by a single conference in 2011 (McCook Reference McCook2018)). We also only retain papers retracted between 1990 and 2015 to allow for five years of retraction window to study post-retraction outcomes. After filtering, we merged RW and MAG records based on DOI. For papers that did not find a DOI match, we used fuzzy matchingFootnote 7 to match paper titles.Footnote 8 This resulted in a total of 6,188 retracted papers, or the “filtered dataset” henceforth.
3.1. P1: Studying the change in citation post-retraction
Our hypothesis posits that the effect of retraction on subsequent citation behavior is modulated by a paper’s pre-retraction visibility, operationalized as the number of citations received prior to retraction. Specifically, we expect that papers with greater pre-retraction visibility experience a larger decline in citations post-retraction.
To test this, we employ matching, which is a statistical method for assessing treatment effects (e.g., retraction). For each retracted paper in our filtered dataset, we identify one or more non-retracted papers matched exactly on: (a) publication year, (b) journal or conference, (c) scientific discipline, and (d) total number of citations received from publication up to the retraction year (inclusive). Scientific discipline labels are assigned based on the MAG field-of-study hierarchy using methodology from prior work by Memon et al. (Reference Memon, Makovi and AlShebli2025). This procedure yields an analytical sample of 5,256 retracted papers, matched to 651,829 non-retracted control papers, with an average of 155, and a median of 27, matched controls per retracted paper. This matching allows us to make predictions for what the citations of retracted papers might have looked like had they not been retracted, and then compare the citations of matched retracted papers to these predictions. Figure 2 shows, conceptually, how this matching works.Footnote 9
Highly cited retracted papers experience greater drops in post-retraction citations. (a) Absolute post-retraction citation difference between each retracted paper and its matched control (outcome 1). (b) Post-retraction citation ratio between retracted and matched papers (outcome 2).
To quantify the impact of retraction across different levels of pre-retraction visibility, we define two outcome metrics, operationalizing the prediction in different ways. As will become clear, each metric has limitations. Both together present a fuller picture of how pre-retraction attention influences success of retraction.
3.1.1. Outcome 1: Absolute post-retraction citation difference
This outcome captures the absolute difference in post-retraction citation volume between a retracted paper and its matched control:
$${\rm Outcome}_1 = {\rm Post}_r - {\rm Post}_m.$$
We limit our post-retraction citations to five years, which is typical for bibliometric analysis, to allow for a consistent window for all the papers regardless of their publishing year. Because retracted and control papers are exactly matched on their pre-retraction citation counts, this outcome can be interpreted as a simplified version of the classic difference-in-differences estimator
$$({\rm Post}_{r} - {\rm Pre}_r) - ({\rm Post}_m - {\rm Pre}_m)$$
with the pre-retraction terms effectively controlled for via matching. A more negative value indicates a larger real-world citation penalty for the retracted paper, relative to its matched counterpart. We predict a larger absolute drop in future citations for papers highly cited before retraction.
One limitation of this outcome is that since pre-retraction citation rates can be dramatically different across papers, even a minor percentage drop in citation rates for highly cited papers can be quite large in terms of absolute citations compared to a large percentage drop for a paper with few citations. For that reason we calculate a second metric.
3.1.2. Outcome 2: Post-retraction citation ratio
This outcome compares the post-retraction citations of retracted papers to their matched controls:
$${\rm{Outcom}}{{\rm{e}}_2} = {{{\rm{Pos}}{{\rm{t}}_r}} \over {{\rm{Pos}}{{\rm{t}}_m}}}.$$
We estimate this using the log-transformed ratio of post-retraction citations as follows:
$${\rm{Outcom}}{{\rm{e}}_2} = \log \left({{{\rm{Pos}}{{\rm{t}}_r} + \varepsilon } \over {{\rm{Pos}}{{\rm{t}}_m} + \varepsilon }}\right),$$
where ε is a small constant (10-5 in our case) added for numerical stability to avoid division by zero. This tracks the relative, or percentage, change in citations for retracted papers versus their matches. In other words, it computes the proportional change in citation compared to what we would predict. Because citation counts are often highly skewed, we use a logarithm scale to compress extreme ratios and make comparisons more interpretable. We predict larger drops in this outcome for papers more highly cited pre-retraction.
While the first metric arguably exaggerates the success of retraction for widely known papers, the second arguably does the opposite. For a paper with one expected post-retraction citation, losing one citation amounts to a 100% reduction. A paper with 100 expected citations can lose 99 of them, and still not achieve that level of reduction. For this reason we think both metrics are useful in more fully understanding post-retraction citation reductions.
The above outcomes are computed at the level of individual retracted paper–control paper pairs. For papers with more than one match, we take their average outcome. To evaluate whether the effect of retraction varies systematically with pre-retraction visibility, we stratify papers into citation groups based on their total citations prior to retraction. More concretely, citation groups were defined based on quantiles of the pre-retraction citation distribution across all retracted papers in the filtered sample. Specifically, we used the 25th, 50th, 75th, and 90th percentiles to divide papers into the following visibility-based citation tiers: [0, 1), [1, 9), [9, 31), [31, ∞).
For each pre-retraction citation group we summarise outcome 1 with the mean across all retracted–control pairs. Outcome 2, which is the logarithm of the post-retraction citation ratio, is summarized with the median. Because the denominator (post-retraction citations of the matched control) can be close to zero, the log-ratio has a long right tail. The median therefore provides a robust measure of the typical effect, while we report the mean and maximum in the online appendix table 2 to show the full spread of the distribution.Footnote 10
3.2. P2: Studying the association between pre-retraction citations and retraction-related attention
Our second hypothesis posits that highly cited papers receive more retraction-related attention. To operationalize attention, we further merge another dataset, Altmetric, with our filtered RW-MAG sample using DOI. Altmetric tracks over 191 million mentions of research outputs across social media, blogs, news media, and knowledge repositories, covering more than 35 million scholarly works. Altmetric provides aggregate scores for the entire time period of the paper. To isolate attention related to retraction, we only look at a twelve-month window surrounding the retraction date. Details on how the score was computed and validated are based on prior work by Memon et al. (Reference Memon, Makovi and AlShebli2025). Along with the Altmetric score, which is calculated using weights over all mentions,Footnote 11 we estimate a model of total mentions as a second outcome for this hypothesis.
To study the relationship between pre-retraction citations and attention, we estimate a linear regression model, controlling for the year of publication, number of years between publication and retraction, journal ranking, the reason for retraction, the number of authors on the paper, and subject area (as assigned by RW). After filtering for complete cases across all control variables, we retain a final analytical sample of 3,271 observations.
4. Results
Regarding P1, for outcome 1 we compute the difference in post-retraction citations between retracted and matched papers across all citation groups. As evident in Figure 2(a), there is a clear and significant trend: more highly cited papers experience larger drops in absolute citation counts. These results are significant on an aggregate level (p < 0.001) using omnibus Welch ANOVA. All post hoc pairwise comparisons between citation groups are also significant (p < 0.001) using Welch t-tests.
For outcome 2, we measure the ratio of the post-retraction citation counts between retracted and matched papers. We employ the median as our summary statistic for this comparison. As can be seen in Figure 2(b), except for the papers with zero citations, there is a consistent drop in post-retraction citation ratio as we move from low- to high-cited papers. The difference in citation groups is found to be significant on an aggregate level using the Kruskal–Wallis test (p < 0.001). Pairwise Dunn’s tests with Holm adjustment show that the zero-citation group differs significantly from all others (p < 0.001). The 1–8 citation group differs from the >30 group (p = 0.03), but not from the 9–30 group (p = 0.29), and there is no significant difference between the 9-30 and >30 citation groups (p = 0.29). See our online appendix for more details related to the summary statistics related to the two outcomes.
Regarding P2, we reproduce Figure 3(a) from Memon et al. (Reference Memon, Makovi and AlShebli2025) showing that Altmetric scores peak around the first month of retraction. We isolate a twelve-month “retraction window” and characterize how attention varies across different citation groups within that time window. As Figure 3(b) and (c) show, the raw Altmetric mentions and Altmetric scores were much higher for the more cited papers. This effect is especially notable in the most highly cited papers. We further control for various confounders, as detailed in the methods section, and estimate a linear regression model. We find that papers with more pre-retraction citations are significantly more likely to receive retraction-related attention and mentions. More concretely, each additional pre-retraction citation is associated with a 0.168 unit increase (p < 0.001) in the raw altmetric score, and a 0.195 unit increase (p < 0.001) in the number of mentions, all else equal. More recent publication years are positively associated with both outcomes, while other factors such as number of authors, journal rank, and time between publication and retraction show no significant effects. Details related to regression are available in our online appendix.
Highly cited retracted papers receive more attention during retraction. (a) Shows the distribution of log-transformed Altmetric scores in the six months before and after retraction. The x-axis represents monthly time windows relative to the retraction date, where 0 corresponds to the month of retraction (omitted for clarity), −1 indicates the month immediately preceding retraction, and +1 the month immediately following. The y-axis displays the log-transformed Altmetric score for each paper within a given month. (b) and (c) respectively show the average Altmetric score and the average number of mentions at the time of retraction for different citation groups.
Figure 3. Long description
Panel A: A combination of scatter plot and line graph shows the distribution of log-transformed Altmetric scores in the six months before and after retraction. The x-axis represents monthly time windows relative to the retraction date, where 0 corresponds to the month of retraction (omitted for clarity), -1 indicates the month immediately preceding retraction, and +1 the month immediately following. The y-axis displays the log-transformed Altmetric score for each paper within a given month. The scatter plot points are color-coded, with red points representing scores before retraction and orange points representing scores after retraction. The line graph shows the trend of average Altmetric scores over time. Panel B: A vertical bar graph illustrates the average Altmetric score at the time of retraction for different citation groups. The x-axis categorizes the number of citations between publication and retraction year into four groups: 0, 1-8, 9-30, and >30. The y-axis shows the average Altmetric score. The bars are color-coded, with dark blue, teal, yellow, and pink representing the respective citation groups. Panel C: Another vertical bar graph depicts the average total mentions at the time of retraction for the same citation groups as in Panel B. The x-axis and color-coding are identical to Panel B, and the y-axis represents the average total mentions.
5. Discussion
We take our data analysis to largely support the prediction that more widely known findings are easier to retract. As noted, we see more significant drops in total citation rates post-retraction for highly cited papers, indicating that a larger number of authors who would have cited these papers do not. We see the same pattern with relative drop in citation rate, though some comparisons in this analysis are not significant.Footnote 12
In addition, more highly cited papers receive more attention during retraction. The observation that people are more likely to share relevant information drives modeling assumptions in LaCroix et al. (Reference LaCroix, Geil and O’Connor2021), and also the expectation that more attention will be paid to retractions of widely known results. In this way, the Altmetric data further validates model predictions.
One limitation is our dependence on citation count as a proxy for (i) familiarity with and (ii) trust in a scientific result and its retraction. Of course, there are cases where authors know a result, but do not cite it. In studying retraction this may be especially relevant, since there is evidence that scientists are good at predicting which findings replicate (Holzmeister et al. Reference Holzmeister, Johannesson, Camerer, Chen, Ho, Hoogeveen, Huber, Imai, Imai and Jin2025). This might mean that pre-retraction citation counts are systematically impacted by the very features that lead to retraction—that a study is fraudulent, or low quality, or contains significant errors. In addition, sometimes researchers cite papers they know to be retracted, including in cases of self-citation (Madlock-Brown and Eichmann Reference Madlock-Brown and Eichmann2015). So citation drop may not perfectly correlate with exposure to the fact of retraction.
Another limitation is our dependence on comparisons to matched papers. This allows us to approximate what citations we might expect if the target papers had not been retracted. This said, these matches may not be perfect predictors for future citation count of retracted papers. We account for a set of covariates (e.g., publication year, journal, discipline, citations), but there may be other covariates critical to this matching, and, even if we are accounting for the key covariates, there could be imbalances that lead to biased estimates. In addition, for obvious reasons we cannot match on the fact of retraction itself. Papers that are eventually retracted presumably differ systematically from those which are not, on features like the use of fraud, poor-quality methods, and error. These differences might have led to systematic differences in future citations even if the relevant papers had not been retracted.
One last issue is that our dataset does not differentiate approving citations from ones that acknowledge retraction, or even are about retraction. There may be systematic differences in the degree to which widely known papers are cited as examples of retraction.
There are some relevant policy take-aways. First, it might be important to more actively spread information about retraction of less influential work. Previous authors have advocated that journals, pre-print servers, and scientific search engines do more to communicate about retraction. We support these reforms, and point to a place where they may be especially important—in cases where information about the retraction itself is less likely to spread endogenously.
Second, it may be effective to increase the relevance of retractions themselves. Packaging a retraction with information about why it may be important to an area of study may help it spread. For instance, journals issuing a retraction, or science journalists writing about it, might explain how claims in the original paper can potentially lead future researchers astray.
A more general take-away is that in thinking about retraction policy, dynamics of social information spread should be taken into account. These dynamics seem to impact whether and how researchers receive information about retraction.
This project also has take-aways related to the combination of diverse methods in the study of philosophy of science and metascience, and related to the use of simple social models. The models presented by LaCroix et al. (Reference LaCroix, Geil and O’Connor2021) do not provide sufficient evidence to support policy proposals, but do direct attention to an unexpected phenomenon—the possibility that high-impact work may be easier to retract. This possibility has not been previously studied, and without the models would not have been a natural hypothesis. The integration of models and data analysis, then, was crucial to this investigation. Our results find support for the models, and also clarify the relationship between prominence of a result and success in retracting it.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/psa.2026.10255.
Acknowledgments
We thank the PSA 2024 audience, and fellow symposiasts. Thanks to Travis LaCroix and Anders Geil for their modeling work. We thank the Knight Foundation for funding support to S.M. and J.D.W. We thank the High Performance Computing Center at NYU Abu Dhabi for their technical support and resources. We are grateful to Ivan Oransky, Adam Marcus, and The Center for Scientific Integrity for maintaining and sharing Retraction Watch data. We also acknowledge Altmetric.com and Microsoft Academic Graph for providing data for our study. Finally, many thanks to Bedoor AlShebli and Kinga Makovi for access to their processed data, as well as their feedback.
Funding statement
S.M. and J.D.W. receive funding from the John S. and James L. Knight Foundation (G-2019-58788) and the UW Center for an Informed Public that supported this research.
Declarations
J.D.W. is on the Board of Consensus.app.
In addition, J.D.W. received the following restricted funds in the 36 months preceding submission of this paper:
Institute of Museum and Library Services (IMLS). Misinformation Media Literacy: Supporting Libraries as Hubs for Misinformation Education (LG-255047-OLS-23)
NIH: Title: Developing User-Centric Training in Rigorous Research: Post-Selection Inference, Publication Bias, and Critical Evaluation of Statistical Claims. (5UE5NS132949-03)
NSF: Convergence Accelerator Track F: Co-Designing for Trust: Reimagining Online Information Literacies with Underserved Communities. Phase II (2230616)
NSF: Collaborative Research: SaTC: CORE: Large: Rapid-Response Frameworks for Mitigating Online Disinformation (2120496)
Open access
