Hostname: page-component-7b7567c8c-w7t8q Total loading time: 0 Render date: 2026-11-19T04:56:43.300Z Has data issue: false hasContentIssue false

Hindsight2020: Characterizing Uncertainty in the COVID-19 Scientific Literature

Published online by Cambridge University Press:  25 July 2023

Kinga Dobolyi*
Affiliation:
George Washington University, Department of Computer Science, Washington, DC, USA
George P. Sieniawski
Affiliation:
Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
David Dobolyi
Affiliation:
University of Notre Dame, Indiana, USA
Joseph Goldfrank
Affiliation:
George Washington University, Department of Computer Science, Washington, DC, USA
Zigfried Hampel-Arias
Affiliation:
Los Alamos National Laboratory, Los Alamos, New Mexico, USA
*
Corresponding author: Kinga Dobolyi; Email: kinga@gwu.edu
Rights & Permissions [Opens in a new window]

Abstract

Following emerging, re-emerging, and endemic pathogen outbreaks, the rush to publish and the risk of data misrepresentation, misinterpretation, and even misinformation puts an even greater onus on methodological rigor, which includes revisiting initial assumptions as new evidence becomes available. This study sought to understand how and when early evidence emerges and evolves when addressing different types of recurring pathogen-related questions. By applying claim-matching by means of deep learning Natural Language Processing (NLP) of coronavirus disease 2019 (COVID-19) scientific literature against a set of expert-curated evidence, patterns in timing across different COVID-19 questions-and-answers were identified, to build a framework for characterizing uncertainty in emerging infectious disease (EID) research over time. COVID-19 was chosen as a use case for this framework given the large and accessible datasets curated for scientists during the beginning of the pandemic. Timing patterns in reliably answering broad COVID-19 questions often do not align with general publication patterns, but early expert-curated evidence was generally stable. Because instability in answers often occurred within the first 2 to 6 mo for specific COVID-19 topics, public health officials could apply more conservative policies at the start of future pandemics, to be revised as evidence stabilizes.

Information

Type
Concepts in Disaster Medicine
Copyright
© In-Q-Tel, Inc. and the Author(s), 2023. Published by Cambridge University Press on behalf of the Society for Disaster Medicine and Public Health
Figure 0

Figure 1.Flowchart of construction of our dataset.

Figure 1

Table 1.Annotator rating scale

Figure 2

Table 2.Annotator rating scale examples

Figure 3

Figure 2.Timing patterns of CORD-19 matches of original research from start week of pandemic (week 0, March 2020) through January 2021. The y-axis lists the density, over time, of CORD-19 evidence sentences that matched the DHS-cited evidence sentences for each of the 16 questions.

Figure 4

Figure 3.Timing patterns of DHS evidence vs. CORD-19 close matches of original research from start week of pandemic (week 0, March 2020) through January 2021. The y-axis lists the density, over time, of evidence sentences (either DHS or CORD-19 matched) for each of the 16 questions.

Figure 5

Figure 4.Timing patterns of results contradicting DHS claims from start week of pandemic (week 0, March 2020) through January 2021. The y-axis lists the density, over time, of DHS-cited evidence sentences, that were contradictions with earlier results, for each of the 16 questions.

Figure 6

Figure 5.Uncertainty evolution over time between MQL claims and matching CORD-19 evidence from a later date, by week from pandemic start. The y-axis shows the potential evolution (or lack thereof) between C-C, C-U, U-C, or U-U as defined above.