Infectious disease modeling has played an unprecedented role in the coronavirus disease 2019 (COVID-19) pandemic. Mathematical models have informed, and sometimes driven, decision-making on the timing and severity of social distancing mandates, establishment of alternate care facilities, and the prioritization of vaccine distribution. The CDC uses an ensemble of over 30 individual models to make its COVID-19 case and fatality forecasts, Reference Ray, Wattanachit and Niemi1 and more than 5000 reports on COVID-19 models have been published in peer-reviewed journals. Reference Becker, Grantz and Hegde2 Even the phrase “flatten the curve,” ubiquitous in spring 2020, is a reference to infectious disease model predictions.
Modeling experts have provided constructive critiques of particularly influential COVID-19 models, Reference Jewell, Lewnard and Jewell3 identifying which models are more or less appropriate for answering different scientific or policy questions. Experts have also provided guidance for modelers and decision-makers about how to engage with those models. Reference Jewell, Lewnard and Jewell4 Users of a model are advised, for example, to understand a model’s basic assumptions and its treatment of uncertainty to avoid drawing incorrect conclusions from the model’s results.
However, in our experience working with state and local governments, we have found that public health and emergency management decision-makers generally do not have the time or training to critically engage with the technical specifics of infectious disease models, such as their precise underlying assumptions or the quantitative derivation of error bars. Instead, decision-makers engage with modeling experts, who interpret mathematical models and their results. For example, early in the pandemic, the Virginia state government engaged external modeling experts to synthesize various models’ vastly disparate predictions about the timing and magnitude of peak intensive care unit hospital bed demand in the state. State leadership later partnered with in-state academics to build a model specific to Virginia.
Even when local governments have the in-house expertise to develop and interpret models, successful engagement between decision-makers and modeling experts is not guaranteed. For example, mismatched expectations between leadership and technical staff in a city we supported during the pandemic led to delays and friction when communicating modeling results. In these cases, the key challenge in using model results to inform decision-making was the interaction of decision-makers with modelers, not poor interpretation of models by modelers or decision-makers.
To bridge the gap between guidance by modelers about models and the dynamics of public health decision-making we observe, we hypothesize that following guiding principles, which focus on the interaction between modelers and decision-makers, will improve the utility of modeling for public health decision-making, especially at the state and municipality levels. Although we use COVID-19 as an example, these principles could apply to any infectious disease.
Be Prepared to Partner With External Modelers
Some health departments have dedicated modeling staff, and these experts should be integrally involved in infectious disease planning and response. However, because infectious disease modeling is such a dynamic discipline and requires such specific expertise, developing or interpreting models often requires outside partnerships. Even at the US Federal level, which likely has the greatest amount of resources for modeling, dedicated modeling staff run only into the dozens, and partnerships with outside groups, especially academics, are common. 5
Trust and good working relationships should be built ahead of a crisis. Partnerships with academics may require special attention, as academics and decision-makers have different motivations for and assumptions around modeling. For example, academic researchers may be particularly motivated to partner with governments if the partnership involves access to data that have not been described in the scientific literature.
Key administrative bottlenecks should also be addressed ahead of a crisis. For example, data use agreements and memoranda of understanding, especially around personally identifiable information, should be established in advance of a crisis. Otherwise, the time required to formulate and approve the data sharing may be longer than the window of opportunity for external partners to be useful in a crisis.
Tell Modelers How Modeling Will Contribute to Decision-Making
In our experience, during a crisis, decision-makers prefer having somewhat less precise estimates sooner rather than more correct estimates later, while modelers sometimes fear that giving imprecise results sooner will lead to decisions with negative outcomes or decreased confidence in modeling in general. We conjecture that this mismatch is due in part to differing assumptions about the role modeling will play in decision-making. Modelers may have the perception that model results will be a key driver of an important decision and so will hesitate to present results unless and until they are entirely confident in their results. However, public health decisions are almost always made based on several factors, notably resourcing, statutory requirements, and political considerations. Developing a common understanding of the role that modeling will play in decision-making could help streamline interactions between modelers and decision-makers.
State Your Needed Precision and Timeliness
In our experience, what a decision-maker needs is often much simpler than what a modeler thinks must be built. As an example, a modeling expert once told one of us that they were being pressed by their government to build a model for a novel infectious disease. The modeler protested, to no avail, that there was so little data available that it did not make any sense to build a model. In the end, to demonstrate the futility of any “real” modeling, the modeler sketched 3 very rough scenarios: best guess, optimistic, and worst case. The decision-makers were delighted: the 3 rough scenarios were exactly what they needed. In this case, the mismatch between the decision-makers’ needs and the expert’s assumptions about the model’s specifications led to delays and friction.
Mismatches between decision-makers’ needs and a model’s complexity are also important because more complex models take more time to develop. Perhaps confusingly, complex models also tend to be less robust than simpler ones. Because academia in particular rewards intellectual novelty over tried-and-true solutions, this concern is especially important for outside partnerships.
Tell Modelers About Your Options
It is easier to make a plausible model of the relative impact of a certain intervention than to make an accurate model of the future course of an epidemic. Mathematically, this is because modelers need to make a large number of assumptions about the model’s design and its parameter values, each of which will affect the model’s output. Asking about the effect of changing one part of model, while keeping everything else the same, can help “cancel out” the effect of any mistaken assumptions. Thus, working with modelers to evaluate and compare particular interventions may produce more accurate results compared with trying to predict the absolute magnitude of a disease outbreak’s burden.
Get a Second Opinion
When a doctor, lawyer, or car mechanic gives advice, we often seek a second opinion, not necessarily because we distrust the first opinion, but because different points of view can lead to different conclusions. Decision-makers should apply the same logic to infectious disease modelers. Different perspectives on the same problem may emerge if different modelers take different approaches to the same question. Technical errors in model implantation can be detected if 2 separate groups, using the same methods and data, arrive at the same conclusions.
Even though modeling relies on mathematical formulae, modelers’ actionable predictions cannot typically be “proven” in the same way that other scientific results can. Even when modelers present results with confidence intervals or error bars, those estimates of uncertainty are products of the model itself. They usually do not account for the possibility that a modeler’s assumptions, made using the best available knowledge, will ultimately prove inaccurate.
Even if time or resources do not permit consulting a second opinion, modelers can serve as their own second opinions. Consider asking modelers what other ways they could have approached the question and how those other approaches would have changed the conclusions.
Closing Guidance for Decision-Makers: Adapt
Ideally, ahead of a crisis, decision-makers will have developed working relationships with partner modelers and laid out expectations for how they will interact with one another. Decision-makers will have found ways to articulate their needs to modelers. Modelers will be prepared to specifically address decision-makers’ potential courses of action on a timetable that keeps modeling relevant.
During a crisis, decision-makers and modelers should be attentive to factors that are limiting modeling’s utility. If modeling is lagging behind decision-making, perhaps data sharing must be streamlined, or expectations around models’ technical precision must be relaxed. A second opinion can be sought to improve accuracy or to bring in new perspectives. If modeling is of dwindling relevance, then decision-makers may need to clarify their needs to modelers, or modelers should be asked to advise on a wider portfolio of questions.
In all circumstances, we advise keeping the relationship between decision-makers and modelers front and center. If modeling is not useful, it is likely not because of the model itself, but because of the human context.
Closing Guidance for Modelers: Sustainably Support Research Into the Modeler–Decision-Maker Interface
There have been multiple calls to expand infectious disease modeling research to include the interaction between modelers and decision-makers. Reference Rivers, Chretien and Riley6–Reference Rosenfeld, Fox and Kerr11 These reports have outlined agendas for research in this area and emphasized its importance, but they note the lack of sustained research activity. We hope that groups of funders, researchers, and practitioners, such as the new Center for Forecasting and Outbreak Analytics in the US Centers for Disease Control and Prevention, 12 can provide sustained support for research in this area.
Acknowledgments
Helpful conversations with Austen Hawk and Tschuna Patterson.
Author Contributions
S.W.O and E.T. conceived and designed the narrative. S.W.O. wrote the first draft of the manuscript. S.W.O. and E.T. revised the manuscript.
