2. Methodological decisions and meta-analysis

In the late 2010s, a scientific dispute emerged over the safety of a drug that had been withdrawn from the British market 40 years prior. There is much that could be said about the events that led to the withdrawal of the drug (Olszynko-Gryn et al. Reference Olszynko-Gryn, Bjørvik, Weßel, Jülich and Jean2018), but what I wish to focus on is the contemporary dispute. Specifically, in assessing the strength of the evidence, present-day researchers split over the acceptability of methodological choices in the original studies and whether they invalidated the contemporary use of meta-analysis to form an all-things-considered judgment of the historical evidence. Because the current dispute turns on details of the original studies, we must first revisit the research that ultimately precipitated the withdrawal.

Prior to the introduction of the stick tests, which have now become a ubiquitous means of detecting a new pregnancy, the methods for early detection were substantially different. In the 1950s, the best available option was to provide one’s general practitioner with a urine sample, which would be sent to a laboratory and injected into a frog to see if she laid eggs (a positive indication of pregnancy). In 1958, a new option became available in which a doctor could prescribe a woman a large dose of estrogen. If she were not pregnant, the estrogen would induce menstruation. The ability for a woman to determine whether she was pregnant privately was especially attractive prior to the legalization of abortion. Use of the hormone pregnancy test (HPT) became widespread (Olszynko-Gryn et al. Reference Olszynko-Gryn, Bjørvik, Weßel, Jülich and Jean2018).

The first cloud of suspicion arose when a study published in Nature suggested that HPTs might be harmful to the developing fetus (Gal et al. Reference Gal, Kirman and Stern1967). Despite this and other similar studies, HPTs remained on the market until 1978, when a large study organized by the British government confirmed the association (Greenberg et al. Reference Greenberg, Inman, Weatherall, Adelstein and Haskey1977). However, although the evidence was ultimately strong enough to spur regulators to remove HPTs from the market, when families ostensibly harmed by HPTs attempted to sue the manufacturers of HPTs, the evidence was judged to be insufficient to demonstrate causation.

For decades, it seemed that the story would end here. But in the 2010s, the declassification of regulatory documents, as well as the discovery of internal documents from the manufacturers, reignited a patient rights group’s campaign (Olszynko-Gryn et al. Reference Olszynko-Gryn, Bjørvik, Weßel, Jülich and Jean2018). In 2017, the efforts of patients gained national news coverage and prompted the Commission on Human Medicine to form an expert working group (EWG) to reexamine the evidence. The result was not what patient advocates had hoped for. The official report reaffirmed the conclusion that the evidence did not support a causal association between the use of HPTs and congenital malformations (UK Commission on Human Medicines 2017). Dissatisfied with this result, the patients’ rights group reached out to the Oxford Centre for Evidence-Based Medicine and asked it to review the evidence. The team, led by the center’s director, Carl Heneghan, conducted a meta-analysis of the data and found, contrary to the EWG report, that children whose mothers took an HPT were significantly more likely to have a congenital malformation (Heneghan et al. Reference Heneghan, Aronson, Spencer, Holman, Mahtani and Perera2019).Footnote ²

Both the EWG and the Oxford team looked at the same body of evidence, yet they reached different conclusions. Although there were many small disputes, the primary reason for the divergence stemmed from whether it was appropriate to conduct a meta-analysis or whether, as maintained by the EWG, combining the studies provided a misleading result. In support of the latter contention, the EWG raised three issues that threaten the reliability of the Heneghan et al. meta-analysis: recall bias, channeling bias, and confounding (UK Commission on Human Medicines 2019). I will here focus on recall bias and whether the standard argument from inductive risk could have been relied on by Heneghan et al. as a response to the methodological concerns raised by the EWG.

The central issue arises because a significant portion of the original studies were based on retrospective interviews that asked mothers whether they had taken an HPT. The interviews occurred after the child was born and thus (generally speaking) after the mother knew whether or not the child had a congenital condition. The EWG’s concern stems from the fact that “using normal babies as a control potentially increases the risk of recall bias as mothers of children with a malformation may be more likely to remember different exposures” (UK Commission on Human Medicines 2019, 18). In support of this concern, the review cites Nora and Nora (Reference Nora and Nora1975), who noted that “considerable probing” was often required to uncover the fact that women had taken an HPT.

Because we will dwell on the details, we need to get precise about exactly how this concern is playing out in the data and the analysis thereof. The EWG (presumably) grants that the types of interview techniques used by researchers did help some mothers correctly remember taking an HPT. The concern is that this more thorough questioning would interact with the mother’s knowledge of whether her child was born with a congenital anomaly. For example, if women whose child had a congenital malformation were psychologically motivated to find some external cause to blame, they might be more likely to recall taking an HPT than women who had healthy babies. If this methodology did engender a recall bias among women whose children had a congenital malformation, this would increase the risk of a false positive (i.e., of declaring HPTs a cause of congenital malformations when they are actually safe).

This brings us to the central point I wish to consider: Could Heneghan et al. (Reference Heneghan, Aronson, Spencer, Holman, Mahtani and Perera2019) simply justify these methodological decisions by providing a standard argument from inductive risk? After all, even if the EWG is right that thorough questioning introduces a bias, it seems quite clear that whether to use it or not involves a value-laden trade-off. Suppose that the alternative to thorough questioning incorrectly classifies a number of women as not having taken an HPT, but it is equally likely to do so for both women who had healthy babies and women whose children had a congenital abnormality. In this case, the comparison is not biased by the collection method, but because it is less accurate, it has a lower power to detect a difference if it actually exists. In this scenario, there would be an increased risk of a false negative (i.e., declaring that HPTs are safe, when they are actually hazardous).

3. Methodological decisions and the argument from inductive risk

According to the argument from inductive risk, because the researchers face a trade-off between false negatives and false positives, they have an obligation as moral agents to rely on values to make the methodological choice that best balances the trade-off (Douglas, Reference Douglas2009). For example, we should require more evidence that a vaccine is safe before releasing it to the public than we would require to conclude that a belt buckle reliably holds up pants. The societal cost of releasing the vaccine on the market if it is unsafe or ineffective is far greater than a few sagging jeans, so we should demand more extensive testing of the vaccine.

Douglas (Reference Douglas2009) takes this point and then observes that if we accept that values can rightly influence our choice of an evidential threshold, then by the same line of reasoning, scientists should also bring their values to bear on selecting a methodology, characterizing the evidence, and interpreting the data.Footnote ³ Because choices here also generate trade-offs between false positives and false negatives, values should play a role in these decisions too.

Put somewhat differently, consider the methodological decision involved in HPT research of whether or not to engage in thorough questioning of women in ascertaining whether they had taken an HPT. I have already indicated that thorough questioning of women makes false negatives less likely and false positives more likely (relative to the alternative). But this is not quite true, or better to say, this is true assuming that everything else stays the same. In particular, it assumes that the evidential threshold (i.e., the p-value of the statistical analysis) stays the same. But consider another methodological alternative: Instead of engaging in thorough questioning, HPT researchers could have adjusted their p-value to achieve a different balance of false positives and false negatives.Footnote ⁴

Again, Douglas (Reference Douglas2009) observes that methodological choices can have effects equivalent to changing our evidential thresholds. Accordingly, if we accept that values are permissible in determining our evidential thresholds, then values should be acceptable in determining which methodological choice we make. This line of reasoning has been called the “downstream conjecture” (Elliott and Richards Reference Elliott, Richards, Kevin and Richards2017, 263). Accordingly, if one rejects the value-free ideal, it might seem that Heneghan et al. (Reference Heneghan, Aronson, Spencer, Holman, Mahtani and Perera2019) can justify the methodological decision of the original HPT researchers (e.g., Nora and Nora Reference Nora and Nora1975) by asserting that this decision correctly balances false negatives and false positives. But as I will show in the following section, this option is not available.

4. The drawback of tailor-made science

To see why Heneghan et al. (Reference Heneghan, Aronson, Spencer, Holman, Mahtani and Perera2019) cannot justify previous methodological decisions by claiming they properly manage inductive risk, it will be crucial to make a clear distinction between the original studies conducted by various researchers and the meta-analysis conducted by Heneghan et al. It will turn out that if inductive risk is properly managed at the level of the individual study, it will be poorly managed in the meta-analysis. In this section, I will review Stephen John’s (Reference John2019) “value-apt ideal” and then return to our case study to show why Heneghan et al. cannot defend the methodological choices in the underlying studies by appeal to an argument from inductive risk.

John’s (Reference John2019) value-apt ideal is an ostensible replacement for the value-free ideal. John argues that when a scientist asserts some claim, they are simultaneously conveying to their audience that the listener has a good reason to accept the claim as true. However, if the claim is dependent on values that the scientist has but the audience does not, then the audience member may not have good reason to believe the claim. Indeed, asserting a value-dependent claim while obscuring the value dependence violates the autonomy of the audience members. Notice, however, that this line of reasoning does not imply that scientists should never use values in their scientific reasoning, only that they should make sure that the values they use align with those of the recipient of the knowledge claim. Science need not be value-free; it need only be value-apt.Footnote ⁵

So, let’s consider the happiest of scenarios in which the same set of values is shared by the researchers of the initial studies, the EWG, Heneghan et al., and indeed, all of the relevant users of this research. Further suppose that the EWG is correct that recall bias is a problem in the original studies, but also that the shared values of all involved are such that the thorough questioning of mothers involved in the study leads to the correct balance of false negatives and false positives in the original studies. In such a case, the original research will be value-apt, but the meta-analysis will not be.

This is because if there is a bias in the data, then as the sample size increases, the risk of a false negative decreases, and the risk of a false positive increases.Footnote ⁶ If these risks change, then a person whose values justified the trade-offs when the sample size was small must judge the trade-off as undesirable when the sample size becomes sufficiently large. Functionally speaking, a meta-analysis increases the sample size of an analysis. So, if Heneghan et al. were to defend the methodological decisions in the original study by saying that the underlying studies were value-apt, they would also have to concede that their own work was biased.

If Heneghan et al. (Reference Heneghan, Aronson, Spencer, Holman, Mahtani and Perera2019) are to defend the methodological decisions made in the underlying study, it must be, as it in fact was defended, on epistemological grounds (e.g., that thorough questioning is unlikely to create recall bias). However, to reiterate a crucial point, if this debate had occurred at the time the original studies were carried out and focused on a particular study and circumstance, the situation would have been different. In such a case, assuming one is willing to dispense with the value-free ideal, even if one knew with moral certainty that thorough questioning would introduce a nontrivial bias to the data, one might also have a very particular use in mind for the research and argue that thorough questioning is the best way to manage the inductive risk. In so doing, and assuming that the values are used judiciously and correspond to stakeholder values in the appropriate way, one is creating knowledge that is epistemically unimpeachable, given the relevant values, and is trustworthy for the intended users. What it would not be is knowledge that can be reliably used beyond the circumstances for which it was created. The knowledge is bespoke.

5. The trilemma of bespoke knowledge

In the previous section, I illustrated how a standard use of the argument from inductive risk had unforeseen consequences when value-laden knowledge traveled beyond the context for which the methodological choices had been tailored. This problem arose because of how the use of values affected the research produced. In this section, I wish to pull back from the specific details of the HPT case and state these issues more generally. In short, we have a set of three qualities that are typically seen as desirable but are often mutually incompatible: value-aptness, avoiding epistemic distortion, and universality (especially the ability of data to travel).

In some cases, one may be able to generate value-apt, reliable research that can travel by refraining from using values in ways that affect the measure of central tendency. But suppose the only way to create value-apt knowledge is by making methodological decisions that affect the measure of central tendency. If we adopt a lexigraphic ordering that prioritizes universality and epistemic reliability, we will have to settle for an inapt alignment between our science and our nonepistemic values. Such a position entails that, oftentimes, values cannot be allowed to influence methodological decisions. Although some philosophers endorse such “strict standards” (e.g., Steel Reference Steel, Elliott and Steel2017), most hold that values should have wider latitude to influence scientific decisions.

So, a second option is that researchers can produce value-apt research and allow it to travel. As a consequence, they must be resolved to introducing an epistemic distortion into the body of scientific knowledge.Footnote ⁷ This is arguably an unacceptable violation of scientists’ epistemic obligations (Holman and Branch, unpublished). Indeed, in most cases, I suspect that neither of the first two options will be seen as desirable.

Accordingly, we have a final option. Scientists produce value-apt results that are epistemically reliable when confined. The consequence is that inquiry will produce knowledge that is tailored to specific circumstances and cannot travel. It will produce bespoke knowledge.