Two paradigms of scholarship have emerged in recent decades examining the nature of juridical proof that might be termed evidence science and evidence theory. These are not paradigms in the Kuhnian sense of one substituting for another, or as involving the evolution of a discipline, but instead two different approaches to studying what appears to be the same phenomenon. Examples of this process from other disciplines include the study of human behavior by various branches of neuroscience, psychology, or sociology, or the study of markets by Marxist and Chicago School economists. A controversial recent example from the United States is the study of history by Critical Race Theorists and classical historians. These are not Kuhnian paradigms in another sense as well. They are not mutually exclusive, a better metaphor perhaps being overlapping Venn diagrams, in that communication between the adherents of different approaches may not only be possible but fruitful. The objective of this article is to further that possibility.
In the study of juridical proof, the two approaches are exemplified by, on the one hand, scholars interested in understanding, explaining, and only secondarily justifying the actual operation of whatever legal system is under examination.Footnote 1 This is what we mean by “evidence science.” On the other hand are scholars striving for more than empirical knowledge through creative ways to understand legal systems that may in turn lead to deeper truths. As one scholar said, “[w]hen I theorize—about statistical evidence, or about anything else—the aim is to reach knowledge and understanding of important relevant truths.”Footnote 2
An example of this contrast is the work by the evidence scientists that has driven the evolution of the best explanation of juridical proof from probabilistic to explanatory.Footnote 3 One of the objectives of the evidence theorists, by contrast, has been to deepen understanding of the meaning and implications of evidence through examination of the legal system's suspicion of overtly probabilistic evidence in general and rejection of naked statistical evidence in particular.Footnote 4 Typically the stage is set in this literature by assertions to the effect that the law is “very reluctant to ground a finding” on naked statistical evidence;Footnote 5 “[t]he dominant view is that merely statistical evidence cannot provide a proper basis for a legal verdict, even when the odds of error are extremely low”;Footnote 6 “courts are reluctant to base affirmative verdicts – verdicts of guilt or liability – on evidence that is purely statistical in nature”;Footnote 7 and the “intuitive distinction between individual and bare statistical evidence can be found in a large number of court judgments.”Footnote 8 Starting from these assumptions, the task becomes “to interpret the existing legal practice – to outline a standard that makes sense of the way in which statistical evidence is actually treated in the law.”Footnote 9 If all goes well, that explanation may lead to deeper understanding of the nature of evidence and the moral or ethical implications of legal decisions.Footnote 10
This theoretical work has generated a fascinating literature illuminating a number of interesting features of evidence, such as safety, sensitivity, normalcy, and the role of incentives in the structuring of legal systems, all of which are pertinent to at least one legal system of which we are aware—the American legal system—and do play a role in the resolution of disputesFootnote 11 and the structure of liability.Footnote 12 This literature may also succeed on its own terms in searching for a deeper understanding as judged by criteria internal to the relevant disciplines.Footnote 13 Such matters are not within our field of expertise, and we take no position on them.
Whatever its other value, the implications of much of the theoretical work on juridical proof for real legal systems is unclear. That is not surprising if the objective is not to explain them. There is another variable that may be operating here, though, which is that the initial assumptions are false, resulting in the inapplicability of the conclusions drawn to real legal systems. The typical first step in this literature asks a meaningless question, to wit, how “the law” treats overtly probabilistic evidence and naked statistics. The meaningless question is then often given a wrong answer—at least so far as the United States is concerned—that the law is resistant to overtly probabilistic evidence and rejects naked statistics. The mistakes result from mismodeling both the structure of legal systems and the process of common law decision making. It turns out that there is no such resistance or rejection. There are instead a few scattered statements taken out of context from old cases that, whatever their original meaning, have been left behind by developments over the last half century or so. The resistance to statistical proffers that one finds in the American jurisdictions is not a function of its statistical form but lies instead in the aversion to unreliable evidence, like witnesses who literally make things up or evidence insufficient for a reasonable person to conclude that the burden of persuasion has been satisfied—what we will henceforth refer to as “lousy” evidence.Footnote 14 If evidence is based on suitable statistical and methodological models and the data is collected in an acceptable fashion, courts universally find it admissible and sufficient to sustain verdicts. The literature has been motivated to explain a nonexistent condition.Footnote 15
The naked statistics literature also contains perspectives on deterrence and the significance of errors that are unlikely to reflect reality or be influential in the construction of legal systems. Perhaps part of the explanation for this is the ubiquitous methodology of devising quite clever hypotheticals that, whatever their philosophical or other value, bear virtually no relationship to any extant legal system of which we are aware—certainly not any legal system within the United States.Footnote 16 Whatever the cause, much of the theoretical literature on juridical proof features at best a complicated relationship with reality. Our aim here is to examine this divergence between theory and reality in the hopes of stimulating more fruitful interactions between the scientists and the theorists.
We will first explain why asking what “the law” requires is a meaningless question, and then how nonetheless it is often given a wrong answer. Shifting focus to the law in the United States, we next explain how the arguments to the contrary mismodel the legal system that they are examining in various ways, reflect a misunderstanding of the common law by failing to contextualize statements from a handful of cases, and fail to take into account the common law's evolutionary nature.Footnote 17 We then briefly explore the problematic use of deterrence arguments in this literature and the equally problematic approach to errors, each again perhaps caused by unrealistic assumptions. We conclude by pointing out serious questions not addressed where legal scientists could use considerable help, such as the meaning and interaction of plausibility and probability and the balancing of social and private utilities.
A preliminary clarification is necessary before we begin the substantive analysis. The general focus in this literature is on probabilistic and/or statistical evidence but there is no agreement among the commentators beyond that general point. The literature began with claims that verdicts could not or should not rest on statistical evidence alone.Footnote 18 Others claim that a particular kind of statistics—naked statistics—is the problem but there is no agreement on what that means. To some commentators the problem is base rate evidence,Footnote 19 but there is disagreement about what that means as well. Some think it applies to any general statistical evidence of any sortFootnote 20 whereas others wish to make a distinction between statistics attached to trace evidence like DNAFootnote 21 and other forms of evidence that just involve membership in a reference class with no trace left at the scene by a suspect or defendant.Footnote 22 Others at times explicitly do not define what they mean by “statistics.”Footnote 23 We will demonstrate that no matter the focus, there is no evidence in US law demonstrating an aversion to reasonably reliable evidence of any sort. As we proceed, we will also point out when case law permits verdicts based on statistics involving both trace evidence and nontrace evidence.Footnote 24
A related linguistic problem involves the old cases that the proponents of the aversion thesis rely upon that use various “probability” language without specifying what precisely the courts mean. We show that the best explanation of those cases is that they involve lousy, unreliable evidence, and that in any event subsequent developments make clear that there is no support in US law for any kind of “aversion” to probabilistic evidence. In every case, the question is the reliability of the evidence.
First, the meaningless question: Why is the law suspicious of statistical evidence and resistant to naked statistical evidence? This is meaningless because there is no “the law.” To say what “the law” is based on what the law may be in a few jurisdictions is a fundamental mistake about the nature of law. For example, the United States has at least fifty-two different versions of the law of evidence. The law of evidence is a product of the sovereign police powers of the individual states, and each state has its own, as do both the federal government (the famous Federal Rules of Evidence) and the US Military. There are similarities across these various jurisdictions, and as we shall see their handling of the questions of interest here is also highly similar, but nonetheless the particular jurisdiction must be identified to say anything intelligible about the law of evidence.
Gazing beyond the borders of the United States turns up a cacophony of evidentiary approaches. Tanzania has a common law tradition of disfavoring circumstantial evidence,Footnote 25 of which DNA and other statistical proffers are examples. Yet a closer look finds that Tanzania, like virtually every other country where we have found data, relies on DNA and other forensic evidence where appropriate and takes no general position on the use of statistics.Footnote 26 In China, evidence law has specific categories into which evidence must fit and there is not a category of “statistical evidence.”Footnote 27 But there is a category of forensic evidence, and again both DNA and other types of statistical analyses are allowed.Footnote 28 A brief review has not found anything in legal sources from European countries indicating that “the law” is generally suspicious of statistical evidence,Footnote 29 and again we find a robust use of the one form of evidence that some seem to think is the paradigmatic case of naked statistical evidenceFootnote 30—DNA: “According to INTERPOL . . . 120 countries use DNA profiling in criminal investigations, 54 of them have national DNA databases and 26 plan to introduce a national DNA database.”Footnote 31
Although this literature refers to “the law,” it focuses almost exclusively on law in the United States.Footnote 32 For that reason, and because American law is our primary disciplinary home, we focus on it. That the law treats naked statistical evidence with suspicion in the United States is based on occasional statements in a handful of cases expressing various doubts about “probability” and statistical evidence. That handful of cases exists in a huge sprawling collection of systems with over 17 million criminal casesFootnote 33 and close to 60 million noncriminal casesFootnote 34 every year, and as we shall demonstrate, the overwhelming message of that sprawling collection of legal systems is to the contrary. One cannot appraise “the law” in the United States based on the occasional idiosyncratic comment in a case that is often taken out of context. To draw inferences about “the law” in the United States, one must understand what the common law cases are actually saying. The conclusion that “the law” takes a dim view of statistics and probability rests on both mistakes.
The first mistake is exemplified by the reliance on Smith v. Rapid TransitFootnote 35 as the “seminal case”Footnote 36 demonstrating the suspicion of statistical evidence (reliance on Smith also exemplifies the second mistake, as we shall explain below). In legal conversation, a “seminal case” means something like “quite influential in an original way.” Smith was neither. Smith has been cited twice by courts in other states, both times in dissents.Footnote 37 It has only been cited ten times in Massachusetts, but never for the proposition for which it is said to stand. Its holding has been rejected in one Massachusetts case and effectively overruled in two other cases.Footnote 38 It has also been cited by eighteen federal cases, some of which repeat the mantra against statistical evidence or probabilistic reasoning, but apply it only in the context of rejecting unreliable data, as we now develop.Footnote 39
Not even in Massachusetts does Smith stand for what it is cited for in the literature. Instead, the Massachusetts cases break down over the quality of the evidence. If a court concludes the evidence is reliable enough, the evidence is admitted even if probabilistic and critical to the verdict, and in cases without reliable statistics, the evidence is excluded or found insufficient. For example, in Carter v. Yardley & Co., the plaintiff brought a negligence action against a perfume manufacturer for injuries to her skin allegedly from the perfume obtained from a vendee.Footnote 40 The plaintiff needed to show that a “sufficient number [of consumers] are susceptible so that a jury could reasonably say that the defendant ought to have known . . . the danger of injury.”Footnote 41 The plaintiff offered two lay witnesses and one expert-physician witness who each testified that they applied the defendant's perfume to their skin and their skin was irritated.Footnote 42 The court found the evidence properly admitted “to show the probability that the injury to the plaintiff was caused by some harmful ingredient in the perfume rather than by [the plaintiff's] own peculiar and unforeseeable susceptibility.”Footnote 43 The court held that this was enough while citing Smith.Footnote 44 Obviously, Smith is not being read for the proposition that Massachusetts law forbids reliance on probabilistic evidence but instead for the proposition that such a proffer must be adequate for its purpose (whatever one thinks of the application of the principle in the case).
It also stands for rejecting lousy evidence, as occurred in Smith itself. For example, in Tartas’ Case, the lower court awarded damages to an employee's dependents brought under a workmen's compensation claim against the employer.Footnote 45 The employee died of anthrax, and plaintiff was required to show that the employee contracted anthrax as a result of his employment.Footnote 46 However, the employee handled hides for multiple employers on different dates in close succession.Footnote 47 To show causation, the plaintiff set forth expert medical testimony that handling hides could lead to contracting anthrax, but that expert explicitly stated that “it was impossible to determine when the infection took place, [since] the employee handled hides” for multiple employers within a close succession of days.Footnote 48 The Supreme Judicial Court, citing to Smith, explained that “[i]t is not enough that mathematically the chances somewhat favor the existence of a state of facts that would give a right to compensation”; rather, the “evidence must be sufficient to create actual belief in the fact to be proved.”Footnote 49 Since the plaintiff's own expert explained that it was impossible to determine which employer's hides gave the employee anthrax, there was “no evidence warranting a finding that anthrax in the employee arose out of and in the course of his employment” with the defendantFootnote 50—again, lousy evidence.
Highlighting the insignificance of Smith, no case has cited it in Massachusetts in almost three decades, and Smith's supposed holding has been explicitly rejected by the Massachusetts Supreme Judicial Court. Confirming the lousy evidence interpretation as early as 1988, that court commented: “Where courts and commentators have been reluctant to admit statistical evidence, that reluctance has stemmed largely from the fact that the probabilities on which the evidence depended were based on mere speculation or were characterized in such a way as to mislead or confuse the jury.”Footnote 51 In Commonwealth v. Joyner, the same court acknowledged that fingerprint evidence is probabilistic and that a jury may find the necessary element of identification in a criminal case based on fingerprint evidence alone.Footnote 52
The federal cases tell a similar story. There are old federal cases that have skeptical comments about probability and statistical evidence, but all of the federal cases citing Smith involve quite lousy evidence, a trend that accelerates and clarifies with time. For example, in United States v. Jackson, the prosecution needed to prove beyond a reasonable doubt that the defendant had committed a previous felony as an essential element of the crime charged.Footnote 53 The prosecution relied solely on a certificate of conviction from eighteen years prior in New York for a person named “Aaron Jackson.”Footnote 54 According to the Second Circuit, an identification “cannot be made beyond a reasonable doubt unless experience, or statistics (if admissible), teach it is far more likely, given the similarities, that the two are the same person, than that they are two different people.”Footnote 55 The evidence was insufficient because it gave no reason to think that the certificate of conviction was referring to the defendant rather than someone else with the same common name in the state of New York.Footnote 56
Another example that demonstrates the common thread of lousy evidence running through these cases is Hart v. Secretary of Department of Health and Human Services, which cited Smith for that proposition that “courts are hesitant to determine the causation of a past event using statistics.”Footnote 57 However, the court described the actual problem in the case quite differently: “In sum, while these studies certainly indicate that EBV can cause HLH, they provide no reasonable assurance that the observations extrapolated by Dr. Berger and others therefrom could be projected validly to the particular facts of the case sub judice.”Footnote 58 Over the last twenty-five years, no other federal court citing Smith has held that naked statistical/probabilistic evidence is an inadequate basis for a verdict. Instead, the typical case is like In re Whole Foods Market Group, Inc. Overcharging Litigation,Footnote 59 where the district court described the critical collection of data as problematic because it “cherrypicked only lots that had failed [an] initial phase [of data collection] and excluded those that had passed. This methodology was by definition and design unrepresentative.”Footnote 60 In other words, lousy evidence.
Smith's insignificance should not be a surprise. If it had announced a general distaste for statistical evidence and a strict rule against judgments based on naked statistics or probabilistic reasoning, it might have been significant, but instead it involved a standard problem in litigation of the sufficiency of the evidence to support a verdict.Footnote 61 The evidence presented was quite weak. Standing alone and ignoring all other considerations, the evidence might have suggested a “balance of probabilities” favoring the plaintiff in the sense that the evidence increased somewhat the probability that the plaintiff deserved to win, but it did not stand alone. Alternative hypotheses were obvious and not disposed of by the plaintiff.Footnote 62 There was evidence that the bus company ran buses in the area, and one could have been there at the time in question, but no evidence of how likely that was. More importantly, there was no evidence of the presence of other buses, like charter buses, school buses, or things that look like buses, so that it was impossible to discern the probability that the bus belonged to the company rather than someone else. Perhaps reasonable people can disagree, but the court's implicit conclusion that a reasonable person could not find for the plaintiff is understandable.Footnote 63
In addition, allowing the plaintiff's case to proceed based on lousy evidence would have externalized the plaintiff's obligation and cost to investigate (in any of the American jurisdictions) to the defendant.Footnote 64 Shifting costs is more significant than it might appear to those not versed in the procedural niceties of litigation in the United States. Until the advent of modern procedural rules modeled after the Federal Rules of Civil Procedure, the law in effect in most states permitted only limited discovery and required the plaintiff to produce satisfactory evidence of the propositions in dispute through independent investigation. Massachusetts is an example, which did not adopt “modern” discovery rules until 1966.Footnote 65 Allowing a verdict in a case like Smith would have disrupted the procedural rules in place by requiring the defendant rather than the plaintiff to invest its resources in investigating the facts. Not surprisingly, the Massachusetts Supreme Judicial Court did not let that occur.
Smith and the cases that refer to it do not stand for what it is cited for in the naked statistical evidence literature, and neither do the other cases routinely cited as forbidding verdicts based solely on probabilistic or statistical evidence. As we demonstrate next, they are saying, like Smith, that plaintiffs have to reasonably investigate and produce plausible evidence supporting their case. Suppose we are wrong: maybe some judges seventy-five years ago actually thought one could dispense with “probabilistic” evidence (leaving what, the modern imagination wonders). Nonetheless, we will show that the evolution of the common law has left little doubt that there is not a robust principle in the American legal systems that is opposed to probabilistic or statistical evidence even as the sole basis for a critical element.
First, consider Stephens v. StateFootnote 66 and Dorsey v. State.Footnote 67 In Stephens, an expert testified in a child sex abuse case “that statistically eighty to eighty-five percent of child sexual abuse is committed by a relative close to the child.”Footnote 68 Although the court said: “It is difficult . . . to understand how statistical information would assist a trier of fact in reaching a determination as to guilt in an individual case and, had an objection been made, it should have been sustained,”Footnote 69 this is just another case of lousy evidence. There was no showing of how the purported data were compiled, or any reason to infer that they demonstrated a causal relationship rather than just an unreliable accidental correlation. Whatever was in the mind of the judges in Stephen, the same court five years later was presented with the same issue in a DNA case where an adequate foundation was laid for the data and effectively overruled the case.Footnote 70
In Dorsey v. State a police detective testified that 75 to 80 percent of the cases he had previously investigated involved guilty suspects.Footnote 71 The court gave this nonsense the back of its hand, pointing out that “there was absolutely no factual or statistical basis upon which to formulate such an opinion” and that “the officer's opinion, under the circumstances, was completely unreliable and untrustworthy.”Footnote 72 On appeal, the Court of Appeals made a disparaging comment about “probability,” but when reliable statistics come along, the story changes.Footnote 73 The argument today in Maryland is over the foundation that has to be laid for admissibility of statistical evidence.Footnote 74
Reliance on cases like Smith, Stephens, and Dorsey is simply a mistake.Footnote 75 Virtually every other case cited in this literature is similar or the landscape has been changed by subsequent developments. Enoch and Fisher identify five additional cases that supposedly stand for the proposition that “when statistical evidence first began to appear in court, many judges responded antagonistically, deeming it inadmissible and devoid of any probative value.”Footnote 76 That is inaccurate. The cases instead form a stunning array of insignificance with respect to establishing an aversion to statistical evidence:
—People v. Collins.Footnote 77 The court held that “mathematical odds are not admissible as evidence to identify a defendant in a criminal proceeding so long as the odds are based on estimates, the validity of which have not been demonstrated.”Footnote 78
—Commercial Standard Insurance Co. v. Gordon's Transports Inc.Footnote 79 Here, the point was that “where proven facts give equal support to each of two inconsistent inferences . . . neither of them being established, judgment, as a matter of law, must go against the party upon whom rests the necessity of sustaining one of these inferences as against the other, before he is entitled to recover.”Footnote 80
—Evans v. ElyFootnote 81 is essentially the same. The actual holding of the court was that there was insufficient evidence to send a case to the jury. It said in passing: “All courts agree that a case should never be submitted and the jury be permitted to act on mere conjecture or even on probabilities.”Footnote 82 It continued: “To justify submission of a case there must be evidence from which the jury may draw valid inferences, by which we mean there must be ‘something of substance and relevant consequence, and not vague, uncertain, or irrelevant matter not carrying the quality of proof or having fitness to induce conviction.’”Footnote 83 It is obvious that the court is using “probabilities” here in the same way that the Smith court incorrectly (as judged by modern conventions) uses the term.
—Virginia & Southwestern Railroad Co. v. HawkFootnote 84 is yet another case in which the plaintiff offered lousy evidence and the court used probability language carelessly, which is shown by its reliance on Day v. Boston & M.R.R.:
Quantitative probability is only the greater chance. It is not proof, nor even probative evidence of the proposition to be proved. That in one throw of dice there is a quantitative probability, or greater chance, that a less number of spots than sixes will fall uppermost is no evidence whatever that in a given throw such was the actual result. Without something more, the actual result of the throw would still be utterly unknown. The slightest real evidence that sixes did in fact fall uppermost would outweigh all the probability otherwise.Footnote 85
Concluding that the mathematical odds of throwing a six would not dominate, say, eyewitness testimony that a six was thrown cannot seriously be taken as a wholesale condemnation of probabilistic evidence.Footnote 86
—In State v. Carlson, the “concern over this evidence is not with the adequacy of its foundation, but rather with its potentially exaggerated impact on the trier of fact. Testimony expressing opinions or conclusions in terms of statistical probabilities can make the uncertain seem all but proven, and suggest, by quantification, satisfaction of the requirement that guilt be established ‘beyond a reasonable doubt.’”Footnote 87 Lest there be any doubt, the court later clarified that its concern was with how the evidence was presented to the jury.Footnote 88 Sixteen years later, when the Supreme Court of Minnesota encountered reliable statistical evidence (DNA), it was admitted.Footnote 89
We will give one final example of the misreading of the common law cases in this literature, Guenther v. Armstrong Rubber Co.Footnote 90 The plaintiff was injured by an exploding tire as he was mounting it and brought a products liability action against the defendant-manufacturer.Footnote 91 The tire was secured at the scene, later examined by both parties’ experts, and brought to the trial.Footnote 92 The tire was a whitewall admittedly manufactured by the defendant, but the plaintiff, who had been knocked unconscious during the incident, testified that a blackwall caused his injury.Footnote 93 The plaintiff also wanted to offer evidence that 75 to 80 percent of the tires sold where he worked were manufactured by the defendant, and thus that if the tire had been a blackwall it probably was manufactured by the defendant.Footnote 94 No reasonable person could believe the plaintiff's testimony. The plaintiff had suffered injuries that required hospitalization, and more importantly the chain of custody of the tire in question was uncontested. To make matters worse, the plaintiff's own expert had examined a whitewall tire, not a blackwall tire, and thus could not support the theory of a defective blackwall tire. Plaintiff's counsel wanted to submit the case to the jury on both theories—that the tire was either a whitewall or blackwall—to offset the possibly damaging consequences of the plaintiff's obviously mistaken testimony. The court held this evidence insufficient to proceed on the theory that the tire was a blackwall because “[w]hat we have before us is . . . the extremely strong, believable submitted testimony which clearly set out the plenary fact that a particular tire [a whitewall] manufactured by defendant-appellee injured him.”Footnote 95 Guenther is a plain case of insufficiency of the evidence and has literally nothing to do with probability as a basis for decision, other than the loose use of probability talk similar to that in Smith.Footnote 96
Common law cases are decided on their facts. In none of these canonical cases cited for an aversion to statistics is it plausible that the court meant to cast doubt on the utility of probability in general or on the acceptability of statistics, and almost without fail, subsequently those same courts faced with reliable statistical proffers admit the evidence and express no qualms about its sufficiency. The arguments to the contrary mismodel the common law and neglect both how it evolves and how older cases are left behind by more refined notions of evidence and probability. There has been a fundamental shift in the cases away from peculiar (to modern ears) statements about probability to an understanding that all evidence is “probabilistic” in some sense.Footnote 97 That has been accompanied by the wholesale admission of and reliance on reliable statistical evidence, including naked statistics, to which we now turn.
The modern treatment of statistics in American litigation involves overwhelming acceptance of the evidence so long as it reliable and helpful—even when the evidence is as close to being “naked” as possible. Rather than applying some special rule to statistical evidence, courts across the country treat it like any other evidence. That requires the evidence to be reliable and requires that the value of its admission not be outweighed by its costs, the same variables that explain the older cases like Smith, Stephens, Collins, and Dorsey, although expressed in modern terminology. When the data that parties offer is compiled in an appropriate way, accompanied by a plausible explanation for why the data pertains to a material proposition, evidence is readily admissible and sufficient for a verdict even if it is a naked statistic.
The Supreme Court of the United States has noted the probabilistic nature of evidence in general.Footnote 98 If all evidence is probabilistic, it would be problematic to prefer ambiguous probabilistic evidence like witness testimony over evidence that clarifies its limitations through reliable statistics.Footnote 99 Still, the Supreme Court does not speak for states in terms of evidentiary policy, but the ubiquity of statistics in modern American litigation is striking.Footnote 100 For example, a plaintiff can recover in disparate impact cases under Title VII only with an adequate statistical proffer.Footnote 101 Equal protection claims may be proven through statistics,Footnote 102 and statutes often provide for the admission of statistical evidence in other situations.Footnote 103
But what about naked statistics? As we mentioned earlier, there is disagreement about what “naked statistics” refers to, whether it is evidence of base rates, market share, or nontrace predictive evidence, but there seems to be agreement that to qualify it must be the only evidence in the case—as in the Gatecrasher and Blue Bus hypotheticals. This disagreement does not matter because either there is no such thing as naked statistics, or if there is, then such evidence is admitted and can be the only evidence of an essential element of a cause of action so long as the evidence meets the normal conditions of the admissibility of any evidence: relevant, reliable, and not highly prejudicial, misleading, or a waste of time.Footnote 104
First, it is impossible to have naked statistics in an American trial, which leaves what is being discussed in the literature unclear.Footnote 105 The evidence law in the United States is uniform in requiring that a foundation must be laid for the admissibility of evidence.Footnote 106 The proponent of the naked statistic is required to provide admissible evidence of how it came into being in the first place. Someone will testify and can be cross-examined. All of this contextualizes the evidence and dresses the statistic in clothes. To have any application to reality the inquiry should focus on statistical evidence that is critical for judgment rather than a nonexistent abstraction.
Second, put evidentiary niceties aside for now.Footnote 107 When the courts are presented with reliable statistical evidence of a necessary element, they readily admit it and allow judgments to be based on it. The most recent and dramatic example is of course DNA evidence.Footnote 108 Case after case concludes that a conviction may be returned when the only evidence of identification is DNA. A typical example is State v. Toomes, holding DNA evidence sufficient when “the only—and we emphasize the only—evidence connecting the defendant to the victim's rape is the DNA results.”Footnote 109 We have found no counterexamples to these cases.Footnote 110
DNA evidence is not unique.Footnote 111 A number of forensic techniques are probabilistically based and sufficient for convictions,Footnote 112 even when the evidence is necessary for the verdict.Footnote 113 Fingerprints are analogous to DNA in this respect,Footnote 114 yet fingerprint evidence alone is sufficient proof of identity for a criminal conviction.Footnote 115 Although typically not as clean as the DNA and fingerprint cases, for close to one hundred years ballistics evidence has been admissible to show that a firearm was used during the commission of a crime.Footnote 116 Ballistics evidence is probabilistic, yet the evidence is readily admitted and can be critical to the sufficiency of the evidence to sustain a conviction. The same is true of toolmark evidence: “Although the scholarly literature is extraordinarily critical of toolmark pattern-matching, it appears that courts have made little effort to limit or qualify the admission of such evidence.”Footnote 117 And handwriting evidence.Footnote 118
A recent development in this literature, fueled by the effort to find some type of evidence that the legal systems are resistant to, is to distinguish nontrace from trace evidence. The mailing element under 18 U.S.C. §1341 (criminal wire fraud) demonstrates both the sufficiency of statistical evidence for a necessary element based on membership in a reference class alone with no individuating evidence and the evolutionary nature of the common law.Footnote 119 In 1989, the Third Circuit allowed the mailing element of criminal wire fraud to be proven upon the showing of a routine practice of an organization, even though that is admittedly probabilistic, so long as there was also testimony involving a specific reference to the mailing at issue (naked statistics plus). However, the court required evidence that the routine business practice was followed with certainty—100 percent of the time.Footnote 120 Five years after Burks, the same court recognized that the requirement of certainty was overruled by the Supreme Court in Victor and effectively overruled Burks's requirement of proof that the routine business practice was followed 100 percent of the time.Footnote 121 The specific reference requirement was left intact but was largely eviscerated, satisfied simply by testimony that the “invoices at issue in this case were handled in the normal manner.”Footnote 122 Continuing this development, the other circuits now have no specific reference requirement;Footnote 123 they instead rely merely on the probabilistic inference from the general business practice.Footnote 124
Moreover, various rules permit propensity evidence, which would be peculiar if there were an aversion to statistical evidence naked or otherwise.Footnote 125 Pertinent character traits of both defendants and witnesses may be proved in criminal cases to prove action in conformity with the character trait.Footnote 126 In sex or child abuse prosecutions, the state may prove any prior sexual abuse or child molestation to show a propensity in order to prove the alleged perpetrator committed the act in question.Footnote 127 FRE 404(b), although purporting to disallow specific acts to show propensity, suggests that various categories that are difficult to distinguish from propensity are admissible.Footnote 128 A witness's character for untruthfulness and truthfulness is admissible to show that the witness is lying, or not, on the stand.Footnote 129 The so-called doctrine of chances is obviously propensity-based nontrace evidence.Footnote 130 FRE 406 permits “evidence of a person's habit or an organization's routine practice” to prove that the person or the organization acted in a similar fashion at the time in question.Footnote 131 Experts are ubiquitous in American litigation, both civil and criminal, and in many cases experts are simply the means of funneling relative-frequency data to a fact finder.Footnote 132 These are obviously probability relationships with statistical generalizations lurking in the background and sometimes in the foreground.Footnote 133 And on occasion American courts not only allow but demand nontrace relative-frequency data as proof of a necessary element. On remand following Daubert,Footnote 134 the Ninth Circuit held that to prove causation in toxic tort cases requires epidemiological evidence of a relative risk of more than 2:1.Footnote 135 Statistical and probabilistic proffers as well as verdicts based on them are plentiful in the American legal systems.Footnote 136
There is no contemporary support in the various American jurisdictions for the categorical exclusion of naked statistical evidence as the basis for a verdict.Footnote 137 Even before the advent of DNA, the general approach across the United States either was or was evolving to admit evidence critical to a judgment in statistical form where an adequate foundation had been laid and to allow experts to explain the evidence to the fact finder. That evolution has been accompanied by a virtual elimination from the cases of phrases to the effect that “a mere probability is insufficient for a verdict,”Footnote 138 a dynamic the theoretical literature seems to have missed.Footnote 139 Arguments to the contrary deny that the robust use of statistical evidence has anything to do with the problem at hand, including those cases closest to “naked statistics.”Footnote 140 They also assert that any counterexamples are just exceptions to a general principle, while simultaneously providing no contemporary examples.Footnote 141
Ironically, the cases that originally motivated the intuition that something is amiss with base rate, market share, or naked statistical evidence are not within the pertinent category. There was no base rate, market share, or statistical evidence in Smith Footnote 142—that was the problem. In both Stephens and Dorsey, the witnesses were simply making things up, just as the prosecutor did in Collins. The problem in Carson was how to present evidence to the jury. Arguing that Guenther stands for the rejection of base rate evidence is absurd. Whatever the judges were thinking in the past, today “the law's” aversion to naked statistical evidence is a myth.Footnote 143
In addition to the mismodeling of various legal matters, the naked statistical evidence literature is highly dependent on the use of weird hypotheticals that themselves both mismodel American legal systems and make impossible epistemological demands that are designed to abstract the question of naked statistics from all the surrounding circumstances.Footnote 144 It is entirely unclear what to make of such bizarre examples. Whatever the cause, the conclusions drawn bear little relationship to the operation of any American legal system. They involve the deterrent value of evidence and a curious view on the implications of errors. We discuss these in turn.
One argument is that verdicts on a statistical basis would have perverse effects and be morally reprehensible. Consider first the perverse effects possibility that was employed by Richard Posner in his discussion of the Blue Bus hypotheticals:Footnote 145
If B, though responsible in fact for almost half the accidents, is never held liable and A is always held liable, A will have a big incentive to be careful and B little or no incentive to be careful. As a result, over time, more than half the accidents will be caused by B, increasing the error rate resulting from allowing juries to base decisions on the ratio of the companies’ buses on the route in question. Eventually, A, having higher liability costs, will probably withdraw from the route; the rule on burden of proof will have created a monopoly!Footnote 146
Such arguments neglect the adaptive nature of life and lead to absurd deductions:
[T]he consequence Posner fears will occur only if A is remarkably stupid. In the world Posner is hypothesizing, all A need do is take at most three buses out of service. If A takes three buses out of service, the ratio of buses now favors A . . . , and B will henceforth be held liable for all accidents. Of course, another economic fear might now arise: B, also not being massively stupid, might respond by taking two of its buses out of service. Perhaps the prediction would now be that “bus reduction” would become recursive, eventually resulting in only one bus company with only one bus driven by a very careful driver, which would surely not serve the needs of the community! Therefore, the logic would go, allowing verdicts based on naked statistics is not economically sensible.
In the absence of an assumption of massive stupidity:
Neither a monopoly nor the essential elimination of the industry would result from allowing probabilistic verdicts. If such verdicts began to accumulate inaccurately against bus companies, they would invest in precautions, many of which are cheaply available.Footnote 147
In addition to requiring massive stupidity for these hypotheticals to make sense, they also neglect that naked statistics will determine outcomes primarily in those cases with no other evidence. One would sensibly think that to be a rather small set, reducing even further the likelihood of significant social disutility flowing from the use of the evidence. This point has even more significance for the next weird hypothetical we will address—the Gatecrasher:
In cases like the Gatecrasher Case . . . a verdict against the defendant does not provide any incentive for lawful behaviour to people in the same situation as the defendant. In the Gatecrasher Case, a verdict based on naked statistical evidence would not give a person in the defendant's position at future rodeos an incentive to refrain from crashing the gate. Since . . . the evidence against him is not caused by not paying admittance . . . [i]t exists and will be held against him whether he pays admittance or not.Footnote 148
The Gatecrasher scenarios could not possibly occur in the United States. They postulate that the only evidence in the case is a naked statistic, like of one hundred attendees, forty-nine paid for admittance while the other fifty-one crashed the gate. However, the rodeo operator must sue somebody, and there must be evidence that the defendant was at the rodeo. The evidence of identity will not come with a 100 percent guarantee of accuracy, nor would the statistic of gatecrashers and paid customers.
Although it is absurd, take the hypothetical at face value—in some unfathomable way the rodeo operator is suing an attendee who was undoubtedly present and there is no doubt about the numbers. The argument is that allowing recovery is immoral because a future gatecrasher in precisely the same position will not be deterred by this kind of evidence,Footnote 149 and thus the law imposes a harm where there can be no benefit.Footnote 150 This is false except under the absurd assumptions of no other evidence being allowed in the case or massive stupidity similar to that of the operators of bus companies. The only thing the potential gatecrasher can know about “the law” is the evidence rule allowing judgments based on naked statistics. The person must not know that all the hassle of being sued evaporates if he buys a ticket and can prove it. If he knows that, he obviously has an incentive to proceed lawfully, an incentive enhanced by the possibility of the naked statistic being used against him. He must also not know that keeping the receipt would be an essentially irrefutable safeguard against being sued—another incentive to buy the ticket. And he must also not know that the ability to testify in the language spoken where the suit would occur (or have adequate translation) would also invariably result in a verdict in his favor.Footnote 151 Absurd assumptions or massive stupidity. In the real world without widespread massive stupidity, it is quite reasonable to believe there would be an overall beneficial incentive effect encouraging the purchase of tickets and thus reducing the wasted transaction costs of unnecessary litigation.Footnote 152
To be sure, in some instances statistical evidence exists regardless of the defendant's actions, which decreases however slightly the deterrent efficacy of “the law” and increases however slightly the risk of a wrongful plaintiff's verdict. Those effects would be swamped by the other behavioral consequences.Footnote 153 The conclusions drawn about deterrence reflect the weird hypotheticals by which they are motivated. They abstract from a complex reality to a single variable and ignore all the surrounding territory.Footnote 154 They also ignore that the evidence may increase the probability of correct verdicts, and that the actual verdict will be influenced by numerous other variables as well. This is one reason why courts are resistant to lousy statistical evidence, but not to reliable statistical evidence, even when it is the only evidence of a necessary element. Correct verdicts are one of the major aspirations of the law of evidence.Footnote 155
The focus on errors involves another mismodeling of the legal systems in the United States. One argument against probabilistic evidence is that returning a verdict with an identifiable risk of error for a wrongful conviction would do “an injustice” to that person.Footnote 156 Whether this argument is, like the deterrence argument, wielded to justify “the law's” aversion to statistics or rather to complain about its immorality in not doing so is unclear. The claim is that this is an epistemological puzzle, but one that lies in “the near vicinity” of the supposed puzzle over statistical evidence, which suggests that the “epistemological puzzle” bears on its legal cousin. Regardless, the claim of an injustice points to a critically important aspect of the construction of legal systems. That in turn explains why this vision of an injustice has had, and will have, little impact on the structure of legal systems in the United States.Footnote 157
False positives are consistently made in the American legal systems,Footnote 158 but the entire legal system would have to be shut down to avoid errors of convicting innocent people or returning incorrect civil verdicts. The problem is running a system with around 80 million cases a year without making a mistake. Even criminal convictions are allowed in cases of contrasting testimony—the classic he said/she said scenario.Footnote 159 As judges say time and time again, questions of credibility are for the jury, and not because of a belief of infallibility.
As tragic as errors are, they are not the only kind of mistake that can be made, and in a sensible system the number of correct decisions matters too.Footnote 160 A system that in every hundred cases had ninety false acquittals and nine false convictions satisfies the famous Blackstone ratio of a 10:1 ratio of errors. It also results in exactly one correct outcome out of every one hundred cases, which surely is unacceptable. One of the difficult tasks of constructing legal systems is to optimize the four possible outcomes at trial (correct and false decisions for either side),Footnote 161 and the task does not end there. The behavior of legal systems affects which cases are settled and which cases are brought, which can also generate errors. If a woman feels that she will be humiliated if she brings a charge of rape, for example, she will be less likely to bring it. That will generate an error—the failure to hold an offender accountable.Footnote 162
There remains the problem of deadly dilemmas. Obviously, legal systems attempt to channel behavior through the promise of sanctions for unlawful behavior, but some governmental action simply shifts the costs and benefits of the regulated activity. If legal systems shut down because of the risk of error, the deterrent efficacy of the law would be diminished, generating orders of magnitude more harm than that caused by the occasional wrongful verdict.Footnote 163 A dramatic example: if executions deter, then shutting down the death penalty involves the trade of a possible few innocent people for many more who would otherwise not have been murdered. Even if there is no deterrent effect, the alternative is long prison stays and more people being killed in prison—all wrongful deaths—than have been executed since the death penalty was reinstated. Moreover, it will come as no surprise that prison with few exceptions reduces life expectancy, and all those lost person-years are analogous to the execution of an innocent person.Footnote 164 Obviously reducing errors matters but so too does the distribution of the costs of doing so.
The construction of a legal system has at least two large and interrelated optimization problems: optimizing primary and litigation behavior and correct and erroneous decisions. Hovering over all of this is the problem of deadly dilemmas. The difficulty of these tasks plainly explains much of what one observes in the American legal system whereas an aversion to statistical evidence does not. Any sane legal system has to manage the problem and not be obsessed with eliminating one kind of mistake at any cost. At the same time, the promise of relatively accurate verdicts is essential to the operation of the rule of law. Thus, what “the law” actually resists is not reliable statistical evidence of any kind but rather the exclusion of probative evidence.
Perhaps the most serious cost (to us) of the fascination with the nonissue of naked statistical evidence is that it may have deflected attention from issues deserving of careful scrutiny in the construction of legal systems. For example, at the root of the explanationist theory of juridical proof is what is known as the relative plausibility theory, which has come to supplant conventional probability as the best explanation of juridical proof. It is explicitly positive and not normative, but there are critical components of it that could be shored up (or shown to be inadequate, casting doubt on the power of the explanation). What plausible means is perhaps inadequately explained and would benefit from the attention of the professional epistemologists, among others. The extent to which one can forgo probability theory for other cognitive tools and remain rational is another, and does the real-life phenomenon of how legal systems function shake the commitment to probability as being at the base of rationality or lead to the conclusion of systemic irrationality? Nor have the complex optimization problems noted above been subjected to sustained philosophical inquiry. Although some work notes the optimization problem of mistakes, there is very little attention paid to the significance of correct decisions or the problems cast by the shadow of the law on the decisions made to litigate (or not). Another issue is the viability of the argument in the legal literature that the aversion to well-formulated statistical evidence would be problematic because it favors worse evidence over better evidence.Footnote 165 With reliable statistical evidence, one precisely knows its limit; with nonstatistical evidence, one does not.
We realize that this may sound like the legal analysts asking others to do our work for us, but perhaps there is room for fruitful engagement on issues that affect the reality of the law. There is no doubt that the evidentiary scientists need all the help that we can get.
