2. Subjective judgments in forensic science

Forensic science examinations use more subjective judgment than commonly recognized. A forensic scientist's decision to declare a ‘match’ is a subjective judgment in many forensic science disciplines. (The NAS report discusses the variety of terms used to describe what is commonly thought of as ‘matching’ − see page S-15.) According to the NAS report, these disciplines include ‘impression evidence’ such as shoeprints and tire tracks, toolmarks, and firearms identification (the later commonly called ‘ballistics’), traditional hair microscopy, the handwriting comparisons of questioned document examiners, bloodstain pattern analysis, and fingerprint examinations (NAS, Reference NAS2009). Clearly, the last item is the most surprising, especially considering the existence of automated fingerprint identification systems such as those now commonly built into laptops. It is probably appropriate, therefore, to spend some time discussing the subjective nature of latent print identification.

Original research by Francis Galton, Henry Faulds, and others supports the hypothesis that friction ridge patterns on human hands and feet are unique and essentially unchanging over time (Beavan, Reference Beavan2001). Apparently, the friction ridge patterns of an individual will change over time only to the extent that cuts, diseases, other accidents, or willful manipulations such as surgery or acid burnings induce a permanent change. Thus, both human examiners and automated systems can reliably decide whether two sufficiently clear and distinct images of friction ridge skin have a common source. The subjectivity of fingerprint analysis derives from the ambiguity of many ‘latent’ fingerprints gathered at crime scenes. These latent prints may be smudged, distorted, covered with other latent prints, or otherwise unclear and indistinct.

Figure 1 illustrates. The clear and distinct image on the left is a rolled print taken from Brandon Mayfield by law enforcement officers. The unclear and indistinct image on the right is latent print taken from the scene of the 2004 Madrid train bombing. In 2004, the FBI declared a ‘100 percent match’ of Mayfield to the latent lifted from the Madrid crime scene. The Spanish authorities objected this identification. They seem to have been correct as the FBI later withdrew its identification and released Mayfield (Office of the Inspector General, 2006). The Mayfield error is one of a growing list of known false positive fingerprint errors (Cole, Reference Cole2005; NAS, Reference NAS2009; Williams, Reference Williams2007; Rubin and Winton, Reference Rubin and Winton2008). On 19 October 2007, Maryland judge Susan Souder cited the Mayfield case and other evidence in her decision to exclude incriminating fingerprint evidence from a capital case, State of Maryland v. Bryan Rose (In the Circuit Court for Baltimore County, Case No. K06–545). She described the standard technique as ‘a subjective, untested, unverifiable identification procedure that purports to be infallible’ (p. 31).

Figure 1.

Ambiguous and unambiguous fingerprint images. The clear and distinct image on the left is a rolled print taken from a suspect by law enforcement officers. The unclear and indistinct image on the right is a latent print taken from a crime scene by law enforcement officers. FBI examiners declared a ‘100 percent match’ between them.

The NAS report contains little to suggest that ‘nuclear DNA analysis’ may often involve substantial use of subjective judgment. But, ‘DNA tests sometimes produce ambiguous results that are subject to multiple interpretations’ and ‘[w]hen interpreting ambiguous results . . . human analysts rely heavily on subjective judgments to distinguish signal from noise, explain anomalies, and account for discrepancies’ (Thompson and Cole, Reference Thompson, Cole, Costanzo, Krauss and Pezdek2007: 34). Subject judgment is likely to enter when more than one person has contributed to the DNA sample, or the sample is contaminated, degraded, or very small.

Figure 2 illustrates unambiguous DNA evidence. The biological sample is prepared and run through a genetic analyzer, which produces data that are represented as an electropherogram, which is then interpreted by a forensic scientist. As Figure 2 reveals, an electropherogram is a squiggly line. The figure shows only three loci, whereas most standard tests in the US examine 13 loci. Thompson and Cole (Reference Thompson, Cole, Costanzo, Krauss and Pezdek2007), from whom Figures 2 and 3 are lifted, explain. ‘As can be seen, the profile of Suspect 3 corresponds completely to that of the crime scene sample, hence it is a match that indicates Suspect 3 is a possible source of the blood at the crime scene. Suspects 1, 2, and 4 are eliminated as possible sources because one or more of their alleles differs from the crime sample’ (p. 34).

Figure 2.

Unambiguous DNA evidence. The electropherogram of the crime scene evidence is clear and distinct. It matches suspect 3, who is included as a possible source. Suspects 1, 2, and 4 are clearly excluded.

Figure 3.

Ambiguous DNA evidence. The electropherogram of the crime scene evidence is not clear and distinct. It is not clear whether the suspect should be included or excluded as a possible source.

Figure 3 illustrates some of the ambiguities that can enter DNA profiling. When DNA is degraded or the original sample contains a small quantity of DNA, it can be difficult to distinguish between signal and noise. The figure, Thompson and Cole explain, ‘shows a comparison between the DNA profile of a saliva sample from the skin of a sexual assault victim and the profile of a suspect. Experts differed over whether these two profiles match. For example, some experts thought the peak labeled “12” at locus “D3S1358” was a true allele, others thought it was merely noise in the system. The experts’, they continue, ‘also differed over whether the peak labeled “OL allele” at locus “FGA” was a spurious anomaly that could be safely ignored, or whether it might be hiding another allele. When interpreting ambiguous results like those shown in [Figure 3] human analysts rely heavily on subjective judgments to distinguish signal from noise, explain anomalies, and account for discrepancies’ (p. 34).

Figures 2 and 3 illustrate a common practice in American crime labs. The forensic scientist typically examines the genetic profile of the police suspect(s) before characterizing the unknown sample. They ‘call the alleles’ only after they have seen the answer key. As Figure 2 illustrates, the DNA examination is sometimes a multiple-choice test. Krane et al. (Reference Krane2008) object to observing the genetic information of suspects and victims before interpreting the genetic information of the unknown sample; they say it may induce ‘observer effects’ that bias the outcome of the examination.

3. Dependence on law enforcement

The NAS report says, ‘The majority of forensic science laboratories are administered by law enforcement agencies, such as police departments, where the laboratory administrator reports to the head of the agency’ (p. 6–1). We do not have good quantitative evidence on how much work is done in police labs rather than independent labs. (In some counties in California, the crime lab is organized directly under the prosecutor's office.) It is clear, nevertheless, that the statement in the NAS report is right.

Crime lab budgets and performance reviews often come from the police, the FBI, or the sheriff's office. Recent literature (Baicker and Jacobson, Reference Baicker and Jacobson2007; Benson and Rasmussen, Reference Benson and Rasmussen1995; Benson et al., Reference Benson, Rasmussen and Sollars1995; Makowsky and Stratmann, Reference Makowsky and Stratmann2007) recognizes that law enforcement agencies have an incentive to expand their budgets in more or less the same way described in the public choice analysis of Niskanen (Reference Niskanen1971), notwithstanding the presence of competing motives. As I have noted elsewhere, ‘If a crime lab's budget and performance reviews come from a budget maximizing law enforcement agency, personnel in the lab have an incentive to serve the interests of that agency, rather than abstract and pristine truth’ (Koppl, Reference Koppl and Lopez2010).

4. Monopoly

Forensic science today is characterized by a twofold monopoly. First, evidence is typically examined by one crime lab only (Koppl, Reference Koppl2005a, Reference Koppl2005b). In this sense, the crime lab receiving a bit of evidence has a monopoly on examination of that evidence. Second, that same lab will normally be the only one to offer an interpretation of the results of the examination it performs. No other experts in forensic science will be asked to judge what the evidence means. When expert witnesses are called in civil cases, it is common that each side will have its own expert or experts. The same is not true of criminal trials. Typically, only the prosecution will have expert witnesses testifying on forensic evidence (NAS, Reference NAS2009: S-8; Giannelli, Reference Giannelli2004). Monopoly in examinations may allow errors and even fraud to go undetected. Monopoly in interpretation may allow false interpretations to go unchallenged and alternative hypotheses to go unexamined.

The twofold epistemic monopoly in forensic science, the biases sometimes induced by dependence on law enforcement, and the ambiguity of much forensic evidence seem to have produced a needlessly high rate of ‘intentional and unintentional’ (NAS, Reference NAS2009: 1–8) errors of test and interpretation in forensic science (Koppl, Reference Koppl2005a; Saks and Koehler, Reference Saks and Koehler2005; Garrett and Neufeld, Reference Garrett and Neufeld2009). The organization of forensic science as a part of law enforcement encourages interpretations favorable to the police theory or the prosecutor's case (NAS, Reference NAS2009: 6–2; Giannelli, Reference Giannelli1997; Risinger et al., Reference Risinger, Saks, Thompson and Rosenthal2002). Whitman and Koppl (Reference Whitman and Koppl2010) develop the logic of such institutional bias with a Bayesian model of rational choice by forensic scientists. The biases, or ‘observer effects’, induced in this way need not be conscious (Dror and Charlton, Reference Dror and Charlton2006; Dror, Charlton, and Péron, Reference Dror, Charlton and Péron2006; Risinger et al., Reference Risinger, Saks, Thompson and Rosenthal2002; Krane et al., Reference Krane2008).

5. Objective performance criteria for law enforcement

Judge Posner notes the use of objective performance criteria to evaluate police agencies. He notes an agency problem with policing, saying, ‘when agents can appeal to local knowledge as the basis of their decisions, the center, which lacks that knowledge, cannot monitor their decisions effectively’ (p. 32). He imagines the problem is mostly solved by objective performance measures. ‘When the task is criminal investigation, this problem is not acute, because the outputs of FBI agents engaged in criminal investigation – number of arrests, prosecutions, convictions, length of sentences, and amount of property recovered – are quantitative, relatively hard to manipulate (at least legally), and therefore feasibly measured.’ He imagines these objective measures permit the successful use of high-powered incentives.

The criteria Posner lists, with the partial exception of ‘amount of property recovered’, make suspects fungible. Law enforcement agencies should attempt to discriminate between the guilty and the innocent. Performance criteria such as number of convictions, however, place no weight on correctly discriminating between the innocent and the guilty. The police can produce arrests, trials, and convictions of innocent persons by arresting persons who are poor, uneducated, or mentally weak, or by inducing false confessions. As Holmstrom and Milgrom (Reference Holmstrom and Milgrom1991) point out in a classic contribution to organization economics, high-powered incentives may be inappropriate when relevant aspects of the agent's performance are unobservable, hard to measure, or subject to significant measurement error (Dewatripont et al., Reference Dewatripont, Jewitt and Tirole2000 provide a helpful survey.) If we may consider false convictions to be low quality output, we would seem to have an example of their ‘result that it may be optimal to provide no quantity incentives when quality is poorly measured’ (Holmstrom and Milgrom, Reference Holmstrom and Milgrom1991: 28).

It is difficult to gather data that would reveal how many poor, uneducated, and mentally weak persons are wrongfully convicted. The literature on false confessions, however, supports the notion that some police investigators in at least some cases may unjustly target such persons. Drizin and Leo (Reference Drizin and Leo2004) assembled the largest set of ‘proven’ false confessions, consisting of 125 cases. They identify four groups who, compared to others, ‘are more vulnerable to the pressures of interrogation and therefore less likely to possess or be able to muster the psychological resources or perspective necessary to withstand accusatorial police questioning’ (p. 919). They are children under the age of 14 (pp. 963–968), juveniles under the age of 18 (pp. 968–970), the ‘mentally retarded’ (pp. 970–973), and the mentally ill (pp. 973–974). (The issue at hand is not related to the debate, reviewed by Novak and Engel, Reference Novak and Engel2005, on ‘criminalizing mental illness’.)

Techniques readily available to the police allow them to do easily what Posner says they can do only with difficulty, namely manipulate the performance criteria. In some cases, such manipulation is illegal. Long standing ‘allegations . . . that Chicago police detectives routinely tortured murder suspects’, for example, may be legitimate given the perjury and obstruction of justice charges filed against a retired police commander in October 2008 and the appointment of a special prosecutor in April 2009 for ‘the cases of five murder suspects seeking new trials’ in connection with the commander's alleged abuses (Coen and Mills, Reference Coen and Mills2008; Wallberg, Reference Wallberg2009). The techniques may also be applied legally. Police brutality is not required to extract a false confession from many vulnerable persons. The poor are less able to marshal resources to defend themselves even when innocent. And so on. It seems likely that unconscious biases could cause police to concentrate attention on persons who are poor, uneducated, or mentally weak, even when the police have no conscious desire to manipulate the system.

Monopoly forensic science is an important tool for police investigators and prosecutors seeking (consciously or unconsciously) to maximize convictions. As we have seen, the current organization of forensic science encourages crime labs to support the police and prosecution. When forensic evidence is ambiguous, this disposition may lead to wrongful arrest and conviction.

Public records reveal at least some use of several inappropriate practices that support the epistemic monopoly of crime labs. These include (1) evidence destruction (Greene and Moffiet, Reference Greene and Moffiet2007); (2) resistance to case review when errors are revealed or suspected (Thompson and Dioso-Villa, Reference Thompson and Dioso-Villa2008: 283–289); (3) failure to comply with Coverdell oversight requirements (Office of the Inspector General, 2008); (4) over-claiming in forensic science testimony (Cole, Reference Cole2007, especially pp. 821–825; Garrett & Neufeld, Reference Garrett and Neufeld2009, especially pp. 55–60); (5) resistance to redundancy and oversight (Gestring, Reference Gestring2009); (6) strong demarcations between outsiders and insiders (Gestring, Reference Gestring2009); (7) asymmetric models of defense and prosecution experts (CLPEX, 2007); (8) resistance to blind testing (Wells, Reference Wells2009); (9) reliance on subjective techniques (NAS, Reference NAS2009); (10) efforts to exclude defense experts (Kreeger, Reference Kreeger2002).