Two years ago, a research team comprising two political scientists, Andrew Martin and Kevin Quinn, and two legal academics, Pauline Kim and Theodore Ruger, set out to forecast the votes cast and outcome reached in each case argued before the U.S. Supreme Court during its 2002–3 term. To generate the predictions, the researchers turned to approaches to decision making dominant in their respective fields. The political scientists devised a statistical model, which assumes, in line with the vast disciplinary literature on the subject, that judicial decisions are largely a function of politics and case facts. The legal academics went in a different direction. To tap a common belief in their field—that Court decisions reflect law and jurisprudential principles—they asked appellate lawyers and legal scholars (“experts” in particular areas of the law) to predict the outcome of each of the term's decisions. The researchers then posted all the forecasts on the Project's Web site (http://wusct.wustl.edu), along with the actual votes and outcomes as the Court handed down its decisions. As it turned out, the statistical model produced far more accurate predictions of case outcomes than the experts (75 percent versus 59.1 percent), while the experts did marginally better at forecasting the votes of individual justices (67.9 percent versus 66.7 percent).

Judging by the quantity (and origin) of hits to the Web site during and after the Court's term, as well as the voluminous number of emails the research team received, this “friendly interdisciplinary competition” generated a great deal of attention in legal circles—both in Washington, DC, and in the faculty commons of the nation's law schools. Political scientists, though, remain largely unaware of the project. This is unfortunate since it raises intriguing theoretical and methodological questions—at least some of which transcend the field of law and courts. Flowing from the project too are numerous normative and policy implications of no small consequence.

In what follows, four distinguished panelists—Suzanna Sherry, Gregory Caldeira, Linda Greenhouse, and Susan Silbey—explore these matters. Drawing on the Web site, along with a description of the forecasting project prepared by its developers for this symposium, they offer a range of commentary—some supporting the research endeavor, some expressing concerns, and all searching and illuminating.

Many of the panelists address theoretical questions emanating from the project but perhaps none more pointedly than Suzanna Sherry. Despite the research team's occasional claims to the contrary, she—along with several other participants (myself included)—envisages the project as generally pitting explanations of judicial decisions that stress the importance of precedent and other legal principles against those that emphasize the political ideologies (or attitudes) of the justices. Of course, this is not a new competition: for at least five decades now, political scientists who specialize in law and courts have debated whether it is “law,” “politics,” or some combination of the two that best accounts for why judges, especially Supreme Court justices, decide cases the way they do. Law professors, too, have joined the debate, though many tend to emphasize legal principles in their arguments.¹

What the results of the project seem to suggest is that legal academics who stress principles and neglect politics in their explanations of case outcomes do so at their own peril. Complete explanations require consideration of the justices' political preferences—not simply their jurisprudence. Sherry argues, however, that first appearances can be deceiving—that law played and will continue to play a crucial role in adjudication. She offers a good deal of support for her claim, but particularly compelling is the relative success of both approaches in forecasting cases of high and low political salience. While the model performed particularly well in the headline-grabbing areas of civil rights and criminal procedure, the experts, as Sherry tells us “swept the field, outpredicting the model on every justice …” in disputes raising technical and often arcane questions of legal procedure. This is a crucial point because it suggests that political scientists concerned with explaining the range of judicial decisions can no more afford to neglect law than law professors can ignore politics. At the least, omission of either amounts to underspecification; at most, it serves to perpetuate myths about judging in both disciplines: that it is a phenomenon largely about politics or law. It is about both, and only by characterizing it as such—perhaps through deeper collaborations between law and political science—are we likely to develop truly accurate accounts of how justices operate.

The theoretical debate about which Sherry writes is of keen interest to specialists in judicial politics. Of wider and more general concern are the methodologies the project invokes to assess the observable implications of those theories. Beginning with the statistical model, it is not of the garden variety that populates the pages of our journals: rather than generate “forecasts” from information that emerges after the event has occurred, the investigators used nonlinear classification trees designed to identify patterns based on variables they could observe prior to the event (oral arguments in this case).²

Juxtaposed against the technical complexity involved in generating these predictions comes the simplicity of the tack taken to assess the legal model: the use of experts to render predictions.

Certainly in their general forms these approaches are not unknown in political science. For years now, scholars have relied on statistical models of one sort or another to forecast the outcomes of presidential and congressional elections. And the use of expert judgments is a methodology that analysts in comparative politics, among others, have long exploited to capture the preferences of political actors and organizations—with the goal, for example, of predicting the composition of governing coalitions. But rarely, if ever, have analysts invoked the two approaches simultaneously to study the same phenomenon. The forecasting project does so—and provides a unique opportunity to compare their relative merits.

And it is precisely this comparison that forms the centerpiece of Gregory Caldeira's contribution. That the model generally outperformed the experts hardly surprises him. Quite the opposite. In light of a long line of literature purporting to demonstrate that “human judges are not merely worse than optimal regression equations; they are worse than almost any regression equation …” Caldeira would have been astonished had the competition come out any other way. This is especially so, Caldeira argues, since the project relied on “human judges” with real expertise, and not “novices,” who might have paid greater heed to information conflicting with their jurisprudential and ideological commitments. Had the project tapped humans with less expertise, Caldeira hypothesizes, its developers would have observed fewer errors.

Maybe so. But at least one astute student of the Court, Linda Greenhouse of the New York Times, generated forecasts as accurate as the computer. In 27 of her reports on the Court last term, she offered predictions in 16 cases. She correctly forecast 12 (75 percent); she was wrong in two cases and had mixed success in the remaining two.

Why Greenhouse fared far better than the project's experts and ran a virtual dead heat with the model is an interesting question—with an answer, at least according to Greenhouse—that may surprise Court scholars. While both the statistical model and the experts generated their predictions prior to arguments before the Court, Greenhouse relied heavily on oral arguments to produce hers. Unlike many political scientists,³

she finds them a font of valuable information on the justices' thinking, as well as a harbinger of the outcomes and rationale to come. Her success at prediction should cause many of us to reconsider explanations of judicial decisions that fail to take notice of these presentations.

I have outlined but a few of the many theoretical and methodological issues that the commentators say the project raises. At the same time, because it is aimed at assessing well-established accounts of judicial decisions before the Court actually makes those decisions public, the project provides an all-too-rare occasion (at least in political science) to consider the implications of scholarly research. An obvious set centers on the confirmation process: if scholars can accurately forecast Court decisions—and it appears that they can—should political actors draw on this information to make choices over potential nominees? Would it be politically tenable for, say, a senator to make use of forecasts to oppose (or support) a particular candidate for the Supreme Court on the grounds that a case would almost certainly “come out differently” if that nominee is appointed?

Other implications are easy enough to summon (e.g., what political, social, and economic consequences would emerge if interest group leaders, corporate executives, and lawyers possessed advance information on Court decisions?). And Martin and his colleagues, along with several panelists do just that, though it is Susan Silbey who most squarely confronts normative concerns implicated in the research endeavor. To Silbey, the project may advance more democratic law-making and legal practice, just as the research team hopes, but looming large also is the possibility that it could promote “Weber's nightmare of imprisonment by our own increasingly masterful rationality.” Silbey's response is “neither entirely one nor the other,” but rather one of caution. Whether readers agree or disagree with her is less the point, I think, than that she raises the question in the first instance. For surely, as she recognizes, if the forecasting project provides any fodder for general contemplation, it is over the uses and consequences of “our” social science.