1. Introduction: Metaphysics and Methodology

Is systematic predictive success in social science possible? Many have given reasons why it is not, such as the fact that social systems are open systems, or that they exhibit reflexivity, or simply that there are too many variables needing to be modeled (Taylor Reference Taylor1971; Giddens Reference Giddens1976; Hacking Reference Hacking, Sperber, Premack and Premack1995; Lawson Reference Lawson1997). In this article I examine a notable case of predictive success so far relatively neglected by philosophers—namely, election prediction by means of opinion polling—that seems to contradict these reasons.

Next, if successful prediction is possible, what makes that so? The lesson from the opinion polling case is that the most fruitful answer to this question is not metaphysical but rather methodological. In particular, success or the lack of it was not predictable from the metaphysics of elections, which indeed in many respects remain unknown.Footnote ¹ Rather, what was crucial was a certain methodological approach.

One popular methodological view, borne in part from pessimism about the possibility of prediction, has argued that the main aim of social science should instead be explanation. The latter can be achieved via the discovery of causal mechanisms, as urged by ‘new mechanists’ (Lawson Reference Lawson1997; Brante Reference Brante2001), or else via the development of underlying theory (Elster Reference Elster1989; Little Reference Little1991). Moreover, much of mainstream practice in economics and other social sciences is implicitly committed to the latter view: while all accept that rational choice models, for instance, might not be predictively successful, nevertheless they are held to provide ‘understanding’ or ‘underlying explanation’.

A contrary view rejects this methodological emphasis on mechanisms or underlying theory (Cartwright Reference Cartwright2007; Reiss Reference Reiss2008). One strand, motivated in part by detailed case studies of other empirical successes, has emphasized instead context-specific and extratheoretical work. Theories and mechanisms play at most a heuristic role; empirical success requires going beyond them (Alexandrova Reference Alexandrova2008; Alexandrova and Northcott Reference Alexandrova, Northcott, Ross and Kincaid2009).

The details of the opinion polling case turn out to endorse this second view. The reason is that, roughly speaking, while the extratheoretical approach achieved prediction but not explanation, the theory-centered approach achieved neither. That is, a search for explanation not only yielded no predictive success but also yielded no explanations. Hence, the first view gives exactly the wrong advice.

I focus on the 2012 US presidential election, in which Barack Obama defeated Mitt Romney. I begin by describing the predictive success achieved by aggregators of opinion polls (sec. 2), before examining how this success was achieved (secs. 3 and 4). In contrast, theoretical approaches to election prediction fared much worse (sec. 5). I then discuss their failure also at furnishing explanations (secs. 6 and 7).

2. Predictive Success and Metaphysical Criteria

The 2012 presidential campaign featured literally thousands of opinion polls. The most successful of all election predictors were some aggregators of these poll results. Famously, several correctly forecast the winner of all 51 states in the 2012 election, as well as also getting Obama’s national vote share correct to within a few tenths of a percent.Footnote ² This was a stunning success, arguably with few equals in social science. Nor was it easy—no one else replicated it, although many tried.Footnote ³ On the morning of the election, the bookmakers had Romney’s odds at 9/2, that is, about 18%. Political futures markets such as InTrade had Romney’s chances at about 28%. These market prices imply that common opinion was surprised by the outcome.Footnote ⁴

Moreover, it is unreasonable to declare this success a mere fluke. First, the same poll aggregators have been successful in other elections too. And within any one election there have been many separate successful predictions, such as of individual Senate races or of margins of victory, which are at least partially independent of each other. Second, the aggregators’ methods are independently persuasive. It therefore behooves philosophers of social science to understand them.

Meanwhile, do elections satisfy the metaphysical criteria allegedly necessary for predictive success? It seems not. Presidential elections are clearly open systems in that they are not shut off from causal influences unmodeled by political science. They undoubtedly feature many variables. And they are clearly subject to reflexivity concerns: sometimes the mere publication of a poll itself influences an eventual election result; indeed, there were several examples of this during the 2012 campaign. Yet despite such troubles, highly successful prediction proved possible nevertheless.

3. The Science of Opinion Polling

In any opinion poll, the voting intentions of a sample serve as a proxy for those of a population.Footnote ⁵ How might things go wrong, such that the sample will not be representative? The most well-known way, a staple of newscasts and introductory statistics courses alike, is sampling error: small samples can lead to misleading flukes. But sampling error is not the only, or even the most important, source of inaccurate predictions.Footnote ⁶ Awareness of this crucial point lies at the heart of any serious election prediction.

To begin, a major issue for pollsters is to ensure that their samples are appropriately balanced with respect to various demographic factors. Suppose, for example, that two-thirds of interviewees were women. Since there was good reason to think that women were disproportionately likely to vote for Obama, it follows that such a woman-heavy sample would give misleadingly pro-Obama predictions. Polling companies would therefore rebalance such a sample, in effect putting greater weight on men’s responses. Notice several things about such a rebalancing procedure.

First, it is quite different from sampling error.Footnote ⁷ In particular, if our sampling procedure overselects women, then this problem will not be alleviated just by making the sample larger.

Second, sample rebalancing is clearly unavoidable if we wish to predict accurately. For this reason, every polling company performs some version of it.

Third, a poll’s headline figures are therefore heavily constructed. They are certainly not the raw survey results. Exactly what and how much rebalancing is required depends on assumptions about the actual turnout on Election Day. For instance, in recent American presidential elections typically there have been slightly more women than men voters, so it would be a mistake to rebalance the sample to exactly 50:50. The correct figure may not be obvious, it needing to be inferred from imperfect polling data about past elections, and moreover with some assessment of how patterns of turnout may change again in the upcoming election. Accordingly, different polling companies may reasonably choose slightly different rebalancing procedures. The result is the phenomenon of ‘house effects’, when any particular company’s results may systematically favor one or other candidate compared to the industry average. When assessing the significance of a poll for election prediction, it is vital to be aware of this.

Fourth, the rebalancing issue is pressing because it applies to many other factors besides gender, such as age, income, race, likeliness to vote, education, ownership of cell phones but not landlines, and home access to Internet. Not only is the precise rebalancing procedure for each of these factors arguable, but it is also arguable exactly which factors should be rebalanced for in the first place (see below).

In addition to random sampling error and systematic sampling bias, there are several other potential sources of error as well. There is space only to mention a couple here. One is the phenomenon of herding: at the end of a campaign, pollsters—it is widely suspected—‘herd’, that is, report headline figures closer to the industry mean, presumably to avoid the risk of standing out as having missed the final result by an unusually large margin. Some sensitivity to this turns out to be optimal for accurate election prediction.Footnote ⁸ A second worry is simply that voters may change their minds between a poll and Election Day. This is the main reason why polls taken, say, 6 months before an election have a much poorer predictive record than those taken later. Election predictions must therefore take into account a poll’s date too.

4. Poll Aggregation

Turn now to the aggregation of polls, which represents a second layer of method, quite distinct from that required to conduct a single poll. Historically, poll aggregation has had a better predictive record than using individual polls alone. One obvious reason is that aggregation increases effective sample size and therefore reduces sampling error. A typical individual poll may have 95% confidence intervals of 3% or 4%; those for an aggregation of 8–10 polls, by contrast, are typically 0.75% or 1%.Footnote ⁹ But it is a different story for the other possible sources of error. Mere aggregation is no cure for those, because they may bias all polls—and hence the aggregate of polls—in a similar way.Footnote ¹⁰

What, then, does account for aggregation’s superior predictive success? In addition to the reduction of sampling error, sophisticated aggregation can mitigate the other sources of error too. This explains why the best aggregators beat simple averaging of the polls. It is instructive to consider a couple of examples of methodological issues in more detail.

5. Failure of the Theory-Centered Approach

The details of poll aggregation show clearly the case-specific nature of its methods. The alternative is to focus instead on ‘fundamentals’, that is, on variables that might shed light on election results generally, not just case-specifically, such as economic conditions, the perceived extremism of candidates, incumbency, and so forth. There is a literature in political science on election prediction that aims to furnish just such generalizable models.Footnote ¹⁵ How does it fare?

Conveniently, it too has focused on US presidential elections. The sample size is relatively small, as fewer than 20 elections have good enough data. This creates a danger of overfitting. In response, models typically feature only a small number of variables, most commonly economic ones such as growth in gross domestic product (GDP), jobs, or real incomes.Footnote ¹⁶ Sensibly, they are estimated on the basis of one part of the sample and then tested by tracking their predictive performance with respect to the rest of the sample.Footnote ¹⁷ Even then, there remains a risk of overfitting—if a model predicted the first few out-of-sample elections quite well, will its success continue in future elections? Moreover, even if a model does successfully predict past elections, there is no guarantee that the political environment is so stable that the model will remain correct in the future too.

With these caveats noted, it is true that the models do have a little success. On one estimate, the best ones’ average error when predicting the incumbent party’s share of the vote is between 2% and 3%.Footnote ¹⁸ But this is not quite as impressive as it might sound. First, for our purposes it is something of a cheat, in that one of the variables in by far the highest weighted model—Abramowitz (Reference Abramowitz2008)—is a polling result, namely, presidential approval rating. Hence, the success is not achieved purely by fundamentals. Second, a 2%–3% average error corresponds to an average error when estimating the gap between the leading two candidates of about 5%. Third, vote shares rarely deviate all that much from 50% anyway, so they are quite an easy target—indeed, another estimate is that economic variables account for only about 30%–40% of the variance in incumbent party vote share.Footnote ¹⁹ Overall, the models do not predict individual election results very reliably. On many occasions, they even get wrong the crude fact of which candidate won. For accurate prediction, it is necessary to incorporate the results of opinion polls.

6. Explanation

Why did Obama win? Answering this requires identification of his victory’s causes.Footnote ²⁰ That, in turn, requires a verified theory or causal model. The problem is that nobody—from either approach—has managed to produce one.

On the polling side, in a trivial sense Obama’s victory is ‘explained’ by the fact that, as revealed by aggregators, on the eve of the election a majority of the electorate were minded to vote for him. But, of course, for most investigative purposes a deeper explanation is required, in particular one that might apply to other elections too. Poll aggregation provides none.Footnote ²¹

On the fundamentals side, if their models had fared better, they would have provided the very explanations that polling aggregation does not. After all, that is precisely the motivation for theory-centered methodology. Thus, we might have been able to explain that Obama won because of, say, positive GDP and jobs statistics in the preceding two quarters. Unfortunately, though, the fundamentals models are not predictively accurate.

Can they nevertheless provide us with explanations anyway? The argument would be that they have truly identified relevant causes. It might be postulated, for instance, that GDP or stock market growth does causally impact voter preferences and thus election outcomes. True, other causes impact too, and so the models do not explain the outcomes fully or predict them accurately, but that still leaves room for the claim that they explain them ‘partially’ by correctly identifying some of the causes present.Footnote ²²

But, alas, even this claim is dubious. First, the different models cite different variables. Abramowitz’s, for instance, cites GDP growth, presidential approval rating, and a complex treatment of incumbency; Hibbs’s, however, cites growth in real disposable income and the number of military fatalities abroad. Even among economic variables alone, some models cite GDP, some household incomes, some jobs data, some stock market performance, and so on. There are many different ways to achieve roughly the same limited predictive success, which shakes our faith that any one way has isolated the true causal drivers of election results. Perhaps the small sample size relative to the number of plausible variables makes this problem insoluble.

A second reason for pessimism is that, elsewhere in science, a standard response to predictive failure is to test putative causes in isolation. As it were, at least we achieve predictive success in the isolated test. Unfortunately, such experiments are impossible in the case of election predictions. So in addition to predictive failure at the level of elections as a whole, the causal factors picked out by the models have not earned their empirical keep by other means either.

The upshot is that we have no warrant for asserting that we have found even some of the causes of election outcomes, and therefore no warrant for claiming even partial explanations. Thus, the basic conclusion stands: we have not achieved any explanation of election outcomes, and so the original motivation for turning to fundamentals models is frustrated.

7. Transportability

Are the predictive successes of one election transferrable to another? That is, will a similar polling aggregation strategy work elsewhere? For US presidential elections, it seems that the answer is “yes”—witness the success of many of the same polling aggregators in 2008. However, it is a different story for other elections, such as US congressional elections or elections in other countries.Footnote ²³

The reason is precisely the case-specific nature of polling aggregation—for the best aggregation does not rely only on polls. It must also factor in features such as whether an election is a single national vote or split into many smaller constituencies, whether there are two or many major political parties, whether the voting system is first-past-the-post or proportional representation, one-shot or multi-round, and so on. The implications of a poll for election prediction depend on just such factors. It has also proved predictively profitable to moderate polling results by considering what result should be ‘expected’, given various local demographic and historical factors. The details of just how to do this are important—and inevitably highly case specific.

Perhaps even more significantly, the earlier nuances, namely, adjudicating state versus national polls and whether polls might be systematically skewed, could also only be resolved by case-specific knowledge. There are many similar examples, such as the extent of regression to the mean to be expected after ‘bumps’ from party conventions or presidential debates, or if one candidate is ‘surprisingly’ far ahead at an early stage of the campaign. Such knowledge is crucial, but typically it is transferrable to new elections only imperfectly if at all.

So a serious polling aggregator must build a new election prediction model each time. This lack of transportability is really just the flip side of two facts familiar from above: first, that no one has achieved satisfactory causal explanations, not even the poll aggregators; and second, that predictive success requires case-specific knowledge rather than a search for generalizable causal mechanisms or theoretical underpinnings.

8. Conclusion

How can we make progress, that is, predict election results even better? It is clear that improving the models of fundamentals is an unpromising route. Rather, progress will be made in the same way as it has been made in the past few years—by doing polling aggregation better. This might involve getting better polling data, analyzing that data better, or understanding better how the implications of that data depend on local peculiarities—in other words, by developing the case-specific, extratheoretical components of prediction for each application anew. Although transportable explanations are elusive, predictive success need not be. What is clear, though, is that context-free general theory offers neither: it seems that there are no shortcuts in social science.