Forecast Accuracy

By the metrics of accuracy I offered in the October symposium (Campbell 2004b), both the Labor Day trial-heat and economy and the convention bump forecasts measured up well this year. Of the more than 121 million Americans who voted for a major party presidential candidate, 51.2% voted for President Bush.¹

The 2.5% error of the Labor Day model and the 1.5% error of the convention bump model are much smaller than the mean errors of a naive guess (4.8%) or of polls conducted at the time of these forecasts (3.9%). The forecast error of the main model was just slightly greater than the average error in the final Gallup Poll conducted within hours of Election Day (about 2%) and the convention bump forecast was more accurate than this benchmark. While the Labor Day trial-heat and economy forecast error was less than one point greater than this year's final vote projections of Gallup (1.2%) and Zogby (1.1%), the accuracy of the convention bump forecast was comparable to or better than either. Both the Labor Day and convention bump forecasts were also much more accurate than the exit polls circulating on the Internet the afternoon of the election. Those exit polls indicated a 2.6% spread in favor of Senator John Kerry.

Both forecast models also measured up to their performances in previous elections, either actual forecasts or those simulated through out-of-sample testing. Although the Labor Day error this year was larger than the model's mean out-of-sample error (1.6%), the forecast was about as accurate as it was in 2000 and more accurate than four of the 14 out-of-sample tests. The convention bump forecast was more accurate than its mean out-of-sample error (2.1%) and half of its out-of-sample tests. While the forecasts were reasonably accurate this year, there remains error to explain. How much of the error was a consequence of model weakness and how much might be attributed to unanticipated effects of the campaign?

The Model and the Data

Although there are no readily apparent model weaknesses exposed by the 2004 election, this year's tests of the models strongly indicate that these forecasts should reconsider use of the likely voter division of the Gallup poll rather than the registered voter division. The likely voter division was used because Gallup put this forward as the best measure of preferences among voters and, in light of increasingly easier voter registration, a substantial difference between the preferences of the registered and those who actually turned out to vote seemed increasingly likely. In hind-sight, as laudable as the effort is to identify those who are likely to vote, the current method of identifying likely voters appears flawed. The likely voter screen, as currently applied by Gallup, not only increases volatility in the preference distribution (Erikson, Panagopoulos, and Wlezien 2004) but diminishes accuracy.

Table 1 compares the use of the likely voter and the registered voter poll divisions in the convention bump and the trial-heat and economy forecasts from Labor Day through to five days prior to the election. In four of the five forecast timings, the use of registered voters produced a more accurate forecast. If I had used registered voters in the two models, both forecasts would have been within 1.1 points of the actual vote.²

In short, using the likely voter division of the Gallup poll produced perhaps as much as half or more of the forecast error this year.

The Unanticipated in the Campaign

In addition to the poll problem, a portion of the difference between the forecast and the vote may have been caused by unanticipated aspects of the campaign. Two factors in particular may have worked to Kerry's advantage: a strong performance in the presidential debates (particularly the first) and the economic issue. Between the first and third debates in early October, Bush's poll lead over Kerry dropped between 2 and 4.5 points in most polls. Some of this loss was regained quickly, some was not. On the economy, while economic growth overall was about average for an election year and this is generally good enough to help the in-party, polls consistently indicated that voters favored Kerry to Bush on the issue. The Gallup polls in the last month of the campaign had Kerry with a 7-point lead over Bush in handling the economy. The focus on the war on terror issue by the Bush campaign may have allowed Kerry to frame the economy to his advantage. Bush countered much of this advantage, however, by an extraordinary get-out-the-vote effort: the RNC's 72-hour plan. This, combined with referenda in 11 states banning same-sex marriage, mobilized those who found Senator Kerry to be too liberal (about half the nation according to Gallup, compared to 40% who found Bush too conservative) and erased most of the gains that Kerry had made through the fall campaign (Campbell 2004c; 2005).