ASSESSING THE 2010 STATE LEGISLATIVE ELECTION FORECASTING MODELS

This brief note reports the accuracy of my two forecasts for the 2010 state legislative elections, one made on July 22 and reported in the October 2010 issue of this journal (the “PS model”; Klarner Reference Klarner2010a), and the other made on September 18 and reported in the October issue of Forum: A Journal of Applied Research in Contemporary Politics (i.e., the “Forum model”; Klarner Reference Klarner2010b).Footnote ¹ Both models used presidential approval, the state of the economy, and midterm loss as national-level predictor variables, while the later forecast also used Gallup's generic ballot question asking respondents which party they would vote for in the upcoming U.S. House election. The PS model predicted the Republicans would pick up 11 chambers, while the Forum model forecast a 15-chamber pickup. In actuality, the Republicans picked up 21 chambers, in contrast to the average 3.2-net chamber shift in party control toward one party or the other from 1962 to 2008.Footnote ² While both forecasts understated the extent of the Republican wave, the July forecast especially did. Overall, the Forum forecast did a good job of predicting the Republican wave, calling about three-fourths of its magnitude.

The forecast made in PS called party control of 82% of chambers correctly, while the prediction made in the Forum called 89% of the chambers correctly.Footnote ³ These results may sound like a high degree of accuracy, but merely predicting that chambers would not switch party control (a “naïve model”) would yield a 76% degree of accuracy. The proportionate reduction in error over this naïve model was therefore .24 for the PS model and .52 for the Forum model, a considerable amount in the latter case.

Alan Abramowitz also used the generic ballot question to make predictions about the number of state legislative chambers that would be gained by the Republicans, using a simple time-series model (Abramowitz Reference Abramowitz2010). Using table 2 of his article to plug in the Republicans' generic ballot advantage in early September (+5) yields a forecast of 48.5 Republican chambers, or a 12.5-chamber pickup. This forecast therefore understated the extent of the Republican shift more than did my forecast in the Forum, which was based on the same generic ballot percentage.

Aside from Abramowitz's prediction, my two forecasts were the only quantitative models used to predict the 2010 state legislative elections. However, using journalistic reporting methods, Louis Jacobson of Governing magazine classified chambers into one of seven categories representing party control (e.g., “safe Republican,” “toss-up,” etc.) on September 29th (Jacobson Reference Jacobson2010). Eleven of the 12 chambers he classified as “toss-ups” were won by the Republicans, with the remaining chamber resulting in a split. Furthermore, six chambers classified as Democratic wins were ultimately not. If we make the charitable assumption that half of the toss-up chambers that were won by the Republicans were accurately called (and, if the New York state senate is won by the Democrats, that that toss-up was accurately called), we get an accuracy rate of 87%, which is virtually identical to the accuracy rate of the Forum forecast that was made at the same time. This similarity indicates that quantitative models can compete very well with other approaches to calling state legislative elections.

My later forecast in the Forum was more accurate than my prediction in PS. One likely reason is that the generic ballot question asking respondents how they would vote in the upcoming U.S. House election was used in the Forum model but not the PS model. Furthermore, the national-level variables used in the second forecast were more pro-Republican at that later date in 2010 and therefore more accurately measured the extent of the Republican wave. Aside from that variation, a crucial difference between the two forecasts is that the PS forecast adopted the state legislative district as the unit of analysis, while the Forum forecast relied on the legislative chamber. Although there were other differences between the two models, the variables that would influence the relative fortunes of the Democrats across states were overshadowed by the unit of analysis discrepancy. When the percentage of legislative seats in a chamber held by the Democrats is regressed on the forecast percentage of seats from both the PS and the Forum models, the percentage from the PS model attains statistical significance (p < .001), while the percentage from the Forum model narrowly misses significance (p < .065, one-tailed test).Footnote ⁴ This finding provides some evidence that district-level models are more accurate than chamber-level models, although a research design that holds all other factors constant is necessary to answer this question. Furthermore, I found no evidence that the superior ability of the district-level model to order chambers from least to most Democratic is concentrated among legislative chambers in which only half the seats are up for election.

My two forecasting models were some of the first attempts at forecasting state legislative elections. Although it is hard to assess how well these models did when few models exist against which to compare them, they did a reasonably good job of calling both the Republican wave and the outcome of individual chambers.

The October 2010 issue of PS published a symposium of midterm election forecasts made in the summer and fall leading up to the election. In the following articles, the forecasters assess the accuracy of their models.