DESIGN OVERVIEW

This study considers two outcome variables. The first is whether members of Congress retained their seat through 2012. Members who won reelection are considered successful, whereas those who were not reelected (whether by losing the primary or the general election or by retiring) failed. For this outcome, the population includes every incumbent who voted on the debt ceiling. If a member’s vote deterred challengers (making reelection easier) or drew sufficient opposition to result in strategic retirement, this outcome captures the effect. ^{Footnote 3}

The dichotomous indicator of retaining office offers an important global outcome, but a policy position’s effect is likely to be more subtle. Therefore, the second outcome variable examined is the incumbent’s share of the two-party vote. Thus, the treatment effect asks not whether a policy stance like this can help an incumbent win in general but rather how such a stance might shape the marginal vote share when the incumbent faces a general-election challenger. Together, these two outcomes enable an assessment of the consequences of this controversial vote for the dynamics of a general election and whether incumbents retain office.

The treatment variable is whether House members voted against raising the debt ceiling on August 1, 2011; the control observations are members who voted in favor. I assume that the treatment was administered at the time of this vote, at which point incumbents’ recorded positions on the issue became public. As an issue stance, the electorate receives this treatment and then decides in what proportion to vote for the incumbents and whether to reelect them.

In modeling the treatment effect, it is essential to account for other factors known to explain incumbents’ vote shares, including their ideology, the district ideology, the incumbent’s prior vote share, and campaign funding (Abramowitz Reference Abramowitz2004; Abramowitz and Segal Reference Abramowitz and Segal1992). For the incumbents’ ideology, I matched members by their first-dimension DW-NOMINATE score for 2011 (Poole and Rosenthal Reference Poole and Rosenthal1997). I measured district ideology using President Obama’s share of the two-party presidential vote in 2008 (Ansolabehere, Snyder, and Stewart Reference Ansolabehere, Snyder and Stewart2001; Brady et al. Reference Brady, Cogan, Gaines and Rivers1996; Erikson and Wright Reference Erikson and Wright1980). An incumbent’s prior vote share is simply the share of the two-party congressional vote in 2010. ^{Footnote 4} Campaign money was recorded from the cash on hand that was listed in incumbents’ July 2011 Federal Election Commission (FEC) report. This was the most recent FEC report before the debt-ceiling vote and therefore the best measure of pretreatment financial standing. Subsequent measures may reflect easier fundraising in response to the debt-ceiling vote, thereby inducing a post-treatment bias.

To isolate the treatment effect of a House member’s debt-ceiling vote on each outcome, I matched treated and control units using Coarsened Exact Matching (CEM) (Iacus, King, and Porro Reference Iacus2012). This method divides continuous covariates into a discrete number of categories and applies exact matching to the coarsened data. CEM is a monotonic imbalance-bounding matching method; therefore, for a given level of coarsening, the degree of covariate imbalance between treated and control observations cannot exceed a certain level (Iacus, King, and Porro Reference Iacus, King and Porro2011).

Using this strategy, I generated two matched samples. The first sample was drawn from the population of 423 incumbents who voted on the debt ceiling. The matched sample includes 175 members with which to evaluate the probability of incumbents retaining their seat. The second matched sample was drawn from the population of 329 members who faced a challenger from the other major party in the 2012 general election. This yielded a matched sample of 129 members with which to assess the debt-ceiling vote’s effect on an incumbent’s share of the two-party vote for contested elections. ^{Footnote 5} Both matched samples were listed in the registered research design. I assessed each causal effect by calculating the local sample average treatment effect on the treated (Iacus, King, and Porro Reference Iacus2012, 3).

Figure 1

Predicted Probabilities of Various Intermediate Outcomes Based on Whether the Incumbent Voted Yes or No on Raising the Debt Ceiling

For the final attribute of the preelection design, I estimated a multinomial probit model of various intermediate outcomes that may befall an incumbent (Imai and van Dyk Reference Imai and van Dyk2005). The baseline outcome was that the member would compete in a contested general election; the four other possibilities included standing in an uncontested general election, losing the primary, retiring, and seeking higher office. The results included all covariates used in the matching analysis, presented in table format in the registered research design. Using their observed covariate values, Figure 1 shows the average predicted probabilities if incumbents all shifted to voting against the debt ceiling and then all voting in favor (Hanmer and Kalkan Reference Hanmer and Kalkan2013). Whereas the distribution of probabilities does not differ markedly, those who voted against the debt ceiling were discernibly more likely to coast to an uncontested general election. Although this describes the incumbents’ eventual fate, the real test came after observing vote shares.

THE ELECTORAL BUMP OF A NO VOTE

Having already established the research design before the election, I now assess the effects with official returns. The first question is whether the debt-ceiling vote affected members’ ability to retain their seat. Figure 2 shows the percentage of House incumbents retaining their seat, contingent on whether they voted Yes or No on Bill S. 365 to raise the debt ceiling. The figure lists percentages for the matched sample, which is used for the actual hypothesis test, as well as percentages for all incumbents for comparison. The results do not support the hypothesis that voting against the debt ceiling would help an incumbent to retain office in 2012. As figure 2 illustrates, retention percentages are actually slightly lower for No voters than for Yes voters. This non-effect is not entirely surprising because the research design explained that any effect would likely be more subtle. ^{Footnote 6}

Figure 2

Percentage of Incumbents Retaining Their Seat by Whether They Voted Yes or No on Raising the Debt Ceiling

Notes: The two bars on the left show the percentages for all incumbents. The two bars on the right show the percentages for the matched sample.

To capture a finer effect, I also analyzed incumbents’ percentage of the two-party vote in contested general elections. Figure 3 illustrates the findings for this outcome variable. The vertical axis shows each incumbent’s percentage of the two-party vote in the 2012 general election. Crosses represent treatment units (i.e., No voters) and dots represent control units (i.e., Yes voters). The horizontal axis presents each member’s first-dimension NOMINATE score to illustrate ideology: the most liberal members are farthest to the left and the most conservative members are farthest to the right. All positive scores belong to Republicans and all negative scores belong to Democrats. The top panel shows all incumbents who competed in a contested 2012 general election. The bottom panel shows the preelection matched sample that is used to test the hypothesis.

Figure 3

Proportion of the Two-Party Vote Earned by the Incumbent Member in the 2012 Election against the Member’s NOMINATE Score from 2011

Notes: A black cross indicates that the member voted against raising the debt ceiling. A red dot indicates that the member voted in favor of raising the ceiling. The top panel shows data for all incumbent members who competed for reelection. The bottom panel shows the matched sample.

(a) Full Data

(b) Matched Data

As illustrated in the bottom panel of figure 3, the share of the two-party vote tends to be higher among those who voted against raising the debt ceiling than for those who voted for the bill. In the matched sample, the average No and Yes voters earned 64.8% and 62.4% of the two-party vote, respectively. Hence, the local average treatment effect on the treated is estimated as a 2.4-percentage-point bump. ^{Footnote 7} Members who voted against raising the debt ceiling were able to pad their margin of victory with a positive treatment effect on voters.

IMPLICATIONS

This study makes both methodological and substantive contributions. Methodologically, it offers an applied example of how preregistered studies can be implemented in political science, complementing the theoretical notions presented in my article titled “Research Preregistration in Political Science: The Case, Counterarguments, and a Response to Critiques.” For an example of how this approach can be useful to prevent fishing, consider the result that voting against the debt ceiling raised an incumbent’s share of the two-party vote. If a researcher wanted to report a larger finding than the estimated 2.4-percentage-point bump, the results could be manipulated to achieve this. For example, an alternative and theoretically reasonable method would have been to use propensity-score matching (Rosenbaum and Rubin Reference Rosenbaum and Rubin1983). Using propensity scores, I created a matched sample of 220 members (i.e., 110 treated and 110 control). In this alternate matched sample, the average No voter earned 67.4% of the two-party vote and the average Yes voter earned 64.4%. Hence, the estimated treatment effect for this alternate model is a 3-percentage-point bump. ^{Footnote 8} Another design example would entail a difference-of-means test over the full population with no control variables. However, this naive approach produces an estimated treatment effect of a 4.5-percentage-point bump by voting against the debt ceiling. ^{Footnote 9} Hence, alternate models could have generated a larger estimate of the treatment effect that passes higher confidence thresholds. If a researcher chose the largest result from all possible methods, the true treatment effect might be over-represented. By assessing the balance of the matched sample in advance and concluding that CEM produced a satisfactory matched sample before observing the outcome, I committed to a test that should have isolated the treatment effect without an opportunity for fishing.

Substantively, this study shows that incumbent House members could improve their standing with the electorate by siding with a majority of the public to vote against the highly publicized debt-ceiling deal. Although the 2011 debt-ceiling vote did not affect a House member’s overall win–loss rate, it could raise an incumbent’s share of the two-party vote by a 2.4-percentage-point bump. This finding is concerning: on matters of essential importance to the nation’s economy and long-term fiscal standing, it appears that members of Congress can gain electorally by voting with the popular will and against expert consensus on the national interest. Given the difficulty of negotiating the deal and the fact that 161 members voted against the bill, many incumbents must have acted on these electoral incentives. As the federal debt again reaches the legal ceiling in the future, we must be concerned about whether incumbents will continue to make a major issue of raising the ceiling and cast their vote to benefit their electoral interests.

In the 112th Congress, the Republican majority in the House was frequently rolled by the Democratic minority, which means that bills passed with majority support from the minority party and with minority support from the majority party. Bill S. 365 was not an example of this case, but it did require substantial Democratic support to pass. It is possible that, in private, many members preferred to enact the economically sensible policy but wanted to make a public stand of opposition that would be electorally popular. With moderates from both parties forming a winning coalition, other members could vote to their own electoral benefit. However, this still implies that members faced cross-pressures due to issue-based constituent pressure. For difficult issues that require considerable time and attention to develop a meaningful opinion, such as the financial details that this bill considered, public opinion may not be sufficiently informed to guide legislators. If constituents instead held their representatives to a standard of dedicating substantial time to learning the problem’s details and using their best judgment, then ideological pressures that lead to economic problems could be weakened. It may be that delegatory rather than trustee-based representation on this issue poses a potential problem for the nation’s long-term financial health.

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit http://dx.doi.org/S1049096515000177.