Roll Calls and the 2004 Presidential Election (So Far)

Using roll call votes to track lawmakers' behavior is prevalent both in the political arena and in academia. Politically motivated groups such as the Americans for Democratic Action, the American Conservative Union, the Sierra Club, the National Federation of Independent Business, the U.S. Chamber of Commerce, the National Rifle Association, the National Right to Life organization, NARAL, and the AFL-CIO all routinely score legislators according to whether legislators vote in accordance with their respective policy agendas. These resulting scores are used not only to mobilize members and target incumbents in elections (e.g., the Americans for Democratic Action “Zeros and Heroes” list), but the groups also seek to influence lawmakers' votes by publicizing which votes are going to be scored ahead of time. As CQ Weekly reports: “interest groups—and members of Congress themselves—use legislative scorecards for much more than reflecting an assessment of each lawmaker's record or mobilizing supporters at the grassroots level at election time. Scorecards, interest groups and lawmakers agree, have become an effective tool of the lobbying trade” (Cochran 2003, 924).

Non-partisan organizations and publications such as Project Vote Smart, the National Journal, and CQ Weekly also routinely and independently score legislators. Unlike the scores produced by partisan organizations (which are intended to serve political purposes), the scores of nonpartisan publications aim to present an objective assessment of lawmakers' voting behavior. While the voting scores given by partisan groups reveal the extent to which a lawmaker agrees with the groups' policy agenda, the scores of non-partisan groups rank a legislator within the political spectrum between two extremes: liberal or conservative. In so doing they provide an important public service. For instance, an interested citizen might well rely on the nonpartisan scores as reasonably unbiased summaries of the ideological positions of their congressional representatives, perhaps with a view to assessing for whom they should vote in the next election.

The scores of ostensibly neutral arbiters like National Journal shape political discourse in less passive ways. Precisely because the source is neutral, noteworthy features of the National Journal scores attract the attention of the press and the campaigns. In particular, candidates seeking to define one another as extreme and “out-of-step” with voters may use the scores to validate their claims. It is well-known that a “problem for members of Congress seeking the presidency is their extensive voting records, which can—and usually are—deciphered, deconstructed and even distorted by opponents looking to given those positions a negative spin” (Martinez 2004, 458). Evidence from impartial sources often provides the basis for such claims.

The Democratic presidential nominee, Senator John Kerry (MA), is susceptible on this score, if only because his voting history is so long: 6,310 recorded votes cast over 20 years of service in the Senate. Since the candidacy of Lyndon B. Johnson, only two other presidential candidates have had lengthier roll call records (and therefore recorded positions on national issues): Bob Dole and Gerald Ford. As CQ Weekly reports: “Republicans are combing through the 6,310 votes Kerry has cast in the Senate to date, looking for this or that position as an indication that the likely Democratic challenger to President Bush is out of step with mainstream America” (Martinez 2004, 458).

Precisely this opportunity presented itself when the non-partisan National Journal issued its ratings of lawmakers for 2003 in January of 2004. The ratings revealed that Kerry was the most “liberal” senator in 2003. In interpreting this result, the National Journal wrote, “To be sure, Kerry's ranking as the No. 1 Senate liberal in 2003—and his earning of similar honors three times during his first term, from 1985 to 1990—will probably have opposition researchers licking their chops” (Cohen 2004, 618).

The national impact was immediate. On February 28, 2004—the day after the scores were released—Kerry's ranking was reported in the Boston Herald (“Liberal Label Sticks; GOP Jumps on Kerry Over Senate Ranking”), the New York Post (“Survey: Kerry A Flaming Liberal”) and the Washington Times (“List Says Kerry Top Senate Liberal”). Kerry's ranking also surfaced during the Democratic presidential debate sponsored by CBS and the New York Times held on February 29. Elizabeth Bumiller, a reporter for the New York Times, questioned Kerry about his rating (FDCH 2004):

In addition to Kerry's dismissal of the National Journal scores, Senator John Edwards responded: “I don't think anybody in America cares about what some inside-Washington publication says about your ideology.” Perhaps. But there is no doubt that the charges as to the “extremism” of both Kerry and Bush will play a critical role in the campaign for president this year.

Since roll call scores permit both the press and partisan groups to characterize (or validate characterizations of) opposition candidates, an understanding of roll call scores is essential in determining the extent to which they should influence political discourse. For example, how much weight should be given to the ratings of the National Journal (using a methodology developed by Bill Schneider—a political scientist and senior political analyst for CNN and frequent commentator on contemporary politics) in light of Kerry's criticisms; is Kerry's reaction mere political spin or does it speak to a more serious criticism of the scoring procedure?

Also, how does President Bush rate relative to Kerry? Even if Kerry is among the most liberal senators, the political relevance of this characterization depends upon the extremism of his opponent. For instance, the choice between a liberal and a moderate is much different than the choice between a liberal and a conservative.

Identifying the Most Liberal Senator(s)

The National Journal scores are based on 62 key roll calls from the Senate, and 73 from the House. In reporting its methodology, the National Journal notes:

In addition to publishing the results in a February 28 cover story entitled “How They Measured Up,” the scores are posted on the National Journal web site. The most newsworthy result in the rankings is that John Kerry is reported to be more liberal than 97% of the Senate, and more conservative than 4% of the Senate. No other Senator is ranked “more liberal” in the National Journal rankings. John Edwards (NC) is ranked “second most liberal” by this measure; Edwards is estimated by National Journal to be more liberal than 95% of the Senate, and more conservative than 6% of the Senate.

One problem with the rankings is that there are no confidence intervals for the reported scores. It is well-known that the results from public opinion polls are inexact measures of true public opinion and that the error associated with the results must be acknowledged. The same is also true for roll call scores. In fact, the uncertainty that accompanies a voting score (or ought to) is most easily explained using analogies from the realm of public opinion polling. Uncertainty results from using a small number of roll call votes, just as a small sample size induces uncertainty in the results of an opinion poll. There is also the question of validity: the legislator's decision on any analyzed vote may well have been influenced by party pressure, presidential pressure, and/or lobbying by interested groups, and is not a perfect reflection of the legislator's ideology (analogously, survey questions are imperfect, and do not tap respondents' attitudes the way we think they might). Finally, the lawmaker may have missed some votes (i.e., surveys are subject to bias from non-response).

These sources of error have important political consequences for the proper interpretation of Kerry's voting record (indeed, any voting record) and the 2003 National Journal scores. As Kerry noted in the CBS/New York Times debate, as a result of his presidential campaign, he (and the other Democratic presidential candidates) missed a sizable fraction of the votes that the National Journal analyzed. For the 62 key roll calls in the Senate in 2003, the average abstention rate is 2.9%—with 56 senators voting on every analyzed vote and another 23 senators voting on all but one. Interpreting the announced positions of the Bush administration on the 62 votes (as reported in CQ Weekly) as evidence of how President Bush would have voted were he in the Senate allows us to compare the percentage of recorded votes by Bush and the Democratic presidential candidates. Table 1 summarizes the findings.

As is immediately evident from Table 1, the Bush administration and the presidential candidates took positions on only some of the analyzed votes. In fact, the two presidential candidates—Bush and Kerry—announced positions or voted only 40% of the time. Kerry was right to suggest in the CBS/New York Times primary debate that his absenteeism is consequential—there is far less data with which to estimate his voting score (and those of his fellow presidential candidates) than for the typical senator. Using fewer votes (i.e., smaller sample size) to score Kerry means that Kerry's score will have a larger confidence interval than those of other candidates. Any conclusion about Kerry's voting score and his relative rank ordering in the Senate should be sensitive to this important feature of the roll call data.

To demonstrate both the validity of Kerry's reaction and the importance of reporting confidence intervals for vote scores we first replicate the analysis of the National Journal using the method outlined in Clinton, Jackman, and Rivers (2004). The intuition underlying statistical models of legislative voting such as that employed by the National Journal and Clinton, Jackman, and Rivers is that each roll call presents each legislator with a choice between a “yea” and a “nay” position. Legislators are presumed to vote for the position most similar to their own ideal policy position/outcome. As a matter of practice, a legislator's voting record probably reflects a number of different influences, including: personal ideology, the ideology of the legislator's constituency, lobbying by interest groups, and pressure from party leaders. Without considerably more data than is available here, the effects of each these plausible sources of influence can not be ascertained.¹

We present our results in a series of graphs. Figure 1 shows our estimated voting scores for the 100 senators (solid points) along with a 95% confidence interval for each voting score (horizontal bars). Negative scores are associated with more liberal preferred positions and positive scores represent more conservative preferred positions. Solid points indicate Democratic senators, generally to the left of Republican Senators (open points)—the notable exception being Zell Miller (GA).

Point Estimates and 95% Confidence Intervals using 62 National Journal Key Votes

We see immediately that some senators' voting scores are estimated with considerably wider confidence intervals than other senators. This is a product of two factors: (1) the relative lack of data resulting from absenteeism—all else equal, the less data we have the wider the confidence intervals—and (2) extremism- all else equal, as ideological extremism increases the confidence interval widens because the voting record becomes less informative (e.g., while the data suggest George Allen (VA) is relatively conservative, since we seldom observe Senator Allen voting for the more liberal position on any given roll call we do not know precisely how conservative he is).²

Figure 2 shows the ranks of the estimated voting scores (and associated 95% confidence intervals) for the Senate. Senators have been arranged according to their estimated ideal points. By definition, the ranks and their confidence intervals must lie within 1 and 100 inclusive. Again, solid (open) points indicate the most likely rank for the indicated Democratic (Republican) senator, while the horizontal bars indicate 95% confidence intervals. Given that the voting scores graphed in Figure 1 are estimated with uncertainty, so too will be any rank-ordering based on these scores. Accordingly, we estimate the probability that a given senator is the most liberal senator (has a rank of 1): only three Senators have any appreciable chance of being the most liberal Senator, namely Kerry (0.38 probability), Senator Paul Sarbanes of Maryland (0.28), and Senator Jack Reed of Rhode Island (0.24).

Rank Ordering and 95% Confidence Intervals using 62 National Journal Key Votes

We can also compare any two senators to assess the probability that they are statistically distinguishable. For instance, the probability that Senator Edwards has a voting score more conservative (greater) than Senator Kerry's is .90, short of the traditional 95% standard used in social-scientific research but moderately strong evidence nonetheless. The probability that Senator Joseph Lieberman (CT) has a more conservative voting record than Senator Kerry is an overwhelming .998 (which is the same probability that Lieberman's voting record is more conservative than Edwards').

Note also that Kerry's voting score is indistinguishable from the other senators with nontrivial probabilities of being the most liberal, Senators Sarbanes and Reed: the probabilities that these senators have voting scores more liberal than Kerry's are both .51, or roughly 50–50. That is, while these senators each have a slightly lower probability of being the most liberal senator than does Kerry, Kerry's voting score is indistinguishable from their voting scores (for the 25 roll calls in which Kerry casts a vote—out of the 62 key roll calls analyzed by the National Journal—Kerry's voting record is indistinguishable from Sarbanes' and Reed's. Any of these three senators could be the most liberal senator of 2003.

Comparing Bush and Kerry

A second problem with the analysis thus far is that it is silent on the comparison that is arguably most politically consequential—the relative positions of President Bush and Senator Kerry. This comparison is straightforward to make: we compute a voting score for President Bush treating Bush as an additional legislator in the statistical analysis, with a voting record on the National Journal key votes given by Bush's announced positions on these votes as reported by CQ Weekly. Table 1 reveals that both Kerry and Bush failed to take a position on approximately 60% of the 62 analyzed votes, and so we expect to recover an implied voting score for Bush that has a wide confidence interval.

Figure 3 presents our estimated relative rank ordering for both the 108th Senate and President Bush using the 62 key votes of 2003 identified by the National Journal. Bush's implied voting record is at least as conservative as any Republican senator. In fact, 17 Republican senators have voting records that are in complete agreement with Bush's announced positions (ignoring abstentions). The fact that Bush announced positions relatively infrequently (on 24 of 62 key votes) generates considerable uncertainty in the voting score we assign him. In turn, this generates considerable uncertainty in the rank orderings among conservative Republicans, just as Kerry's high rate of absenteeism, coupled with a relatively extreme voting record, generates uncertainty in any assessment of the degree of Kerry's liberalism. The probability that Bush lies to the right of the entire Senate is .41; other contenders for the “most conservative senator” include Senator Mitch McConnell of Kentucky (probability of .18), Senator George Allen of Virginia (.16), and Senator Conrad Burns of Montana (.11).

Rank Ordering and 95% Confidence Intervals including President Bust using 62 National Journal Key Votes Notes

It is worth repeating that the combination of short and extremely one-sided voting histories means that estimates of legislators' voting scores (and rank orderings) are imprecise. Although the most probable rank ordering supported by the data locates Bush as more conservative than the entire Senate, a conventional 95% confidence interval on Bush's voting score extends far into Republican ranks: in fact, there is better than a 5% chance that Bush is less conservative than as many as 22 senators. Likewise, although the most probable rank ordering supported by the data locates Kerry as more liberal than the rest of the Senate, there is a better than 5% chance that Kerry is less liberal than as many as 16 senators. These confidence intervals are large when one considers that the Senate consists of 100 legislators, underscoring the limits of the available data.

A Longer Horizon: Bush and Kerry in the 107th Congress

The analysis above highlights the difficulties of estimating voting scores for incumbent legislators who are actively seeking the presidency. Campaigning takes these legislators away from Washington, generating considerably high rates of abstention in their voting records in the run up to an election. Indeed, the problem is probably more pernicious: the roll calls that do draw candidates back to Washington to cast votes are not a random subset of roll calls, but are on issues where their votes might have utmost importance for procedural reasons. Party loyalty rather than a genuine ideological position might explain some of these votes. In fact, there is some evidence that supports this notion:

To determine if analyzing more votes in a non-presidential year is (both statistically and politically) consequential we analyze the 498 non-unanimous roll calls of the 107th Senate. Again, we include President Bush in the analysis, adding his announced positions to the roll call data (Bush publicly announced positions on 63 of the 498 non-unanimous roll calls). In contrast to the National Journal key votes of 2003, Kerry misses just 15 of the 498 non-unanimous roll calls in the 107th Senate.³

Figure 4 presents the rank-ordering and 95% confidence intervals that result from the analysis of this larger data set. Taking this longer time horizon and analyzing more votes in a nonpresidential election year reveals a slightly different conclusion. The probability that Kerry is the most liberal senator in the 107th Senate is infinitesimally small. Our best guess is that Kerry is the 16th most liberal senator, and a 95% confidence interval on his rank ranges from the 10th to the 20th most liberal. In contrast, the probability that President Bush is the most conservative lawmaker in the analysis is 31%; other contenders include Republican Senators Jesse Helms (NC, with probability 0.24), John Kyl (AZ, 0.20), and Phil Gramm (TX, 0.17). Our best guess is that Bush is more conservative than the entire Senate, but since the president did not take a position on every vote we cannot estimate his score (and rank) with much precision. As a result, a 95% confidence interval over Bush's rank extends from the most conservative to the 27th most conservative.

Rank Ordering and 95% Confidence Intervals including President Bust using all non-unanimous roll calls from the 107th Senate. National Journal Key Votes

In summary, with this larger set of data, the conclusion that Kerry is extremely liberal simply is not supported. To be sure, Kerry is more liberal than most senators, and we would unambiguously locate him among the most liberal quintile of the Senate. But Kerry is far from being the most liberal senator in the 107th Senate; the late Paul Wellstone (D-MN) is unambiguously to the left of the rest of the 107th Senate. In fact, the only conclusion that is supported by analyses of both the 107th Senate and the 62 votes analyzed by the National Journal in the 108th Senate is that Kerry and Bush are among the most liberal and most conservative quintiles, respectively.