Data: Eliciting Perceived Ideology via Pairwise Comparisons

Prior research has measured ideology based on congressional voting, political donations, or other elite-level behaviors. We focus instead on how political activists perceive senators’ ideologies (see also Eady and Loewen Reference Eady and Loewen2021). Our rationale is that activists are high-knowledge political observers; if they cannot distinguish senators’ ideologies, it seems unlikely other citizens will (Stone and Rapoport Reference Stone and Rapoport1994).

Specifically, we use surveys of political activists conducted for the Huffington Post by YouGov. There was no deception or intervention in political processes; respondents were aware they were participating in a research study and consented to voluntarily provide anonymous responses at empanelment, aware that they could decline participation. In consultation with the authors, the Huffington Post administered three surveys during 2016. The surveys interviewed three separate but potentially overlapping samples of approximately 500 Republicans and 500 Democrats. They were fielded January 14–20 (n = 989), July 11–18 (n = 972), and October 28 to November 5 (n = 1,068). Table A1 provides descriptive statistics, demonstrating that the 2016 samples are disproportionately white (85% to 87%). Respondents are relatively evenly divided by gender (48% to 50% female), and between 46% and 49% have a bachelor’s degree. See the Appendix for details.

Potential respondents were drawn from YouGov’s panel but had to meet our definition of activists in initial screener questions. One way to qualify was to report having done at least two of the following in the prior four years: donated to a candidate, attended a campaign event, volunteered for a campaign, or made campaign calls to voters. Alternately, respondents were considered activists if they reported having ever been at least one of the following: a paid staffer for a campaign or elected public official, a candidate for elected office, or an official in a political party (such as a local party chair). Table A2 reports the fraction reporting each activity/experience. Thus, those who qualify only through the first set of criteria (62% in 2016) report having done significant activism. Those meeting the second criterion (38%) are significant political actors, albeit at the bottom of the hierarchy.

The outcome of interest is respondents’ answers to five questions about the ideology of U.S. senators. Specifically, they were asked, “we will be providing you with the names of two members of the U.S. Senate. We would like you to indicate which senator of the pair is more liberal/conservative than the other.” The senators were drawn from the 114th Congress (2015–16). Figure A1 illustrates a sample question, showing that senators were listed by state but not party affiliation. Respondents could choose either senator or reply they weren’t familiar with one or both.

We structured the questions as pairwise comparisons based on extensive research in survey methodology and psychology that emphasizes the relative ease of pairwise comparisons (Hainmueller, Hangartner, and Yamamoto Reference Hainmueller, Hangartner and Yamamoto2015). Respondents may well struggle to provide cardinal measures of ideology or rank many senators. But for a given pair, the task is more straightforward.

Democratic activists were asked who in a pair was more liberal, with pairs drawn from all Democratic senators as well as the 10 most liberal Republicans, identified via nominate. Republicans were asked which of a pair from all Republicans and the 10 most conservative DemocratsFootnote ¹ was more conservative.Footnote ²

We initially limit the pairs to senators from the respondents’ own ideological sides on the grounds that they will be more familiar with their copartisans. This also avoids wasting time asking about comparisons between extreme members of opposing parties, where answers may follow straightforwardly from partisanship. Nonetheless, to confirm that this choice does not unduly influence the estimates, we removed this restriction for the 2021 survey.

In all, respondents were asked to make five comparisons each. Once we removed pairs in which respondents were unfamiliar with one or both senators, we retain 9,030 rated pairs.

Modeling Pairwise Comparisons

Our data are in the form of records of how often a given senator was judged more or less conservative than the senator with whom they were randomly paired. This might be analogous to a sports team’s win–loss record. One simple way to sort the senators would be to calculate how often they are identified as more liberal. However, this ignores the senators against whom they were compared. In sports, beating a strong opponent provides more information than beating an average opponent.

Instead, we employ the Bradley–Terry (Reference Bradley and Terry1952) model to estimate a latent ideological trait (see also Loewen, Rubenson, and Spirling Reference Loewen, Rubenson and Spirling2012). The model assumes that the outcome of any pairing is probabilistic, with the base probability determined by the relative position of the compared senators. In other words, the probability that the i^th senator is seen as more conservative than the j^th senator is

(1) $$ P\left(i>j\right)\hskip1.5pt =\hskip1.5pt \frac{p_i}{p_i+{p}_j}. $$

Bradley and Terry parameterize this model with an exponential form, allowing for a convenient interpretation of its parameters:

(2) $$ P\left(i>j\right)\hskip1.5pt =\hskip1.5pt \frac{e^{\lambda_i}}{e^{\lambda_i}+{e}^{\lambda_j}}. $$

(3) $$ logit\left(P\left(i>j\right)\right)\hskip1.5pt =\hskip1.5pt {\lambda}_i-{\lambda}_j. $$

In this model, λ_i can be interpreted as the i^th case’s latent trait. In our application, this is each senator’s perceived conservatism. The exact values of the estimated λ’s will depend on which senator is the omitted reference point (John McCain here), but we rescale the measure to the unit interval. We term these rescaled λ_i ’s “perceived pairwise ideology.”

Results: Perceived Ideology in 2016

The procedure produces measures of perceived ideology that broadly fit our expectations. Figure 1 presents the estimated pairwise ideology scores, with standard errors discussed in the Appendix. Table A3 lists each senator’s score. As expected, there are two broad clusters, one (left) for Democrats and one (right) for Republicans. However, there is overlap between senators from the two parties, something that has disappeared from twenty-first century nominate scores. Notably, Democratic activists are able to more clearly distinguish ideological distinctions among Democrats than Republican activists are among their copartisans—whereas the perceived ideology measure has a standard deviation of 0.14 for Democrats, it’s 0.09 for Republicans.

Figure 1.

Distribution of Estimated Pairwise Ideology, 2016

Note: Democrats in blue. Republicans in red.

Even knowledgeable activists aren’t likely to have strong opinions about every senator: respondents answered “don’t know” or said they were unfamiliar with at least one of the senators 40% of the time. For that reason, the measure we present does not differentiate a number of less prominent senators, especially when its uncertainty is taken into account (see Figures A2 and A6). Still, extremity is not simply a function of familiarity, as Senators Harry Reid and John McCain are well known and estimated to be relatively centrist. See the Appendix and Table A5 for a discussion of which activists are less likely to rate pairings.

One way to explore this new measure is through a comparison with nominate. Figure 2 plots nominate against perceived pairwise ideology. The two measures are closely related. They are correlated at 0.90, with correlations of 0.63 among Republicans and 0.76 among Democrats. Nevertheless, there are significant divergences.

Figure 2.

First Dimension NOMINATE vs. Estimated Pairwise Ideology in the 114th Senate

Note: Democrats in blue. Republicans in red. Independents in green.

As Figure 1 shows, while there is no overlap between the parties in nominate, there is in the pairwise measure. This is because Democrats like Joe Manchin, Jon Tester, and Joe Donnelly are estimated to be more conservative than Republicans Susan Collins, Mark Kirk, Kelly Ayotte, and Shelly Moore Capito. Such scores are defensible. Both Manchin and Donnelly are pro-life, while Collins is pro-choice; Tester is pro-gun rights. And while our surveys were conducted in 2016, notice that the list of centrist GOP senators includes some who voiced opposition to the various Republican health care proposals in 2017.

There are two related reasons for the divergence between our metric and nominate. First, respondents may care more about some issues than others, and hotly contested social issues like abortion and gun rights may be both better known and more important to their evaluations of senators’ ideologies. These issues rarely appear on the senate agenda.

Second, these senators will often vote “against” their ideology on party votes. At the bare minimum, our respondents may not be keeping up with how often senators vote with their party on procedural votes (Lee Reference Lee2010; Theriault Reference Theriault2008). Moreover, our respondents would probably be correct in determining that those votes are not principally about ideology and so should contribute less to senators’ perceived ideology.

There are other divergences. There are several senators who are estimated to be much more moderate by the respondents than by nominate and others who are much more extreme. For instance, Ben Sasse and Jeff Flake were among nominate’s most conservative senators. But our respondents do not view them as such. In Flake’s case, the senator’s high-profile support for immigration reform may be part of the explanation. But Flake and Sasse also shared their vocal opposition to then-Republican nominee Trump. Others who opposed Trump, like McCain and Kelly Ayotte, are also perceived to be more moderate than expected.

Meanwhile, Jeff Sessions had only a moderately conservative voting record according to nominate but was the first senator to back Trump. Other outspoken Trump supporters like Tom Cotton are also rated more conservative than their voting records would suggest. Just as conservative respondents adopt policy positions when they are told Trump supports them (Barber and Pope Reference Barber and Pope2019), conservative activists evaluate politicians in terms of their relationship to the former president.

In Table 1 and Figure 3, we use regression to estimate these differences systematically. We regress the perceived ideology measure for each senator on the first-dimension nominate score for all senators and then separately by party. One thing to note is the substantial differences by party, with nominate having markedly more predictive power for Democrats (β = 0.77) than Republicans (β = 0.33).

Table 1.

Ordinary Least Squares Models Fit to Different Subsets of the Dataset Predicting Senators’ Perceived Ideology Scores

Note: *p < 0.05

Figure 3.

Relationship between NOMINATE Scores and 2016 Perceived Ideology Scores for GOP Senators, with Notable Pro- and Anti-Trump Senators Labeled

Using the model from Table 1’s fourth column, Figure 3 illustrates that for Republicans, perceived ideology appears to deviate from nominate in that anti-Trump senators are estimated to be less conservative than their voting records, whereas pro-Trump senators are estimated to be more so. In Table 1’s sixth column, we formalize this test by showing that, among Republicans, a measure of senators’ relationship to Trump in 2016 is a meaningful predictor of perceived ideology beyond nominate.Footnote ³ For example, as compared with a GOP senator who clearly supported Trump, an anti-Trump senator had a lower perceived ideology score by 0.10, more than a standard deviation. Table 1 shows a relationship with nominate’s second dimension (Column 5), but it loses significance when also including Trump support (Column 7). Such results may be consistent with the interpretation that the second dimension reflects an insider–outsider dimension (Enten Reference Enten2015). However, more research is needed. The change in the second-dimension coefficient across models is neither large nor statistically significant, and more recently, the second-dimension scores place Mitt Romney far higher than pro-Trump senators like Josh Hawley.

2021 Survey

The surveys employed above were from 2016, before Trump took office—and they primarily asked activists about senators from their own party. To see how stable these results prove, we conducted a follow-up YouGov survey in late April 2021 (n = 1,110). The set-up was similar to that described above, with a few exceptions: activists assessed 12 pairs of politicians from both parties, including not just senators but 14 other prominent figures such as Trump, Ron DeSantis, Joe Biden, and Kamala Harris.

Figure 4 depicts the results. In broad strokes, they confirm and extend the patterns observed in 2016. First, we detect meaningful overlap, with Republicans such as Collins (and Lisa Murkowski and Mitt Romney) again more liberal than some Democrats.Footnote ⁵ Also, we can see how Trump’s influence persists, with Trump himself rated more conservative than 82% of the Republicans queried and Trump’s allies (DeSantis, Hawley, Tuberville, Cotton, and Pence) more conservative still. Kamala Harris is perceived as the most liberal politician, outpacing Elizabeth Warren and Bernie Sanders and raising questions about the role of race and gender in such perceptions, which we will pursue in future research (Huddy and Terkildsen Reference Huddy and Terkildsen1993; Sanbonmatsu and Dolan Reference Sanbonmatsu and Dolan2009).

Figure 4.

Perceived Ideology Scores, April 2021 Survey

These results confirm that the core findings from 2016 were not the product of asking Democrats and Republicans primarily about in-party senators. Here, too, we see clear evidence of overlap between the parties. These results also strongly reinforce the substantive claim that Trump reoriented definitions of “conservative.”