Subtraction and Opinion

When testing the relationship between ideological disagreement and some political outcome at the individual level, scholars often rely on the subjective ideological placements of survey respondents. In these surveys, proximity is not directly expressed by the respondents but is instead derived by the analyst by calculating the difference between the respondent’s own ideological position and that of the candidate, policy, or institution she is asked to rate. Then, if the analyst is interested in the degree of ideological proximity rather than the direction, he or she will calculate the absolute value or the square of the difference between the respondent’s stated ideological location and the perceived location of the candidate, policy, or institution. If respondents reliably and validly provide the two ratings, then the measure of ideological proximity that results is an appropriate one for scholars to use. If, however, those ratings are error prone, then there is reason to doubt their use and, by extension, the substantive conclusions they generate.Footnote ⁴

This is exactly the approach taken among scholars of judicial legitimacy interested in the relationship between ideological disagreement and diffuse support (e.g., Bartels and Johnston Reference Bartels and Johnston2013; Gibson and Nelson Reference Gibson and Nelson2015). Common among all of these studies is their use of a subtraction-based measure of subjective ideological proximity with the US Supreme Court as a key explanatory variable for legitimacy. But, unfortunately, the measure of ideological proximity used both differs across studies and is closely associated with the substantive conclusions the study reaches. The two studies that find a substantial relationship between ideological proximity and legitimacy, Bartels and Johnston (Reference Bartels and Johnston2013) and Christenson and Glick (Reference Christenson and Glick2015), both rely on the following question: “Judging by its recent decisions, do you think the Supreme Court is generally liberal, generally conservative, or is it making decisions more on a case-by-case basis?” (emphasis added). Gibson and Nelson (Reference Gibson and Nelson2015) criticize this question, arguing that it creates a heterogeneous middle category, housing respondents who believe that the Court is legalistic rather than ideological (and therefore decides cases on a “case-by-case” basis), who believe that the Court’s policy making is ideologically moderate, or who simply do not know where the Court stands and use the middle category on the response set to disguise their ignorance.

Gibson and Nelson (Reference Gibson and Nelson2015), who find that the relationship between ideological proximity and institutional support is, at best, weak, drew their measure from the following question: “Thinking about the United States Supreme Court in Washington and the decisions that it has been making lately, would you say that the Supreme Court is a very liberal court, a somewhat liberal court, a somewhat conservative court, or a very conservative court.” This question has also come under fire for its omission of a middle category, the lack of which fails to provide respondents a stated option for “moderate” policy making (see Bartels, Johnston, and Mark Reference Bartels, Johnston and Mark2015, 777 n. 13).

In addition to these wording-specific criticisms, there are reasons to doubt the extent to which a simple subtraction-based procedure results in an accurate measure of ideological proximity. Most obviously, respondents may guess, rationally satisfice by relying on heuristics to judge institutional performance, or generalize from a few salient decisions to their assessments of an institution’s global performance (Bartels and Johnston Reference Bartels and Johnston2013; Lupia Reference Lupia2015). This likelihood raises two important points. First, respondents’ subjective beliefs about institutional performance may well differ from an objective measure of institutional performance, especially when respondents care more about some issues than others.Footnote ⁵ For example, a conservative respondent might judge the Roberts Court as very liberal because of the Court’s decision on same-sex marriage even though the institution has been, by objective measures, relatively conservative. Second, it suggests that respondents’ views of the institution may well not fall neatly on a simple unidimensional space. Indeed, survey researchers continue to debate the unidimensional nature of ideology, with Treier and Hillygus (Reference Treier and Sunshine Hillygus2009, 697) finding that “failing to account for the multidimensional nature of ideological preferences can produce inaccurate predictions” of political behavior (see also Carmines, Ensley, and Wagner Reference Carmines, Ensley and Wagner2012). Respondents may respond that an institution is “liberal” based off of its action on a single issue of importance (and salience) to the respondent, even though the respondent is aware of other, countervailing decisions that happen to be of a lesser degree of importance to her or him.

Additionally, these measures rely on the assumption that respondents are rating themselves and the institution or policy they are asked to score on the same unidimensional ideological scale. A good deal of survey research (e.g., Krosnick and Presser Reference Krosnick and Presser2010) has shown that even the most minute changes to survey design, such as changing the endpoints of a survey scale, can have drastic effects on the responses given by respondents, in large part because it affects the anchor from which respondents judge their answer. It is unclear, when respondents are asked to rate the ideological position of a candidate, whether their judgment is based upon a comparison between themselves and the candidate, between the candidate and the opposing candidate, between the candidate and the institution, or between the candidate and something else. Yet, subtraction-based measures of disagreement uniformly assume the first of these possibilities to be the correct one.

To compound the issue, the extent to which respondents use a simple, subtraction-based distance calculation is unclear. The literature on directional voting (e.g., Rabinowitz and Macdonald Reference Rabinowitz and Elaine Macdonald1989), for example, suggests that respondents do not engage in a simple distance calculation when deciding for whom to cast their vote. Instead, the theory argues, voters privilege direction over distance. Yet, despite this potential for additional systematic measurement error, subtraction-based measures of ideological proximity abound.

Given that those scholars who claim a strong linkage between perceived ideological location and legitimacy have relied upon the “case-by-case” question while those who have failed to find such a strong connection have relied upon a question that does not offer respondents a middle category, this controversy provides a valuable testing ground for reexamining the use of subtraction-based measures of ideological dissatisfaction. By assessing the accuracy of the dissatisfaction measure across alternative question wordings (including those that remedy the criticisms of the original measures), we are able to (1) assess the utility of subtraction-based measures of ideological proximity and (2) determine whether these concerns about measurement error are exacerbated or mitigated by question-wording effects. To this end, the central hypothesis guiding this research is that subtraction-based measures of subjective ideological proximity are contaminated with measurement error and therefore threaten the substantive conclusions scholars draw about spatial models of politics.

Research Design

This article relies on data collected by the American Panel Study (TAPS), a monthly online survey. Panelists were first recruited as a national probability sample with an address-based sampling frame in the fall of 2011 by Knowledge Networks for the Weidenbaum Center at Washington University in St. Louis. Individuals without internet access were provided a laptop and internet service at the expense of the Weidenbaum Center. More technical information about the survey is available at taps.wustl.edu.

The respondents in the TAPS survey were randomly assigned to one of four unequally sized conditions. Condition 1 was the three-category, “case-by-case” version of the ideological location question (and follow-ups): $N=678$ . Condition 2 was the four-category version (and follow-ups): $N=342$ . Condition 3 was the “case-by-case” question and then the four-category version (with follow-ups): $N=257$ . And condition 4 was the four-category question, followed by the “case-by-case” question (with follow-ups): $N=276$ . For analytical purposes, we combine some of these categories of respondents. We do so only when the respondents were asked the same questions in the same order to eliminate question-order effects.Footnote ⁶

An important innovation of our research design is the use of the survey instrument to directly validate the measure of ideological proximity. After the respondents were queried about both their placement of the Court in ideological space and their own ideological self-placement, the survey software translated the respondent’s own self-placement and the perceived location of the Court into a five-point scale. The two measures were subtracted creating a difference score. Using these scores, respondents were asked the following question:

The respondents were asked whether that placement “represents my views extremely well,” “represents my views fairly well,” “does not represent my views very well,” or “definitely does not represent my views.” The respondents’ answers to this question constituted their reported self-validation. Because this technique queried respondents directly about the validity of the measure, we can determine (1) the extent to which respondents judged subtraction-based measures of ideological proximity to be accurate, (2) whether particular types of respondents were especially likely to object to their calculated difference score, and (3) whether misclassification using this method might affect substantive conclusions scholars draw. Below, we investigate all three issues.

Measures of Key Concepts

Validating Reported Self-Validation

Our research design relies upon a comparison between a derived, subtraction-based measure of subjective ideological proximity and respondents’ answers to the self-validation question. Some may be concerned with the use of the self-validation as a comparison—in essence, as a criterion validity variable. Perhaps this measure itself suffers from systematic measurement error, suggesting that any differences we find between the measures are due to error in our criterion variable rather than in the calculated measure of ideological distance. We defend this approach on both theoretical and empirical grounds.

First, as Bartels and Johnston (Reference Bartels and Johnston2013) have documented, the key concept that these studies measure is respondents’ perceptions of ideological distance. Because subjective and objective judgments may differ, validating the calculated measure of ideological distance with an objective measure of the concept would not achieve our ends (for a study with that goal, see Jessee and Malhotra [Reference Jessee and Malhotra2013]).Footnote ¹¹

Second, our approach is analogous to experiments that use a manipulation check to gauge whether an experimental stimulus was received as it was intended. The “stimuli” in our validation approach are the separate measures of ideological placement, and our “manipulation check” is the reported self-validation question.

Third, we offer several empirical tests of the validity of the reported self-validation measure based on a widely employed measure of ideological agreement as a criterion variable. Scholars of institutional legitimacy have long included in their surveys a question that asks respondents whether they believe the Court’s policies are “too liberal,” “too conservative,” or “about right” (e.g., Gibson and Caldeira Reference Gibson and Caldeira1992; Jessee and Malhotra Reference Jessee and Malhotra2013). This measure has the virtue of asking respondents directly about their subjective agreement with the Court’s policy making rather than relying on many of the assumptions outlined above. However, it is a directional measure, lacking—in its commonly asked form—information about the degree of congruence with the Court’s policy making. Still, it captures respondents’ subjective opinion about the placement of the Court’s policy making compared to their own preferences.

We compare the strength of the relationship between respondents’ answers to the trichotomy question and the difference score within self-validation categories. If the self-validation measure is valid, the relationship between the respondent’s difference score and the trichotomy measure should be stronger for those respondents with higher levels of reported self-validation. That is, individuals whose difference score indicates that the Court is “a great deal more conservative” than they are should be more likely to answer the trichotomy by saying the Court is “too conservative” if they have high levels of reported self-validation than respondents who have low levels of reported self-validation.

Across a variety of tests, this is exactly what we find. First, in the bivariate case, the correlation between the calculated difference score and the trichotomy is .52 for respondents who say the calculated classification “definitely does not” represent them. For respondents who say the score “does not” represent them very well, the correlation increases to .59, while the correlation balloons to .71 for those who say the calculated score “somewhat” represents them. For those saying the score “definitely” represents them, the correlation skyrockets to .83. In other words, the correlation between the difference score and the trichotomy answer is much higher for respondents who say they are well classified by the difference score than for those who reject the classification.

The same pattern holds in the multivariate case. Here, we compare the squared prediction errors from separate linear regression models run at each level of reported self-validation. If the trichotomy performs better at higher levels of reported self-validation, then the mean squared error from a regression model should be lower for models run for respondents with higher levels of reported self-validation. Model estimates are available in appendix B.Footnote ¹²

Again, the results are supportive. The mean squared errors decrease as reported self-validation increases. As figure 1 demonstrates, the mean squared error is twice as large for those respondents who reject their classification as it is for those who say it describes them very well. This difference is highly statistically significant ( $p<.001$ ). Similarly, the standard deviation of the squared errors decreases from .54 to .31 as reported self-validation varies across its range, and the predictive power of the linear regression models increases with reported self-validation.

Figure 1.

Mean squared error, by reported self-validation. The estimates are from separate linear regressions run for each level of reported self-validation. The dependent variable is the trichotomous measure of ideological proximity. Error bars, 95% confidence intervals.

Substantive Results

Who Is Misclassified?

Having established that nearly 40% of the respondents reject the accuracy of a subtraction-based measure of subjective ideological proximity, we ask whether misclassification is systematic. To this end, we conduct an ordered logistic regression analysis. The dependent variable is the four-category reported self-validation (with higher values indicating more reported validity). We assess the extent that reported self-validation is related to (1) ideological disagreement and partisanship, (2) political sophistication, including education and Supreme Court knowledge, (3) the respondent’s belief in legal realism, to account for the possibility that individuals who reject ideological judicial decision making in favor of a legalistic approach will report less validity from the measure, and (4) standard demographic factors.Footnote ¹³

The results are shown in table 1. The most obvious initial finding from table 1 is the fact that reported self-validity is related primarily to political sophistication.Footnote ¹⁴ The extent to which political sophistication, here measured by level of education and knowledge of the US Supreme Court, predicts reported self-validation is quite large. Indeed, increasing knowledge and education from the 25th quartile to the 75th quartile triples (from .06 to .18) the probability that the respondent reports that the calculated measure “represents my views extremely well,” while decreasing the probability that a respondent reports that the measure “definitely does not represent my views” drops the corresponding measure from .14 to .05. What is more, for a respondent at the first quartile on both political sophistication measures, the probability she rejects the measure’s reported self-validation is .51, suggesting that the respondent is more likely than not to reject the classification. This finding dovetails well with Jessee and Malhotra’s (Reference Jessee and Malhotra2013) finding that more sophisticated respondents were better able to locate the ideological position of the US Supreme Court accurately and highlights the primacy of political sophistication in conditioning the level of systematic measurement error inherent in these measures.

Table 1.

Ordered Logistic Regression Analysis of Respondents’ Reported Self-Validation

Notes. The dependent variable in the analysis is reported self-validation. Higher values of the dependent variable indicate that respondents ascribe more accuracy to the measure. $N=\mathrm{1,447}$ . b = unstandardized regression coefficient. BIC = Bayesian Information Criterion.

Table 1 also provides some evidence of an ideological effect. Respondents who were estimated to believe that the Court is more conservative than they are tend to believe they are better classified than those who were scored that the Court is more liberal than they are. All else equal, a respondent with a difference score that indicates the Court is much more liberal than the respondent has a .50 probability of agreeing with the stated classification. For respondents who are calculated as in complete agreement with the Court, the same probability is .61. That probability rises to .71 for respondents who have a calculated difference score at its maximum (indicating that the Court is much more conservative than the respondent).

One might worry that many Americans reject the view that the Court is ideological and therefore reject their proposed classification because they also reject legal realism. We considered this possibility, including in the model a measure of a respondent’s belief in legal realism; if it were the case that a belief in legal realism leads respondents to systematically reject their classification, this variable’s regression coefficient should be negative and statistically significant. That is not the case. In this model, there is no relationship between one’s belief in legal realism and her self-validation score. In other words, those who believe that the Court decides cases entirely on the basis of legal factors are no more or less likely to reject attempts to classify the Court in ideological space.

Additionally, demographic characteristics have no statistically significant relationship with reported self-validation, providing no evidence that such measures are problematic for specific subgroups of respondents. Instead, political sophistication reigns supreme.

We draw one major conclusion from this analysis. As one might expect given the cognitively difficult nature of many of the assumptions in these measures, subtraction-based measures of subjective ideological proximity contain systematic measurement error. Indeed, our analysis suggests that the most unsophisticated respondents are more likely than not to reject the subtraction-based score assigned by this methodology.

Does Misclassification Matter?

If misclassification affects about 40% of the respondents and is correlated with political sophistication and ideological placement, it is reasonable to wonder about the substantive ramifications of misclassification. Put differently, does the relationship between subjective ideological proximity and Supreme Court legitimacy—the substantive debate at stake—vary depending on how valid respondents judge the measure? To consider this question, we conduct several regression analyses. The dependent variable in each analysis is the diffuse support index, and the independent variables are (1) the difference score, (2) the measure of the reported self-validation of the calculated score, and (3) the interaction of the difference score with reported self-validation. Full model results, including results by question wording, are provided in table B3.

Because we aim to determine whether the effect of subjective ideological proximity on legitimacy varies based upon reported self-validation, figure 2 plots the marginal effect of ideological proximity across the four-point reported self-validation measure. In other words, each bar represents the effect of ideological proximity on diffuse support for a respondent with a particular level of reported self-validation.

Figure 2.

Marginal effect of ideological disagreement on diffuse support, by reported self-validation. Full model results are available in table B3. The measure of ideological proximity is the difference score. Error bars, 95% confidence intervals.

The figure demonstrates that the relationship between ideological disagreement and legitimacy increases in magnitude as reported self-validation increases. For those respondents who are dissatisfied with their score, ideological disagreement has no effect. But for those respondents who have high levels of reported self-validation, ideological disagreement has a negative and statistically significant effect on diffuse support.

How large are the effects of subjective ideological disagreement on diffuse support? The vast majority of respondents viewed themselves as relatively close ideologically to the court; only 15% fell into the two most extreme categories on this measure of ideological disagreement. The modal respondent in the survey has a value of .25, meaning that—for even those respondents who are well classified by the measure—the effects of ideological disagreement are far from overwhelming. A respondent who agrees with her classification and has a modal score on ideological disagreement reduces her support for the US Supreme Court by a mere −.06. Because diffuse support is scaled from 0 to 1 in this analysis, ideological disagreement therefore accounts for about a 6% decrease in support for the Court for a modal well-classified respondent.

How do the estimates derived from the difference score compare to those derived from each question wording? Figure 3 provides additional analyses of the conditional effect of self-validation on ideological proximity across a variety of question wordings. The results are striking. Most importantly, for the two Bartels and Johnston–based measures (figs. 3a and 3b) and the Christenson and Glick measure (fig. 3c), the effect of ideological proximity on legitimacy for respondents who judge the classification as poor is positive, meaning that more dissatisfaction equates to more institutional support.Footnote ¹⁵ For all of the measures, with the exception of the original Gibson and Nelson measure (fig. 3d),Footnote ¹⁶ ideological proximity has the expected negative and statistically significant relationship among well-classified respondents, although the effect size markedly increases with reported self-validation.

Figure 3.

Marginal effect of ideological proximity on diffuse support, by reported self-validation. a, Bartels and Johnston original measure; b, Bartels and Johnston variant measure; c, Christenson and Glick original measure; d, Gibson and Nelson original measure; e, Gibson and Nelson variant measure. Full model results are available in table B3. Error bars, 95% confidence intervals.

The implication of this finding is that whatever question-wording effects that might appear come not in terms of the distribution of the variable but instead with their conditional relationship with diffuse support. Still, we are inclined to conclude that any question-wording effects are negligible since the overall conclusion holds across all question wordings: the expected negative and statistically significant relationship between diffuse support and ideological proximity only holds for those respondents who have higher levels of reported self-validation.

These analyses raise a fundamental question: Why should we take evidence that the model fits better for those who were well classified than it does for those who were not well classified as evidence that the subtraction-based measure is not valid? A reasonable alternative might be to suggest a validity problem with the self-validation measure.

Of course, were it the case that the self-validation score was itself full of random measurement error, the self-validation score would have no significant conditional relationship with ideological disagreement when accounting for a respondent’s level of diffuse support. But it does. Moreover, this systematic relationship suggests a possible spurious relationship in previous studies: more political sophistication is associated with higher levels of diffuse support (as has been well documented); at the same time, these results suggest that more sophistication also is associated with less measurement error associated with a respondent’s level of subjective ideological disagreement. While we acknowledge the possibility that the self-validation measure itself may suffer from some systematic measurement error, that it has the systematic and strong relationship we expect with ideological disagreement and that it is a direct measure of self-validation leaves us optimistic about this measurement approach.

Thus, these results (1) suggest that subtraction-based measures of subjective ideological proximity are not valid for a substantial proportion of the respondents whose views they purport to represent and (2) provide some new insights into why dissatisfaction has such minor consequences for Supreme Court legitimacy. To wit, the fact that the relationship between ideological proximity and diffuse support increases in magnitude with respondents’ reported self-validation suggests that a measure with less systematic measurement error might lead to a stronger relationship between the two concepts. Still, as we saw above, the effect of ideological disagreement on diffuse support for the modal respondent—even when she believes she is well classified by the measure—is far from overwhelming.

Discussion and Concluding Comments

Our analysis leads us to a series of conclusions. First, and most important, our data raise concerns about the validity of subtraction-based measures of subjective ideological proximity. Across a variety of different question wordings, we find that about 10% of respondents say that such measures “definitely” do not describe them, while about 40% of respondents reject to at least some degree their classification using such a measure. These results, which rely upon a novel survey experiment that queried respondents directly about measurement accuracy, suggest that the validity of these measures is far from certain.

Second, we show that the measurement error that arises from these invalid classifications is far from random, a necessary condition for a conclusion that these measures suffer from validity threats. Indeed, our analysis predicts that a respondent at the first quartile of political sophistication is more likely than not to judge the measure of subjective ideological proximity as inaccurate. In other words, these measures likely do a particularly poor job of judging subjective ideological proximity among individuals with low levels of political information and attention than they do the opinions of political sophisticates. The error is systematic, not random.

Third, we demonstrate that misclassification has important substantive consequences, with the effect of subjective ideological disagreement on diffuse support failing to reach statistical significance for those respondents who have low levels of reported self-validation. This suggests that one reason that subjective ideological proximity is so poorly related to diffuse support is measurement error in the independent variable used in earlier analyses.

If two-fifths of the respondents are misclassified by a subtraction-based measure of subjective ideological proximity, and individuals who are misclassified differ significantly from those who are correctly classified, is there a better way to measure subjective ideological proximity? While we hesitate to make a strong recommendation without additional data, we note that an expanded version of the trichotomy measure we used to validate the reported self-validation measure may well be a better measure of subjective ideological proximity than the subtraction-based measures discussed throughout this article. By asking respondents to make the comparison directly, this measure relies on fewer assumptions about respondent preferences. While the trichotomy measure itself is a directional measure of preferences, an item that asked respondents whether they believed the Court’s policy making was “much too liberal,” “somewhat too liberal,” “about right,” “somewhat too conservative,” or “much too conservative” would give respondents the opportunity to directly weigh in on their subjective ideological disagreement with the Court’s policy making. Simplicity is another virtue of this measurement approach.

There are important limitations to our analysis. For example, there may also be measurement error in the reported self-validation score, an issue that our experimental design allows us only some leverage to detect. However, as we have shown, the relationship between the self-validation score and respondents’ answers to the trichotomy score provides us with compelling evidence that the self-validation score is accurate.

Additionally, although we sought to present evidence of generalizability across a variety of question wordings, we analyze only a single substantive application: the relationship between subjective ideological proximity and the legitimacy of the Supreme Court. Here, we conclude that question wording has only minor, at most, effects on the measurement of the concept, providing evidence to settle a major debate in the literature.

As far as generalizability to other applications is concerned, some might argue that the debate over the attitudinal versus legal policy-making tendencies of the Court might make our findings sui generis. We have acknowledged this possibility through the inclusion of a measure of belief in legal realism throughout the analysis. Indeed, we tested for such differences, finding no evidence that those respondents who rejected the notion of legal realism had systematically lower levels of self-validation. Thus, the analysis gives us no reason to conclude that the legal nature of the Court’s policy making in the eyes of some undermines the generalizability of our findings.

Moreover, subjective measures of ideological disagreement are used beyond the study of judicial legitimacy. Such measures are commonplace in studies of presidential voting (Boatright Reference Boatright2008) and congressional elections (Adams, Bishin, and Dow Reference Adams, Bishin and Dow2004). We do not expect a lack of knowledge of the Court’s policy making to be anomalous; just as Bartels and Johnston (Reference Bartels and Johnston2013) and Heatherington and Smith (Reference Heatherington and Smith2007) note that respondents’ judgments of the Court’s ideological location differ from an objective placement of the institution, respondents’ judgments of the president and Congress are similarly skewed (e.g., McCarthy Reference McCarthy2016; Manchester Reference Manchester2018), implying the possibility of substantial and systematic measurement error in these settings as well. Our measurement validation approach is generalizable to these contexts, and our results advise that scholars studying these important concepts would do well to pause and consider whether they are measuring subjective disagreement with as little measurement error as possible.

At a minimum, however, our findings raise questions about the validity of a variety of findings related to support for the US Supreme Court. In this and other applications, in areas of political science ranging from voting behavior, representation, and public attitudes toward political institutions, obtaining estimates of respondents’ positions with minimal measurement error is as vital of a task for subjective measures of proximity as it is for objective measures of the concept. This article provides a step toward this goal, raising questions and providing a simple solution that is applicable to the US Supreme Court and, perhaps, beyond.