Public opinion research of the mid to late 20th century portrayed American voters as poorly educated about their own political system and uninformed of current political events. In survey after survey, researchers verified the electorate’s ignorance of political facts. Many scholars and political pundits interpreted this to mean that voters do not possess the requisite knowledge to make informed democratic decisions (Lippmann Reference Lippmann1922; Campbell et al. Reference Campbell, Converse, Miller and Stokes1960; Converse Reference Converse1964, Reference Converse1970, Reference Converse1990). For some scholars, voter incompetence implied that representative democracy was broken (e.g., Schumpeter Reference Schumpeter1950; Converse Reference Converse1964; Bennett Reference Bennett1988; Entman Reference Entman1989; Somin Reference Somin1998).
A recent strand of research, however, has questioned whether standard survey measures of political knowledge can adequately assess voter competency (see, e.g., Jerit, Barabas and Bolsen Reference Jerit, Barabas and Bolsen2006; Lupia Reference Lupia2006; Prior and Lupia Reference Prior and Lupia2008; Delli Carpini Reference Delli Carpini2009; Johnson Reference Johnson2009; Pietryka and MacIntosh Reference Pietryka and MacIntosh2013). Notably, Lupia (Reference Lupia2006) argues that surveys such as the American National Election Studies are not measuring useful political knowledge. Rather, the standard battery of knowledge questions only measure an individual’s proficiency in civics. It is hard to judge, he concludes, whether knowledge of basic civics helps voters make competent decisions on the ballot since most of the questions are not relevant to the choice at hand.
We share the concerns of scholars who question how previous research has defined and measured political knowledge, and seek to assess the construct validity of these measures. By using political knowledge as a covariate, most research in political behavior implicitly assumes that the standard measures of political knowledge capture a unique aspect of an individual’s knowledgebase—in particular, one that offers a direct way to measure that individual’s political efficacy. Further, we question whether political knowledge is a unique construct, and not just a type of knowledge that is related to many other types of knowledge.
We take a step toward establishing whether political knowledge is unique by examining survey results we collected in 2011 wherein we asked a pool of undergraduates to answer a broad range of factual knowledge questions about politics, sports, popular culture, rules of the road, economics, geography, and consumer products. We find that political knowledge largely falls on the same dimension as does knowledge of other subjects. We will argue that this finding calls into question whether political knowledge is measuring the relative efficacy of voters.
The construct validity of political knowledge
Recent studies have questioned the validity of how surveys measure political knowledge. These critiques highlight problems with both measurement and construct validity. Some scholars argue that simple survey questions about political knowledge are flawed and wrought with measurement error (see, e.g., Achen Reference Achen1975; Mondak and Davis Reference Mondak and Davis2001; Ansolabehere, Rodden and Snyder Reference Ansolabehere, Rodden and Snyder2008; Gibson and Caldeira Reference Gibson and Caldeira2009; DeBell Reference DeBell2013; for an exploration of how partisanship influences knowledge, see Jerit, Barabas and Bolsen Reference Jerit, Barabas and Bolsen2006; for an exploration of “don’t know” options, see Luskin and Bullock Reference Luskin and Bullock2011).
Inquiries into voter knowledge have begun to focus more on what makes a voter competent, and whether the validity of current survey measures that purport to assess political knowledge are up to the task. Lupia (Reference Lupia2006; see also Luskin Reference Luskin1987, Reference Luskin1990) argues that the common questions researchers use to measure political knowledge (e.g., identifying the Vice President) lack relevance to the voter’s ability to make a competent ballot choice. Similarly, Johnson (Reference Johnson2009) argues that humans gather information in order to achieve a goal (see Anderson Reference Anderson1993; but for a discussion of what incentives promote learning see Lupia and McCubbins Reference Lupia and McCubbins1998). Our research expands on this growing literature that considers the validity of the construct of political knowledge. Our goal is to assess whether political knowledge is a unique type of knowledge, or if political knowledge is foundationally similar to other kinds of knowledge.
Hypotheses, research design and data
We begin with the premise that the tree of knowledge has many branches that share the same root system. In a more technical sense, we argue that knowledge of many subjects resides on a single latent dimension. We assume, of course, some individuals will know a great deal about a specific topic—likely because these individuals derive some enjoyment or benefit (i.e., knowledge related to their job) from expert-level knowledge on a specific topic. On average, however, our expectation is that individuals who have extensive knowledge about politics will also know a great deal about other subjects. Further, we predict that there is a single, underlying latent dimension that explains subjects’ ability to answer factual knowledge questions correctly. We therefore have two hypotheses:
Hypothesis 1: Individuals who know a good deal about politics will also know relatively more about other subjects.
Hypothesis 2: Political knowledge will largely reside on the same dimension as other types of knowledge.
We conducted a survey to explore our hypotheses. In this survey, we asked college students enrolled in lower-division political science and criminal justice courses to answer factual knowledge questions about seven topics: sports (ten questions), popular culture (ten questions), the rules of the road (five questions), economics (five questions), geography (five questions), consumer products (ten questions), and American politics (ten questions). We conducted the study from September 5–19, 2011. We chose to administer the surveys toward the beginning of the semester to minimize the potential testing threat of students learning as the semester continued. We provide a detailed description of the randomization process and recruitment methods in the Online Appendix.
All questions employed the multiple choice format and included a “don’t know” option. The list of multiple choice questions we used and their correct answer as well as the distribution of answers chosen by the subjects are available in the Online Appendix. Of course, we cannot establish equivalence of difficulty across question subjects with certainty. Indeed, most of our questions have not appeared on any survey before. We did, however, select questions based on our expectations of difficulty, attempting to create variation across categories. It is worth noting that our hypotheses do not require equivalence in difficulty across subjects. Indeed, our methodological approach—a within-subjects design—is agnostic to this potential confound. Having a range of difficulty, however, does provide greater variation across our measures.
We use two distinct analytical approaches. First, we examine broadly how political knowledge compares to other types of knowledge. We calculate each individual’s political knowledge score by dividing the number of correct answers by the number of questions. We also calculate scores for our other knowledge categories. We then compare scores in political knowledge to scores in all other subjects.
Second, we use Optimal Classification (Poole Reference Poole2005)—a nonparametric scaling method—to examine whether political knowledge exists on a latent dimension (or dimensions) that is separate from other types of knowledge. Given the weight that some scholars have placed on political knowledge as a necessary condition for effective democratic representation, one might expect political knowledge to be a domain-specific (i.e., unique) type of knowledge. We, however, predict that fact-based questions will largely load onto a single dimension. Poole originally designed Optimal Classification for the analysis of roll call voting, which can uncover the latent dimensions that influence voting behavior in a legislative setting (in the classic example, the first dimension is most closely related to ideology). In our application, we wish to uncover the underlying dimension (or dimensions) that predicts correct responses to factual knowledge questions across all seven of the topics under consideration. One of the strengths of Optimal Classification is that it does not treat any question as belonging to any knowledge category, nor does it make any assumptions about a question’s difficulty. Instead, it minimizes the number of classification errors. For example, if most people know that Barack Obama is president usually know that Joe Biden is vice president, Optimal Classification will minimize the number of times it incorrectly predicts this relationship.
In our analysis, we analyzed 55 questions for 367 respondents. We allowed Optimal Classification to construct ten dimensions. In essence, Optimal Classification reduces the number of classification errors across as many as ten dimensions for all 367 respondents based on their responses to the 55 questions on our survey. We can then analyze which dimensions most accurately predict knowledge for all seven categories. The Online Appendix provides an intuitive explanation of how Optimal Classification uncovers these dimensions.
Results
Our research explores whether political knowledge is domain-specific—that is, if political knowledge is a wholly different construct from other knowledge subjects. Figure 1 is our first attempt to discern whether this is the case. For this figure, we plotted scores for political knowledge against all other kinds of knowledge, adding a LOWESS smoothing line to each plot with 95 percent confidence intervals. If political knowledge is domain-specific, we would expect that a high score on political knowledge questions would not be related to higher scores on other subjects. As the data reveal, however, there is a positive—albeit noisy—relationship between higher scores on the political knowledge questions and higher scores on other subjects. Interestingly, the relationship between political knowledge and the rules of the road is fairly flat, perhaps because most of the respondents score similarly on the rules of the road questions (most subjects probably drive a car).
Comparison of political knowledge to other knowledge categories
The second approach we use to examine whether knowledge of politics is a unique construct provides significantly deeper analysis than the plots we present in Figure 1. Specifically, we explore the construction of political knowledge questions by using Poole’s (Reference Poole2005) Optimal Classification algorithm to analyze the underlying structure of the latent knowledge dimensions in our data for all 55 knowledge questions. With our 55 knowledge questions, we analyze our data by having Optimal Classification produce ideal points—which we conceive as ability scores—on ten latent dimensions for each of our 367 respondents. This approach takes into account each individual’s response to every question, fully utilizing the individual-level variation in responses—something that aggregate-level analysis cannot achieve. In essence, Optimal Classification finds the best location for each respondent on the specified number of latent dimensions based on each respondent’s choices relative to every other choice in the data.
While we had allowed Optimal Classification uncover ten dimensions, not every dimension turned out to be important. As seen in Figure 2, the scree plot shows that the “elbow” in the eigenvalues for the ten dimensions Optimal Classification estimated begins with the second dimension.Footnote 1 As such, our Optimal Classification results indicate that the propensity of a respondent to provide a correct answer to a question from any and all of our knowledge categories largely falls on a single dimension. Corresponding with the eigenvalues, the first dimension accounts for 82.6 percent of the variance in the answers, while the second dimension and third dimension explains just 3.0 and 2.2 percent of the variance, respectively. The amount of variance explained by the fourth through 10th dimension is even smaller than the second and third. Overall, the ten dimensions explain 95 percent of the variance. In other words, the second through tenth dimensions explain a meager 12.4 percent of the variance in our respondents’ choices—compared to 82.6 percent for the first dimension—further demonstrating that the first dimension is dominant.Footnote 2
Scree plot of optimal classification dimensions
If, in fact, all categories of knowledge fit best on a single dimension, a simple correlation between our respondents’ uncovered ideal points and their respective scores in each knowledge category will bear this out. That is, if only one dimension is the best predictor of our respondents’ choices then we expect that our respondents’ knowledge scores will be the most strongly (and positively) correlated with their first dimension ideal point when compared with any other dimension. As Table 1 shows, this is indeed the case: a respondent’s first dimension ideal point is the best correlate with her knowledge on every topic we asked about, with the lone exception, again, being rules of the road. Importantly, the correlation to the first dimension is positive for all knowledge categories, and is the only such dimension to achieve this distinction.
Correlation Matrix of Optimal Classification Dimensions and Knowledge Categories
Note: Bold denotes largest positive correlation with dimension.
To provide a visual sense of fit, Figure 3 plots the first dimension estimates against scores for all knowledge subjects and a combined score for all questions on the survey. The figure also includes a LOWESS smoothing line and 95 percent confidence intervals. These plots clearly show a positive relationship for all knowledge subjects and first dimension estimates. Taken together, Figures 2 and 3 and Table 1 demonstrate that all types of knowledge—political knowledge included—fit well on one dimension.
Optimal Classification (OC) first dimension estimates compared with knowledge scores
While the first dimension has the strongest correlation with political knowledge, it is worth highlighting that the second dimension has a notable positive correlation with political knowledge (0.37). Does this imply that political knowledge does, in fact, reside on multiple dimensions, or there is a domain-specific element to political knowledge? It is plausible, given the results in Table 1. It is also plausible, however, that another explanation is correct (e.g., students are too homogenous of a subject pool). As Figure 4 shows, while there is indeed a positive relationship between political knowledge scores and the second dimension estimates, it is far less pronounced than it is for the first dimension’s estimates.
Comparison of first and second dimension estimates and political knowledge Note: OC=Optimal Classification.
Discussion
Our results show that a single latent dimension of knowledgeability explains the overwhelming majority of variance in responses to all types of knowledge, including politics. While we computed ten possible dimensions, the remaining nine dimensions explained very little of the variance. Although political knowledge does correlate positively with a second dimension, it is worth noting that the second dimension explained just 3.0 of the variance in responses. On the whole, then, it appears that political knowledge largely resides on the same dimension as knowledge of other subjects. Whereas political scientists often treat political knowledge as a distinctive predictor of political behavior, our analysis leads us to question this assumption: political knowledge is not remarkably different than any other kind of knowledge.
As we note earlier, this does not imply that some individuals will not have domain-specific knowledge that vastly exceeds what the average person knows about a topic. Indeed, the correlations in Table 1 suggest that some dimensions beyond the first dimension may be positively or negatively correlated with specific domains. In fact, we would argue that there is room for both general and domain-specific knowledge. Our overall point—one that that data strongly support—is that a single dimension is the strongest predictor of correct answers on all types of knowledge since a single dimension accounts for 82.6 percent of the variance and correlates the strongest with all knowledge categories, except rules of the road.
Since we are using a student sample, we present our results knowing that our data may suffer from a loss of generalizability. It is possible to conceive that students at a large university may be knowledgeable about a variety of topics and the results simply reflect that. Indeed, college students often share a common background and are generally high-achievers. We suspect, however, that our results would likely hold with an adult sample. Even though college students are somewhat homogenous creatures, we still expect their interests to be varied, thus creating the potential for multiple dimensions to be significant based on those interests. We did not find such a result. Further study using an adult sample would be a worthwhile addition, however.
One implication of our findings is that if political knowledge was unique, then standard political knowledge questions are inadequate measures of politics-specific knowledge. That is, our current measures of political knowledge are largely capturing what a non-politics-specific measure of factual knowledge would capture. Indeed, as others have noted (see, e.g., Johnson Reference Johnson2009; Lupia Reference Lupia2006, Reference Lupia2016), these basic measures do not adequately measure meaningful political knowledge. We agree that the need to create measures of this “experiential knowledge” as it relates to politics is the next step in understanding how voters perceive and react to the political world. We believe that creating measures of experiential knowledge represents the next frontier in research on political knowledge. While Delli Carpini (Reference Delli Carpini2009) may be correct when he noted that factual knowledge and experiential knowledge may be strongly related, without adequate measures of experiential knowledge we cannot be sure what, if any, relationship exists between the two.
Supplementary Materials
To view supplementary material for this article, please visit https://doi.org/10.1017/psrm.2018.43
Acknowledgements
The authors thank Daniel Enemark, Vlad Kogan, Arthur Lupia, Ellen Seljan, and two anonymous reviewers for helpful comments. The authors also thank Mike Best and Joseph Petrack for research assistance.
