1. Introduction
Forms of government, regimes, and regime change are major topics of interest in political science. Recent electoral politics in both the United States and elsewhere have added to an already extensive literature on the nature of democracy, including widespread investigation of democratic backsliding or erosion (e.g., Bermeo Reference Bermeo2016; Diamond Reference Diamond2015; Haggard and Kaufman Reference Haggard and Kaufman2021; Levitsky and Ziblatt Reference Levitsky and Ziblatt2019; Lührmann and Lindberg Reference Lührmann and Lindberg2019).Footnote 1 Credible claims about the extent to which democracy is under threat require evidence to support them. Such evidence frequently revolves around assessment of just how democratic nations are and so involves measurement—specifically the use of democracy indices like those produced by the Polity Project, Freedom House, and the Varieties of Democracy Project (V-Dem). Not surprisingly, there are debates about which indices are best for which purposes and what these various indices show. Each approach uses a somewhat different mix of indicators and different methodologies. In recent years, V-Dem’s indices, the newest of these, have been widely, although not universally, adopted by researchers, replacing Polity as the most frequently used.
Early in 2023, political scientists Andrew Little and Anne Meng circulated a draft of a paper that proposed that worries about democratic backsliding may stem from researchers using indices that rely on experts to code the indicators of democracy. They suggested that this subjective element might result in an exaggerated evaluation of the extent to which democracy has eroded globally. The paper sparked a lively conversation among political scientists because so many make use of these indices in their research. The conversation resulted in a 2023 American Political Science Association (APSA) meeting panel and a special issue of PS: Political Science and Politics (57, no 2 [2024]) that published Little and Meng’s paper (Reference Little and Meng2024a), responses to it, and Little and Meng’s reply (Reference Little and Meng2024b).Footnote 2
The discussion in both venues raises a number of issues that are philosophically interesting. In this article, I explore one such issue—the question of the objectivity of measures of democracy. I consider the challenge that Little and Meng (Reference Little and Meng2024a) pose to democracy measurement projects that make use of expert coders, and I use the debate to offer an alternative account of objectivity for measurement: coherence objectivity (Crasnow Reference Crasnow2021). I argue that this account provides a better way of thinking about the measurement of many social science concepts, such as democracy.
In section 2, I outline Little and Meng’s critique and give some background on V-Dem. I focus on V-Dem as an example of a measurement project that makes use of expert coders and so is a target of Little and Meng’s critique. Section 3 explores three aspects of objectivity relevant to the debate (agent, methodological, and ontological) and focuses on the third as the key concern for this discussion of measurement. Section 4 proposes coherence objectivity as an alternative account of objectivity, and section 5 argues for its virtues in the context of this debate.
2. Little and Meng’s challenge
Our primary contrbution is to highlight and explore the consequences of a key distinction between different democracy indicators. Some are objective and factual, such as whether the incumbent party loses and accepts defeat in an election. Other indicators are subjective and rely on the judgment of expert coders to answer questions such as whether a particular election can be considered “free and fair.” (Little and Meng Reference Little and Meng2024a, 149)
Little and Meng (Reference Little and Meng2024a) begin by noting what they take to be a significant difference among various indices that they believe may damage the conclusions that scholars draw about democratic backsliding. Although they acknowledge that “subjective” indicators that use expert coders have some advantages in that a much wider variety of information can be gathered, they argue that the downside is a greater potential for subjective bias.
To investigate this concern, they take a “quasi-minimalist” conception of democracy that relies on indicators that they deem objective—such as whether there are elections that produce a turnover in government. Little and Meng also consider constraints on executives, free media, and rights and protections, thereby expanding the minimalist conception, because a strictly minimalist account, such as one that identifies democracies with regimes that have contested elections, is generally considered to be too inclusive.Footnote 3 They use indicators from available sources of data, such as National Elections Across Democracy and Autocracy and the Database of Political Institutions, and present an analysis in which these objective indicators do not show the level of backsliding claimed by scholars who use indices that have data coded by country experts, as V-Dem does.
Little and Meng aggregate the data from these various sources to create what they term an “objective index,” which they use as a guide to gauge the trend in democracy over the covered years.Footnote 4 Their index shows a rise in the average level of democracy after the Cold War, as do all the indices they examine. There is a slower but steady increase from around 2010, diverging from a slight decline in democracy for Freedom House and V-Dem, followed by a brief upturn and slight decline during the next decade. V-Dem shows a downward trend during this same period. Though Little and Meng note that the differences are really quite small, they argue that even these small differences can affect the conclusions scholars draw.
Little and Meng point to two possible explanations for the differences in indices. The first is coder bias. They speculate that an increase in media attention to cases of backsliding and a general media hype about the issue may have influenced the perceptions of coders and that ultimately this bias is transmitted to the indices in question. Another possible explanation is that their objective indicators reflect a conception of democracy that is too thin and so does not capture the real phenomenon of backsliding, whereas these other indices do. They note that these explanations are not mutually exclusive.
To better understand the nature of Little and Meng’s critique, some background information on indices that use expert coding is useful. To simplify, I focus on V-Dem. The project website describes V-Dem in the following way:Footnote 5
Varieties of Democracy (V-Dem) is a unique approach to conceptualizing and measuring democracy. We provide a multidimensional and disaggregated dataset that reflects the complexity of the concept of democracy as a system of rule that goes beyond the simple presence of elections. The V-Dem project distinguishes between five high-level principles of democracy: electoral, liberal, participatory, deliberative, and egalitarian, and collects data to measure these principles.
The five principles noted are derived from the theoretical literature on democracy (political theory), and V-Dem identifies them with five different types of democracy: electoral, liberal, participatory, deliberative, and egalitarian (Coppedge et al. Reference Coppedge, Gerring, Lindberg, Skaaning and Teorell2017, 42). These five conceptualizations of democracy are captured through surveying country experts for each of the countries represented in V-Dem. The survey questions are designed to assess the extent to which a state has institutions, structures, and practices that cohere with the associated principles, and the answers to the questions result in values that are treated as indicators of the extent to which a regime qualifies as democratic under each principle (Coppedge et al. Reference Coppedge, Gerring, Lindberg, Skaaning and Teorell2017, 25). For example, as one question used to code for free expression, experts respond on a Likert scale to “Does the government directly or indirectly attempt to censor the print or broadcast media?” (207).
The country expert ratings are not the only indicators determining the resulting index values, however. Altogether, V-Dem has more than six hundred indicators. The values for the expert-rated indicators are aggregated with the other indicator values—described as “objective” or “factual”—to produce the higher-level indices corresponding to each of the five principles. The result is that V-Dem produces not just one measure of democracy but five, corresponding to each of the five principles.Footnote 6 V-Dem is in its fifteenth iteration—all country experts code for V-Dem each year—and each iteration has included an expanded number of and additional types of indices that are available.Footnote 7
These are the basics of V-Dem. It differs from previous indices in the number of indicators it uses, the techniques it uses to produce the values from the coding, and the way it makes use of experts as coders. Responding to Little and Meng, contributors to the special issue from V-Dem noted that the project uses a combination of subjective and objective indicators and that it relies on a wide range of checks on intercoder reliability (Knutsen et al. Reference Knutsen, Marquardt, Seim, Coppedge, Edgell, Medzihorsky, Pemstein, Teorell, Gerring and Lindberg2024, 164).
3. Three aspects of objectivity
This debate raises a variety of issues for political scientists working on these topics and using these indices. The papers in the special issue responding to Little and Meng and the accompanying panel discussion at the 2023 APSA meeting cover many of them, but the issue that I will focus on in this section is at the core of the Little and Meng critique—concerns about objectivity that stem from the worry about bias resulting from the use of subjective measures.
The approach that Little and Meng take depends on a distinction between objective and subjective indicators. Most of the participants in this discussion acknowledge that a sharp or clear distinction is not always possible because subjective judgments are always involved in making choices about how to conceptualize democracy, which indicators to use, and how to aggregate indicators. Subjectivity of some sort appears to be ineliminable. Those who want to maintain the distinction acknowledge that such judgments need to be made but argue that indicators such as whether there are elections that result in turnover, how widespread suffrage is, and whether and how much legislation puts restraints on executive power are at least more objective than expert responses to survey questions like the V-Dem questions.
Two main arguments are given for maintaining the distinction between subjective and objective indicators. The first is negative: that abandoning the distinction would result in nihilism—the inability to claim that any measure is better than any other. But nihilism is not an inevitable outcome of a failure to make a distinction. Another, perhaps more conservative approach would be to seek out better objective measures of the sorts of features of democracy that country experts provide information about.Footnote 8 A more radical tactic would be to reconceptualize objectivity—a project that a number of philosophers of science have engaged in (i.e., Longino Reference Longino1990; Montuschi Reference Montuschi2021; Koskinen Reference Koskinen2020).
The many ways in which objectivity can be understood has been a topic of analysis for many philosophers and historians of science (Lloyd Reference Lloyd1995; Janack Reference Janack2002; Douglas Reference Douglas2004; Daston and Galison Reference Daston and Galison1992, Reference Daston and Galison2007). Rather than detailing the various complexities of this discussion, I offer the following three senses of objectivity intended to capture the main ideas that have emerged from this literature (Crasnow and Intemann Reference Crasnow and Intemann2024).
Agent objectivity has to do with the qualities of epistemic agents or communities who are producing knowledge. In the case under examination, the epistemic agents are in the first case the coders, although those who use the data for research and ultimately those who are consumers of that research are also epistemic agents. Traditionally, objective epistemic agents are thought to be detached, or able to distance themselves from any emotions, personal involvements, or other nonrational, outside influences that might bias their investigation.
A second sense of objectivity has to do with the procedures or methods used to produce knowledge. Some methods, practices, and procedures are thought to be more reliable in achieving epistemic ends than others are. This is methodological objectivity.
A third sense of objectivity might be called ontological objectivity, as it concerns the existence or status of that which we have knowledge of. When we say that belief or knowledge is objective, we often mean that it gets something right about the object under investigation—that it latches on to the phenomenon in some way. In summary, objective is an adjective that modifies (1) epistemic agents (agent objectivity), (2) the methods or procedures that we use to gather evidence (methodological objectivity), and (3) our accounts of the world (ontological objectivity). In this debate, all three come into play.
Consider the arguments discussed thus far. When Little and Meng question the objectivity of indices that use expert coders (like V-Dem), they begin with the subjectivity (and potential bias) of the expert coders and so are challenging the agent objectivity of the approach. But the use of coders is just one piece of V-Dem’s methodology, which also includes indicators that are considered objective by the lights of Little and Meng.Footnote 9
In addition, V-Dem aims to addresses expert bias—agent objectivity—through a variety of techniques intended to produce methodological objectivity.Footnote 10 V-Dem attempts to address this through assessing the validity and reliability of their measures—standard criteria for assessing measurement. Measures are valid if they get the target of the measurement right and are reliable if they do so consistently. But validity and reliability are ultimately the standards of ontological objectivity—getting it right.
The question of whether V-Dem (or any other set of indices) gets democracy right is difficult to assess for a variety of reasons but most importantly because democracy is a multidimensional and contested concept. Any account of ontological objectivity for measures of democracy should address the problem of how best to conceptualize and, consequently, operationalize the concept.
4. Coherence objectivity
Although people do have some intuitions about what it means for a nation to be democratic, such intuitions are not on their own consistent enough to provide a standard of correctness of measures of democracy appropriate for much political science research, which relies on quantitative analysis. On the other hand, neither are such intuitions irrelevant to the assessment of democracy.
Reliability is also problematic. A measurement procedure is reliable if it consistently yields correct results, which brings us back to validity. Consequently, I focus on validity for the remainder of the discussion and address it through an analysis of ontological objectivity. I propose an account of objectivity that I call coherence objectivity. The main idea is that measures are ontologically objective when their use coheres with scientific practice.
I turn to (Hasok Chang’s Reference Chang2004, Reference Chang2022) discussion of epistemic iteration to assist in this project. Chang’s (Reference Chang2022, 239) historical and philosophical account of the standardization of temperature centers around a process that he calls “epistemic iteration”: “In the process of epistemic iteration, we knowingly start inquiry on the basis of an imperfect starting point, and use the outcome of that inquiry in order to improve its own starting point.”
There are important differences between measuring temperature and measuring democracy. Temperature is a measure of a physical phenomenon, whereas democracy is not. Temperature has a basis in physical sensation—heat—and is implicated in a network of physical laws. Democracy names a social/political concept that is neither directly connected to physical sensation nor directly observable. In addition, as is the case for most concepts in social science, it is not embedded in a system of laws. Yet, because they are both measurement projects, they have important similarities.
Cartwright, Bradburn, and Fuller (Reference Cartwright, Bradburn, Fuller and McClimans2017, 78) describe measurement as requiring three “steps”: identification of the boundaries of concept to be measured (characterization), identification of a metric to which that concept will be matched (representation), and rules or procedures through which the matching is to be done (procedures). For temperature, the sensation of heat—the empirical observation that identifies the phenomenon to be measured—is not consistent or reliable. It varies from person to person and differs depending on context. The background concept (heat) needs to be systematized (temperature) and operationalized—linked to indicators. The metric needs to be of the appropriate sort (in the case of temperature, an interval scale) to make use of the measure in conjunction with physical laws (such as Boyle’s law). The iterative process through which an appropriate measure is established is both conservative and progressive as Chang sees it: It is conservative in that it begins from what we know (although we recognize that knowledge to be imperfect), and it is progressive in that among its goals is the improvement of knowledge—that is, to make knowledge more adequate to our aims.
Scientists (researchers) start from some existing body of knowledge using what Chang calls the “principle of respect.” They adopt what is currently accepted as known. However, this respect for background knowledge does not require that they wholeheartedly embrace it or steadfastly hold it to be true. The principle of respect may sometimes be in tension with what Chang calls the “imperative of progress.” We build on what we know to improve our knowledge and, in the process, may revise or even discard what we thought we knew.
I propose that epistemic iteration might be a useful way to understand democracy measurement projects. Chang’s account begins with the sensation of heat—an observable. For measuring democracy, epistemic iteration may begin with some other accepted bit of knowledge—for example, theories of democracy. The starting point is not a foundation but rather a point from which to begin. It is open to reevaluation and revision.
V-Dem starts with the five principles that delineate five forms of democracy. In this way, normative theories of democracy constrain measures of democracy by providing a theoretical framework. Theories of democracy are not purely normative, however. They are also empirically informed through the study of regime types as they have appeared around the globe over the course of history. Consequently, theories of democracy are shaped by both normative and empirical considerations—iteratively. Which features of regimes should serve as appropriate indicators depends on what political scientists observe as well as what they theorize. Both the connection to theory and the realities of the political world (at any particular moment) serve as (temporarily) fixed points. They are part of the (current) body of knowledge that we affirm (principle of respect). The reliability (consistency) of the measure with theory provides a means to construct valid measures—measures that cohere with theory, with what is observed, and with the aims of research.
Debates within the discipline of political science about the nature of democracy—for instance, the question of whether democracies can be identified solely through objective indicators—rely on theoretical resources. But they also look at the consequences for knowledge projects, the aims of research, and the policies research informs. While conceptualizing democracy is a starting point for developing measures, concepts may alter in response to changes in regimes around the world, the results of empirical research on the causes and effects of democracy, debates about the values democracies exemplify, the effects on policies of different ways of conceiving of and measuring democracy, and the goals of research.
Empirical generalizations produced by political science do not typically take the form of laws as they do in physics. Nonetheless, results of research using measures of democracy may produce robust empirical generalizations or middle-range theories. An example is the democratic peace hypothesis—the generalization that democracies do not go to war with each other. The way these empirical generalizations, the measures produced through the model, and the overall success of our knowledge when it is put into practice all fit together provides a form of validation of the measures that the research relies on. Adopting a measure of democracy that would result in countries that were deemed democratic going to war with each other would need to be accompanied by a compelling account of why the democratic peace generalization was violated. It is the coherence among previously accepted knowledge, the results of research using the measures, and the successful application of knowledge produced through their use that amount to objectivity—an objectivity that is assessed more holistically than it is through objective indicators of the sort that Little and Meng use (Reference Little and Meng2024a). Put another way, on this account, a measure of democracy may be understood as objective when it is subject to theoretical, empirical, and pragmatic constraints. The effect of these constraints can be seen in the way measures function in the practice of the discipline. The measures are objective insofar as they form part of a coherent practice of the relevant science. This is what I refer to as coherence objectivity.
5. Conclusions
How does coherence objectivity help us think about this debate? As I have framed the discussion, one alternative to reconceptualizing objectivity is to capture a thicker conception of democracy through finding better objective indicators. These would be indicators of aspects of democracy, such as deliberation, civil liberties, and free expression, that are among those V-Dem uses. Such an approach remains focused on agent objectivity and methodological objectivity as a foundation for the adequacy of measures. Problematic about this focus is the implicit understanding of knowledge as dependent on individuals. Thinking about objectivity of measures in terms of their coherence with scientific practice more clearly acknowledges the social nature of knowledge.
The idea that measures need to be objective in Little and Meng’s sense also evokes a questionable foundational approach to measurement. Recall epistemic iteration, which tackles the problem of starting from a fixed point when all aspects of a process are interdependent. A thermometer gives us an interval measure of temperature, but to construct and calibrate a thermometer, we must first understand the relationship between temperature, pressure, and volume, but these are interdependent. Such dependencies mean that assessing validity—getting temperature right—requires the coherence of the entire system of practice.Footnote 11 Similarly, the connections among various indicators of democracy are not independently known or transparent. We do not have knowledge of the extent to which contested elections that result in turnover depend on freedom of the press, for example. Seeking individual indicators that are taken to be privileged or foundational may obscure such relations.Footnote 12
Another feature of many social science concepts is problematic for measurement. Concepts like democracy give rise to what Anna Alexandrova (Reference Alexandrova2017, 80) has called “mixed claims”—claims that mix the moral and the empirical. Political science researchers are dominantly from democracies of the West and treat democracy as a good, yet this is rarely explicitly acknowledged in the discussion of measurement. Because coherence objectivity treats objectivity as theoretically, empirically, and pragmatically constrained, it calls for attention to the various goals of research and the values that shape them. Debates about how to conceptualize democracy are, in part, debates about values. The APSA panel discussion reflected this reality when one of the participants pointed out that Little and Meng’s work had been cited by autocratic rulers as evidence that backsliding had been exaggerated. Because the assessment of objectivity as coherence calls for attention to the goals of research, it reminds us that values are indeed involved in such debates. The conception of objectivity as agent objectivity that Little and Meng work with obscures the value valence of democracy and its indicators. Addressing issues of agent objectivity through a focus on methodological objectivity does no better. Though awareness of objectivity as coherence does not directly address the role of values in democracy studies, it at least opens the way to doing so by acknowledging values and restructuring the discussion of measures of democracy.
The discussion of coherence objectivity offered here neither vindicates V-Dem’s approach nor sheds light on whether there is or is not erosion of democracy. The participants in the discussion addressed a number of other important issues—such as how to conceptualize democracy and whether average measures of democracy globally or examination of specific cases of backsliding tell us more about democratic erosion—but they have not questioned Little and Meng’s use of objective. I have argued that it would be useful to do so.
Acknowledgment
I thank Stephan Haggard for alerting me to this debate.
