Introduction
In 2011, I published an article called Pitfalls of Measuring the Rule of Law, which wrestled with some of the conceptual and measurement challenges involved in the project.Footnote 1 The rule of law, I noted, was a central political ideal that crossed political systems, and even dictatorships claimed to uphold it; like democracy itself, there is normative agreement on its desirability even if the underlying conceptualizations of what the rule of law requires are incompatible. This leads to demand for concrete, tractable cross-national measures, which is a deeply fraught enterprise because analysts often ignore basic social science standards of conceptualization and measurement. With Mila Versteeg, I have occasionally returned to this issue.Footnote 2 In this short update, I review developments since then and argue that, while the rule of law remains an essentially contested concept, concrete measures have not advanced much. The revolution in large language models (LLMs) offers promise, but also new pitfalls, mainly in reifying the relatively poor measures that already exist.
The Many Versions of the Rule of Law
The measurement challenges flow from the complexity of the rule of law as an ideal, which has very many different definitions on offer. As Aziz Z. Huq points out, it is less a settled normative concept than a contested cluster of aspirations about how legal norms can and should constrain the exercise of political power.Footnote 3 Standard accounts of the rule of law vary in terms of how “thin” or “thick” they are, meaning the amount of substantive content incorporated into the definition. A thin version, for example, might require only that everyone obeys the law, including the government, but have no stipulations about the content of the law, the role of courts, or the relationship of the law to other mechanisms of social ordering. A thick version, in contrast, might assert that the rules be adopted in ways that are transparent, that they be enforced by independent courts, and that they protect a set of fundamental rights.
Because of its flexibility as a concept, the rule of law is also a universal value, invoked by development agencies, the Chinese Communist Party, political theorists, and constitutional courts in many countries. Its ubiquity reflects not only slipperiness, but also that there is some normative power in the phrase. It is, as we Americans say, motherhood and apple pie, but there are many recipes and they are not all healthy.
One influential account, to which I am partial, is associated with the legal scholar Lon Fuller, who identifies several ways in which the law can fail.Footnote 4 Fuller argues that the law must be general, publicly announced, consistent, prospective, clear, stable, capable of being complied with, and enforced as written. This view is sometimes described as procedural, because it does not require that the law have any specific content. The rule of law, as fellow traveler Joseph Raz put it, does not mean the rule of good law.Footnote 5 Note that these procedural definitions focus on the law and legal system itself. They do not require democracy, human rights, or a market economy. They are not minimally thin, but not so thick as to incorporate elements from outside the law. (Fuller, somewhat confusingly, attributed the term morality to his definition, but we can set aside the implications of that label for our purposes.)Footnote 6
Other concepts, such as that promoted by the World Justice Project (WJP), include protection of fundamental rights in the definition of the rule of law.Footnote 7 In its description, the WJP states that the laws should be just, which is a thicker concept (and in fact an essentially contested one, since justice is what many of our political disagreements are about.) Its multidimensional measure seeks to capture some of the complexity.
In short, the rule of law is an important and popular ideal, which means there is great demand for cross-national governance indicators, with new offerings all the time to help scholars and policymakers compare legal systems.
The Pitfalls of Measurement
Measuring the rule of law is both necessary and deeply fraught. The challenges arise in part because analysts often ignore basic social science standards of conceptualization and measurement. The quality of indicators varies widely, and comparability is more often assumed than demonstrated. This situation can lead to drawing invalid inferences and generating bad policy advice.
The first set of pitfalls, foreshadowed in the prior section, concerns conceptualization: defining what, exactly, is being measured. The large range of conceptual definitions requires the analyst to specify exactly what they are focusing on, and why. It also requires that one think rigorously about the relationship between conceptualization and measurement.
A good social-science concept should be coherent, parsimonious, distinct from neighboring concepts, and in principle measurable, but many rule-of-law indices fail these tests. An example of a bad practice is the World Bank’s Worldwide Governance Indicators (WGI) rule-of-law index, whose definition folds together contract enforcement, courts, and even crime and violence, thereby conflating procedural legality with substantive security outcomes that may depend heavily on social or economic conditions rather than legal institutions.Footnote 8 Because some of these elements lie outside direct government control and overlap heavily with other governance notions, the result is both conceptual confusion and extremely high correlations with other WGI dimensions such as control of corruption and government effectiveness, suggesting that the indices may not be measuring distinct constructs.
In their contribution to this symposium, Adam Chilton and Mila Versteeg explain how the proliferation of rights can actually undermine the thinner view of the rule of law. Thickness of the concept leads to the aggregation of measurements of many elements of the rule of law, but this means that each element loses its relative importance.Footnote 9 This does not mean that only minimally thin versions are appropriate. The Fuller view, to which I am partial, at least has the virtue of conceptual clarity, and coherence, and does not require (for the most part) looking outside the legal system itself. But its eight elements provide some thickness.
The second set of pitfalls involves technical problems of measurement and aggregation. This involves the availability of data sources. The Fuller view, in principle, could be measured if there were sufficient data about the legal system to measure its internal clarity, prospectivity and other aspects. However, data availability is not to be taken for granted.
Core criteria of validity, reliability, and bias are often violated in rule of law measures. They may be susceptible to problems of asymmetry, as Janka Deli has argued.Footnote 10 Expert or investor surveys can be biased and may confound effective regulation with an “unfriendly” business environment, while objective proxies such as “contract-intensive money” capture only some facets of institutional quality and cannot disentangle different institutional channels.
A third pitfall previously identified concerned the possibility that widely publicized indicators can generate perverse feedback effects that undermine the very phenomena they are meant to improve. “Name and shame” strategies can reshape expectations and behavior: if citizens come to believe corruption is pervasive, they may become more willing to pay bribes, reinforcing a corrupt equilibrium, and negative ratings can also deter new, cleaner entrants from investing.Footnote 11 A similar mechanism is possible for rule-of-law indicators, where low scores might discourage citizens from using courts and encourage reliance on informal networks, or scare away foreign investors, thereby entrenching weak formal institutions. Conversely, countries can “game” the reporting system so as to develop a reputation for high performance that is not wholly deserved. These dynamics make it especially important that indicators be conceptually sound and empirically accurate.
Existing measures tend to either use surveys, whether of experts or the public, or objective data. Subjective measures are plagued with problems, as they measure perception rather than reality and so can simply replicate biases that have formed about particular countries or environments. Objective data on the degree to which the rule of law is present are difficult to obtain: cross national measures of, say court effectiveness are challenging to conceptualize and develop. Each has its problems, but better objective indicators are clearly what we need to make progress.
The Missing Variable Problem
In another article from 2017, Mila Versteeg and I surveyed four of the leading cross-national measures of the rule of law, namely the World Bank’s Worldwide Governance Indicators (WGI), Freedom House, the World Justice Project’s Rule of Law Index (WJP), and the Varieties of Democracy (V-Dem) project.Footnote 12 We noted that these indicators embody different underlying conceptualizations of the rule of law and rely on distinct data sources, ranging from expert surveys to coded legal texts and public opinion polls. From a conceptual perspective, therefore, one would expect the indices not only to measure different things but also to diverge in how they rank countries’ adherence to the rule of law. Empirically, however, we found that the four indicators were highly correlated with one another, with pairwise correlations between three of them exceeding 0.95. This strong convergence persisted despite differences in the indicators’ formal definitions, operationalization, and sources, and the measures also correlate tightly with widely used corruption indices. Based on these patterns and borrowing from important work by the political scientist Bo Rothstein, we argued that the indicators are not capturing the rule of law as a narrow, normatively contested legal ideal but rather a broader, thicker construct we called impartial administration.Footnote 13 In our view, what these datasets share is an emphasis on the extent to which public authority is exercised in a non-arbitrary, predictable, and unbiased fashion, which may overlap with but is not identical to the rule of law as understood in jurisprudential debates.
The article then turned to a methodological critique of the measurement enterprise, arguing that the very features that make the indicators tractable also contribute to conceptual slippage.Footnote 14 Heavy reliance on expert perception surveys limits the ability of the indicators to distinguish among different dimensions of legality and encourages evaluators to collapse a wide range of governance attributes into a single holistic judgment about “good government.” Information constraints—especially in data-poor environments—reinforce this dynamic, as coders resort to proxy variables and generalized impressions that blur lines between legality, effectiveness, low corruption, and state capacity. As a result, the composite indices risk becoming self-referential reputational scores that are only loosely linked to specific, normatively defensible accounts of the rule of law.
One might argue that this gap between conceptualization and concrete measurement is perhaps unavoidable and may not matter much in the end. An indicator that captures perceptions of overall good governance is a valuable thing; it is just not a measure of the rule of law. But we are now entering a different world.
LLMs and Rule of Law Measurement
In principle, large language models would seem to offer great promise for complex measurement challenges such as those confronting scholars of the rule of law. By aggregating all existing data and subjecting it to complex new forms of manipulation, LLMs might allow us to develop novel indicators. For example, Diego Zambrano’s proposal in this symposium to look at government win rates may be easier with LLMs than reported data, since “winning” is a complex idea.Footnote 15 LLMs may help us in our efforts to think outside the box of cross-national measures.
Return to Fuller’s criteria. Being able to evaluate large amounts of legislation and enforcement data might allow us to come up with cross national indicators of clarity, prospectivity, and publicity, for example. Because LLMs work with language, they will be best applied to legislation and judicial decisions, rather than enforcement data, which varies across countries in incommensurable ways. For the latter, there is a real risk of “garbage in, garbage out.” But for the linguistic quality of law, and the timing of its promulgation, LLMs can actually do a pretty good job, and thus offer promise.
As I have repeatedly argued, any definition that acknowledges that the rule of law has multiple dimensions raises measurement issues of aggregation.Footnote 16 One needs measures of each of the subcomponents and a mode of aggregating them into a single feature. Weighting various subcomponents can be a source of bias in the measures. For example, if 90 percent of laws produced in one legal system are prospective, and all of them are clear and easy to understand, does it then approximate the rule of law more than another legal system whose laws are always prospective yet difficult to understand? Furthermore, some of the institutional features that are necessary for the rule of law might not be absolute goods, but rather goods for which we should think in terms of optimal rather than maximal values. Judicial independence is one example: we want judges to be independent of politicians, but also to be broadly accountable to the society as a whole. And multiple subcomponents might interact in complex ways.
Any LLM queried about the rule of law as a whole will end up looking at the easily available, highly correlated rankings made available by existing projects. Many of these use ordinal rankings on various dimensions. The WJP, for example, rates Mongolia as seventy-second in open government, eighty-ninth on corruption, and fifty-sixth on order and security. Is there a meaningful difference between Mongolia and the Kyrgyz Republic (seventy-third on open government), the Dominican Republic (ninetieth on corruption), and St. Kitts (fifty-seventh on order and security)? Ordinal rankings are designed to convey information, but by pretending that there are meaningful differences among countries, they actually obfuscate.
LLMs may help us with the creation of new and imaginative proxy variables. One of the most ingenious proxies I have ever come across is that developed by Christopher Clague and others called “Contract-Intensive Money” as a measure of institutional quality.Footnote 17 The measure is elegant: it is simply the proportion of the money supply kept in formal institutions such as banks, which is argued to be a proxy for trust in institutions and the state. After all, if banks can steal deposits in violation of contracts, no one will put money in the bank. Furthermore, bank deposits are transparent to the state, and so those afraid of expropriation will not make them.
To be sure, the measure is not a good one for capturing the rule of law per se. It does not allow us to distinguish among the various factors that would incentivize or disincentivize bank deposits. Bank regulation, the quality of lenders, tax rates, and other factors will also matter, along with contract enforcement, or the rule of law in determining the level of such money. But as a measure of good governance, the measure is strong indeed.
We are in the early days of LLMs, and the fact that they are informed by all existing published data means that they will be biased in the same directions as prior literatures. This is something to be very cautious of. But they also offer great promise, particularly in allowing us to try to develop new indicators of the core procedural criteria identified by Lon Fuller sixty years ago. This would give us the ability to test whether his framework for what he called the “Morality of Law” really was associated with better outcomes in terms of development and democracy.
Conclusion
As I put it in an earlier paper “Social science rules constrain our ability to measure such an abstract and multifaceted concept as the rule of law, but following those rules is the best way to proceed if the goal is inference.”Footnote 18 Rigorous attention to the basic rules of conceptualization and measurement will be essential for the field to progress as new tools come online. They offer promise toward what has been an intractable goal.
Open access
