One of the trends sweeping the world of global governance since 1990 has been the proliferation of rating and ranking indices gauging state performance on diverse topics, from corruption and democracy to happiness, gender equality, budget transparency, “good” governance, press freedom, conflict, environmental sustainability, and human rights. Although this phenomenon has been a welcome boon for scholars, particularly enriching data in the fields of international relations and comparative politics, the reasons for its rise, and systematic studies of its consequences, remain undertheorized.
This volume edited by Alexander Cooley and Jack Snyder brings together a distinguished range of contributors to examine these issues. Chapters focus on comparisons of corruption rankings, measurements of state fragility, democracy indices in the former Soviet Union, and related topics.
The editors argue that the development of these indices reflects three related trends: the use of techniques of performance evaluation in new public management, the strengthening of global governance networks, and the proliferation of new information technologies and open data sources. Cooley and Snyder identify several functions of rating and ranking organizations, including as expert judges of state performance, establishers of standards and benchmarks, and agenda setters advocating specific issues.
The editors consider several issues concerning this phenomenon, including the sources of authority of the rating and ranking organizations, the methods employed, and how the indices alter political relationships. Their introduction suggests that the second-generation phase has seen increased sophistication and transparency in data-collection procedures, conceptual validity, and aggregation and scale-construction methods.
In conclusion, however, the editors seem to have a change of heart as they highlight several serious criticisms of the whole enterprise, suggesting that the ratings are often incoherently defined, measured idiosyncratically, and subject to manipulation, leading to unintended consequences. Cooley and Snyder bemoan the fuzziness of the underlying normative assumptions, and the general lack of methodological rigor in processes of measurement and aggregation. They end by offering ratings and ranking organizations a series of commonsensical recommendations for “best practices,” such as the need for greater transparency and conceptual precision.
Overall, this is a useful volume reviewing many of the questions and concerns common in discussions about the topic. It could have been strengthened, however, by several steps.
Firstly, the framework used to organize the separate chapters is very loose. Each chapter provides informative case studies of specific types of indices, but the reason for their selection, and the questions they address, could have benefited from tighter editorial direction. This is a common flaw in many edited volumes, but this one could have been tied together more strongly to present more than the sum of its parts.
Secondly, perhaps surprisingly given the critique and the contributors, little systematic evidence is presented in the volume to test the measures which are compared in terms of their external and internal validity, reliability, and legitimacy. Thus, the volume presents many pages of descriptive charts, for example of measures of democracy in Central and Eastern Europe, and a couple of (unlabeled) scatter-plots, which can be eyeballed, but not more rigorous statistical analysis.
Thirdly, the volume features contributions by social and political scientists, but there appears to have been no dialogue with the producers and users of the indices in the international community. The work is written by, and for, social scientists. We are talking to ourselves. What we need, however, is less a one-sided dialogue of the deaf and more a debate among practitioners and scholars about the construction and use of the evidence. The incentives facing both are not identical. For example, while social scientists desire precision and conceptual clarity, there are also considerable benefits in the world of diplomacy and international relations for fuzzier understandings and permeable boundaries, especially for gaining global agreement on highly sensitive issues. The diversity of views within the practitioner community is also important, such as the contrasting constraints and opportunities in the use of indices by nongovernmental and intergovernmental organizations.
Finally, the list of recommendations presented in the conclusions is primarily of the “do this” preaching-to the-choir variety, rather than considering how these goals could best be accomplished. One way that the indices could potentially be improved might be to strengthen and institutionalize the informal ties and networks connecting both producers and users of the data. Just as market research companies have published agreed-upon codes of ethics and standards of practice, so it would be useful for the rating and ranking organizations to develop a professional network, including links with academic associations of social scientists, which could serve several formal functions in monitoring indices.
