3.1 Normative and sociological concepts

There is a rich literature on legitimacy in many academic fields. This literature often makes a broad distinction between legitimacy as a normative concept (i.e. legitimacy as a goals) and legitimacy as a question of sociology (i.e. legitimacy as a means). To be sure, both of these perspectives also overlap, which, as we will see, has particular relevance for the legitimacy of global social indicators.

A common starting point for the normative variant of legitimacy is the introduction of Rousseau's Social Contract:

Thus, the question about legitimacy is what justifies why persons in power have the authority to make decisions that affect others. Here, Beetham (Reference Beetham1991) suggests that the ‘legitimation of power’ requires that (1) the power confirms to established rules, (2) that these rules need to be based on shared beliefs by both parties and (3) that the subordinate party has consented to this power relationship.

Another conceptualisation distinguishes between forms of ‘input’, ‘output’ and ‘throughput’ legitimacy (Beisheim and Dingwerth, Reference Beisheim and Dingwerth2008; similar Bodansky, Reference Bodansky1999: ‘source-based’, ‘substantive’ and ‘procedural’). Input legitimacy refers to aspects such as the expertise of the law-making authority and sufficient forms of accountability; output legitimacy may be assessed by theories of justice as well as criteria of efficiency; and throughput legitimacy refers to fair and transparent procedures. Often, it is also said that democratic legitimacy should be the main foundation of such an assessment (Barnard, Reference Barnard2001).

The sociological (or empirical) perspective of legitimacy appears to be very different. The usual starting point is here a reference to Max Weber, who stated that ‘social action … may be guided by the belief in the existence of a legitimate order’ (Weber, [1921] Reference Weber1968, p. 81). Thus, having laws that the public believes to be legitimate is important, since the state may not be able or willing to enforce those by force and since other reasons for compliance (such as habit and expediency) may not always be sufficient.

It follows that this ‘relevance of people's perceptions of the rightfulness and appropriateness of authority for their acceptance and support for political and social order’ (Quack, Reference Quack2010, p. 8) poses the empirical question how far individuals really act in such a way. Hyde (Reference Hyde1983) is doubtful; however, recent psychological research suggests that:

As far as state law is concerned, the sociological and the normative positions are linked because a high degree of voluntary compliance means that the state is best able to pursue the long term goals of society (Tyler, Reference Tyler, Caldeira, Kelemen and Whittington2008, p. 716). An even stronger connection exists for pure private forms of governance, since – due to the lack of state enforcement – the success of such a private scheme crucially depends on its perception as legitimate (Beisheim and Dingwerth, Reference Beisheim and Dingwerth2008, p. 3). This observation will also be important for the discussion of the legitimacy of global social indicators in the subsequent sections.

3.2 The international context in particular

Another consideration relevant for the present paper is that the recent literature has frequently discussed the relevance of ‘legitimacy’ in the context of international, transnational, and global law and governance (e.g. Brunnée and Toope, Reference Brunnée and Toope2010; Karlsson-Vinkhuyzen and Vihma, Reference Karlsson-Vinkhuyzen and Vihma2009; Black, Reference Black2008; Buchanan and Keohane, Reference Buchanan and Keohane2006; Clark, Reference Clark2005; Zürn, Reference Zürn2004; Bodansky, Reference Bodansky1999). To some extent, the general concepts about legitimacy simply apply here. For example, in the transnational context, the distinction between input, output and throughput legitimacy can be understood as a distinction between inclusiveness of wide stakeholder participation, expertise-based effectiveness in solving international public policy problems, and validation of transnational governance through procedural fairness and impartiality (summary in Quack, Reference Quack2010, pp. 6–8).

But it can also be said that concepts of legitimacy of domestic laws require more fundamental adjustments. For example, researchers have explored how to implement democratic legitimacy in a global constitutional order (O'Donoghue, Reference O'Donoghue2014; Zumbansen, Reference Zumbansen2012), how shared social norms based and adherence to a practice of legality can produce legitimacy in international law (Brunnée and Toope, Reference Brunnée and Toope2010) and how the ‘democratic, results-based, order-derived, systemic, deliberative, and procedural’ values of a global administrative law can create legitimacy at a supranational scale (Esty, Reference Esty2006).

The complexity of the many interests in the international sphere can also lead to further complications. For instance, it can be noted that ‘too much’ input legitimacy can be counter-productive for output legitimacy, since numerous diverse interests make it difficult to achieve a meaningful consensus (Kelly, Reference Kelly2008, p. 623). The question ‘legitimacy for whom?’ is also more complex here, since it requires to give consideration to more than the representatives of one state (Karlsson-Vinkhuyzen and Vihma, Reference Karlsson-Vinkhuyzen and Vihma2009, pp. 408–409). Indeed, to put it in more general terms, in the international and transnational sphere ‘mandates are uncertain, and it is not clear on whose behalf they purport to act and to whom accountability should be owed’ (Black, Reference Black2008, p. 143).

It is suggested that the various dimensions of legitimacy make it a useful concept for the analysis of global social indicators. It also raises a number of questions such as: What type of legitimacy can such indicators have? Do indicators turn the goals of normative legitimacy into the means to sociological legitimacy (or vice versa)? Who are the relevant groups that need to be considered for their legitimacy (markets, targets, users, etc.)? How does it compare to alternative ways of achieving the same functions? The following will address these and other questions based on considerations of social theory and the perspective of indicators ‘in action’.

4.1 The nature of global social indicators

At a theoretical level, a preliminary question is when and how statistics and data can be said to be transformed into indicators. This is not an entirely new discussion. Ewald (Reference Ewald1990), using examples from insurance and industrial standardisation, reflects on ‘normalisation’ as a process: it means that, today, norms emerge not simply through formal expressions (constitutions, codes and laws), but through a process where common standards form a normative and objective basis for judgment. More concretely, Shore and Wright observe that:

A concrete example for the way ‘rankings’ and ‘audits’ can transform data into indicators is the World Bank's Doing Business Report (DBR) (see Table 1). Its origins go back to academic work by La Porta et al. and Djankov et al., which ‘merely’ tried to understand the relationship between legal rules and financial development. But, when these studies were incorporated into the DBR, they were transformed into a ranking system and became one of the benchmarks for World Bank funding.

This transformation can also be seen as an example of a process whereby the underlying information becomes less meaningful and, possibly, illegitimate. Many countries care about their performance in the DBR ranking and some, such as Georgia (cf. Schueth, Reference Schueth2011), implement policies in order to rise in the ranking. The problem is then that the DBR indicators are only proxies for good law and can only cover a small proportion of relevant laws for doing business. Thus, a country that changes its law according these indicators quickly rises in the ranking, but this is irrelevant as far as the remainder of their legal system remains unchanged.

Conceptually, such a process can be described as a version of ‘Goodhart's law’ – a variant of Heisenberg's uncertainty principle in the social sciences. According to Goodhart's law, originally developed for questions of macro-economic policy, an indicator that is employed as a target for a particular policy makes it unsuitable to fulfil this role (Chrystal and Mizen, Reference Chrystal, Mizen and Mizen2003). This is a problem for the DBR, but potentially other indicators as well, since measurement and impact of indicators are inherently interlinked – thus, to a larger or lesser extent allowing the ‘gaming’ of the corresponding results.

Another way to describe the paradox of global social indicators is that they are torn between their belongings to the fields of ‘sociology of knowledge’ and ‘regulation and governance’ (e.g. Davis et al., Reference Davis, Kingsbury, Merry, Merry, Davis and Kingsbury2015, pp. 1–3). It is an inherent feature of indicators that they aim to be ‘successful’ due to their persuasive power and the expertise and power of the drafting institutions. It can also be asked how to conceptualise the governance function of global social indicators – and how far this should reduce, or accept, the complexity of the new global landscape (cf. Goodwin, Reference Goodwin2013, p. 696).

A possible approach is Krisch's ongoing project on ‘liquid authority’.Footnote ⁴ The concept of liquidity aims to characterise the dimensions of ‘informality, ideationality, dynamism, and multiplicity’, in particular referring to forms of governance ‘without a clear centre, dispersed over a multitude of actors and institutions (public and private) without a formal ascription of authority’. Typically, indicators can said to belong to this category. For details, of course, the precise shape of the indicator needs to be considered, such as whether it is drafted by public or private institutions; whether it is aimed at states or private bodies; how the global, transnational, national and local levels interact with each other; and, therefore also, whether it is really truly global or connected to a particular local context (see also Section V, below).

4.2 Contrasting Habermasian and Luhmannian positions

An understanding of indicators can also benefit from juxtaposing positions by Habermas and Luhmann.Footnote ⁵ It can be suggested that, for Habermas, the problem posed by indicators would be connected to the need to get beyond what he calls systematically distorted communication. For Luhmann, on the other hand, the issue would be rather how indicators illustrate the necessary limits of communications in conditions of highly differentiated social systems.

This difference is particularly relevant for the relationship between the contestation and the legitimacy of indicators. According to a Habermasian, the role of indicators can be justified if it helps the community come to terms with essentially contested concepts, such as if the indicator channels the discussion of how an indicator about ‘rule of law’ is to be understood. But repoliticisation through contestation may also be seen as problematic. The very appeal of indicators can be that they are presented as mere techniques of measurement, thus staying away from controversial political choices, complex social realities about which there is little agreement and the corresponding risk of ‘gridlock’ in global governance (for such gridlock, see Hale et al., Reference Hale, Held and Young2013).

From the point of view of Luhmannian systems theory, the maintenance of social complexity produces and requires differentiation of codes and programmes. On this view, there is little to be gained by repoliticising what law and indicators have depoliticised. Public consensus is a chimera at a time of increasing (and competing) types of expertise – nor are the resources available that would be needed to allow members of civil society to double check on indicators. In other words, too much contestation can have bad effects, because, as with law, indicators need to have some means of imposing closure at some point if they are to be of practical use. Whether and when contestation can be seen as a sign of success – or of failure – will therefore depend on how far a given indicator needs to provide certainty to users and targets.

4.3 What does ‘law’ have to do with it?

A further question is whether and how law-makers should react to global social indicators. In general studies of ‘law and regulation’, it is often said that law can play an important role for legitimacy claims, though, in the international context, it may mainly be the case that it can structure regulatory conversations (Morgan and Yeung, Reference Morgan and Yeung2007, pp. 244, 330–331). While there are some similarities between indicators and international legal instruments, it is also clear that there are differences, such as at the production stage (Davis et al., Reference Davis, Kingsbury, Merry, Merry, Davis and Kingsbury2015, pp. 18–19). Here, an important consideration is that legitimate indicators need to ‘get the numbers right’: thus, at the level of soft law, international codes of good practice and ethics for statistics may be needed (Lievesley, Reference Lievesley2001). It may also be possible to regulate some of the procedural aspects of indicators (Nelken, Reference Nelken, Merry, Davis and Kingsbury2015, p. 328), aiming to improve their procedural legitimacy.

Alternatively, it may be said that law-makers should simply not allow the proliferation of global social indicators. For example, critics of indicators refer to a lack of democratic participation and accountability and, in the international sphere, a ‘risk of imperialism’ (Scamardella, Reference Scamardella2015). They also argue that indicators camouflage political considerations and provide false specificity (Merry, Reference Merry2016, pp. 19–20) and may therefore be seen as a form of ‘expert rule’ that is detached from its distributive consequences (Kennedy, Reference Kennedy2016). It is also possible to relate indicators to the growing use of mathematical models in modern life, the problem being that such models are said to be ‘opaque, unregulated, and incontestable, even when they're wrong’ (O'Neil, Reference O'Neil2016, as ‘weapons of math destruction’). Thus, a possible response may be that law-makers themselves should take responsibility for such matters of public concern.

However, the reason global social indicators have emerged is often that local and national legislation would not be sufficient and that agreement on international legal norms is not feasible. Thus, in those circumstances, the choice of formal law is not available. In addition, some of the criticism may go too far. As the next section will discuss, power imbalances are possible but not inevitable for all indicators. It is also contentious to criticise the fact that indicators give more power to experts than conventional law-making: the political events of 2016 show rather the opposite problem, namely that of a ‘post-truth’ political climate in which people are said to have ‘enough of experts’.Footnote ⁶

Finally, it is not just the case that law can regulate or restrict indicators but that legal rules may foster their success as they can provide an ‘aura’ of legitimacy (Bernstein, Reference Bernstein2004, pp. 10–14: ‘legitimacy as legalisation’). This has both a sociological and a normative dimension. Once a particular indicator is legally embedded, users and other stakeholders are more likely to consider it as legitimate. Law-makers also contribute to the normative legitimacy of indicators, since, with ‘legalisation’, it is then not simply the power or persuasion of the originator of the indicator that matters, but also the law-maker who acts on behalf – and, in democratic societies, with representation – of the users of those indicators.

5.1 Previous research

It has been said that ‘approaches to legitimacy have generally paid little attention’ to the study of such claims and ‘beliefs in real-life situations’ (Quack, Reference Quack2010, pp. 7–8). To be fair, some of the previous studies have addressed the legitimacy of specific indicators. Beisheim and Dingwerth (Reference Beisheim and Dingwerth2008) assess the Global Reporting Initiative (GRI) and identify ‘the development of ownership based on inclusive, fair and representative participation; social learning and persuasion based on deliberative procedures; and social control based on transparency and accountability’ as factors of procedural legitimacy. Dubois and Nowlan (Reference Dubois and Nowlan2010) put the initiatives of the World Bank in the context of global administrative law, which is said to explain legitimacy as a function of the bank's transparency and structured decision-making. Niemann (Reference Niemann2013) deals in more detail with the World Bank's Doing Business Report (DBR) and identifies how the enrolment of experts, professionals and academics creates legitimacy and how the bank keeps tracks of how ‘prestige and media attention plays a large role in the popularity and legitimacy of the DB programme’.

These three studies address important issues. However, considering the wider range of indicators of Table 1 (see Section II, above), the following identifies further dimensions of global social indicators in action and their implications for legitimacy. Doing so, it suggests that we have to consider that indicators differ in many respects, starting with the data, drafters, addressees and beneficiaries, and then dealing with further dimensions.

5.2 Data, drafters, addressees and beneficiaries

Indicators differ in the way they make use of objective data (e.g. the Indicators of Employment Protection), subjective data (e.g. the Corruption Perception Index) or both (e.g. the Worldwide Governance Indicators (WGI)). Regardless of the method, many indicators have been criticised due to problems of measurement and aggregation (e.g. Arndt and Oman, Reference Arndt and Oman2006, pp. 49–76, for the WGI) or simple arithmetic flaws, even in the most high-profile reports (Pogge, Reference Pogge2004, for the UN-Millennium Development Goals). Naturally, such flaws are detrimental to the legitimacy of the indicator in question. Apart from such general measurement problems, subjective data may have a slight advantage in terms of procedural legitimacy, since surveys can involve the potential users of the indicator in an early stage.

As we have seen, the effectiveness of an indicator is correlated with its sociological legitimacy. A further distinction can therefore be made between indicators by drafters that are economically and politically powerful (such as the UN or the World Bank, cf. Darian-Smith, Reference Darian-Smith2016; Perry-Kessaris, Reference Perry-Kessaris2011) and by others that merely have to rely on persuasion (e.g. the Good Country Index). However, from the perspective of normative legitimacy, it can also be said that these latter indicators may be preferable, since they raise fewer concerns about the deficit of democratic governance structures at the global level.

The drafters of the indicators can be international public bodies (such as the World Bank, the UN, the OECD, the EU) or private ones. The former can have an advantage as far their structures reflect the input legitimacy of their members. However, some private bodies may also benefit from their high reputation (e.g. the ISO). It can also be the case that the independence and neutrality of a private body foster its credibility and therefore the input and output legitimacy of the indicator (e.g. this may be the case for the Press Freedom Index by Reporters without Borders).

Next, many indicators take the country as the unit of comparison while others address private entities such as universities or corporations (e.g. the global university rankings; the GRI guidelines). In terms of legitimacy, the former indicators may be criticised as far they bypass principles of international law and democratic participation at the domestic level. Similar concerns about the latter indicators may be more indirect, such as when the indicator determines the behaviour of the unit and there is little that the state in which the unit is based can do to prevent it (e.g. one may think about the global university rankings).

Indicators can also be divided into those that aim to benefit specific groups and those that have a wider range of potential beneficiaries. Often this may derive from the content of the indicators: for example, the Global Competitiveness Report is more focused on economic interests while the Happy Planet Index is a wide-ranging one. Drafters influence how narrow or wide they design their indicators. For example, having a wide scope can be beneficial to the procedural and substantive legitimacy of an indicator as far as it manages to balance all of the relevant interests. However, there is also a trade-off, since an indicator that tries to please everyone may become too diffuse to be of use for any of the beneficiaries.

5.3 Further dimensions

Going beyond the specific drafters, addressees and beneficiaries, some indicators are embedded in a broader network of institutions which support the indicator, while others have a ‘slimmer’ support structure (such as many of the indicators drafted by academics). An example of the former are the World Bank indicators, since the main shareholders of the World Bank, other donor institutions, financial institutions, corporations, non-governmental organisations (NGOs), etc. all influence the role these indicators play in donor countries. The legitimacy of such indicators may benefit from such a network as far it makes the indicator more effective and, possibly, accountable to a larger constituency.

Next, while all of the indicators discussed here have a global ambition, not all of them can said to be relevant at a global level. For example, the Index of Legal Certainty has the main aim to defend the French civil law model of the world, the global university rankings are mainly focused on Anglophone institutions and the Failed State Index is mainly relevant for the assessment of countries at the bottom of this list. Broad indicators such as the Rule of Law Index are ‘more global’ in their ambition but face the challenge of international comparability. There is also the issue of possible mismatches between the indicator and the local context: while it is unrealistic to expect full agreement about the values that underlie global indicators (Frydman and Twining, Reference Frydman and Twining2015, p. 4), the legitimacy of an indicator is bound to suffer if it is frequently not accepted at the local level. Indeed, it may therefore be said that indicators need to be ‘paired with context-rich qualitative accounts grounded in local knowledge’ (Merry, Reference Merry2016).

A further distinction can be made according to the focus of indicators on economic or wider social goals, such as the Global Competitive Report on the hand and the UN Global Compact on the other. From those examples, it can be seen that indicators are not necessarily ‘left-wing’ or ‘right-wing’. This is not to say that there may not be a differences: for example, indicators that support economic interests have a natural advantage as far as they are supported by influential multinational corporations (Krever, Reference Krever2013).

Some indicators are rather unique while others have to compete with indicators that cover similar topics. Examples of the first group are the Global Slavery Index and the Index of Legal Certainty, while the indicators on rule of law and university rankings compete in a crowded field. The general expectation is that indicators that operate in a well-functioning market for indicators are more likely to be legitimate than ‘monopolistic indicators’, since their drafters face pressure to fix possible technical flaws. But it is also possible that multiple organisations that work on the same topic operate in a cartel-like structure, with the result that flaws are not fixed – or even proliferate (parallel to the indicators of the ‘big three’ credit rating agencies; see Infantino, Reference Infantino and Cassese2016, p. 363).

Finally, indicators differ in the extent to which they have a system of checks and balances. For example, the World Bank's DBR initially included an employment law sub-index that penalised countries with high levels of employment protection. After considerable protest, the World Bank decided to drop this sub-index from its DBR rankings (see McCann, Reference Mccann, Bogg, Costello, Davies and Prassl2015). In preparation for the recent versions of the DBR, the World Bank also engaged with some of the critics of its report (including one of the authors of this paper). This openness to criticism is certainly beneficial for the legitimacy of an indicator. However, it is also rare. In particular, it is unlikely that the drafters of indicators will listen to radical critics who reject, for example, the ‘mathematical turn in law’ (for the discussion, see Restrepo Amariles, Reference Restrepo Amariles2015, pp. 14–19) or the possibility of university and journal rankings (for the discussion, see van Gestel, Reference Van Gestel2015).

Overall, it can be seen that the legitimacy of global social indicators ‘in action’ shows various nuances which drafters and users of indicators – as well as domestic and international law-makers – need to consider. It should also be noted that the dimensions identified here are not independent of each other; often there may be trade-offs between increasing the legitimacy in one of those dimensions while decreasing it in another one.