1. Introduction
Evaluation of legal institutions is necessary for making good decisions when creating and reforming them. In recent decades we have relied on indicators based on surveys eliciting the perceptions of experts and citizens, as well as metrics based on assessments of formal rules. Prominent examples are the Governance Indicators of the World Bank (Kaufmann et al., Reference Kaufmann, Kraay and Mastruzzi2010); specific measures focusing on the rule of law, such as the World Justice Project Rule of Law Index (WJP, 2017) or, more narrowly, on corruption, such as Transparency International's Corruption Perception Index (TI, 2018); or individual assessments of formal rules, such as those of Doing Business (World Bank, 2003–2018).
This reliance on general information makes sense if knowledge is held by experts and the general public. However, experts and the public are often asked for their perceptions without considering their position regarding the institution being evaluated. This differs starkly from the established practice for all sorts of organizations. The latter – from business firms to churches to universities – routinely collect the opinions of users and customers, who are assumed to be better informed on how the organizations satisfy their demands.
Can we do better with institutions? Given that information is costly, one should not expect the relevant information on institutions to be evenly distributed among members of society. Moreover, much information – in particular ‘knowledge of the particular circumstances of time and place’ – is costly or even impossible to transfer (Hayek, Reference Hayek1945: 521). This ‘specific’ information (Jensen and Meckling, Reference Jensen, Meckling, Werin and Wijkander1992) is likely prevalent for legal institutions, but eliciting it is particularly difficult: in legal institutions, the main channel for transferring information – prices – is often missing or distorted.
Moreover, as opposed to markets, artificial systems to gather information require identifying who knows most, and this is harder for institutions than for organizations. For instance, in a sense all citizens ‘use’ general institutions such as the political system or the rule of law. However, even if all citizens benefit from the law, some of them are likely to be better informed on its functioning. In fact, the rule of law operates through supporting organizations such as the judiciary, the police, and administrative bodies, and all citizens are unlikely to be equally well informed on how these organizations work.
This paper assumes that those who interact most closely with such organizations are better informed about the corresponding set of institutions. In particular, those who contract with them, experience contractual conflicts, and litigate with them can be expected to have more information on the performance of contract law and litigation. Similarly, both victims and criminals should know more than the rest of the public on the functioning of the police and the courts and, therefore, on criminal law; those who most often deal with public bureaucracies should know more on administrative law; and parties to employment contracts should know more on labor law. In short: those with experience should know more because the experiences considered in this paper imply proximity to the public organizations and institutions being evaluated and are, therefore, likely to reflect greater knowledge on how they work. In a sense, experienced individuals are considered here as ‘institutional customers’. Basing performance indicators on their perceptions is akin to considering users' satisfaction with their service providers.
To the extent that this proximity assumption holds water, it opens up many questions on how to design information systems to evaluate legal institutions and even on how to manage them. If some users or constituencies hold specific information about institutional performance, should we still focus our information-gathering efforts for evaluating those institutions on the whole population (as is often done) or, instead, on those particular users or constituencies? Should we apply, as the WB Governance Indicators and WJP Rule of Law Index do, a mixed methodology, asking both the whole population and a sample of experts? Should experts be replaced by users? In any case, how should we combine information from such mixed sources? More generally, should not we design our information-gathering efforts with a more explicit focus on the actual distribution of knowledge?
As a first step, this paper focuses on more modest questions: How does experience affect perceptions of institutional quality? How does considering experienced subsamples modify performance indicators, both at the individual level and in terms of country scores, and what drives such changes? What are the consequences for our understanding of the distribution of knowledge across the population? Does experience affect perceptions more or less than, e.g. education and other demographic variables?
To answer them, using the World Justice Project general surveys (WJP, 2014), the paper explores how indicators of performance in four areas of the rule of law systematically vary depending on which persons we consider – the general population or experienced individuals – , and how experience may be more informative than variables such as education, age, or wealth.
In doing so, the paper is related and contributes to several strands of the literature. By pointing out how perceptions vary with experience, it adds a more nuanced view on indexes built on perceptions. To the best of my knowledge, the role of experience has until now been discussed in part of the literature on corruption, which, since it aims to assess the true extent of corruption, is concerned about the differences between estimates of countries' corruption obtained (1) from individuals' or experts' perceptions and (2) from personal recollections of direct experience of corruption. Most of these studies rely on recollections of bribes to directly estimate the extent of corruption. The lack of correlation between perceptions and recollections of experience observed by many of these studies suggests that perceptions by the general population and even by experts may have serious systematic weaknesses.Footnote 1 The present paper follows a different strategy by focusing on knowledge-related experiences and using them to, first, identify allegedly better-informed individuals and, then, analyze how their perceptions on institutional quality differ from those of the inexperienced – and allegedly less-informed – individuals.
The paper also relates to works examining the methodologies and difficulties for measuring institutions (Voigt, Reference Voigt2013, Reference Voigt2018) and, in particular, the rule of law (Ginsburg, Reference Ginsburg2011; Voigt et al., Reference Voigt, Gutmann and Feld2015). These include discussions on the comparative advantages of subjective versus objective measures of governance (Kaufmann et al., Reference Kaufmann, Kraay, Mastruzzi and Rose-Ackerman2006), the different results obtained when focusing on citizens or experts (Arndt and Oman, Reference Arndt and Oman2006; Kaufmann et al., Reference Kaufmann, Kraay and Mastruzzi2007; Kurtz and Schrank, Reference Kurtz and Schrank2007; Razafindrakoto and Roubaud, Reference Razafindrakoto and Roubaud2010), the potential conflict of interest posed by relying on lawyers (Arruñada, Reference Arruñada2007), and the interpretation of lobbying as a way of using information as a tool for gaining influence (Grossman and Helpman, Reference Grossman and Helpman2001: 223–318). The paper is also linked to the literature concerned with tax salience (Chetty et al., Reference Chetty, Looney and Kroft2009; Goldin and Listokin, Reference Goldin and Listokin2014; Zheng et al., Reference Zheng, McLaughlin and Kaiser2013), as some of the experience indicators could be taken as salient incidents and be subject to a similar debate.
It proceeds in two main stages corresponding to Sections 2 and 3.
Section 2 identifies for each legal area – i.e. for contract, criminal, administrative, and labor law – at least one experienced and allegedly better-informed group of respondents, develops performance measures for each legal area, and estimates the ‘impact’ of experience on such measures. Experience shows a consistently negative correlation with stated perceptions of performance in all four legal areas. Moreover, it is the main observed driver of perceived performance, well above other popular variables, such as education, wealth, gender, age, and participation in economic activity, some of which are shown to exert their influence only through inexperienced individuals, therefore casting doubt on their reality: e.g. educated individuals hold stronger and more positive opinions which may be unfounded.
Given the observational nature of the data, any claim that experience is causing the observed differences in perceptions must be subject to the assumption that there are no unmeasured confounding variables. Under this assumption, the results pass several robustness checks using matched samples, diverse (additional as well as narrower) experience indicators, and an alternative survey with additional and slightly different questions.
Section 3 calculates the corresponding country scores and rankings, and examines how the effect of experience varies across countries. Differences between performance scores based on the responses given by the general population and by the corresponding experienced subsample cause substantial changes in how countries are ranked by both sets of indexes. Moreover, the alleged informational advantage of experienced subsamples increases with some plausibly exogenous country features, such as countries whose legal system originated in France and regulatory quality (both of them arguably connected to the efficacy of the state), and positively with ethno-linguistic fractionalization (possibly through legal discrimination). Moreover, while the estimated effect of experience does not vary with the percentage of experienced respondents in each country sample, the estimation error decreases significantly as this percentage rises.
Section 4 concludes.
2. How individual perceptions on rule-of-law performance differ between the general public and experienced individuals
Model
To examine how experience with rule-of-law services correlates with perceptions of institutional performance and how this experience interacts with some relevant variables such as education, wealth, gender, age, and involvement in economic activity, I estimate several variants of the following equations:
whose dependent variables, i_Law, are the individual (respondent-level) scores of performance indexes in different legal areas (with i = Contract Law, Criminal Law, Administrative Law, and Labor Law), while the independent variables of interest are the experience variables Exp j identifying the experienced subsamples. Controls are: X, a vector of individual characteristics including respondents' Education, Wealth, Male, Age, and being Economically active, as well as Country r, a vector of country dummies (r = 1, 2,…102). Some equations will also include experience interactions. In other equations, country controls will be replaced by GDP to easily observe possible differences at different stages of economic development.
Variables
Experience indicators
The data from the WJP surveys of the general public (the ‘General Population Polls’ or GPPsFootnote 2) allow to define four legal areas – contract, criminal, administrative, and labor law – with their corresponding indicators of experience and performance.
Experience indicators are built as follows:
• For contract law, the variable Has had a contractual conflict is equal to one for respondents who have suffered a conflict with someone who refused to fulfill a contract or pay a debt in the past 3 years; zero, otherwise. Complementarily, I also use the variable Has litigated a conflict, a binary variable equal to one if the conflict exists and was litigated before a third-party enforcer (be it a court, commercial arbitrator, or traditional ruler); zero, if there is no conflict or, in the case of a conflict, it was not litigated.
• For criminal law, the variable Has been a victim, detained, or abused is equal to one for those who, either themselves or those living in their households, have been victims of crime or have been detained or abused by the police or the military; zero, otherwise. Complementarily, I also use the variables Has been a crime victim and Has been detained/abused to identify those who have been victims or detained/abused; zero, otherwise. In addition, I also consider how the perceptions of crime victims are affected by their having reported the crime and by the perpetrator having been caught and punished.
• For administrative law, the variable Has requested a permit is equal to one for those who, either themselves or anyone living in their household, in the past 3 years have requested a government permit or license in a local authority; zero, otherwise. Complementarily, I also use the variable Has paid a bribe, which is one if, according to the respondent, the request led to payment of a bribe; zero, if the respondent did not ask for a permit or, after asking for it, did not pay a bribe.
• For labor law, the variable Has had employment experience is equal to one for those who have employment experience either as employers or employees (independent professionals, business owners, salaried employees, or retired/pensioners); zero, otherwise.
Panel A of Table A.1 in the online Appendix explains in more detail how these indicators have been built.Footnote 3 Panel A of Table A.2 presents summary statistics for the experience variables. The relative size of the four main subsamples ranges from 11.31% for Has had a contractual conflict to 56.81% for Has had employment experience, with intermediate values for Has requested a permit (22.01%) and Has been a victim, detained or abused (30.21%).
Performance indexes
The WJP Rule of Law Index survey also asks respondents to assess different institutional services. I use these responses to build four corresponding indicators of perceived performance, which are built as follows:
• For the contract law area, the Contract Law indicator is a composite index built as the first principal component of three variables, calculated by codifying responses to the WJP survey asking whether judges are captured by private interests, to what extent there is corruption in the judiciary, and to what extent the courts guarantee everyone a fair trial.
• For criminal law, the Criminal Law indicator is the first principal component of seven variables related to the extent of self-help, punishment of homicides, punishment of corrupt police chiefs, safety when walking at night, police acting according to the law, the police respecting basic rights of suspects, and punishment of the police when it violates the law.
• For administrative law, performance is measured with the index Administrative Law, built as the first principal component of three survey questions related to punishment of a nepotistic government officer, extent of corruption in local government, and punishment of corrupt government officers who unlawfully issue licenses.
• For labor law, its performance indicator, Labor Law, was built from the response to a single question asking if workers can freely form labor unions and bargain for their rights with employers.
Panel B of Table A.1 in the online Appendix lists the questions used to build these performance indexes.Footnote 4 For easier interpretation, all these indexes are built so that they rise with greater lawfulness. Panels B and C of Table A.2 present summary statistics of these performance indicators for the whole sample and for each experienced subsample.
Covariates
The covariates X were obtained as follows: Education uses directly the zero-to-five scale in the survey. Wealth is composite index measuring respondents' wealth standardized within each country to capture relative individuals' wealth, built as the first principal component of 23 wealth binary variables asking respondents if they or any member of their household have a variety of assets. Male is a binary variable equal to one for male respondents; zero, otherwise. Age is a continuous variable measured in years. Economically active is a binary variable equal to one if the respondent is an independent professional, a business owner, an informally self-employed person, a salaried employee or is temporarily out of work; zero, otherwise. (This variable is omitted in the analyses of labor law performance and experience, given that it is correlated with the Has had employment experience variable.) GDP is the log of the 2015 gross domestic product per capita, based on purchasing power parity, as estimated by the International Monetary Fund.Footnote 5
Panel C of Table A.1 explains how these variables have been built. Panels D and Ei of Table A.2 present summary statistics of the covariates for the full sample and for each experienced subsample.
All non-binary independent variables were standardized.
Results
Most of the mean scores and variances of the performance indexes are significantly different for the experienced and inexperienced subsamples. Moreover, the mean scores and the variances are predominantly and significantly lower for the experienced subsamples (Table 1). According to the estimated equations presented in Table 2, experience is associated with statistically significant negative evaluations of performance, which are also substantively significant. For instance, having suffered a contractual conflict in the last 3 years lowers the perception of contract law performance in 0.128 standard deviations. Similarly, having being a victim of crime or having being detained or abused by the police account for a fall of 0.141 standard deviations in the perception of how well criminal law institutions work.
Summary statistics of individual scores for the inexperienced and experienced subsamples
The effect of specific experience on perceptions of institutional performance in four areas of the rule of law
Note that experience variables differ across columns.
Notes: Dependent variables are the corresponding indexes of institutional performance. OLS models with constants and country controls (omitted).
*p < 0.10, **p < 0.05, ***p < 0.01. Clustered standard errors are shown in parentheses.
Furthermore, these negative impacts of experience are around 3.6 times bigger – in terms of the average of absolute values for significant coefficients – than those of other independent variables. In addition, the coefficients associated with active economic participation (those of the Economically active variable), which can be seen as a source of knowledge across different legal areas, are also significantly negative. Overall, experience seems to be the main observed driver of performance perceptions.
Some interactions of experience with the dependent variables are also significant (presented in the even columns in Table 2). This suggests that estimating the effect of these variables without considering experience (as in, e.g. Behrer et al., Reference Behrer, Glaeser, Ponzetto and Shleifer2019) might lead to mistaken inferences. For instance, column 2 of Table 2 suggests that the positive effect that education and age seems to have on how respondents evaluate the performance of contract law is concentrated among subjects without experience. Intriguingly, this unfounded optimism of educated – therefore, arguably, influential – individuals poses the possibility that individuals' general education may exert a negative externality on how well societies evaluate and eventually decide on their institutions.
To see how these effects of experience may change with economic development, Table 3 replaces country fixed effects with country-level controls (GDP and its interactions with X j). As expected, countries with greater GDP show better performance (in all but labor law). However, this is less so according to experienced individuals – in other words: experience has a significantly more negative impact on high-GDP countries.Footnote 6 Moreover, the effects of other variables, such as education or relative wealth, are substantially different for poor and rich countries, hinting again that inferences about these variables should be made with great care and probably not based solely on their single effects.
The effect of specific experience on perceptions of institutional performance with GDP interactions instead of country controls
Notes: Dependent variables are the corresponding indexes of institutional performance. OLS models with constants (omitted).
*p < 0.10, **p < 0.05, ***p < 0.01. Clustered standard errors are shown in parentheses.
The almost consistently negative effect of experience is remarkable considering that the four indicators of experience refer to quite different phenomena and relate to four different legal areas. It is therefore tempting to explore a causal inference on the impact of the experience (i.e. ‘treatment’) variables on perception of institutional quality. Moreover, given that experience is associated with specific knowledge and the other independent variables used as controls (Education, etc.) are closer to general knowledge, their relative impact might have consequences for institutional evaluation.
Discussion and robustness
However, given the nature of the data, it is important to keep in mind that the analysis is merely observational. In particular, the negative correlation consistently observed between experience and perception might be spurious. It might be the case that experience does not affect perception but that experienced individuals have particular features that also affect perception – i.e. some hidden variable may affect both the treatment assignment (experience) and the outcome (perception). In other words, any claim that experience is causing the observed differences in perceptions must be subject to the assumption that there are ‘no unmeasured confounders’ or ‘no omitted variables’ – that is, the treatment assignment is independent of the potential outcomes: P(Exp|X, i_Law) = P(Exp|X) or, in words, controlling for all covariates, the probability of experience remains unchanged when we know the perception of performance.
The consistency of results across quite diverse types of experience makes it hard to imagine confounders which could be driving the results; however, they cannot be ruled out. Working, therefore, under this strong ignorability assumption, I check for robustness with respect to the functional-form restrictions in simple ordinary least squares (OLS) estimation using coarsened exact matching (Blackwell et al., Reference Blackwell, Iacus, King and Porro2009), to replicate a randomized experiment by building ‘treated’ (i.e. experienced) and control (inexperienced) groups of respondents with similar distribution of covariates.Footnote 7 Table 4 presents the results of four equations that adjust for the remaining imbalance in the matched data. This matching exercise hardly alters the estimated coefficients of contractual and crime experiences. It only substantially reduces the coefficient linked to requesting a permit, but without changing its sign or statistical significance, and also makes the coefficient associated to employment experience insignificant.
The effect of specific experiences on perceptions of institutional performance, matched samples
Notes: OLS equations in columns. Dependent variables are the individual-level scores of the corresponding indexes of institutional performance. Matched samples with importance weights as well as demographic (Education, Wealth, Male, Age, and Economically active [but for Labor_Law]), constants and country controls (all of them omitted).
*p < 0.10, **p < 0.05, ***p < 0.01. Clustered standard errors in parentheses.
Second, the complementary indicators of experience provide additional robustness checks because of their different characteristics. While having litigated a conflict and having paid a bribe are additional experiences, having been a victim of crime or having being detained or abused by the police are narrower experiences with respect to having been a victim or a criminal (the Has been a victim, detained, or abused variable). Moreover, they also differ in the extent to which most of the experiences of each type are decided by respondents (having litigated) or by service providers (having paid a bribe). They should therefore clarify the nature and the impact of diverse types of experience.
In this vein, the additional-versus-narrower difference between the complementary indicators of experience calls for different models. Table 5 reports results for additional experiences by adding experience variables as determinants of performance. We observe that those who have litigated a conflict hold a better perception but the size of this positive effect is approximately one-third of the negative coefficient associated with having a contractual conflict. However, the significance of the litigation coefficient disappears with proper sample matching, suggesting a selection effect in the raw subsample. At this point, it is worth saying that those who in the last 3 years had a contractual conflict but chose not to litigate are still likely better informed in two dimensions: first and foremost, about the preventive efficacy of contract law institutions; second, about the quality of the litigation process if the conflict encouraged them to gather information before deciding not to litigate.
The effect of additional specific experience on perceptions of institutional performance in two legal areas
Note that experience variables differ across sets of columns.
Notes: Dependent variables are the corresponding indexes of institutional performance. OLS models with constants and country controls (omitted).
*p < 0.10, **p < 0.05, ***p < 0.01. Clustered standard errors are shown in parentheses.
On the contrary, the negative effect observed in those who had requested a permit seems to be concentrated among those who paid a bribe for it. The experience of having paid a bribe after asking for a permit seems to greatly worsen perceptions on the lawfulness of administrative law. The greater impact we find for this experience than for individual characteristics such as education, age, wealth, and economic status contrasts with the conclusions reached by Donchev and Ujhelyi (Reference Donchev and Ujhelyi2014) with respect to corruption experience. (Note that concentration of the experience effect on those who paid a bribe does not make the having-requested-a-permit indicator less relevant, as the information which those requesting a permit convey includes not only the perception effect of having paid a bribe but also the social incidence of bribes, which seems to be the key aspect.)
Table 6 reports results for narrower experiences determining perceptions of performance in the criminal law area (being a victim versus being detained or abused by police), which hardly change with respect to the combined experience variable (i.e. being a victim, being detained/abused, or both). This is consistent with the interpretation that what drives the change in stated perceptions is proximity to the institution, rather than the outcome of the experience, which would predict different effects for victims than for detainees. Moreover, punishment of perpetrators is significantly related to a more positive perception among victims of crime (columns 3 and 4). This suggests that, in a similar fashion to what happened with paying bribes in administrative law, considering the corresponding experience (victimhood) weights both dimensions of performance (that is, the effect of punishment on perception and the probability of punishment).
The effect of specific experience on perceptions of institutional performance on criminal law, relying on complementary indicators of experience
Note that experience variables differ across columns.
Notes: Dependent variables are the corresponding indexes of institutional performance. OLS models with constants and country controls (omitted).
*p < 0.10, **p < 0.05, ***p < 0.01. Clustered standard errors are shown.
Third, the overall conclusion that the type of experience being considered in the paper correlates negatively with perceptions of institutional performance is also consistent with the results obtained by replicating the estimations with different WJP (2017–2018) data, as presented in Table 7. We observe again that having experienced problems and disputes is negatively associated to the perception of contract law institutions, while having litigated a conflict has a positive but much smaller impact. Moreover, both effects are also significant for the matched samples.
The effect of conflict and litigation experiences on perceptions of institutional performance, relying on the WJP 2017–2018 surveys
Notes: Equation in columns. The dependent variable is the index of institutional performance in contract law. OLS with constants and country controls (omitted). Full sample in columns 1 and 3; matched samples in 2 and 4.
*p < 0.10, **p < 0.05, ***p < 0.01. Clustered standard errors are shown in parentheses.
Another possibility to consider is that experience does change perception but does not improve the knowledge held or transmitted by respondents. For instance, stated perceptions might worsen when the legal institution works well. According to this hypothesis, respondents are sour as a consequence of having been treated impartially for the benefit of the general public. However, even if this possibility might drive some individual responses, it seems unlikely to explain the observed negative aggregate effects. For a start, contractual conflicts affect two parties with opposite outcomes and, therefore, opposite inclinations to fall prey to this bias. The argument sounds more sensible for criminals than for victims but, as we have just seen, both victims and criminals hold similar views and these are significantly more negative than those of the rest of the population. Moreover, the negative coefficient for those requesting a permit is not driven by those who were denied a permit (possibly for the sake of positive externalities) but by those who had paid a bribe (hardly a socially-beneficial event).
Lastly, experience might also improve perception simply because being in touch with legal institutions is costly for individuals. To make use of some institutions, respondents may have to pay professional services, fees, and taxes, in addition to spending time and effort. This is plausible. However, to the extent that such costs reflect social opportunity costs, experienced respondents would indeed be better informed. This might be particularly important if inexperienced individuals suffer the illusion that costs are zero. Furthermore, if such costs are salient, respondents' reactions should help reduce the bias toward political correction seemingly suffered by surveys of stated preferences on some types of public policies (e.g. Funk, Reference Funk2016).
3. Impact of considering experience on country scores and rankings
How experience alters country rankings
To analyze the impact of considering experience on country scores, I calculate the scores for each country in the four legal areas using the responses given both by the full sample of respondents (i_Law r with i = Contract, Criminal, Administrative, and Labor Law and r = 1, 2,…,102), and by different inexperienced and experienced subsamples – that is, i_Law r/Exp jr. (Summary statistics are offered in panels F and G of Table A.2 of the online Appendix.)
With respect to the corresponding indexes based on responses by the general public, correlation coefficients of average country scores based on their corresponding experienced subsamples are between 0.7269 for administrative law and 0.9922 for labor law (Table 8). Kendall's rank correlation coefficients are between 0.5411 for the index based on respondents who paid a bribe and 0.9123 for those with employment experience. Rankings are therefore more sensitive to methodological changes, which adds doubts as to the effects of using them to encourage competition and emulation.
Correlations between the pairs of country scores and country rankings calculated with responses given by the population and by the corresponding experienced subsamples of respondents
Notes: Column 3 presents Pearson correlation coefficients between country scores calculated with the full population for the index defined in column 1 and the corresponding subsample defined in column 2. Column 4 reports Kendall's rank correlations of country rankings. For all coefficients p < 0.01. Observations in parentheses.
Despite these positive correlations, country ranks drastically change when estimated with allegedly better-informed subsamples. Table 9 summarizes the changes that countries experience in the four main rankings. Rankings change more for contract law and less for labor law, with criminal and administrative law in between. Some countries move much more than average: the first quartile of countries concentrates 47% of the total movements. Country details are reported in Table A.3. The maximum number of changes is reached by Vietnam with 68.
Summary of country rank jumps and drops in the main four indexes
Why does the experience effect vary across countries?
To tentatively explore the determinants behind these changes and therefore identify how the effect of experience varies across countries, for each main performance index, I estimate a version of model (1) adding experience-country interactions (η ijr Expj*Country r) so that the estimated coefficients $\hat{\eta} _{ijr}$
measure the impact of each type of experience in each country. These impacts are then regressed against a set of country characteristics, estimating the following cross-country equations:
whose independent variables X r are country characteristics and whose dependent variables are the estimated experience-country interaction coefficients (descriptive statistics are presented in panel H of Table A.2). Assuming that particular subsamples are better informed, these experienced-country interaction coefficients, $\hat{\eta} _{ijr}$
, measure the informational advantage that experienced individuals have over the general population in a given country.
Table 10 presents the results of estimating OLS regressions with a set of independent variables selected by applying the procedure in Treisman (Reference Treisman2000).Footnote 8 For contract, criminal and administrative law, this informational advantage increases significantly – i.e. given that such advantage involves a more negative perception, it is significantly more negative – in countries with French legal origin. This may be the result of these countries having worse institutions, as argued in the literature pioneered by LLSV (La Porta et al., Reference La Porta, Lopez-De-Silanes, Shleifer and Vishny1997, Reference La Porta, Lopez-de-Silanes and Shleifer2008), but one may also conjecture that this type of statist action in legal matters worsens the (allegedly optimistic) perceptions of the inexperienced, perhaps because they become more inclined to rely on statutory law. A similar argument might account for the negative coefficients of regulatory ‘quality’: in countries with allegedly better (perhaps simply more intrusive) regulation, only the experienced individuals know its full costs.
Determinants of the ‘experience effect’ across countries for each legal area
Notes: OLS equations in columns. Independent variables are country characteristics; constants omitted [sources: Treisman (Reference Voigt2015) but for GDP (IMF)].
*p < 0.10, **p < 0.05, ***p < 0.01.
We also observe in Table 10 significantly positive coefficients for ethno-linguistic fractionalization,Footnote 9 which are consistent with the possibility that in more divided societies even the inexperienced have greater incentives to know, and therefore do know, more about the rule of law, a version of the ‘complacency’ argument proposed for French legal origin and regulatory quality. One would also expect that better politics, by increasing personal involvement, would reduce the advantage of the experienced respondents and therefore lead to positive coefficients. Indeed, we observe a slightly positive coefficient for political rights.
Robustness checks confirm these results. First, considering influential outliers through robust regression, following Li (Reference Li, Hoaglin, Mosteller and Tukey1985), modifies some coefficients and significance levels (such as fractionalization) but hardly affects most of them. Second, most coefficients change little when the model is estimated using as dependent variables the estimated coefficients of experience-country interactions in regressions run on individual data using matched samples, similar to those reported in Table 4.
As shown in column 4 of Table 10, results for labor law and employment experience depart from the pattern observed for the other three types of experience, which is consistent with the greater correlation shown for labor law between their general-population-based and experienced-sample-based indexes. Even if they are more in line with that pattern when the model is calculated using matched samples, the (fewer) variables behind the labor law index and employment experience seem to be less effective for capturing cross-country differences. In other words, performance indicators built with more underlying information generate stronger results.
Assessing how the effect of experience varies across countries by estimating experience-country interactions is similar to calculating the differences between (1) the average score obtained by each country from the responses given by allegedly better-informed subsamples minus (2) the score that each country obtains for the same index from the responses given by the whole sample. Correlation of both sets of scores is 91.80, 90.38, and 90.65%, for contract, criminal and administrative and labor law, respectively. Labor law again departs from this pattern, showing a significant but lower correlation (62.13%).
An advantage of using interaction coefficients is that they provide estimation errors (panel I of Table A.2 in the online Appendix contains their descriptive statistics), which decrease significantly with the percentage of experienced respondents in each country (Figure A.1 and panel B of Table A.4). However, this percentage shows no significant effect on the level of the interaction coefficient, as can be seen in panel A of Table A.4. (For labor law, the unmatched sample shows a positive coefficient, which again turns out non-significant in the matched sample.) Contrary to the contention in Michelson and Read (Reference Michelson, Read, Woo and Gallagher2011), based on observing Chinese surveys, the percentage of experienced individuals in each country does not therefore bias perceptions in a particular direction but does increase the variance, as might have been expected given that, for constant sample sizes, the percentage of experienced respondents is the size of the experienced subsample.
4. Concluding remarks
Individuals who have had experience with rule-of-law institutions evaluate them more negatively than the general population. Subject to the ignorability assumption, this ‘experience effect’ holds two important consequences. First, indexes based on the assessments of experienced individuals rather than those of the general population rank countries very differently. Second, based on available information, experience is more influential on stated perceptions than alternative variables often considered as proxies of knowledge, such as education, gender, wealth, or age.
To this extent, if confirmed and all other things being equal, these findings support focusing indexes of institutional performance more on experienced individuals. This could be achieved by selecting respondents, weighting the data toward experienced respondents or, at least, asking additional experience questions that would help in evaluating respondents' knowledge. These experience-related questions would also provide additional evidence for clarifying many key issues, such as the presence of possible confounders and the effects of different types of experience, including the role of recurrent and expected experience, as well as the tradeoffs of relying more on experienced individuals.
In this vein, a likely advantage of experience is that it does not rely on informational intermediaries, but is more factual, governed by transactions and less centralized. On the contrary, perceptions by the inexperienced might be reflecting stereotypes that experts and the media create and educated individuals help to disseminate. Hopefully, there is a lesser risk that experienced respondents suffer the biases that have been pointed out in the literature with respect to subjective evaluations, which might be tainted by theories, ideologies and prejudices (Voigt, Reference Voigt2013): e.g. basing their responses on countries' reputations (Treisman, Reference Treisman and Heywood2015); suffering a ‘halo effect’ that might lead them to consider economic growth as evidence that institutions are good (as hinted by Glaeser et al., Reference Glaeser, La Porta, Lopez-de-Silanes and Shleifer2004, and discussed by Kurtz and Schrank, Reference Kurtz and Schrank2007); or basing their assessments on the same observed variables (e.g. authoritarian political regimes) that theories had been claiming to be driving institutional quality (Treisman, Reference Treisman2007).
Acknowledgements
This work has greatly benefitted from valuable comments and exchanges with Albert Satorra, as well as Lee J. Alston, Mircea Epure, Marco Fabbri, Dean Lueck, Ethan Michelson, Dan Rockmore, Pablo T. Spiller, Gustavo F. Torrens, John Wallis, three anonymous referees, and participants at the World Justice Project's Scholars Conference at Duke University, the Ostrom Workshop at Indiana University, the Quality of Governance Institute at the University of Gothenburg, the Stockholm 23rd Conference of SIOE and the University of Vigo. I am grateful to Juan Botero, Alejandro Ponce, and the WJP for providing me with data. The project received support from the Spanish Government through grant ECO2017-85763-R and the Severo Ochoa Program for Centers of Excellence in R&D (SEV-2015-0563). Usual disclaimers apply.
