Efficiency versus effectiveness

Once the importance of judicial institutions for economic development has been made clear, it becomes necessary to shed some light on how these institutions work and what are the dimensions of their performance. If one's aim is to try to reform and improve the functioning of the judiciary, it is crucial to know through which channels it operates, the determinants of its performance, and the measures that best capture its variability. Staats et al. (Reference Staats, Bowler and Hiskey2005) identify five different levels through which one might try to measure judiciaries’ performance: (1) independence, (2) efficiency, (3) accessibility, (4) accountability, and (5) effectiveness. Each of these five aspects has been proven to be a key element of the proper functioning of judicial systems and thus to have an impact on countries’ economic performance. Yet they all refer to distinct dimensions and ought to be properly distinguished. It is not the aim of this work to survey the existing evidence showing the importance of all the aforementioned dimensions (see Voigt (Reference Voigt2016) for this purpose), but rather to focus our attention on some (rather controversial) aspects that, in our opinion, have not been handled with sufficient attention by previous literature. More specifically, we wish to clarify the difference between judicial efficiency and judicial effectiveness. These two dimensions have been the focus of the majority of scholarly work so far and have been very often misleadingly treated as if they were referring to the same phenomenon.

As mentioned above, the lexical, but yet theoretical, problem is mainly due to the fact that the term “efficiency” is used very often indistinctly to refer to both. However, (technical) efficiency and effectiveness (sometimes referred to as efficacy) indicators measure different aspects of judicial institutions’ performance. Efficiency, in line with an industrial conception of judicial activity as a production process, essentially refers to the optimal use of resources for attaining a given result. Efficacy refers to the ability of a judicial system to match the demands of justice. While both pertain to economic analysis, the former focuses only on the dispute resolution technology, while the latter looks at the equilibrium in the “market” of justice. Of course the two indicators may be correlated: an efficient system (producing the maximal output given its inputs) might be able to effectively satisfy the demand for justice (solve cases promptly). However this is not necessarily the case. Efficacy can be achieved even by a highly inefficient system devoting excessive resources to the judiciary. At the same time, an efficient yet undersized judiciary can turn out to be ineffective when trying to meet the current requirements of justice.

As a consequence, these metrics might not necessarily always move in the same direction. But even more importantly, their inner meaning and their consequences for stakeholders might be different. One can object that, to a certain extent, policy makers and entrepreneurs do not necessarily expect the same thing from judicial institutions. Policy makers and managers in the public sector are usually concerned more about the efficiency dimension of the justice sector, while entrepreneurs are mainly affected by courts’ effectiveness. Of course this proposition is not to be considered as an absolute statement, as we will explain below, but it is nonetheless necessary to disentangle this difference and to pick up the indicator that better captures the dimension of judicial performance that one needs to measure. Policy makers and bureaucrats are asked to employ the resources available optimally within the public budgets they manage. This puts the emphasis mainly on the efficiency dimension, as they are supposed to supply an efficient level of service with available endowments. This idea can be viewed according to a duality principle: goals might be achieved either by maximizing the production of justice with given resources (output-oriented efficiency) or by producing a given level of justice by minimizing the amount of resources employed (input-oriented efficiency). Within this framework, the production of justice usually refers to the number of cases disposed by a given judicial unit within a certain time period (usually a year). Given the labor-intensive nature of judicial activity, the productive factors are limited mainly to the labor force (magistrates and administrative workers), although also physical capital (IT and buildings) might be taken into consideration. Bureaucrats should thus optimize the number of solved cases, conditional on the public resources they are asked to manage.

The problem derives from the fact that what we have defined as a well-performing judiciary in the eyes of a bureaucrat (an efficient judiciary) might not necessarily be equally beneficial to an entrepreneur. This is because, as we have seen in section 2, economic agents are affected primarily by the effectiveness of judicial institutions rather than their efficiency. In other words, what really matters is the capacity of the judicial system to respond promptly (quickly) to the extant demand for justice. Unlike efficiency, measures of effectiveness do not take into account how the productive factors required to supply justice are employed. Accordingly, effectiveness is measured by the speed in solving cases and thus granting contract enforcement and the protection of property rights. As a consequence, entrepreneurs will benefit more from a fast justice system rather than from an efficient one. This is because economic agents do not primarily care about the optimal use of public resources, but rather that disputes are brought to a conclusion as soon as possible, thus limiting uncertainty and post-contractual opportunistic behaviors.

Of course things are not so absolute and, to a certain degree, one can say both that bureaucrats will be interested in effectiveness and entrepreneurs in efficiency. Policy makers are usually elected and thus subject to voters’ satisfaction. Accordingly, they cannot focus solely on the efficiency dimension while neglecting the judiciary's reaction to the demands of justice in terms of delay. On the whole, policy makers will be incentivized to find a balanced solution between guaranteeing a reasonable outcome (effectiveness) with a sound use of public resources (efficiency).

Also entrepreneurs cannot completely disengage from the efficiency dimension, as this could have downsides in terms of opportunity costs vis-à-vis other services and forms of public intervention. For example, one might imagine that in order to guarantee a very effective (fast) delivery of justice, the judiciary could employ an unreasonably excessive level of resources. In this case we might have an effective yet highly inefficient judicial system. It is fairly intuitive to see that, while beneficial to litigation, a similar allocation of resources at the expense of other sectors – e.g. education, health, or infrastructure – would be detrimental in the long run for a nation's economic growth. The positive effect on economic transactions derived from an excessive, and thus inefficient, allocation of resources to the judiciary could be offset by shortages in other fields.

Making this clarification is of the utmost importance from a public policy perspective, especially at a time when governments all around the world have to struggle with public finance “austerity.” If one can agree that long delays might be a symptom of an ineffective judiciary, this is not necessarily caused by an inefficient use of public resources. In this respect, different measures might not necessarily yield similar results: a given judiciary might be very effective and thus solve cases quickly by meeting the demand for justice, but at the same time turn out to be inefficient because oversized. This means that a given judiciary, although providing a “good” service to society, is not yet acting efficiently, thus yielding a waste of public resources. As a consequence, any policy attempt to reform the judiciary should carefully identify the issue at stake and the index that best captures it.

Different indices

With some exceptions (Ippoliti et al., Reference Ippoliti, Melcarne and Ramello2015), when trying to estimate the impact of judicial institutions’ performance on the economy, previous literature has mainly employed effectiveness measures, rather than efficiency ones, albeit often still referring to them as judicial (in)efficiency (Chemin, Reference Chemin2009, Reference Chemin2012; Jappelli et al., Reference Jappelli, Pagano and Bianco2005; Safavian and Sharma, Reference Safavian and Sharma2007; Visaria, Reference Visaria2009). But what are we really capturing with judicial delay? And to what extent do measures of effectiveness also capture the efficiency dimension?

The main indicator of effectiveness used in literature is judicial delay (or disposition time). This metric simply measures the time needed for a case to be solved from the moment it is filed by the plaintiff. Judicial delay (JD) has the merit of being an extremely intuitive metric, as it is measured usually in days or years. The way this index is calculated might vary depending on the information available to the Ministry of Justice's statistical offices. Ideally, one would like to know for every case c, the date of start (τ⁰_c) and the date of conclusion (τ¹_c) in order to estimate the delay of case c as the number of days between these two events. Accordingly, for a given country/court i in a given time t the expected delay would be:

$$\begin{equation*}JD_{i,t} = {1 \over c}\mathop \sum \limits_{c = 1}^n (\tau _c^1 - \tau _c^0 ),\end{equation*}$$

with c = 1,. . .,n, being the number of cases dealt in the considered timespan. However, it is often very hard for both developing and developed countries to keep a record of this information for massive volumes of cases. As a consequence, statistical offices usually employ other metrics based on aggregated data. For example, the European Commission for the Efficiency of Justice (CEPEJ) has promoted the use of standardized methods for indirectly estimating delay among member states (CEPEJ, 2014). Given the number of pending, incoming, and solved cases in a judicial organization i in a time period t,

\begin{equation*}JD_{i,t} = \displaystyle{{\,pending_{i,t - 1} + pending_{i,t}} \over {incoming_{i,t} + solved_{i,t}}}.\end{equation*}

Another measure of judicial effectiveness often used by governmental agencies trying to evaluate judicial performance is the clearance rate (CR). This indicator tries to capture judiciaries’ ability to react to the current demand for justice and it is measured as the ratio of solved cases to incoming ones:

$$\begin{equation*}CD_{i,t} = \displaystyle{{solved_{i,t}} \over {incoming_{i,t}}}\end{equation*}$$

(CEPEJ, 2014). In fact CR has a slightly more complicated interpretation of delay. For CR = 1, the judiciary solves all the lawsuits filed in the given timespan. According to this metric, CR < 1 implies the inability to fully handle incoming cases, thus increasing the stock of pending lawsuits. On the contrary, CR > 1 means that the judicial system is not only able to fully satisfy the current demand, but it might also solve a share of previously pending cases (if they exist). The limitation of both these measures is that neither of them takes explicitly into account the inputs used. Hence, we cannot really grasp from a microeconomics viewpoint whether the resources are used in an efficient way. For example, a CR > 1 might be explained on the basis of either an efficient or an oversized, inefficient judiciary.

These shortcomings can be redressed by estimating technical efficiency scores (TE), a much more complex indicator obtained by applying data envelopment analysis, a non-parametric technique relying upon linear programming and applied widely to many different contexts in order to compute the technical efficiency of specific “decision making units” (Ippoliti et al., Reference Ippoliti, Melcarne and Ramello2015). In its output-oriented version this index measures the ability of a given judicial unit to maximize the number of cases disposed with given endowments: both labor (number of judges or staff workers) and capital (number of computers or other physical endowments) might be considered. Efficiency scores (TE ∈ [0,1]) are thus estimated as the radial distances of each unit from a deterministic production frontier that represents the ideal efficiency. Interpretation of these measures works as follows: in an output-oriented specification, if a given judicial unit obtains a TE = 0.7, this implies that, with given inputs, it is only able to solve 70% of the cases that an otherwise ideally efficient unit would solve.

Some empirical evidence

We now propose some very basic empirical evidence to demonstrate how the previously described metrics of judicial efficiency and effectiveness might not necessarily measure the same dimensions of judicial performance. In this perspective we use data supplied by the European Commission for the Efficiency of Justice (CEPEJ, 2014) and the World Bank's Doing Business Program (World Bank, 2016). Unfortunately, due to scarcity of data, it is not possible to estimate technical efficiency scores for countries other than those participating in the CEPEJ Commission. Our measure of judicial efficiency (TE) is estimated according to the methodology proposed by Melcarne and Ramello (Reference Melcarne and Ramello2015) and using data from CEPEJ (2014). We then employ two measures of judicial effectiveness: judicial delay (JD) and clearance rate (CR). Our measure of judicial delay (JD) is the “time-enforcing contract” variable supplied by World Bank (Reference Voigt2016), whose constructive methodology was originally conceived by Djankov et al. (Reference Djankov, La Porta, Lopez-de Silanes and Shleifer2003).Footnote ² The clearance rate (CR) values are the same as those estimated in Ippoliti et al. (Reference Ippoliti, Melcarne and Ramello2015) using data from CEPEJ (2014).

The first thing we wish to highlight is the empirical relationship between measures of judicial efficiency and measures of effectiveness. Trying to establish a causal link between these two metrics is no mean feat. In this paper, we avoid any causal interpretation of our results. On the contrary we simply investigate to what extent these metrics are significantly correlated and, if so, whether they carry the expected sign. Table 1 presents pairwise correlation coefficients between different measures of judicial efficiency and effectiveness. We also employ partial correlation in order to measure the degree of association between efficiency and effectiveness, with the effect of a set of controlling random variables removed.

Table 1.

Judicial efficiency versus effectiveness, effectiveness versus effectiveness: correlation coefficients

Notes: TE is a measure of technical efficiency extracted from Ippoliti et al. (Reference Ippoliti, Melcarne and Ramello2015). JD is the “time-enforcing contract” variable supplied by World Bank (2016). CR is an indicator of clearance rate estimated with data extracted from CEPEJ (2014). Controls#1 include GDP per capita and government expenditures (IMF's World Economic Outlook). Controls#2 include Rule of Law index, education and life expectancy (World Bank's Governance Indicators).

As emerges from column (1) of Table 1, JD and TE are only weakly correlated (p-value = 0.0549), although with the expected (negative) sign. This means that a slow judiciary (high values of JD) tends to be also less efficient (low values of TE). In columns (2) and (3) we present the estimated coefficient of partial correlation between JD and TE, while controlling for the effect of other variables. In column (2), the partial correlation coefficient turns out to be even weaker and barely significant at the 10% level once we include a first set of controls: GDP per capita and level of government revenue (Controls#1). No significant correlation can be found in column (3) where a second set of controls is also used: rule of law, average education and life expectancy (Controls#2).

From Table 1, we cannot conclude that an efficient judiciary will also be a timely one in solving cases. Figure 1 might further help us understanding how these two indices refer to different dimensions of judicial performance and how this relation is a non-trivial issue. As mentioned above, efficiency provides a static measure comparing inputs and outputs, while effectiveness refers to the equilibrium between the demand and supply of justice. The case of Italy is emblematic. Despite being over 90% efficient with respect to the ideal production frontier, the Italian judiciary turns out to be the second slowest (after Slovenia), with more than three times the disposition time than a country like France. The reason behind this apparent inconsistency seems to be simply that the Italian judiciary turns out to be under-dimensioned with respect to the demand for justice it has to face. As a consequence, despite employing (relatively) efficiently the resources available, courts are not able to react effectively to such demand, and thus solving cases takes more time. Hence, a immediate policy implication might be in favor of simply shifting the supply curve upward, that is to say, devoting a larger share of the public budget to the judicial system in order to hire more judges. Yet a deeper analysis shows a much more complex picture and suggests a more cautious use of public resources. In fact, for the sake of clarity, it must be said that public resources devoted to the Italian judiciary are not so scarce and one might claim that it is possible to allocate them better. For example Italian judges are, on average, paid more than their European colleagues.Footnote ³ In addition, the efficiency of Italian courts might benefit from different organization in order to better exploit economies of scale (Forsund et al., Reference Forsund, Kittelsen, Melcarne and Ramello2018).

Figure 1.

Judicial efficiency versus effectiveness.

All in all, efficiency might still have a role in improving effectiveness without increasing public expenditure.Footnote ⁴ However, to a large extent, long delays in a country like Italy seem mostly to originate in two concurring factors that, by impacting on the demand side, lead to the congestion of the judiciary. First, the highly skewed demand for justice (in terms of the number of lawsuits filed) is the immediate consequence of an oversized number of practicing attorneys (Buonanno and Galizzi, Reference Buonanno and Galizzi2014). Among European countries, Italy has the leadership in lawyer density, with around 290 practicing attorneys per 100,000 inhabitants, almost double that of Germany (168), more than three times that of France (76) and ten times that of England (CEPEJ, 2014). In addition, most of them are allowed to litigate at all the hierarchical levels of the national jurisdiction, that is, all the way up to the Supreme Court (Corte di Cassazione). Consequently, most cases are “automatically” appealed (regardless of the merits) and thus litigation usually reaches the court of last resort, which is less usual in other countries. Second, Italian procedural law also seems to be unable to restrain frivolous lawsuits. Given the mandatory jurisdiction of Italian judges, there is no filter limiting non-meritorious cases to access the judicial system. All in all, the scarce effectiveness of the Italian judiciary seems in itself to be the outcome of an institutional imperfection artificially boosting an excessive demand for justice, whose solution might reasonably rely on pushing demand downward rather than shifting supply upward.

The Italian example moreover shows that, while the pursuit of judicial efficiency can be clearly promoted by working on dispute resolution technology, effectiveness is a much more sophisticated target that in general refers to a broader picture. Accordingly, it should be handled on a case-by-case basis and with a deeper understanding of the real practice represented by official statistics. In a similar framework of apparent low effectiveness, the policy implications might be opposite to the Italian case. If, for example, low effectiveness depends on an inefficient judiciary, then limiting demand and thus restricting access to justice could also prevent meritorious cases from entering the courts. Likewise, as we learned from section 2, while in the Italian case less judicial delay would reasonably increase certainty and in turn serve economic activity, in the latter case a similar policy in a different setting would not help the reduction of uncertainty and consequently would not foster economic development.

Once it is highlighted that measures of judicial effectiveness and efficiency might not necessarily be correlated in a significant fashion, we now try to investigate whether at least different measures of judicial effectiveness are correlated. In columns (4)–(6) of Table 1, we show both pairwise and partial correlations between judicial delay (JD) and clearance rates (CR) of European national judiciaries. Surprisingly, under no specification are we able to find a statistically significant correlation. Similar results suggest that, when trying to address the issue of judicial performance from a policy perspective, scholars need to be extremely careful and possibly try to explore the causal link between judicial institutions and the economy with the support of various measures.

Some empirical evidence

In this section we wish to provide some preliminary empirical evidence regarding the possibility that qualitative and quantitative indicators of judicial performance might be substitutes. To achieve this, we employ data supplied by the World Bank and explore the correlation between proxies of quantitative and qualitative justice: respectively in terms of judicial delay (JD – World Bank's Doing Business) and rule of law (ROL – World Bank's Governance Indicators). In order to supply further evidence, we also employ another measure of judicial quality, JQ, a composite index provided by World Bank's Doing Business project and based on objective features of judicial systems.Footnote ⁵ We are able to observe these variables for a sample of 187 countries. As in section 3, our empirical analysis avoids any causal interpretation of the interplay between the quality and the quantity of justice. Rather, we just provide pairwise and partial correlation coefficients.

Our very preliminary results suggest that the alleged trade-off between quality and quantity seems not to bind. In columns (1) and (4) of Table 2, the pairwise correlation coefficients are highly significant and with an opposite sign, regardless of the measure of judicial quality used. That is, slow/ineffective judiciaries are also the ones experiencing poor qualitative indices. In columns (2) and (3) we estimate the partial correlation coefficient between JD and ROL with the effect of a set of controlling random variables removed. In column (2) we include a set of basic financial controls (GDP per capita, government expenditures, investments), while in column (3) we also account for the legal origin of a country and the procedural formalism of its judicial system. Once controlling for these factors, (partial) correlation coefficients remain negative (no trade-off), but turn out not to be statistically significant: depending on the proxy of judicial quality used we obtain a p-value of 0.2268 in column (3) and 0.1265 in column (6). These preliminary estimations might suggest some arguments against the quality–quantity trade-off. Judiciaries’ celerity in solving cases does not appear to be negatively correlated with respect for the rule of law or with respect for a general sentiment of justice.

Table 2.

Judicial quality versus quantity: correlation coefficients

Notes: JD is the “time-enforcing contract” variable supplied by World Bank (2016). ROL is the rule of law index by the World Bank's Governance Indicators. JQ is a composite index of judicial quality supplied by World Bank (2016). Controls#1 include: GDP per capita, government expenditures, foreign investments and demographic changes (IMF's World Economic Outlook). Controls#2 include: dummy for belonging to the Common Law tradition and the number of judicial procedures necessary to dispose a case as supplied by World Bank (2016).

Figure 2 plots values of JD and ROL and seems to suggest a slightly more complex situation. For simplicity we divide the picture into four quadrants using the median values of JD and ROL. In the upper-right quadrant we find “ideal” countries. These judiciaries are at the same time effective (lower delays) and able to grant more respect for the rule of law. However, in the lower-right quadrant we can equally see another cluster of countries in which there seems to be a trade-off between delay and rule of law. In these (mainly developing) countries we can witness a relatively fast judiciary but at the cost of poor respect for the rule of law. While many factors could in fact hide behind this correlation, it could potentially be problematic: in these countries, quantitative measures of judicial institutions’ performance (like delay) might hide substantial violations of the rule of law. As a consequence, empirical analyses failing to account for this issue could produce biased estimates of the impact of judicial institutions’ performance on economic development.

Figure 2.

Rule of law and judicial delay.