Introduction
The efficiency of public procurements plays a significant role in the EU Member States, as such procurements made up around 20% of GDP in the European Union in the period 2009–2011.1 Hitherto, public procurements have received substantial attention from researchers aiming to improve both the efficiency of public procurements and the measurement scale.Reference Bovaird2–Reference Jovanovic, Joksimovic and Milosavljević4 Alongside the scholarly efforts, public authorities, as well as national and international agencies have also been active in the field. For instance, the European Commission regularly conducts a six-item Single Market Scoreboard analysis of the efficiency of public procurements within European countries and provides various recommendations for improvements.5 However, these and other metrics are biased with subjectivity and suffer from omissions that could mislead policy makers, decision makers, and other interested parties. Consequently, composite indices based on the holistic, international ranking of public procurement efficiency have been out of the scope of current research and policies. Bearing in mind the importance of composite indices for research nowadays,Reference Chabova6 any lack of evidence in this field is a strong motivation for an in-depth analysis.
This study aims to fill the lacuna in the extant body of knowledge by ranking European countries in an unambiguous manner. Measuring the entities in order to evaluate, compare, and rank them is an important cause and objective.Reference Dobrota, Martic, Bulajic and Jeremic7 ‘A natural instinct for humans is to make comparisons, […] These comparisons are important because rank position effects [sic] an individual’s beliefs about themselves and their abilities’.Reference Murphy and Weinhardt8 Likewise, the objectivity in measuring is also important. Any subjectivity in the creation of composite indicators may affect the measurement to a great extent.Reference Maricic and Kostic-Stankovic9 Consequently, the main purpose of this study is to measure in an objective manner, using the Composite I-Distance Indicator (CIDI) methodology,Reference Dobrota, Bulajic, Bornmann and Jeremic10 the efficiency of public procurements, progress in procurement practices, differences among European countries, and development potentials.
To the best of the authors’ knowledge, a study of this kind has never been conducted before. First, there are only a few scholarly attempts to assess and rank public procurement policies and effects across the European continent, but their emphasis is not holistic.Reference Hessami11 Second, no study attempts to challenge the current public procurement policy scoreboard set by the European Commission. Moreover, there is a paucity of quantitative studies of any kind in the field.Reference Pîrvu and Bâldan12, Reference Brianzoni, Coppier and Michetti13
The remainder of this article is structured in the following order: The next section draws on theory, policies, and existing public procurement efficiency measures. The third section elaborates on the methodology used in this paper, placing emphasis on the CIDI methodology. The fourth section presents the results of an international analysis, thoroughly discusses the results, and puts a focus on strengths, limitations, implications, and recommendations of the study. Finally, the fifth section is reserved for concluding remarks.
Public Procurement Efficiency Measures
Public procurements are public agencies’ purchases of goods and/or services from an outside body.Reference Arrowsmith14 The primary objective of public agencies is ‘to obtain goods and services of the required quality at a competitive price.’Reference Allen and Tommasi15
Public procurements have received strategic importance in the EU by the 2011 Single Market, which has put a new emphasis on liberalized and integrated public procurements.Reference Bovis16 The rationale for a strategic focus on public procurements is clear – they can serve for numerous policy ends. To name a few, public procurements are important for the convergence and integration of the European market,Reference Sanchez-Graells17 they shape economic development,Reference Peck and Cabras18 stimulate the development and diffusion of innovations,Reference Edler and Georghiou19–Reference Vecchiato and Roveda22 foster sustainability,Reference Uttam and Roos23 and affect market favouritism towards large, domestic, or any other classes of suppliers.Reference Van Long and Stähler24
In spite of supranational legal and organizational stimulus, the value and importance of public procurements significantly differs across the European continent – accounting for 10.5% in Cyprus to 30.6% in the Netherlands.Reference Schulten, Alsos, Burgess and Pedersen25 Still, there is no country where they are viewed merely as a technical purchase of services and goods; instead, they generally serve as a nationwide resource allocator and a means of capital transfer.Reference Ayhan and Üstüner26 Not surprisingly, public procurement systems have been a focal point of regulators, policy makers, decision makers, and other interested parties. The system is heavily regulated and monitored, both nationally and through international agencies.Reference Mukhopadhyay27
Most commonly, public sector buyers are forced by regulation to use fixed-price open auctions in order to secure a competitive bidding.Reference Tadelis28 The legacy of public sector contracting is to adopt an arms-length approach, foster market mechanisms, and avoid any partnering.Reference Caldwell, Walker, Harland, Knight, Zheng and Wakeley29 Procurement procedures with one or only few bidders can raise the question of efficiency and possibly of corruptive behaviour. Academic voices are not unanimous in promoting competitiveness by simply increasing the number of bidders. By challenging public and private-sector behaviour patterns in procurements, they argue that more suppliers means higher transaction costs and greater inefficiency.Reference Cox and Furlong30 Accordingly, the number of bidders seems to be a solid measure of the competitive efficiency of public procurements, but the role of such indicators should always be viewed with a dose of professional scepticism.
Public procurements should be shaped in order to enable economies of scale, which can only be achieved with the adequate level of procurement aggregation. This implies that procurements should not be heavily decentralized and departmentalized. However, this is probably the most ambiguous determinant of public procurement efficiency. Even though aggregation can boost economies of scale, EU Member States are advocated to decentralize public procurements so that local small and medium-sized enterprises can more easily access tender procedures and benefit from the stability, reliability, certainty of payment, and marketing opportunities.Reference Peck and Cabras18 Aggregation is important for efficiency, but should not be isolated as the only efficiency factor.
Award criteria are another important driver of the efficiency of public procurement. The lowest price is most often used as a single criterion for public purchases.Reference Bergman and Lundberg31 However, the lowest procurement price can jeopardize the quality of purchased goods, services and works. As indicated in Verdaux,Reference Verdeaux32 the EU is promoting a combination of price and quality as award criteria in public procurement. This practice is endorsed by scholars who tend to provide theoretical and empirical arguments for the diversified usage of different criteria in the procurement decision-making process.Reference Bajari and Lewis33 These arguments are sometimes referred to as ‘economies of quality’, as they can deliver the economic, environmental, and social benefits of sustainable development.Reference Sonnino34
The procedure of tendering for goods, services, and works is supposed to be as transparent as possible, in order to minimize information asymmetries. Evenett and HoekmanReference Evenett and Hoekman35 explain that ‘opaque procurement practices may result from either administrative inefficiencies, the absence of hard budget constraints and oversight by the Ministry of Finance, or rent-seeking and corruption’. A transparent procurement regime improves procedures, limits manoeuvring space for corruption, and increases the potential for the development of control mechanisms. However, this might not always be the case. For instance, LenneforsReference Lennerfors36 finds that in a corruptive environment, purchaser and supplier are more informed parties than regulators (i.e. courts), and transparency, in this case, will not lead to the efficiency.
Methodology
Single Market Scoreboard for Public Procurement Efficiency Measurement
A paucity of efforts has been undertaken so far with regards to the evaluation of public procurement efficiency among European countries. Strand et al. Reference Strand, Ramada, Canton, Muller, Devnani, Bas and Dvergsdal37 examined various factors, such as patterns of use, cost and effectiveness of public procurements in the EU. Nevertheless, this study does not provide any aggregated scores for the analysed countries. Milosavljević et al. Reference Milosavljević, Milanović and Benkovic38 used the Data Envelopment Analysis approach to estimate the input–output efficiency of public procurements among European countries, thus reflecting on the relationship between resources invested and the effects of public procurement systems. Although the study ranks countries, the ranking is biased by the selection of inputs.
The most prominent composite metrics for the evaluation is proposed by the European Commission Single Market Scoreboard.5 The Scoreboard incorporates six indicators, listed and explained as follows.
(1) One bidder – measures the proportion of contract awards with just one bidder. Owing to different reporting patterns, framework agreements must be excluded. A higher value of this indicator leads to better value for money, offering more options to public buyers.
(2) No calls for bids – measures the proportion of procurement procedures that were negotiated with one selected company without a call for the tender being announced. Calling for bids contributes to the transparency of the bidder selection process and encourages competition.
(3) Aggregation – measures the proportion of procurement procedures with more than one public buyer. An aggregated network of buyers provides better prices and fosters the exchange of know-how.
(4) Award criteria – measures the proportion of procurement procedures where the lowest price was the only criterion for the bidder selection process. Excessive reliance on price indicates that opportunity for finding better quality criteria is being missed.
(5) Decision speed – measures the time between the deadline for receipt of offers and the award of the contract. Slow decision-making leads to uncertainty and increases the cost of procurement procedures.
(6) Reporting quality – measures the proportion of contract awards without information about the awarded contract’s value. This indicator excludes framework agreements. A higher value of this indicator represents better bidding decision-making and citizens’ insights into the way their money is spent.
Using the biased weights and thresholds that are mostly based on expert judgements, the scoreboard delivers a ranking for 30 European countries and categorizes them into three tiers (above average, average, and below average). However, this scoreboard is subject to many omissions. A notorious one is related to the use of subjective weights and thresholds, which creates numerous biases in the interpretation of results. For instance, the highest weight (tripled score) is given to two indicators – Number of bidders and Number of calls for bids, assumingly due to their indication of competitiveness efficiency. The second two indicators – Aggregation and Award criteria – are doubled, as they reflect on the system level efficiency. Finally, Decision speed and Reporting quality are seen as the representation of technical efficiency with minimal weights.
Composite I-distance Indicator
A significant number of various publicly known ranking methodologies, regardless of their subject area, have been criticized because of their obvious bias and subjectivity. This problem has been successfully addressed by some authors who propose the I-distanceReference Išljamović, Jeremić, Petrović and Radojičić39–Reference Jeremic, Bulajic, Martic and Radojicic41 and CIDI (Composite I-distance Indicator) methodologies.Reference Dobrota, Bulajic, Bornmann and Jeremic10, Reference Zornic, Bornmann, Maricic, Markovic, Martic and Jeremic42–Reference Dobrota and Jeremic44
The main idea of the CIDI methodology is to construct a composite indicator that is unbiased and impartial. CIDIs are formed using the additive aggregation method, but instead of using biased weights, the used weights are created in an objective manner. Namely, most of the ranking methodologies use weights that are assigned by experts and which are formed according to their subjective opinion on the importance of individual indicators. Weights created by CIDI are based on the I-distance methodology, which has the specific property of sorting indicators in their order of importance for a total ranking.
I-distance relies on calculating the mutual distances between the processed entities and with regard to the one entity that is set as the reference.Reference Dobrota, Jeremic and Markovic45–Reference Seke, Petrovic, Jeremic, Vukmirovic, Kilibarda and Martic48 The I-Distance D 2 is calculated by the following formula:Reference Ivanovic49, Reference Ivanovic and Fanchette50
$$D^{2} (r,s){\equals}\mathop{\sum}\limits_{i{\equals}1}^k {{{d_{i}^{2} (r,s)} \over {\sigma _{{i{\equals}1}}^{2} }}} \prod\nolimits_{j{\equals}1}^{i{\equals}1} {(1{\minus}r_{{ji.12...j{\minus}1}}^{2} )} $$
where d i(r,s)=x ir−x is is the distance between the values of the variable X t(i = 1,2,…k) for entities 
$$e_{r} {\equals}\left( {x_{{1r}} ,x_{{2r}} ,...x_{{kr}} } \right)$$ and 
$$e_{s} {\equals}\left( {x_{{1s}} ,x_{{2s}} ,...x_{{ks}} } \right)$$, 
$$\sigma _{i} $$ is the standard deviation of X i, and 
$$r_{{ji.12...j{\minus}1}} $$ is a partial coefficient of the correlation between X i and X j, 
$$\left( {j\,\lt\,i} \right)$$.
After calculating the I-distance, it is possible to establish the correlations between its values and each of the individual compounding indicators. Correlations are used because of a particular feature of the I-distance method: it can determine the relevance of input indicators.Reference Dobrota, Martic, Bulajic and Jeremic7, Reference Dobrota, Bulajic, Bornmann and Jeremic10 Weights are formed by weighting empirical correlations: values of correlations are divided by the sum of correlations. The final sum equals 1, thus creating a novel appropriate weighting system.Reference Dobrota, Bulajic, Bornmann and Jeremic10
$$w_{i} {\equals}{{r_{i} } \over {\mathop{\sum}\limits_{j{\equals}1}^k {r_{j} } }}$$
where 
$$r_{i} \,\,\left( {i{\equals}1,2,...k} \right)$$ is a Pearson correlation between X i and the I-distance value. One of the most important features of the CIDI thus defined is that it is widely applicable to any ranking methodology if the goal is to overcome the negative influence of bias.
Results and Discussion
Country Rankings
The first step in creating a new original ranking of countries in the field of public procurements is the normalization of the data.Reference Nardo, Saisana, Saltelli, Tarantola, Hoffman and Giovannini51 The six indicators used to measure public procurement efficiency are measured differently and even have different directionalities. The directionality of the original indicators One bidder, No calls for bids, Award criteria, and Decision Speed is such that the lower the value the better the placement, while the situation for Aggregation and Reporting quality is the opposite. All indicators are converted to the same directionality and normalized using the Min-Max methodology.Reference Nardo, Saisana, Saltelli, Tarantola, Hoffman and Giovannini51
As described above, the new unbiased and impartial weights are calculated using the CIDI methodology. They are given in Table 1, in comparison with the original weights defined by experts.
Individual indicators, original weights and unbiased CIDI weights.
As displayed in Table 1, the largest weight of 21.5% is obtained for the indicator Reporting quality. Next in line is One bidder with 19%, followed by No calls for bids at 16.8%. The indicators Aggregation, Award criteria and Decision Speed are weighted very similarly according to CIDI.
It is instructive to compare and discuss the differences in the original weights and weights obtained according to the CIDI methodology. As mentioned above, original weights are not given in percent. They indicate how many times one indicator is more significant than the other. Original weights are assigned to indicators by experts. The situation with the objective and data-driven weights obtained by CIDI is completely different. For example, as opposed to One bidder being three times more significant than Reporting quality, the latter is even more significant than the former given that it has a larger CIDI weight. It is important to note that the values for the original weights are not supported or reinforced by the literature review. In other words, there is no theoretical background to uphold the given weights, while CIDI weights are, as explained in the methodology section, data-driven.
The ranking results obtained by the CIDI methodology are given in Table 2. These results are compared with the original ranks (groups) from the Single Market Scoreboard.
CIDI scores and country ranks compared with original groups.
For the purpose of higher comparability of results, countries were grouped into three tiers (ranks 1–10, 11–20 and 21–30) according to the CIDI rankings. The tiers are provisional and formed exclusively for the purpose of comparing them with the original rankings. The results are displayed in Figure 1.
Illustrative comparison of CIDI and original rankings.
The results indicate that the best ranked country according to CIDI methodology is Iceland, followed by Sweden, the Netherlands, the United Kingdom and Denmark. Iceland scored 85.7 in the overall CIDI values, mainly because of its individual scores. Iceland is ranked first by the indicators One bidder and Decision speed, and second according to the highest-weighted indicator Reporting quality. Sweden, for example, is ranked highest according to No calls for bids and Reporting quality, but has a very low value for Aggregation, etc.
The results presented in this study emphasize the fact that European countries are very different in terms of public procurement efficiencies. This study is in line with other studies in confirming that Northern and Western European countries, particularly Sweden (Rank 2), hold a high rank in public procurement efficiency.Reference Milosavljević, Milanović and Benkovic38 The Swedish system should be considered as a role model. Similar conclusions could be applied to the United Kingdom (Rank 4). However, the objectivity of the approach used in this study clears a path for countries such as Iceland (Rank 1) and the Netherlands (Rank 3) to be considered as paragons of public procurement efficiency. On the other side, countries that joined the EU in this century are ranked low, and none of them holds a position in the first half, excluding Malta. On many factors of this phenomenon can be speculated – from the rule of law to economic development. An interesting explanation may be that public procurement officials in post-communist countries have ‘high bargaining power’ and foster informal practices.Reference Grødeland and Aasland52 Therefore, they can raise de jure competitiveness in public procurements, which is highly important in the scoreboard matrix. Still, a more objective view through the CIDI approach shows real flaws in their public procurement systems.
Strengths, Limitations and Further Recommendations
The first strength of the methodology adopted in this paper is the use of a quantitative and comprehensive approach in the study of public procurement efficiency. The majority of publications in this field are either theoreticalReference Resh and Marvel53 or case-based and partial,Reference Patrucco, Luzzini and Ronchi54 which makes comparisons and rankings rather difficult, if not impossible. However, this also creates a limitation. Being purely quantitative, the study lacks any qualitative assessments that could contribute to an in-depth understanding of the nature and causes of differences in the public procurement practices among European countries.
The second significant strength is the use of the most important variables for the assessment of public procurement efficiencies. All variables are complex and capture different aspects of the efficiency, such as competitiveness, transparency, and comprehensiveness of the explored phenomena. The study uses accurate data from a reliable source. However, some critical aspects are left out of its scope. For instance, a major flaw is the exclusion of corruption from the metrics. The inclusion of corruption in public procurement measurements has been widely advocated in the literature.5, Reference Hessami11, Reference Thai55–Reference Dastidar and Mukherjee57 AuriolReference Auriol58 defines two major types of corruptive behaviour: (1) Capture – a case of active bribery; and (2) Extortion – when a firm complies with the demand for a bribe. Although they do not affect taxpayers in the same manner, each form of corruption distorts public procurement institutions and mechanics. In the context of the EU, anti-corruption efforts are to some extent presented in the European Act on Public Procurement. Although it is not unanimously seen as the best facilitator of efficiency and a boost for competitiveness, it strives to eliminate corruptive behaviour.Reference Lennerfors36 This major omission is, nonetheless, difficult to capture. ChabovaReference Chabova6 states that corruption is a ‘clandestine activity, very often unreported, therefore there is no official statistics on the number of corruption cases’. Accordingly, inclusion of corruption in the analysis is a strong recommendation for any further research in this area.
The third and most important contribution of this paper is the use of unbiased weights and evidence-based methodologies for the assessment of public procurement efficiency among 30 European countries. Unlike other rankings,Reference Milosavljević, Milanović and Benkovic38 composite indices5 and methodologies for improvements in public procurements,Reference Dalen, Moen and Riis59 which are subject to myriad different judgements and inconsistencies, this methodology clears the path for an objective and quantitative assessment of the countries’ public procurement efficiency rating. Thus, the ranking resulting from this analysis could be used for an unbiased rating of public procurement systems among European countries. From the methodological point of view, further research directions of this study could address the uncertainty and sensitivity analysis of the results.Reference Saisana, Saltelli and Tarantola60, Reference Saisana and D’Hombres61 Namely, the ranks of entities could be somewhat unstable, depending on the operating values of the weights for each of the individual indicators. Weights could be susceptible to variations depending on their relative contributions.Reference Saisana, d’Hombres and Saltelli62 According to the values of the relative contributions and their standard deviations,Reference Saisana, d’Hombres and Saltelli62 it is manageable to perform an uncertainty and sensitivity analysis of the CIDI results, and potentially compare them with the uncertainty of the original results.
As mentioned before, the study uses reliable sources, but covers a single period of time, making the analysis cross-sectional. It is highly advisable to use time series in further studies, which would capture the evolution of public procurement systems. In addition, the study encompasses only those European countries with available data on public procurement efficiency variables. Any inclusion of new countries into the analysis would open new aspects for discussion.
Conclusions and Implications
Public procurements are high on the agenda of academics and practitioners. Regulators and other policy makers in Europe have prescribed myriad different indices and matrices for measurement, management and improvement of public procurements across the European continent. This article aims to add to this body of knowledge by creating a novel approach for the measurement and ranking of the efficiency of public procurements. For this purpose, we used a Composite I-distance Indicator methodology. Unlike any previous proposals, the presented approach ranks countries without the need for any subjective judgements. The results were tested among 30 European countries and, accordingly, the ranking displayed significant differences to conventional composite indicator-based rankings.
This article contributes to the better understanding of public procurement efficiencies as a phenomenon and puts a new spotlight on the way in which this efficiency should be measured among different countries. Therefore, the general idea of this methodology is not to replace any current scoreboards, but rather to add to the extant body of research. Using multiple matrices, policy makers will not be incentivized to advocate any single (or very few) dimensions of public procurement efficiency but will rather work on systematic solutions.
About the Authors
Miloš Milosavljević, PhD, is an Assistant Professor at the University of Belgrade, Faculty of Organizational Sciences. He teaches Financial Management and Public Administration. He has coordinated or participated in numerous projects in the field of public procurements, public administration, and controlling. He has published more than 50 articles and chapters in journals, conferences and international monographs.
Marina Dobrota, PhD, is an Assistant Professor at the University of Belgrade, Faculty of Organisational Sciences, Department of Statistics. Her research interests are applied statistics, computational statistics, data analysis and data mining. Most of her research combines large-scale qualitative and quantitative data. She has published more than 40 articles in international and national journals or conferences. She is currently participating in several research projects funded by the Serbian Ministry of Education and by the European Union.
Nemanja Milanović, PhD candidate, is a researcher at the University of Belgrade, Faculty of Organizational Sciences. His main fields of research interests are financial management, management accounting and business analysis. He gained international professional experience through a number of international studies and research projects. He is a member of the European Finance Association and the Youth Committee of the National Petroleum Committee of Serbia – World Petroleum Council.
