Conceptual framework

RIAs are often understood as tools in the hands of a principal seeking to control an agent (Froud et al. Reference Froud, Boden, Ogus and Stubbs1998; Posner Reference Posner2001; Rowe Reference Rowe, Hofman and Turk2006). The principal is the centre of government which introduces the new system of impact assessment with the aim of improving regulatory outcomes, whereas the agents are line ministries which implement RIA when preparing legislative proposals for cabinet decisions. In order to encourage the agent to engage with RIA substantively rather than formally, the principal sets up an oversight body for RIA quality. Renda (Reference Renda2006) ascribes to such an oversight centre a number of powers and duties, such as advocacy, consultation, guidance, coordination, training, annual reporting and maintaining institutional relations. A key hypothesis is thus that the more power the centre of government has in terms of internal oversight, the better the quality of the RIA process and its outputs. Conversely, the weaker the central executive authority, the less oversight in place and the more ministries will treat RIA as an exercise that involves little more than ‘ticking the boxes’.

Resources at ministerial level are also likely to matter for RIA performance. The way in which formal rules for conducting RIAs have been established, the degree to which they correspond to administrative capacity at the level of line ministries and the broader institutional context are, therefore, important factors to consider when one seeks to account for RIA performance. The clearer and more detailed the instructions provided to the administration by the formal framework of rules, the higher the probability of good performance. Moreover, one may expect that the longer formal RIA rules have been in place, the better implementation results are likely to be. However, delay in the institutionalization of RIA need not mean worse performance if the delay is due to a careful phase-in of the framework.

The key expectations are thus as follows: the existence of either a strong centre of government or a RIA coordinating body would be reflected in better RIA performance due to enhanced leadership, oversight and guidance. Also, clear formal rules on how to undertake RIA would have a positive effect on overall results. It may also be expected that the duration of the RIA regime will affect performance because of a learning process at the administrative level.

When it comes to determining RIA quality and how to measure it, Radaelli (Reference Radaelli2005) distinguishes between two approaches: indicators and tests.Footnote ¹ In both, the main aim is to check the validity, reliability and other properties of quality. By contrast, Hahn et al. (Reference Hahn, Burnett, Chan, Mader and Moyle2000) have developed a scorecard that assesses the key assumptions and the appropriateness and application of models used in particular analyses. The present paper takes yet a different approach. It focuses on the quality of information RIAs contained in the explanatory memoranda attached to draft legislation. Thus, in our analysis, RIAs are not judged by their validity, truthfulness or the appropriateness of assumptions and methods used, but simply by, first, the existence of certain types of information contained in the explanatory memoranda, and, second, by indicators of the quality of this information.

The quality of RIA information is evaluated in three principal dimensions: (i) regulatory issues addressed, (ii) types of impact analysis, (iii) consultation process (cf. European Commission 2004). Regulatory issues may include identification of the problem to be solved, objectives to be achieved, policy options considered, positive and negative impacts, comparison of options, monitoring and evaluation. The second dimension relates to the details of the analyses conducted in social, economic and environmental areas. Traditional fiscal impact assessment (impacts on the state budget) is also included to better differentiate between ‘internal’ impact on the state and ‘external’ impact on the society. Thus, this analysis looks at identification of impacts in each area, the provision of qualitative or quantitative data in each area, the identification and quantification of costs and benefits, as well as mutual trade offs. The third dimension – consultation process – concerns the way in which affected parties are identified and brought into the policy-making process as well as how the results of such consultations are presented in the explanatory memoranda.

The study is based on an analysis of explanatory memoranda and RIA statements attached to parliamentary bills that governments submitted to parliaments in 2007 or in the Czech Republic 2008. The empirical analysis omits other material submitted for discussion to the government, such as statements on legislative intent, concept papers, strategies, or action plans. The reason for this empirical focus is twofold. First, most policies in CEE require legislation. Thus, laws typically have a significant impact on the lives of citizens. Second, it is the process of preparing legislation rather than the policy process overall that is usually subject to detailed regulations. The methodology employed in this study is based on previous content evaluations of RIAs (Renda Reference Renda2006, Zubek Reference Zubek, Brusis, Staroňová and Zubek2007, Staroňová Reference Staroňová, George and Kirkpatrick2007), though it differs from them in that the present study is designed to probe the effects of institutionalization on performance.

Institutionalization of RIA

This section looks at how the RIA process has been institutionalized in the five countries studied here.Footnote ² Table 1 summarizes the results.

Table 1:

Institutionalization of Research Impact Assessment

* planned for 2009

** moved from Government Office in 2006

*** in effect from 2008

**** until 2006, unclear afterwards.

Source: compilation by the author

Typically, RIA is often adopted as part of a wider better regulation agenda. The CEE countries have followed a different path in this respect, since in all of them RIA was adopted prior to the development of better regulation programmes. Hungary and Estonia are the forerunners in RIA adoption, in 1994 and 1999 respectively, followed by the Czech Republic and Slovakia. In the Czech Republic, the Legislative Rules of Government introduced a RIA requirement in 1998,Footnote ³ but it only entered into force in 2007. In Slovakia, the first requirements for RIA were introduced in November 2001 via an amendment of the Legislative Rules of the Government, following recommendations by the Audit of State Administration. In 2004, the Slovenian government signed The Act on Cooperation between the National Assembly and the Government in EU Affairs, which obliges the Government to carry out assessments of the impact and implications of draft EU-related measures. Overall, the rhetoric of ‘better regulation’ was largely absent when RIA was initially adopted.

When it comes to the adoption of RIA guidelines, the record of CEE states was patchy. By 2007/8, only the Czech Republic and Estonia had adopted RIA guidelines, Slovakia did so at the end of 2008. In Estonia, well-structured and detailed manuals have been drawn up to help the preparation of explanatory memoranda for draft legislation at both governmental and parliamentary levels as well as a multi-stage system for quality control (Kasemets and Liiv Reference Jacobs2005). The Czech Republic has prepared guidelines on the basis of prior piloting of RIA.

The countries studied here, with the exception of Estonia, have not set up strong centres of government that would act as a coordinating unit vis-à-vis line ministries in the preparation of strategies, in administrative planning or in synchronising the policy process. In Estonia, overall policy, EU and strategic planning has been entrusted to the State Chancellery (OECD SIGMA 2007), which has developed a system of communication with liaison personnel in the line ministries. The centre is well organized, with competent staff with good managerial skills and is strong in coordination. It reviews draft laws to ensure conformity with strategic plans and overall government priorities as well as controlling the quality of explanatory memoranda.

In the Czech Republic and Hungary, there have been some efforts to set up a unit for coordinating RIA, but they have been only partially successful. In the Czech Republic, there used to be a special Department for Regulatory Reform at the Government Office. It operated under direct supervision of the Deputy Prime Minister for Economic Affairs and consisted of analysts who were to assist in RIA development in line ministries. In 2005-06, the department piloted RIA and developed a methodology which was later adopted by the Government (Staroňová, Pavel and Krapež, Reference Staroňová, George and Kirkpatrick2007). The Department was moved to the Ministry of Interior in November 2006, and its responsibilities in quality assurance have become unclear.

In Hungary, overall coordination is fragmented. The Prime Minister'sOffice provides overall inter-ministerial coordination and houses a special unit composed of advisors who summarize opinions on each legislative proposal (OECD SIGMA 2007). But it is the Ministry of Justice that has a general responsibility for law-making and it is here that, in 2002, the Department of Impact Analysis, Deregulation and Registration of Law was established. The civil servants of the Department elaborated and published the Hungarian RIA methodology in 2003–2004 (though it was not adopted by the Government as an official document) and began several training courses in the field of RIA and other aspects of better regulation. Nevertheless, ministerial RIA activities have remained mostly isolated and the integrating role of the Ministry of Justice has not received much backing from the centre of government, particularly the Prime Minister's Office.

Slovakia and Slovenia have never had a strategic centre of government, and only when both governments adopted better regulation programs, did they create better small regulation (rather than RIA) units at the level of line ministries. In Slovenia, the Ministry of Public Administration and, in Slovakia, the Ministry of the Economy became responsible for the implementation of better regulation in 2005 and 2007 respectively. In both cases, these are single-person units with no powers over other ministries. Only in 2008 did the Slovak government adopt a Joint Methodology on RIA, in which it charged four different ministries with oversight for particular areas of RIA: the Ministry of Finance for fiscal RIA, the Ministry of the Economy for economic RIA, the Ministry of Environment for environmental RIA and the Ministry of Labour, Social Affairs and Family for social RIA. Despite these efforts, the individual ministries are likely to face fragmentation and hierarchical problems vis-à-vis other line ministries.

RIA performance

Regulatory Issues Addressed in RIAs

Despite the overall frameworks in place, the presentation of the results of RIA is quite puzzling in several of the countries; the exceptions are Estonia and the Czech Republic, which also possess detailed RIA guidelines. In the other countries, some information that is related to RIA, such as the purpose, objectives and need for the draft legislation, the results of the inter-ministerial review process, references to other studies or organisational support for the implementation of legislation, can be found in different sections of the explanatory memoranda. Some of the same information is requested by the guiding documents themselves in different parts of the explanatory memoranda, which contributes to confusion in presenting information. Sometimes, one finds a summary in the explanatory memorandum, which then refers for details to the Statement of Impacts, at other times, the opposite is the case. In some of the explanatory memoranda, reference is made to detailed calculations to be found in a third document, which is not, however, attached to the explanatory memorandum. It is thus impossible to check the methodology or results of the analysis. In general, comprehensive executive summaries are missing, as are explanations of analytical methods used or sources of data (see Table 2).

Table 2:

Regulatory Issues Addressed in Assessment

Source: own calculations; n/a means that this type of information is not required by formal rules on RIA in a given country. In the Czech republic only 12 RIAs have been submitted; the percentage is calculated from the overall number of draft proposals submitted to Cabinet.

The examination of explanatory memoranda shows that information regarding the purpose of legislation can be found in all of the observed countries. However, the information on purpose does not provide detailed information on problems analyzed, but rather is very general. For example, most of EU transposition related issues simply state ‘EU harmonization’ as the legislative purpose, without any further elaboration. Specification of measurable objectives does not meet RIA standards, except for Slovenia, where measurable objectives are stated in 90 per cent of proposals. Such a non-systematic way of presenting information in explanatory memoranda makes it difficult for both decision-makers and any interested party to check information contained in explanatory memoranda. Only the Czech Republic has started to present data in a new manner: if a statement of RIA is elaborated, it substitutes the explanatory memorandum to avoid confusion.

Not surprisingly, in most of the RIAs, the analysis of effects takes place only for the option already chosen. The Czech Republic conducted only 12 RIAs for all the proposals adopted by the Cabinet; however, 11 of these identified and analysed options as a part of RIA. When it comes to the Statement of Impacts, these are attached to the explanatory memoranda of all proposals in Estonia, Slovakia and Slovenia. In Slovakia and Slovenia, most of these are of a very formalistic nature, with statements such as ‘no impact’ or vague statements such as ‘the changes will be positive for society’, ‘there are no impacts on citizens, the state budget and the environment’ or even ‘non-action will bring sanctions from the EU which are costly’ without providing any further explanations.

The exceptions are Hungary and the Czech Republic, with a large number of cases where the draft laws are submitted to and passed by the Cabinet and do not contain any RIA statement. In fact, in Hungary, only one proposal out of 115 contained a RIA statement and in the Czech Republic only 12 RIA statements were prepared. In both cases, information on costs (particularly on fiscal costs) could be found in the general sections of explanatory memoranda that did not contain RIA statements (see also Table 3).

Table 3:

Type of Analysis in Regulatory Assessment

Source: own calculations

Note: one proposal can contain one type or several types of RIAs at once. In the Czech Republic an RIA is not made for an EU directive related proposals.

Information on the administrative and organisational implications of the implementation of draft legislation is, in general, very limited or absent. Only Estonia regularly discusses implementation measures such as organisational problems that need to be solved, changes in functions and/or structures of ministries or the need for specific training programs. This is even though the discussion of this issue is not, in fact, formally required. As for monitoring and ex post evaluation measures, in none of the countries is such information provided (and is also not formally required).

Analysis of Impacts

This section distinguishes the way in which RIAs identify economic, social and environmental impacts. For EU-related and domestic bills to the extent that this is possible.

As we can see from Table 3, an RIA form (Statement of Impacts) is attached to Explanatory Memorandum to draft proposals approved by the Cabinet in almost every case in Estonia, Slovakia and Slovenia, regardless of whether EU-related or domestic bills are considered. The Czech Republic has prepared and attached altogether only 12 RIAs, all for domestic bills only as RIAs are not prepared for EU related bills. In this regard, Hungary performs worst, as it simply ignores RIA requirement and only in one case an RIA form was attached, although it does occasionally prepare fiscal RIA.

Even though RIA forms are attached in Estonia, Slovakia and Slovenia and thus generally do address future effects, in Slovakia and Slovenia in more than 50 per cent of the cases they are pro-forma and simply state ‘there is no impact’ or ‘overall positive impact is expected’ either on the economy or on society without any further specification or clarification as to how such a conclusion was reached. Estonia stated ‘no impact’ or similar phrases only in one third of the cases, which corresponds to OECD estimates of the percentage of regulations that may not have a significant impact. The Czech Republic, on the other hand, is a different case, as all 12 cases of RIAs attached are of very high quality.

As far as the individual types of impacts are concerned, only Estonia analyzes social impacts (in 21 per cent of cases) and economic impacts (in 32 per cent of cases), followed by Slovenia, which analyzes social impacts in 7 per cent of the cases, even though the RIA document does not ask for social impacts. In Estonia, the economic RIA is frequently related to the assessment of administrative burdens on enterprises, based on the Dutch Standard Cost Model. In Slovakia, RIAs address social impacts in close to 20 per cent of cases, however, all of them limit the discussion to employment strictly within the state and the public service and not the labour market in general. Therefore, they have not been considered as social impact assessments, as they do not assess impacts external to the state. In general, economic impact assessments are more an exception than the rule. Surprisingly, environmental impact assessments in the five countries remained at a very low level, despite the volume and tradition of environmental impact assessments in the EU. This result is striking because Environmental Impact Assessment (EIA) and Strategic Environmental Impact Assessment (SEIA) have already been adopted in four of the countries (but not in Slovenia). Moreover, the Ministries of Environment in the countries have created specific units dedicated to EIA/SEIA. If there is any mention of environmental RIA, it is generally limited to the comment that the bill ‘fulfils EIA’.

The only exception, and a clear pattern across the four countries can be observed in this respect, is the fiscal analysis in the Statement of Impacts. Fiscal impact assessments are addressed in close to half of cases in the countries, except for Hungary, where only 5 proposals contain a fiscal impact assessment. The proportion of fiscal RIA conducted with EU proposals is the same in the relevant countries. Nevertheless, the quality of these varies significantly from case to case, particularly in Slovakia and Slovenia, where in many cases figures are given without providing evidence of how they were arrived at. By contrast, Estonia and the 12 RIA statements in the Czech Republic as well as a few examples from Slovakia and Slovenia have a high quality of fiscal impacts calculated. It seems that RIA, if conducted at all, concentrates exclusively on assessing the financial and budgetary impacts of regulations. Fiscal impact assessment in all of the five countries is equated with expenditures and costs that fall on the state budget (and/or the budget of local authorities in exceptional cases).

When one looks in more detail at calculations conducted in fiscal RIAs (Table 4), one can see that the emphasis is invariably put on the identification and quantification of costs and only to a much lesser extent the identification and quantification of benefits. In fact, RIA with EU related legislation does not provide any benefits only costs. Slovenia is the only country where benefits with EU related legislation are provided, though they are not quantified. Nevertheless, in none of the countries does one see efforts to compare costs and benefits. The absence of such comparisons is mostly due to RIAs focusing either on costs or on benefits and only very rarely on both at once.

Table 4:

Costs and benefits in Fiscal RIA (state budget)

Source: Own calculations;

Parties Affected and Public Consultations

As Table 5 shows, all of the countries identify public administration, e.g. central and local state authorities or state owned enterprises such as railways, as the main bearers of effects, which are usually also quantified. If the citizenry at large is considered, this often amounts to no more than phrases such as ‘the citizenry will benefit’, with no elaboration of how citizens will benefit or any attempt at quantifying benefits. Both Slovenia, in 22 per cent of cases, and Estonia, in 26 per cent of cases, try to identify specific groups of citizens in proposals. Whereas Slovenia only mentions these groups of citizens (e.g. ‘owners of cars will have to undertake extra activities’), Estonia goes further and tries to estimate costs and benefits accruing to the identified parties. A similar practice is found in identified industries or businesses. Whereas other countries only mention a specific industry, Estonia makes efforts to quantify costs and benefits. Some of the information in Estonia comes directly from consultation with affected parties.

Table 5:

Identification of Affected Parties in Assessment Process

Source: own calculations; draft proposals may mention more than one affected party

Consultations have not been conducted in observed countries, with the exception of Estonia and the Czech Republic (see Table 6). In Estonia, consultations have been conducted for nearly half of the proposals (42 per cent), with resulting information provided in half of those cases in the RIA statement (24 per cent). Again, the proportion of consultations conducted for EU related legislation is roughly the same. In the Czech Republic, 11 out of 12 RIAs have utilized consultation processes (thus in all cases where affected parties have been identified) and one proposal presented all the results in the RIA statement.

Table 6:

Use of Consultation in Assessment Process

Source: own compilation

Institutions and RIA performance

The foregoing analysis has revealed interesting cross-country differences in RIA performance across the five ECE countries. How can one account for such differences and what role do institutions have in such accounts?

Estonia has a relatively good performance in RIA implementation. It scores highly on the quality of information regarding public consultations and predicted impacts, where it both identifies and directly consults affected parties and utilizes this information for RIA. It is also the only country of the five in which impacts other than fiscal effects are discussed on a regular basis. When it comes to issues addressed by RIA, it scores highly on two areas: RIA statement and implementation. Like in other countries, in Estonia options are not discussed, however, because formal guidelines do not ask for this.

These results reflect a number of factors. The Estonian State Chancellery performs an important role, despite the fact that it does not coordinate RIA implementation as such. Some issues, such as purpose, implementation measures (which are not required by formal RIA rules), impacts other than financial, and consultations are checked by the State Chancellery as part of overall strategic and administrative efficiency monitoring. In addition, the Estonian State Chancellery initiated, prepared or oversaw the preparation of several guidelines to assist civil servants in carrying out RIAs. For example, it composed the Code of Good Involvement Practice, based partly on the Code of Practice on Consultation in the United Kingdom. The State Chancellery also promoted this code through workshops for civil servants organized by the Network of Estonian Nonprofit Organizations (Lepa et al Reference Lepa, Illing, Kasemets, Kallaste and Lepp2004).

Furthermore, in 2007, each ministry appointed an official in charge of monitoring compliance with the Code of Good Involvement Practice. Moreover, EU officers, who are direct liaison persons of the State Chancellery in the line ministries, assist and coordinate consultation processes for their particular ministry. Many ministries have also described consultation and involvement of stakeholders in ministerial development plans. In 2007, the Estonian State Chancellery opened an official e-consultation portal www.osale.ee to facilitate public consultation. Moreover, the Ministry of Internal Affairs adopted a Civic Action Development Plan 2007–2010, which aims to develop stakeholder and interest-group involvement and public participation and dedicates funds to this purpose. Thus, there are certain spill-over effects in having a strong executive oversight body that oversees the overall strategic performance of line ministries and pushes for reforms in the policy-making process in spite of the fact that this centre is not specifically dedicated to RIA performance.

The Czech Republic is an interesting case as there are only 12 cases in which RIAs were actually performed in 2008, nevertheless, 11 of these were of very high quality: they identified and assessed options, with quantified costs and benefits, and included consultation processes. Why is this the case? When one looks at the sponsors of these RIAs, five came from the Ministry of Interior, five from the Ministry of Information and Telecommunications and two from the Ministry of Finance. The Ministry of Interior is the place where the original RIA coordinating and oversight body from the Government Office under prime ministerial supervision was moved to, along with its entire staff. The Ministry of Information is the one where RIA was piloted and used for the development of RIA methodology. Thus, they were the architects of RIA in the Czech Republic. Nevertheless, other line ministries did not take part in the pilot and do not have the adequate skills and capacities to conduct RIAs on their own. Moreover, with the shift of the body from the Government Office to the Ministry of Interior, the body lost power to supervise and monitor other line ministries. As a result, other ministries simply ignored the elaboration of RIAs.Footnote ⁴