INTRODUCTION
Environmental governance, understood as ‘the establishment, reaffirmation or change of institutions to resolve conflicts over environmental resources’ (Paavola Reference Paavola2007, p. 93), has been through some significant transformations in the former communist countries of Central and Eastern Europe (CEE countries hereafter) (Andonova Reference Andonova2004; Scrieciu & Stringer Reference Scrieciu and Stringer2008; Kluvánková-Oravská et al. Reference Kluvánková-Oravská, Chobotova, Banaszak, Slavikova and Trifunovova2009). In addition to already existing international environmental obligations, new political conditions, such as the European Union (EU) Neighbourhood Policy (Bosse Reference Bosse2009), caused the objectives of national environmental policies to be further extended and management practices to be brought into line with European and international standards (Hicks Reference Hicks2004; Jörgens Reference Jörgens and Lafferty2004). From the beginning of the 1990s, a number of international actors entered the CEE countries. International organizations such as the World Bank, the European Bank for Reconstruction and Development and many other governmental and non-governmental agencies played a significant role and influenced the process of designing and implementing national environmental strategies (Connolly et al. Reference Connolly, Gutner, Bedarf, Keohane and Levy1996; Weimer Reference Weimer1997). This influence was conveyed through two ‘channels’: first, by providing financial aid and, second, by setting new standards in project implementation and policy and technical advice (Derviş et al. Reference Derviş, Selowsky and Wallich1995; Carmin & VanDeveer Reference Carmin and VanDeveer2004). Financial aid became important for a range of state activities (Keohane Reference Keohane, Keohane and Levy1996) and had a power-shifting effect.
The international standards introduced in order to influence national practices can be divided into two groups: those relating to outcomes (such as quality of environment, measures taken or project reporting), and those relating to processes (such as effective organization and management, adequate representation of different groups, interests, knowledge and information). While deviations from standards relating to outcome are relatively easy to recognize, standards for processes and deviations thereof are rather difficult to monitor. Process standards are likely to be very sensitive to subjective factors, such as understanding and interpretation of what a ‘good’ process means for different parties involved (Webler & Tuler Reference Webler and Tuler2006).
The role of international organizations in capacity building and bridging national practices and international standards has been widely discussed (Keohane Reference Keohane, Keohane and Levy1996; Gutner Reference Gutner2002; Sagar & VanDeveer Reference Sagar and VanDeveer2005; for the special case of ‘Europeanization’, see for example Jordan & Liefferink Reference Jordan and Liefferink2004; Paraskevopoulos et al. Reference Paraskevopoulos, Getimēs and Rees2006). Although the intervention of international donor agencies may benefit domestic environmental policymaking, critics argue that this potential largely remains unrealized (Gutner Reference Gutner2002; VanDeveer & Carmin Reference VanDeveer and Carmin2004). Twenty years of experience have demonstrated significant difficulties in implementing new environmental policies and standards (Lavenex Reference Lavenex2006; Otto et al. Reference Otto, Shkaruba and Kireyeu2011). In particular, international organizations have often tended to overlook and underestimate the importance of local conditions (traditions of governance, scientific research and often low social capital), while trying to apply practices developed for the Western democracies (Hallstrom Reference Hallstrom2004; Hicks Reference Hicks2004). The sources of implementation problems may lie within the sphere of operation of international organizations, and also with local implementers, third parties and external conditions (Derviş et al. Reference Derviş, Selowsky and Wallich1995; Keohane Reference Keohane, Keohane and Levy1996). The environmental authorities of CEE faced particular difficulties during post-communist transformation. Lacking the necessary financial support and expertise to develop and introduce innovative practices, the authorities often tended to use readily available approaches based on national natural science and technical expertise (Wolchik Reference Wolchik1991; VanDeveer & Carmin Reference VanDeveer and Carmin2004).
In the late 1990s, the analysts of international aid projects admitted the necessity to observe project results over a sustained period of time, normally between 10 and 20 years (Fairman & Ross Reference Fairman, Ross, Keohane and Levy1996). Nevertheless, very little reflection has taken place on the lessons learned. In this paper we aim to contribute towards filling this gap by analysing implementation of the World Bank project in Belavezhskaya Pushcha National Park (BPNP) in Belarus (1992–1997).
The BPNP is a transboundary ecosystem, covering approximately 120 000 ha in Belarus and 87 600 ha in Poland, various parts of which were protected since the 16th century. The ecosystem of coniferous and broad-leaved lowland forests has remained undisturbed over centuries, and possesses a unique landscape and biological diversity (Luchkov et al. Reference Luchkov, Tolkach, Berwick and Brylski1997; Martsinkevich et al. Reference Martsinkevich, Shkaruba, Falaleeva, Wascher and Pérez-Soba2004). The BPNP provides ecosystem services for larger areas in Europe, but also has significant meaning as a natural and cultural heritage of the Belarusian nation. The area supports the livelihoods of the local population through multiple land use and economic activities: harvesting forest products, hunting, agriculture, and tourism including game hunting. The national park is also the biggest employer in the area.
The Belavezhskaya Pushcha has been subject to various forms of protection throughout its history as a part of Poland, the Russian Empire, the Soviet Union, and Belarus. The last transformation from State Hunting Ground (special kind of nature reserve with mixed use) to State National Park at the beginning of the 1990s introduced new standards for biodiversity protection. The national park regime and zoning implied a certain flexibility of economic use, partly less strict than the former regulations, that facilitated resource exploitation for economic purposes. The management functions had also changed and passed from the Belarusian Ministry of Environment and Nature Protection (MNEP) to the Department of Presidential Affairs (DPA), the latter focusing primarily on economic development and welcoming new possibilities of more intense economic use.
Our study highlights one specific case in one CEE country; yet this case is characterized by a constellation of international donor organizations and local administrations, as well as by a public unused to active stakeholder participation. Its intial setting is therefore typical of other cases in CEE countries.
Evaluation and monitoring are important instruments for reflecting on the effectiveness, failures and achievements of environmental initiatives either at the level of specific projects (Taut Reference Taut2007) or for broad multi-level institutional set up (Shkaruba & Kireyeu Reference Shkaruba and Kireyeu2012). Transparent and well-designed evaluation is increasingly referred to as an integral part of effective project management and planning (Margoluis & Salafsky, Reference Margoluis and Salafsky1998; Stem et al. Reference Stem, Margoluis, Salafsky and Brown2005) and an important part of capacity building within international aid initiatives (Sagar & VanDeveer Reference Sagar and VanDeveer2005).
The mission of international organizations as standard setters for outcomes and processes of environmental management can be closely connected with the debate on two types of policy and project evaluation, namely ‘outcome’ and ‘process’ evaluation (Kaufmann & Kraay Reference Kaufmann and Kray2007; Rauschmayer et al. Reference Rauschmayer, Berghöfer, Omann and Zikos2009). The former, more traditional form of evaluation is often based on quantitative criteria and indicators; it is aimed explicitly at reporting the project results and is widely used in communication and policy processes. Nevertheless, there is a growing recognition that outcome evaluations are unable to take account of the social and environmental complexities and fail to provide an adequate picture of project realities and lessons to be learned (Ferraro & Pattanayak Reference Ferraro and Pattanayak2006). At the same time, an increasing number of studies reflect on the social processes behind the implementation procedures, as well as their effectiveness (Dietz & Stern Reference Dietz and Stern2008). Process evaluation provides a significant management and communication challenge because the criteria for ‘good’ decision making and implementation process are difficult to define (Blackstock et al. Reference Blackstock, Dunglinson, Marshall and Waylen2012) and they depend highly on the actors’ values and perceptions and interpretations of processes.
The aim of the paper is to: better understand the Belavezhskaya Pushcha project implementation process and shed some light on the durability of the project outcomes; and help develop understanding of internationally-funded nature protection projects in CEE countries.
METHODS
Case study description
The ‘Forest Biodiversity Protection Project Belarus’ financed by the World Bank was officially launched in 1992 as a part of the pilot actions undertaken by international aid agencies in the former communist countries (World Bank 1992, 1998). One million US dollars was a significant investment for the period, and one of the largest investments in Belarusian nature protection in the 1990s, aimed at supporting the BPNP, which was suffering from severe social and economic problems typical of the transition period.
The project, designed in accordance with up-to-date international standards of biodiversity protection, contained two groups of objectives (World Bank 1998). The primary group of scientific objectives included: maintaining on-going studies on ecosystem functioning and conservation; establishing a system of air and soil monitoring and a forest gene bank; developing a geographic information system (GIS) for monitoring and management of the forest ecosystem; purchasing equipment (computers and monitoring equipment); and scientific training courses for the Park personnel. In addition, management and social objectives emphasized a new aspect of the project and involved: expanding the area of the BPNP to support the integrity of the ecosystem; developing a complex management plan; fostering participation of the interested groups; enhancing sustainable economic and social development; and promoting contacts with the Polish side.
The World Bank project evaluation documents report advances as having been achieved, alongside with the difficulties in cooperation and different approaches by the World Bank managers and local implementers which led to a delayed implementation (World Bank 2001). The initiative was officially completed in 1997, two years later than planned, and received positive evaluations: in 1997, the BPNP was granted a Council of Europe Diploma as a protected area of international importance. However, several years after completion, the project appeared to be producing rather poor results, as indicated by a number of environmental and social conflicts, such as local unemployment, a relatively low level of economic development, and tensions between the administration and other groups (local population, non-governmental organizations [NGOs] and the scientific community), as well as excessive logging, illegal use of forest resources by the local population, and improper forest management and planning (Dranchuk Reference Dranchuk2004; Datskevich Reference Datskevich2010). From 2007 to 2010, the Council of Europe has annually suspended the Diploma and the BPNP was requested to submit a management plan and to correct present management strategies (Council of Europe 2007).
The present context of biodiversity governance in Belarus significantly retains the features of the 1990s, although shifts have been made towards more flexible and decentralized governance (Otto et al. Reference Otto, Shkaruba and Kireyeu2011). Recent stakeholder interviews indicate continuing dominance of DPA supporting top-down management of protected areas and promoting their economic use (Otto et al. Reference Otto, Shkaruba and Kireyeu2011). Stakeholder participation remains poor as well as the information and experience of participation. Remarkably, international organizations and joint projects are continuing to be seeing as potential agents of change and transformation, which makes reflection on the effectiveness of project implementation necessary and timely endeavour.
Data collection and analysis
We used administrative and scientific documents, as well as interviews and surveys with stakeholders, namely individuals and groups influencing and potentially influenced by the outcome of the World Bank project. In 2005–2006, seven semi-structured telephone interviews were conducted with representatives from the current and former administration and employees of the BPNP, managers of the World Bank, national level agencies, public organizations and the scientific community. In two cases, potential respondents, both formerly involved in the project management, declined to talk. Informal communications in 2005–2008 were used to back up the interviews. In 2006, a questionnaire survey (dealing with the conflict in the BPNP, the composition of stakeholders and their interests, power distribution and the World Bank project implementation; Appendix 1, see supplementary material at Journals.cambridge.org/ENC) was conducted among the local population with the help of local activists. We received and analysed 20 responses from people selected to represent different age and occupation groups. Questionnaires had been spread by a local resident, an academic researcher who had good knowledge of the community, as well as understanding of the purposes and methods of the research and respondent selection. Collected material reflected reasonably well the age and occupation groups, although the actual response rate was difficult to determine due to the uncontrolled nature of the survey. The general tendency was that people were reluctant to give their opinion to a local activist, which was also confirmed by other researchers'experiences (M. Biriukova & E. Shushkova, personal communication 2006). This reluctance can be linked to the current atmosphere of mistrust between stakeholders, which creates suspicion amongst locals about any form of activism on the matter and poses higher demands on the confidentiality of the interviews. These factors partly explain the specificity of our methodical approach (implying also an indeterminate response rate) leading to limitations in the results.
To update the results of formal outcome-oriented self-evaluation by the World Bank project team, we summarized the data from official evaluations of the project outcomes by the World Bank and the Belarusian side, along with data from interviews concerning project implementation and the present state of affairs. The documents from the World Bank present a detailed account of how the planned outcomes were achieved. The indicators were primarily of a quantitative nature including: area covered by the national park, number of animal species and amount of equipment purchased (World Bank 1998, 2001). A similar approach applied by the Belarusian side was presented at scientific conferences and in a book (Luchkov et al. Reference Luchkov, Tolkach, Berwick and Brylski1997; Luchkov & Artuchevsky Reference Luchkov, Artuchevsky, de Bruijn, Kazulka and Okolow2002). Although described as ‘synthesizing the technical and scientific research, a social assessment, and specific management actions’ the book, co-authored by the researchers and official project managers, emphases biological research.
In order to evaluate the process of project implementation, two different perspectives were distinguished: standards and procedures for the process described in the project proposal (World Bank 1992) and its actual implementation (World Bank 1998, 2001). The former reflected the expectations of the World Bank regarding the establishment of new process standards for biodiversity governance in Belarus, while the latter showed whether and how the Bank and local partners succeeded in meeting those expectations. Our analysis used the mentioned criteria and represents an independent expert ex-post evaluation. We analysed the interviews, the surveys and additional background materials using criteria suggested by Wittmer et al. (Reference Wittmer, Rauschmayer and Klauer2006) and further developed by the EU projects IBEFish and GoverNat (Varjopuro et al. Reference Varjopuro, Gray, Hatchard, Rauschmayer and Wittmer2008; Berghöfer et al. Reference Berghöfer, Wittmer and Rauschmayer2008; Rauschmayer et al. Reference Rauschmayer, Berghöfer, Omann and Zikos2009; for a recent comparison of different evaluation criteria see Blackstock et al. Reference Blackstock, Dunglinson, Marshall and Waylen2012). Four groups of criteria include: (1) integration of knowledge and information (considering environmental and social complexity, different types of information and uncertainties); (2) supporting legitimacy (compatibility with the existing regulations, actors’ accountability, representation of different groups, transparency of rules and assumptions to insiders and outsiders); (3) promoting social dynamics (supporting relationships and respect between the actors; providing space for learning and exchanging perspectives; balancing empowerment of different groups; facilitating convergence and illustrating diversity); and (4) cost-effectiveness of the measures taken (effectiveness of the investments made to compare with the results achieved, including in sustaining results in medium and long term).
RESULTS
Our results show that the first, overall positive outcomes of the project could not be sustained, whereas the process objectives had either not been reached at all, or not been sustained.
Outcome evaluation
Expansion of the BPNP as a protected area
This objective was reported to have been ‘achieved’. Approximately 12 000 ha were added to the BPNP, which provided necessary protection to ecologically valuable areas The additional areas have increased the spatial integrity of the ecosystem; however not all these areas correspond to the standards of management and quality of natural protected areas and there have been requests for further improvement, for example extension of the strict protection zone and limitation of economic activity in the new areas.
Maintaining scientific research in the BPNP
Reported to have been achieved in principle, this objective has supported a number of scientific activities (maintainance of gene bank, in-situ and ex-situ forest conservation, long-term monitoring programmes and experiments) and new research projects (optimization of ungulates population, air pollution monitoring, GIS for forest management and spatial planning). Nevertheless, the number of scientific programmes has declined (albeit due to rotation of personnel), cooperation with the Polish side remains limited, and scientific management is strongly oriented toward the interests of economic use.
Support for research infrastructure
This objective, reported as having been ‘achieved’, has led to equipment being provided for pollution analysis and monitoring, computing, transport and GIS software. While the official actors confirmed that the research equipment was still being used, several non-official respondents (for example former employees of the national park and members of the local population whose professional duties are related to the BPNP) reported ineffective use and lack of maintenance. There was no evidence for any significant recent update of the research infrastructure.
Professional development and training
Professional scientific training (workshops, study tours and professional contacts) was evaluated as being very successful, and this objective was considered to have been ‘achieved’. Training on technical and planning issues (sustainable agriculture and nature-based tourism) was reported to have taken place. Yet, there were almost no trained staff employed in the scientific or management divisions of the BPNP and no local respondent mentioned having been involved or having benefited from the management training on sustainable agriculture or tourism.
Development of a management plan for the BPNP and bordering areas
This objective was stated to have been ‘achieved in principle’. Suggestions by the group of planners and scientific objectives were reported in a draft management plan; no other specific planning or policy document has been developed. Nevertheless, the approach has been claimed to be ‘more interdisciplinary and participatory’ than the previous schemes for forest management in Belarus (World Bank 2001). However, at present, there has been no robust policy for national park development. Current policy favours actors who are interested in short-term economic profit rather than in biodiversity preservation; unclear perspectives and responsibilities have contributed to mistrust between the actors, as well as explicit and latent conflicts.
Process evaluation
Integration of knowledge and information
The project explicitly aimed to cope with the complexity of the socioecological system involved and to integrate different types of environmental and social knowledge. An interdisciplinary project team had to conduct scientific research and to cooperate with other stakeholder groups. The Conservation Management Plan was supposed to integrate scientific and management solutions for biodiversity protection, land-use and social development (World Bank 1992). Nevertheless, no specific training on multi-disciplinary projects management, knowledge integration or participatory methods for local managers had been planned or funded. The team had been formed representing the technical and natural sciences expertise of the Belarusian partners. During the series of scientific conferences, these experts had a chance to exchange ideas with the Polish side and with international partners, but otherwise had very limited support to develop and integrate social science knowledge. Stakeholder groups were present during some consultation events, but this participation was very limited in terms of its representativeness and, especially, its influence on the actual decision making process. No wider information about the project goals and possibilities of involvement was given to the public; no mechanisms had been provided for communicating scientific findings and planning solutions to non-experts.
Supporting legitimacy
The project design, planned measures and results were formally compatible with the existing legislation and procedures in Belarus. The project intended to promote the use of additional legal mechanisms that formally existed but were not implemented, for example the right to participate in biodiversity governance and to increase representation, transparency and accountability through participation and knowledge integration. Project implementation initially faced difficulties with the formal procedures of registration and the transfer of funds. The obstacles were resolved by enlisting support from the official actors at government level, thereby handing over to them a significant part of the project ownership and control over the finances. Due to the lack of previous experience and/or information, several stakeholder groups did not realize their legal rights or claim them (for example for participation, transparency and accountability); no suggestions for promoting these rights and monitoring their implementation had been provided by the international experts. Poor accountability resulted in the official actors and scientific experts largely dominating the process.
Promoting social dynamics
The project was intended to facilitate new relationships between the actors, increase trust, support a more balanced distribution of power and increase the capacity of less influential stakeholder groups. Stakeholder involvement, training courses for academics, managers and locals were planned to facilitate these social processes (World Bank 1992). However, the majority of the educational activities were academically oriented, and no training was provided to enhance the capacity of the other groups or for trainers. Mechanisms for sustaining the training results were not provided either. Ineffective involvement and participation as ‘manipulation’ and ‘therapy’ (see Arnstein Reference Arnstein1969) resulted in growing mistrust between the actors. Poor communication across the groups gave little possibility for mutual learning or for changing behaviour. Investments resulted in further empowerment of already powerful actors (administration) and had little positive influence on the other stakeholders.
Cost-effectiveness
The effectiveness and sustainability of the major investments in the scientific part of the project were to be achieved by implementing the Conservation Management Plan, running education activities and ensuring stakeholder involvement (World Bank 1992). The idea was that the management plan should be a document that provided guidance in integrating scientific findings into development strategies on the basis of new standards for biodiversity management, therefore securing investments in the long term. Investments made in the scientific process had a significant positive effect on maintaining biodiversity protection upon project completion. However, failure to achieve strategic objectives, including developing and adopting a management plan, and failure to introduce more effective biodiversity management practices resulted in poorly effective investment in the longer term. A more transparent and balanced administration of the BPNP might have lessened the loss of human capital due to high staff turnover.
DISCUSSION
Comparative analysis of both types of evaluation indicated a significant emphasis on the outcome-related actions reflected in the reporting documents from both sides (Luchkov et al. Reference Luchkov, Tolkach, Berwick and Brylski1997; World Bank 1998). Having been articulated in the planning documents (World Bank 1992), the process-related objectives were significantly overlooked during the implementation phase. Three years after project completion, World Bank (2001) introduced elements of process evaluation for its future operations in Eastern Europe. However, the lack of structured evaluation criteria, along with the internal character of the document, means that it is not possible to communicate the failures identified to the other actors. At the same time, the official Belarusian actors portrayed a fairly positive evaluation of the process in the interviews that may be explained by the fact that the implementation process remained within the limits of their usual procedures. Moreover, the occasional involvement of other stakeholder groups in consultation and training events made it possible to report (perhaps in good faith) that the new standards for biodiversity protection had been successfully met. Nevertheless, 15 years after project completion, the present situation in the BPNP suggests that an unbalanced emphasis on the outcome (purchasing equipment and commissioning additional science-based studies) was an obvious drawback of the management strategy and eventually led to a low level of project sustainability. The lack of management processes (such as preparing a management plan based on stakeholder involvement), is likely to be an important reason why the investments have not been effective in the longer term.
Is the Belavezhskaya Pushcha project a specific case of mismanagement or is it part of a broader ‘web’ of repeated errors in cooperation between international organizations and their local partners?
Operation of international organizations in post-communist countries
The limited success of this international project can be attributed to three principle groups of factors. First, natural-science and technical expertise dominated the project. Although a multi-disciplinary participatory approach was indicated among the priorities of the World Bank initiative in Belavezhskaya Pushcha, many of the interviewees representing Belarusian scientific and management communities, as well as local respondents, admitted that the purely natural-science character of the project corresponded to its initial objectives. This indicates poor communication on the project's objectives, which was thus considered a scientific initiative in line with traditional perceptions.
Second, the prevalence of technical expertise and lack of social science expertise in CEE states is recognized by the international research community (see Otto et al. Reference Otto, Shkaruba and Kireyeu2011). Nevertheless, it seems that very little has been done to address this problem at the management level. Referring to Wolchik (Reference Wolchik1991), VanDeveer and Carmin (Reference VanDeveer and Carmin2004) identified the same problem, stressing the role of the EU and international organizations in fostering an unbalanced use of technical expertise. While the local environmental ministries lack the capacity to implement complex international environmental policies and tend to rely on traditional technical expertise, their international partners seem to follow the same logic in order to avoid possible management problems and to secure a timely and smooth delivery of outcomes.
Knowledge integration is closely connected to the unbalanced actors’ representation and to the unequal cooperation between international organizations and local partners. In their search for effective project implementation, the World Bank managers acquired support at the highest national level. Although this was extremely helpful during the early stages (World Bank 1998), the seizing of control by the official actors decreased transparency and the possibility of involvement (World Bank 2001; personal communications with the former employees of the BNPN, local and national environmental activists 2005–2008). At the same time, a significant number of the respondents pointed towards international organizations as a potential external agency to balance the distribution of power at the domestic level, not least through empowering civic society.
Following Keohane (Reference Keohane, Keohane and Levy1996), the empirical evidence from the case study confirms that in CEE, where traditions of centralized top-down governance are very strong, international aid projects are unlikely to be a success without the active support of governmental authorities. However, the long-term effectiveness of financial investments will also not be great without taking interests into account and developing strong ties between international managers and local recipients (Keohane Reference Keohane, Keohane and Levy1996; Fairman & Ross Reference Fairman, Ross, Keohane and Levy1996). VanDeveer and Carmin (Reference VanDeveer and Carmin2004) stressed how international actors often ignore the opportunity to foster the legitimacy of civil society's involvement. Instead, international managers, much like the domestic officials, tend to prioritize technical expertise and to see civic society groups as a source of competition or, at least, as implementers of ready-made policy recipes at the local level (VanDeveer & Carmin Reference VanDeveer and Carmin2004).
Third, alongside power distribution, the imbalance in representation strongly connects to the lack of knowledge about objectives and methods of participation. Despite the fact that there was no local expertise concerning organization of participatory processes in the Belarusian project, no funds or expert support had been provided to develop this expertise. During an interview, a project manager from Belarus mentioned that the domestic team was finding it extremely difficult to organize the participatory process according to what were perceived to be international standards due to a lack of adequate knowledge.
The effectiveness of directly replicating democratic countries’ standards of participation in CEE remains an open issue and one that has received very little attention from researchers to date (Hutcheson & Korosteleva Reference Hutcheson and Korosteleva2006; Kluvánková-Oravská et al. Reference Kluvánková-Oravská, Chobotova, Banaszak, Slavikova and Trifunovova2009). The objectives and methods of democratic participation may be interpreted differently by foreign managers and local implementers (Hutcheson & Korosteleva Reference Hutcheson and Korosteleva2006). The existing local formal and informal institutions are, in many cases, suited to domestic conditions. This does not mean, however, that they are acting effectively and that innovations in the form of international standards for participation are not needed. These controversies stress even more the need to make a careful selection of participatory methods that build on local realities and that, at the same time, reach the level of management effectiveness currently associated with democratic participatory techniques (see Agrawal Reference Agrawal2000). Such an endeavour requires allocating human and financial resources to build fit-for-purpose institutional structures for participation.
Project evaluation: reflection, learning, communication
A critical issue raised in analyses of the early technical assistance projects is that foreign agencies often tended to ignore the actual reasons underlying the performance of local partners. Instead, international agencies focused on expressions of incapacity, such as the absence of particular technologies (Grindle Reference Grindle1997). This unbalanced approach tends to make local partners mere recipients of foreign standards and practices, while the actual value and suitability of these standards for the local conditions are ignored by the donors (Sagar & VanDeveer Reference Sagar and VanDeveer2005). Poor reflection on past experiences and a lack of institutional coordination between the partners are among the factors detrimental to mutual learning that lead to deficiencies in the implementation and sustainability of international assistance (Sagar & VanDeveer Reference Sagar and VanDeveer2005). Project evaluation can serve as a learning tool helping to achieve such moves.
Links between project evaluation and mutual understanding between the parties relate to the broader debate on the development and interpretation of environmental assessments. Describing environmental assessments as a ‘communicative process’ rather than ‘reports that they often produce’, Farell et al. (Reference Farell, VanDeveer and Jäger2001) stressed the importance of ‘perceptual lenses’ reflecting respective worldviews and determining underlying assumptions of the parties involved and procedures used. For successful communication, it is felt all parties need to share basic perceptual lenses (such as evaluation methods and criteria) or, at least, be aware of the ‘lenses’ that the other parties are applying. International projects add an extra layer of complication due to the differences between international and local partners regarding standards, expectations and related evaluation criteria or, in other words, their impressions about what is good and what is bad implementation. Therefore, effective evaluation needs to build upon a comprehensive system of flexible criteria that enable aspects of reality within and beyond the lenses of each actor to become visible and the development of an appreciation of what is important to the other partners involved.
Whether explicitly (based on specific evaluation criteria) or implicitly (following their idea of common sense), the actors involved in the project often combine outcome and process evaluations when reflecting on project implementation and results. We suggest that both outcome and process evaluations have their advantages and limitations. The approach of combining different perceptual lenses in terms of different approaches (such as outcomes and process) applied by different actors (for example international organizations, local partners or third parties) is likely to provide the most comprehensive picture of the project implementation. Clear definition and communication of the perceptual lenses through evaluation criteria and methods are likely to advance mutual understanding.
Different evaluation systems may be used in an international project (Fig. 1). In one model (Fig. 1, a) both international organizations and local partners act in the project according to their own standards for the outcomes and the process and use their own lenses to interpret the results. In this case, the feedback is likely to be shared by a group that has the same lenses without directly affecting the area of mutual cooperation. For example, the standards delivered by the international organization are unlikely to affect the mind sets of the local implementers. Different evaluations of the project might eventually lead to communication and cooperation difficulties. In another model (Fig. 1, b), the parties partially share their evaluation standards, for example having the same standard for outcomes and different measures to reflect on the processes. The feedback on the outcomes achieved is likely to directly affect the area of mutual cooperation; however there is still no agreement on the processes behind the successes and failures of the implementation. A third model (Fig. 1, c) where both parties adopt a common system to reflect on both the process and the outcomes can potentially provide the most effective feedback to the cooperation process.
Process and outcome evaluation as ‘perception lenses’ reflecting reality for different groups of partners. (a) Different criteria for outcomes and processes for the local and international partners. (b) Different criteria for processes and similar for outcomes. (c) Similar criteria for outcomes and processes for the local and international partners.
The World Bank project in Belavezhskaya Pushcha seems to reflect the situation described in the second model (Fig. 1, b). Shared criteria and techniques for outcome evaluation provided perceptual lenses that were clear and transparent for both local managers and the Bank's experts. The analysis of the process remained at the level of internal reflections by the implementing team, with no clear criteria communicated between the parties, including such important but flexible objectives as ‘fair and competent organization’ (World Bank 2001). The present case study provides insights into how the second model (Fig. 1, b) may manifest itself over the longer term: while both parties agree that the scientific and technical outcomes have been achieved, a failure to communicate and agree on a common vision of the process can be among the important reasons leading to limited long-term sustainability of investment.
CONCLUSIONS
Our analysis of Belavezhskaya Pushcha corroborates the conclusions from several other examples of international aid initiatives in post-communist countries and highlights the problems in setting new standards for environmental governance in CEE. The most obvious obstacles encountered were: dominance of technical and natural scientific approaches implying a lack of social science perspectives; unbalanced representation of different stakeholders; powerful implementation by official high-level actors; underestimation of the role of civil society (NGOs and local actors) as partners for donor organizations; and lack of knowledge about and experience in participatory involvement, with little support for developing this knowledge.
Although these problems are not new, they are still often overlooked by the official project evaluations, which traditionally focus on technical outcomes as reflected in quantitative indicators. Outcome evaluations have significant value in the context of reporting on the targeted objectives achieved (or not) upon project completion. However, our example shows that even a correct and positive outcome evaluation is unable to guarantee the sustainability of project results and investments in the longer term. One reason for this is that project sustainability has a strong link to the effective organization of the implementation process (see Rauschmayer et al. Reference Rauschmayer, Berghöfer, Omann and Zikos2009).
Outcome and process evaluations have different but complementary functions, however to be effective they should occur in parallel with project implementation, as part of an iterative reflection by all parties involved. An unbalanced composition of outcome-related and process-based criteria, or different perceptual lenses, may have significant detrimental effects in long term, introducing systemic error in perception and communication despite well intentioned management on both sides.
ACKNOWLEDGEMENTS
We gratefully acknowledge the support of the Saxonian Ministry for Research, the EU Training and Research project GoverNat (Contract No. 0035536, www.governat.eu), and LUMES International Master Program at Lund University. We are grateful to Nina Hagemann and the Participatory Research Cluster at IVM VU Amsterdam, Nicholas Polunin and three anonymous reviewers for their comments on earlier versions, and to Marina Biruikova, Elena Shushkova, Irina Usova, Yuliana Polishchuk and Heorhi Kazulka for technical support.
