INTRODUCTION
The Consultative Group on International Agricultural Research (CGIAR) has a long tradition of impact assessment research, which dates back to the origins of the system in the early 1970s. The history of impact assessment in the CGIAR has been documented and assessed by numerous authors, including Alston et al. (Reference Alston, Chan-Kang, Marra, Pardey and Wyatt2000), Mackay and Horton (Reference Mackay and Horton2003), Pingali (Reference Pingali2001) and TAC and CGIAR Secretariat (2001). Approximately 600 impact and adoption studies have been produced by the CGIAR System to date (Raitzer, Reference Raitzer2005). Most of these have been ‘ex-post’ studies of completed work, which have employed methods drawn from agricultural economics. Economic methods are generally considered to be the standard for good impact assessment in the CGIAR (Maredia et al. Reference Maredia, Byerlee and Anderson2001). However, as Pingali (2001, p. 24) noted, ‘while CG economists have had a good record in impact assessment research, their ability to communicate their results to a wide audience has been rather poor’.
In addition to communication problems, several other factors limit the use of economic impact assessments on decision making (Mackay and Horton, Reference Mackay and Horton2003). Few assessments have been designed and carried out specifically to meet the information needs of decision makers. In some cases, there has been a lack of confidence in the results of impact assessments, due to a focus on successful cases (Matlon, Reference Matlon2003). Sometimes, impact assessment results have not been provided rapidly enough to influence decision making (Collinson and Tollens, Reference Collinson and Tollens1994). In other cases, impact assessments have been done before enough time had passed to assess impact or its sustainability reliably. As a result of these and other factors, a recent study of impact assessment in international agricultural research indicated that there was little evidence of researchers using ex-post impact assessment (Raitzer, Reference Raitzer2005, p. ii).
In 2002, the CGIAR's Standing Panel on Impact Assessment sponsored a conference to address the question, Why has impact assessment research not made more of a difference? (Watson, Reference Watson2002). Most participants presented mainstream economic impact studies, but a few others presented alternative approaches that aimed to promote learning and programme improvement.Footnote 1 Discussions generated three broad ideas for improving the use of future impact assessments (Watson, Reference Watson2002, p. ix): (1) better matching of impact assessment results to the needs of decision makers; (2) making impact assessments more credible and more understandable, without losing rigour; and (3) improving methods for assessing a broader array of impacts, beyond traditional economic measures.
The 2002 conference served as a springboard for the emergence of an ‘Institutional Learning and Change (ILAC) Initiative’ in the CGIAR – an informal grouping of individuals who seek to foster learning from experiences that lead to improvements in research programmes and changes in professional behaviours. The ultimate goal of ILAC is to expand the contributions of agricultural research to sustainable poverty reduction by fostering learning and change. One promising mechanism to do that is to enhance the learning and change-stimulating roles of impact assessment and evaluation.
The goal of this paper is to provide insights based on our experience to date for complementing traditional ex-post impact assessments with other evaluation approaches that foster learning and change throughout research and development processes.
BASICS OF INSTITUTIONAL LEARNING AND CHANGE
When the CGIAR System was established in the 1970s, its goal was to raise yields and expand food supplies in developing countries through the application of modern agricultural science to major food crops. ‘International centres of excellence’ were set up to carry out strategic research and to generate new technologies that would be tested by national programmes and transferred on to farmers. The first two CGIAR centres – the International Maize and Wheat Improvement Centre (CIMMYT) and the International Rice Research Institute (IRRI) – contributed to substantial increases in per-hectare yields and total production of wheat and rice, particularly in Asia (Evenson and Gollin, Reference Evenson and Gollin2003).
Over time, development goals and processes have become much more complex. Rather than focusing primarily on increasing the yields and production of basic staple foods, the global community is increasingly concerned with resolving complex issues such as poverty and environmental degradation. The CGIAR Centres have found it necessary to modify both their research agendas and their modes of operation to address the new and more complex goals and institutional settings that characterize the developing world today.
The CGIAR is struggling with a difficult transition from a model of ‘centres of excellence’ dedicated to strategic research to a more open and flexible model of ‘facilitators of innovation’ (World Bank, Reference World2006). In this context, the ILAC Initiative is testing approaches to encourage continuous learning and change as essential features of research for development. Central to ILAC are ideas about innovation processes, the role of learning in innovation and the role of evaluation in learning. In the perspective of an innovation system, an international agricultural research centre needs to position itself strategically among the many other inter-related entities responsible for innovations. Learning is an essential feature of innovation processes. Evaluations and impact assessments can be powerful tools for fostering learning and innovation.
Innovation systems and ILAC
In searching for explanations for why some economies – for example the UK – were successful in producing Nobel prize winners, but less successful than say Japan in applying new knowledge to increase international competitiveness, science policy analysts have developed the notion of an innovation system. A central insight of the innovation system approach is to shift attention from ‘research’ by itself to ‘innovation processes’ in which research is just one part (Barnett, Reference Barnett2004; Hall et al., Reference Hall, Bockett, Taylor, Sivamohan and Clark2001).
Focusing attention on innovation as a complex process that involves research as an integral part of the whole rather than as a stand-alone activity has important implications for the way CGIAR Centres should operate and how they should be evaluated. As innovation involves a range of research and non-research tasks, researchers can contribute most effectively to innovation when they collaborate with others in partnerships, alliances and coalitions. Consequently, impact assessments should focus not only on research inputs and outputs, but on the broader system in which research is embedded and the contribution of research to technical and institutional changes.
The crucial role of learning
Innovation studies highlight the importance of learning from testing and improving potential new technologies under field conditions. New technologies produced by research laboratories are very seldom widely disseminated and adopted by economic agents without considerable modification resulting from field testing (Douthwaite, Reference Douthwaite2002; Juma and Yee-Cheong, Reference Juma and Yee-Cheong2005). Management and organizational development studies also show that learning – at the level of the individual, the programme unit and the organization as a whole – is critical for maintaining the vitality of organizations, especially those engaged in research and development activities (Argyris, Reference Argyris1999; McElroy, Reference McElroy2003; Senge, Reference Senge, Kleiner, Roberts, Ross, Roth and Smith1999).
The organizational development studies of Argyris (Reference Argyris1999) and others show that most learning occurs as a result of reflecting on errors (deviations of results from expectations). Consequently, learning requires admitting to problematic situations and failures, which are an inevitable part of innovation. Unfortunately, there is little room to admit to errors in the ‘logframe culture’ of many development organizations (including CGIAR Centres), where overly simplistic and rigid planning paradigms assume that goals will be achieved if activities are carried out as planned (Gasper, Reference Gasper1997). ILAC seeks to help researchers and managers avoid major failures by identifying and resolving problems on a more continuous basis, long before they reach career-, project- or institute-threatening proportions.
Use of evaluation for learning and change
Many specialists in management and organizational development view evaluative inquiry as a potentially powerful tool for learning and change (Patton, Reference Patton1999; Preskill and Torres, Reference Preskill and Torres1999; Sonnichsen, Reference Sonnichsen2000). However, organizations – including those dedicated to research and development activities – often exhibit limited capacity to learn from their own experiences, especially what they consider failures (Carlsson and Wohlgemuth, Reference Carlsson and Wohlgemuth2000).
In the CGIAR, impact assessments and evaluations are geared primarily to providing information for fund-raising and satisfying donor accountability requirements. Learning and programme improvement are considered to be desirable results, but little is done to achieve them. In fact, requirements for external evaluators to work at ‘arm's length’ from projects, programmes and their staffs, to maintain neutrality and impartiality, often inhibit learning.
There is a rapidly growing international community of professional evaluators concerned with designing and implementing evaluations that not only satisfy accountability requirements but also meet the needs of intended users.Footnote 2 Patton (Reference Patton1997) and others advocate a ‘utilization-focused’ evaluation approach that involves intended users in the evaluation process. Participation in an evaluation has been shown to develop skills for reflection, evaluation and learning; to forge common understandings of the programme under review; and to clarify how the programme relates to the organization's broader mission and goals. Consequently, involvement in evaluation processes can have significant and lasting effects on the knowledge, attitudes and skills of people and on their subsequent decisions and actions (Mackay and Horton, Reference Mackay and Horton2003).
The ILAC Initiative seeks to identify potentially useful approaches for fostering learning and utilization-focused evaluation and to promote experimentation with these methods. The goal is not to supplant or replace economic ex-post impact assessment or accountability-oriented evaluations, but to complement them with evaluative activities that promote more learning and programme improvement or to reorient the process by which economic or other methodologies are applied.
EXPERIENCES WITH ILAC IN THE CGIAR
This section describes six cases supported by the ILAC Initiative in which CGIAR researchers are experimenting with new ways to learn and promote change through the use of impact assessments and other evaluative activities.Footnote 3
Promoting learning and reflection processes at CIAT
The ‘Mainstreaming Impact’ group at the International Center for Tropical Agriculture (CIAT) searches outside the agricultural research and development community for methods to foster learning and change, and adapts these methods to promote rural innovation. Such methods include ‘innovation histories’, ‘appreciative inquiry’, ‘social network analysis’, ‘most significant change’ and ‘impact pathway analysis’.
The innovation history method has evolved from being a researcher-led investigation to one in which people involved in innovation processes construct a shared understanding of how innovation occurred (Douthwaite and Ashby, Reference Douthwaite and Ashby2005). At CIAT, researcher-led innovation histories were developed for Local Agricultural Research Committees (CIALs – Spanish acronym) and cassava processing milling technology. The latter produced findings that were highly sensitive and controversial. This experience motivated the CIAT group to develop a more participatory method for developing innovation histories. The Pan-Africa Bean Research Alliance, reviewing work in Uganda through early 2005, learned about the crucial role played by partnerships that they had not realized before. Actor network maps helped participants see that innovation and impact emerges from a complex and messy process of interaction, rather than being delivered by research to extension and then on to farmers.
In late 2005, the CIAT Mainstreaming Impact group helped apply the innovation history method to draw ‘partnership lessons’ from four natural resource management (NRM) projects in India. In a workshop setting, participants constructed ‘partnership histories’ and identified several lessons for partnering. These led to a policy brief prepared for senior research managers and policy-makers. A ‘partnership resource document’ was also prepared to assist research practitioners as they form new partnerships to pursue NRM objectives.
In 2004, a member of the CIAT Mainstreaming Impact group learned about ‘appreciative inquiry’ (Cooperrider et al., Reference Cooperrider, Whitney and Stavros2003) and the ‘most significant change’ monitoring and evaluation approach (Davies and Hart, Reference Davies and Hart2005) at a conference of the American Evaluation Association. Subsequently, an appreciative inquiry workshop was held at CIAT and most significant change was adopted in two CIAT projects in Latin America. In 2005, the CIAT group began leading an effort to construct impact pathways for projects within the CGIAR's Challenge Program on Food and Water. The impact pathways include both (a) causal chains of activities, outputs and outcomes that show how a project achieves its purpose and goal, and (b) network maps that show the evolving relationships between implementing organizations, partners and beneficiaries. The network maps give the approach an innovation systems perspective because they show innovation emerging from a network, not a linear ‘pipeline’. This method borrows components of innovation histories, appreciative inquiry and most significant change. Application of the method made use of facilitation skills learned at an ILAC-sponsored workshop.
The main result of the impact pathway workshops was unexpected: the Challenge Program Secretariat expressed interest in adopting an impact pathway perspective in other Program activities and commissioned the CIAT group to provide methodological support to achieve this. Such a development would mean the incorporation of an innovation systems perspective into the management practices of one of the CGIAR's largest research programmes.
Promoting change through analysis of ‘institutional histories’ at ICRISAT
The International Crops Research Institute for the Semi-Arid Tropics (ICRISAT) has developed a method – institutional histories – to engage scientists on reflecting on institutional arrangements that foster or hinder change. This involved a detailed narrative or story of change with all its complexities and conflicting views of scientific practice of a research group – the natural resource management (NRM) research at ICRISAT for a period of 30 years. The institutional history revealed that the scientists at ICRISAT had developed a novel approach to watershed-based research in order to address the complex conditions found in the semi arid tropics. They also developed new mechanisms of working with partners but were constrained by impact assessment methodologies that only measured success of technological packages and failed to value institutional innovations (Shambu Prasad et al., Reference Shambu Prasad, Hall and Thummuru2006). Despite high estimated rates of return that were observed in research trials, most farmers rejected the technological package. Hindering better technological uptake were institutional constraints that conceptualized innovation as a linear process and that prevented scientists and farmers learning from failure and from the several institutional innovations by the research team. These included use of the ‘watershed’ as a basis for research in semi-arid regions, work in interdisciplinary teams in complex and diversified farming systems and learning how (and how not) to work with the national agricultural research system (NARS) of India and the role of non-research actors in technology uptake.
These tacit but important institutional lessons remained uncaptured, and it was only in the late 1990s that ICRISAT was able to recognize the importance of working with non-research actors leading to a successful consortium approach. The draft institutional history was widely circulated among current and past scientists to elicit reflections on the research process and engage the scientists through a seminar at ICRISAT. Scientists and research managers proposed different reasons for the ‘turnaround’ from failure to repeated success in recent years, but few felt it was due to the technology alone. ICRISAT's partners also reflected on the institutional history, and many of them praised ICRISAT for allowing a critical assessment that was not designed solely to celebrate successes. Some of the NARS partners wanted similar exercises with institutional histories for their organizations and programmes. With increasing diversity of partners and complexity of institutional relations, scientists recognized that a broader type of evaluation, beyond economic impact assessment, was required to obtain insights on linkages and partnerships.
On the basis of the response to the first institutional history, the approach was extended to an ongoing project to foster real-time reflection and learning about institutional arrangements. A project with diverse partners and an unconventional non-governmental Indian donor was chosen. The second institutional history revealed interesting insights on how projects could use actor-oriented tools to explore, reflect and plan for improved relations and linkages in multi-partner projects. Scientists at ICRISAT are beginning to see their own work differently following experience with the institutional histories, and have shown greater sensitivity to institutional arrangements in their own project reports, documentation and journal articles. With greater openness to reflect on both successes and failures and to accommodate different viewpoints, scientists are realizing that ICRISAT's main international public goods are not its technologies on their own, but its approaches to agricultural research, through partnerships, that make its (and others’) technologies relevant and useful.
Facilitating social learning among colleagues in Papa Andina (CIP)
The ‘Papa Andina’ regional initiative at the International Potato Centre (CIP) has developed a participatory evaluation approach, known as Horizontal Evaluation, that fosters programme improvement and knowledge sharing among network participants. The horizontal evaluation approach was developed because of frustrations with traditional external programme reviews and also with unstructured study visits for professionals in research and development organizations in Bolivia, Ecuador and Peru. Horizontal evaluation combines self-assessment and external review by peers (Thiele et al., Reference Thiele, Devaux, Velasco and Manrique2006).
The central element of a horizontal evaluation is a workshop that brings together a group of local participants, who are developing a new research and development (R&D) methodology and a group of peers (referred to here as ‘visitors’) from other organizations and mostly from other countries in the region, who are interested in learning from the experience. The workshop combines presentations about the methodology, field visits, small group work and plenary discussions. Views of the two groups concerning the strengths and weaknesses of the methodology under development are elicited and confronted. Suggestions are generated for improving the methodology, and further development of the methodology is encouraged in other settings.
Four horizontal evaluations have been done so far, and all four have led to substantial learning. For example, in the workshop in Peru, local participants recognized that farmers had not participated much in the participatory market chain approach (PMCA), but attributed this to the distance between Lima (where most activities had taken place) and rural production areas. Visitors saw a more profound problem and noted the lack of mechanisms for empowering farmers and a missing link between product development and poverty reduction. As a result of the evaluation, the Peruvians began efforts to involve and empower farmers. During the same evaluation, Bolivian participants were impressed with the rapid results obtained in Peru. Subsequently, with help from Papa Andina, they began experimenting with and further developing the PMCA in Bolivia.
The horizontal evaluation workshops helped build trust between partners in the three countries and led to the emergence of an informal community of practice. Gradually, the role of Papa Andina has shifted from one of guiding technology development to one of facilitating knowledge management.
Expanding the impact assessment framework at CIMMYT
CIMMYT has a strong tradition of impact assessment (Pingali, Reference Pingali2001). Until recently, work focused on the adoption of improved germplasm and rates of returns on research investments. Little attention was directed at measuring impacts in terms of poverty reduction. In 2005, CIMMYT's Impacts Targeting and Assessment (ITA) unit began to develop and promote a more wide-ranging type of impact assessment.
A comprehensive process was used to change the way that CIMMYT conceptualizes and assesses impact that included: collectively assessing the level of understanding of impact and how it is assessed among CIMMYT's social and biophysical scientists; developing a framework for impact that is more people centred, with a focus on systems and livelihoods improvements; developing a learning and operational platform for CIMMYT-wide impact assessment; and strengthening individual skills and capacity for high-quality impact assessment research (La Rovere et al., in press).
The process comprised workshops to increase staff awareness of the importance of impact assessment and to offer tools to carry out impact assessment. Using a proposed framework as a basis, participants discussed case studies, describing intended users and uses of information generated, potential stakeholders involved, topics to be assessed, critical questions to be asked, expertise required and ways to mobilize resources.
Follow-up actions included discussions to strengthen the management buy-in for impact assessment, project-based training courses conducted in Africa and Asia in 2006, steps to institutionalize impact assessment in proposals and ensure its inclusion in work plans, and a workshop in late 2006 to outline the impact pathways (using an approach described in Douthwaite et al., Reference Douthwaite, Kuby, van de Fliert and Schulz2003) of CIMMYT projects identified in its Medium-Term Plan.
As a result of these activities, participants gained a better understanding and appreciation of the principles of impact assessment and its practical uses. Behavioural change, chiefly for project managers, was often reflected in a much clearer attention to impact assessment in projects and by a common language used in collegial discussions and in developing Medium-Term Plans. Templates for project proposals have also been introduced that contain budget lines for impact assessment. Partly as a result of these activities, impact assessment has received more consideration by CIMMYT management and Trustees.
One example of the use of the new approaches to impact assessment is a study of the impacts on livelihoods of past CIMMYT work on maize technology and capacity building in southern Mexico. This study developed and applied livelihood-based household surveys, and analysed the data by econometric tools integrated with participatory and spatial tools, to assess impacts on livelihoods. A second example is a project to assess the impacts of long-term investments in agricultural technologies by the Sasakawa Global 2000 association and its partners, in Ethiopia and Uganda. A monitoring system was developed to provide feedback on activities and results to local development actors, to allow them to adjust their efforts during the project cycle and improve decision-making, hence fostering the learning function of impact assessment. A third example is the development of impact assessment guidelines for use in CIMMYT to support scientists in developing proposals, designing and carrying out impact studies and properly budgeting for impact assessment.
Building learning and livelihoods into an impact assessment: experiences of Bioversity International and bananas
In 2002, Bioversity International was invited to collaborate in an ex-ante economic impact assessment of genetically modified banana cultivars in Uganda and Tanzania. The economic impact study involved a survey of 400 households over a year, with monthly repeat visits and three additional interviews on special topics. Given previous experiences in the CGIAR with the sustainable livelihoods (SL) approach to assess the poverty impact of new technologies and because of emerging interest in learning and change (Adato and Meinzen-Dick, Reference Adato and Meinzen-Dick2002), Bioversity sought to integrate SL research with the economic analysis and to add a learning component. The intention was to create a multidisciplinary and SL-oriented research team with sociology and economics MSc and PhD students, their advisors, and scientists from national universities and research institutes, IFPRI and Bioversity. While some interaction among different disciplines and organizations was achieved, the ex-ante economic and SL components were not successfully integrated, largely because of inadequate funding support.
A small pilot study sought to test mechanisms for learning and change in the context of the economic analysis. Bioversity organized focus groups and carried out structured interviews in national research institutes, field organizations and communities. Managers from national research organizations explained how they gathered information from stakeholders to prepare annual and medium-term plans and identified potential uses of impact assessment studies. They mapped their organization's approach to achieving impact using cards and markers. Community members described the challenges they have faced in recent decades, major changes that had taken place and how those came about, and identified possible partners in future innovations.
Data from the household survey, focus groups and structured interviews were presented in final workshops held in each country, with banana farmers, NGO representatives, extension agents and researchers to identify lessons for banana development projects and to map potential impact pathways. Three lessons were identified. First, the interests of potential users of results should be taken into account when impact studies are designed and carried out. As one research manager put it, ‘Our research organization has only a limited choice of actions. I am more interested in how to have impact than just measuring whether it has occurred.’ Second, to achieve impact, learning-oriented project implementation is critical. Participants noted that the duration of most projects is too short to achieve much impact, but at the same time expectations that impact will be achieved are unrealistically high. Moreover, many project teams work in isolation from other development efforts and appear not to learn from either their own or others’ experiences. To improve learning by project teams, community members and other local stakeholders, small studies are preferred over costly large-scale surveys. Third, building capacity to implement a learning-oriented approach will be a challenge, because of the time, resources and training required.
Developing an ILAC Unit at Bioversity International
Until recently, there was no coordinated responsibility for planning, monitoring, evaluation and impact assessment at Bioversity. The Director of Research was responsible for monitoring and evaluation, the Assistant Director General was responsible for impact assessment and neither of these functions was directly linked to planning or decision making. In 2005, Bioversity prepared a new strategic plan that put poor people at the centre of its agenda and reorganized to deliver on it. An ‘ILAC Unit’ was established to harmonize, coordinate and support all of the institute's monitoring and evaluation and impact assessment, and much of the planning. The Unit Head reports to Bioversity's senior management team.
Impact assessments are now planned and budgeted within an overall management cycle for Bioversity's programme and are timed to provide results for periodic Centre-Commissioned External Reviews (CCERs). The ILAC Unit is now responsible for annual performance measurement and project reporting. This ensures correspondence between the indicators used in performance measurement and those used in other aspects of planning, monitoring and evaluation, such as CCERs, Medium-Term Plans and annual project reports. When impact assessments and evaluations are conducted, a use-oriented approach is applied whereby scientists and managers are actively engaged in a task force to design the assessment. The roles of task force members include verifying intended impact pathways, helping to identify key questions for analysis and developing recommendations from findings. In addition to assessing the results of work, efforts are made to understand the processes by which results are achieved, and could be improved.
The ILAC Unit also supports many of Bioversity's planning functions, including preparation of the Medium-Term Plan and annual project work plans. The unit also functions as the secretariat for the institute's Priorities and Strategies Committee – the main group responsible for the scientific direction of the institute. Through its work in facilitating the committee's meetings and following up on decisions taken, the ILAC Unit promotes use of results of monitoring and evaluation of all kinds by managers in the institute's major scientific decision-making processes, and helps to ensure that senior management supports impact assessment as a key source of information for decision making.
DISCUSSION
In this section, we review the experiences with ILAC outlined in the previous section in relation to the three major ideas for improving impact assessment, which emerged from the 2002 conference on ‘Why has impact assessment research not made more of a difference?’ and in relation to institutionalizing new approaches.
Idea 1. Better matching of impact assessment research to the needs of decision makers
In many cases we – like traditional impact assessors – have been better at carrying out studies than at influencing the decisions of research managers, scientists and other policy makers. Nevertheless, some notable gains have been made. Among the CG Centres, Bioversity has made notable progress in institutionalizing the link from impact assessment and evaluation activities to decision making and programme improvement through the establishment of an organizational unit with these objectives. Over time, as the uses of evaluation in programme improvement have been demonstrated, Bioversity's impact assessment work has evolved from an initial focus on providing impact information for public awareness to a broad range of tasks linked to the Centre's priority setting, planning and decision making.
Another noteworthy case is the Papa Andina regional initiative, which has successfully built learning and programme improvement into a participatory evaluation approach that it applies periodically to its main areas of work. Participants in Papa Andina's horizontal evaluations consider them to be useful exercises for understanding the work underway, as well seeing its potential uses in their own settings. Participants in the horizontal evaluations have formulated useful recommendations for improving the work in question and peer evaluators have gained insights for improving their own work.
Another case where impact assessment is influencing decision making is CIAT's recent work with impact pathway analysis, which has struck a chord with the Secretariat of the Challenge Program on Water and Food. Impact pathways, particularly the network components, help both the project teams and the Programme Secretariat understand, articulate and improve their strategies for achieving impact. The programme has found that conventional logframe-type reporting formats are inadequate for the way it works, which involves complex R&D networks rather than a single organization responsible for delivery.
A second CIAT example of linking impact assessment to decision making is the recent adaptation of the innovation history method to involve policy makers in workshops for learning policy lessons and developing briefing materials for other policy makers and research managers.
Idea 2. Making impact assessments more credible and more understandable, without losing rigour
Many in the international research and development community have become sceptical of the consistently high rates of return that are commonly reported in impact assessments. Understandably, most impact assessments have focused on successful cases, on the logic that the few successes in agricultural research pay for the many failures. Nevertheless, research stakeholders, including donors and potential beneficiaries, would like to also know about projects that were less successful and what has been learned from them. Many readers of impact assessment reports also complain that they are highly technical, difficult to understand and offer few practical suggestions for action.
The cases presented in the previous section provide some examples of how impact assessments and evaluations can be made more credible and understandable. For example, at CIP, within the Papa Andina regional initiative, engagement of stakeholders through the application of the participatory ‘horizontal evaluation’ approach has greatly facilitated the understanding and acceptance of evaluation results as valid and useful. Similarly, at ICRISAT, the ‘institutional history’ approach has helped to clarify why certain technologies were taken up by farmers at certain times and others were not. It has also shown how institutional factors can limit widespread use of new technologies that initially looked quite promising. Institutional histories prepared at ICRISAT have proved to be of great interest to the scientists and a source of insights for change. This work has also been popular with donors and others outside of ICRISAT, and it has stimulated new interest in this type of work on natural resources management.
Idea 3. Improved methods for assessing a broader array of impacts beyond traditional economic measures
As already noted, most impact assessments in the CGIAR and the broader agricultural research community have employed agricultural economic methods to assess economic returns to research investments. The cases presented above show how a broader array of results and impacts can be assessed with a range of methods, beyond economics. One of the explicit goals of CIMMYT in developing a new framework for impact assessment is to expand the scope of analysis to embrace a range of impacts on livelihoods and the environment. Bioversity's study of new banana materials complemented an ongoing economic impact assessment study with qualitative analysis of livelihoods impacts and with institutional analysis. CIAT's work on innovation histories and actor network mapping has helped inject a network vision into the linear impact pathway model and it has helped participants better understand the importance of partnerships in generating innovation. Network mapping is emerging as an important tool for helping people to visualize what is meant by an ‘innovation system’, as well as to monitor and evaluate changes to the network.
Institutionalizing learning approaches to impact assessment
The case studies illustrate different paths towards institutionalizing learning approaches to impact assessment within the Centres and their partner organizations. For example, the horizontal evaluation approach developed by Papa Andina has become a standard operating procedure with national partners, but CIP has not yet applied it to its core-funded research. The efforts described to develop impact pathways at CIAT have been taken up by a challenge programme and are being institutionalized within that network of organizations. At CIMMYT and Bioversity, work related to ILAC began at the core of the organization rather than on the periphery and focused on the approaches that guide impact assessment across the organization as a whole. At CIMMYT, a decision was taken to change the way impact assessment is done. This was followed by recruitment and planning for a new unit responsible for impact assessment and some related research. In Bioversity, an ILAC Unit was established that reports to the senior management team, with a broad mandate for evaluation of all kinds, including impact assessment, and with a clear learning objective throughout, to complement evaluation for accountability. Institutionalization has to do with embedding new practices within organizational policies and standard organizational routines and procedures, as well as within the collective culture of the members of the organization. It requires adequate budgets, staffing, skills, and unambiguous support from management, donors and other influential forces. Our cases found that the skills and attitudes of senior and middle management for learning-and-change-oriented impact assessment are highly variable. Meanwhile, such forces as the CGIAR performance measurement system (Science Council and CGIAR Secretariat, 2006) are providing incentives for Centres to strengthen ex-post economic impact assessments, but may discourage experimentation with, and institutionalization of, non-traditional approaches and methods.
CONCLUSION
Impact assessments and evaluations offer great, and largely untapped, potential for improving ongoing and future research and development programmes in the CGIAR. At a time of increased scrutiny of investments in agricultural research, we believe that taking a more eclectic, learning-and-improvement-oriented approach to impact assessment could contribute to bolstering confidence among donors and other stakeholders. It would also help ensure that investments in impact assessment pay off by linking assessments more closely with management decision making.
For impact assessments and evaluations to be used in decision making, it is important that they be designed with specific uses and users in mind and that these users be involved in the evaluation process. Impact assessment methods should build in opportunities for learning and reflection by those responsible for managing programmes that are being assessed. They should also foster interactions among managers, beneficiaries and other stakeholders, to promote social learning and arriving at shared understandings concerning impacts and how they have emerged.
The CGIAR faces an important strategic decision. It can either continue to define impact assessment narrowly, in a way that limits its usefulness for learning, decision making and programme improvement, or it can broaden the standards by which impact assessments are judged, to encourage new types of evaluation that foster learning and change.
Progress made by ILAC in these areas suggests that a quiet revolution in the practice of impact assessment is already underway. This requires further nurturing to help improve the relevance of agricultural research for the millions of poor women, men and children in poor families who could potentially benefit.
Acknowledgements
This paper would not have been produced without support from the Rockefeller Foundation, GTZ–BMZ, IFAD and the Netherlands Ministry of Foreign Affairs (DGIS). Special thanks also to Ronald Mackay, Robert van den Berg and Cristina Sette for their comments and suggestions.
