NEW AID MODALITIES IN RWANDA: SOME BASICS

In the wake of the 1994 genocide, the international donor community flocked to support Rwanda. Many donors later made the transition from humanitarian to structural aid, and at present, the country is highly aid-dependent: between 1998 and 2004, aid averaged 20% of the Gross National Income (World Bank 2006). In the immediate recovery period, it was substantially higher (with an average of 61% over the period 1994–6). ‘Genocide guilt’Footnote ² seems to have fuelled, in part, the financial generosity of international donors, and ‘it opened the door to aid with a political dimension’ (Purcell et al. Reference Purcell, Dom and Ahobamuteze2006: 17). At the end of 2000, the World Bank/IMF endorsed Rwanda's interim PRSP, and its final PRSP was published in June 2002. Since its adoption, it has provided the general framework for donor assistance in Rwanda. Between 2001 and 2004, 18–30% of Rwanda's aid was delivered through budgetary support. The main budgetary donors included the United Kingdom, the World Bank, the EU and Sweden. Belgium, France, Germany, the Netherlands, USAID, Japan and Canada were among the main non-budgetary donors (Purcell et al. Reference Purcell, Dom and Ahobamuteze2006). By mid 2006, Rwanda began preparing for the second round PRSP (this time called the ‘EDPRSP’, with ED standing for Economic Development). The Rwanda Development Partners Website (DPCG 2007) described the drafting process and anticipated a new version by May 2007. By December 2007, however, the document had not been released.

Rwanda is a harmonisation pilot for both the Strategic Partnership for Africa (SPA) and IMF initiatives (Development Gateway 2008). Donors have been regrouped as ‘Development Partners’, and together with the Government of Rwanda (GoR) they have set up an Aid Coordination, Harmonisation and Alignment framework. It is the highest-level coordination structure in Rwanda, drawing its membership from government representatives and heads of cooperation in bi- and multilateral agencies (MINECOFIN 2006). The Development Partners website includes resources and information, and may help to coordinate and improve development activities. Aid harmonisation in Rwanda is, however, hampered by the divide between budgetary and non-budgetary donors, with each camp strongly defending diverging opinions, particularly regarding political and governance conditions in the country (Purcell et al. Reference Purcell, Dom and Ahobamuteze2006; Uvin Reference Uvin2001). On top and in line with this division, is the cleavage between francophone and anglophone-oriented donors (Browne Reference Browne2007; Kanyarukiga et al. Reference Kanyarukiga, Van Der Meer, Paalman, Poate and Schrader2006; Purcell et al. Reference Purcell, Dom and Ahobamuteze2006; Uvin Reference Uvin2001). Whereas Rwandan progress on harmonisation is rated ‘low–moderate’ in the 2006 Survey on Monitoring the Paris Declaration, the same survey considers ‘ownership’ (i.e. ‘a country's ability to exercise effective leadership over its development policies and strategies’) particularly outstanding (OECD/DAC 2007b: 27–2).Footnote ³

M&E IN RWANDA FROM A NARROW TECHNOCRATIC PERSPECTIVE

While many of the details described below are peculiar to the Rwandan situation, they are supportive of a bias towards methodological and technical issues, as noted in several reviews of recipient M&E systems and arrangements (Holvoet & Renard Reference Holvoet and Renard2007; Lucas et al. Reference Lucas, Evans and Pasteur2004; World Bank 2002: 20, 2005). In practice, such reviews involve a strong focus on the handling of inputs, coupled recently with a focus on monitoring poverty outcomes and Millennium Development Goals (MDGs), an upsurge of new rounds of data collection, an emphasis on monitoring (to the detriment of evaluation), and an extolment of targets and indicators, sometimes referred to as ‘indicatorism’. This fragmentary approach on the ground seems to match donors' visions of M&E. Over the years, most recipient countries have been bombarded with Public Finance Management (PFM) assessment missions (see SPA-BSWG 2005: 15–18). M&E capacity has often been reduced to improve statistical data capacity,Footnote ⁴ and targets and indicators are crucial for donors in the context of Performance Assessment Frameworks (PAFs). As the World Bank's Evaluation Capacity Development Unit states in its 2002 Annual Report, ‘PRSPs and Bank/Fund Joint Staff Assessments often use the term M&E as a shorthand for financial tracking and national development indicators’ (World Bank 2002: 20).

Within this narrow version of M&E, Rwanda is not doing badly. On the contrary, as the Joint Evaluation of General Budget Support puts it (Purcell et al. Reference Purcell, Dom and Ahobamuteze2006: 204), ‘in recent years excellent progress has been made in constructing a modern public financial management (PFM) system in Rwanda’. Since 1997, two ambitious rounds of PFM reform have been held to rebuild a system that had totally collapsed during the 1994 genocide. Whereas there remain considerable challenges, the GoR, with support of various donors, introduced a Medium-Term Expenditure Framework (MTEF) in 2000, which sets forth the basis for further PFM reforms (Purcell et al. Reference Purcell, Dom and Ahobamuteze2006: 205). Furthermore, Rwanda is widely praised for fighting corruption (Amis et al. Reference Amis, Jackson and Ahobamuteze2005: 6; Purcell et al. Reference Purcell, Dom and Ahobamuteze2006). It scored 55·8 (on a scale of 0–100) on the sub-indicator ‘control of corruption’ in the latest 2007 update of the Kaufmann, Kraay and Mastruzzi governance indicators, whereas the regional average for SSA stands at 30·3 (Kaufmann et al. Reference Kaufmann, Kraay and Mastruzzi2007). On a critical note, however, the evaluation report states, ‘while corruption is generally perceived as not being a problem in Rwanda, risks of subtle forms of corruption, such as exclusion patterns and concentration of economic power, might be increasing’ (Purcell et al. Reference Purcell, Dom and Ahobamuteze2006: S9).

The GoR has shown a keen interest in improving data collection, which is evident from the emphasis given to the Rwandan National Institute of Statistics (OECD/DAC 2007b). This has resulted, so far, in an upsurge of data collection, especially on poverty, and to a renewed interest in household surveys (e.g. the 2001 and 2005/06 Enquêtes Intégrales sur les Conditions de Vie des Ménages, the 2001 and 2005 Demographic and Health Surveys, and the 2001 Core Welfare Indicators Questionnaire). With a score of 71 (scale of 0–100) on the 2007 Statistical Capacity Building Indicator,Footnote ⁵ Rwanda does better than most low- and middle-income countries in the world (average score of 66) as far as statistical capacity is concerned (World Bank 2008). The growing attention to qualitative, difficult-to-measure, issues has also given an impetus to survey approaches which seek to combine qualitative and quantitative methodologies, as well as to the usage of more participatory techniques. In the context of the first full PRSP, Rwanda conducted a Participatory Poverty Assessment (PPA) and introduced a mechanism (Ubudehe) for ‘grassroots collective decision making and action’ (Evans et al. Reference Evans, Piron, Curran and Driscoll2006: 62). Additionally, the University of Butare, in collaboration with the Rwandan chapter of OSSREA (Organisation for Social Science Research in Eastern and Southern Africa), conducted M&E exercises aimed at incorporating the visions of the population. In 2002, it published the results of the Poverty Reduction Policies Relevance Test,Footnote ⁶ and in February 2006, a report based on Citizen Report Cards – a grassroots participatory method of data collection that provides insights into citizens' appreciation of services provided. Various sources highlighted the ‘unprecedented levels of citizen participation’ in these initiatives (Evans et al. Reference Evans, Piron, Curran and Driscoll2006: 62). There are, however, others (Renard & Molenaers Reference Renard and Molenaers2003) who indicated that these processes were largely government-controlled and ignored the involvement of traditional independent M&E demand side actors (e.g. civil society organisations or CSOs).

There is more agreement among various sources on the fact that the significant upsurge in data collection stands in sharp contrast to a serious deficit in analysis of the data gathered (Amis et al. Reference Amis, Jackson and Ahobamuteze2005: 10; Kanyarukiga et al. Reference Kanyarukiga, Van Der Meer, Paalman, Poate and Schrader2006; Mutebi et al. Reference Mutebi, Stone and Thin2001: 35; Purcell et al. Reference Purcell, Dom and Ahobamuteze2006). Data collected in the context of the Poverty Reduction Policies Relevance Test and the Citizen Report Cards are not explicitly linked to specific policies, and disaggregation to relevant social categories is mostly absent. Differential effects of measures on income groups, regions, and along urban-rural and gender lines remain underexplored. Evidently this severely hampers the usefulness of data collection for policy making and evaluation. A noteworthy attempt at moving beyond pure data collection and ‘stocktaking’ is the 2007 Preliminary Poverty Update Report, which is based on a comparison of results of the latest 2005/06 household survey with the 2000/01 exercise. It provides, for the first time, a frank overview of changes in poverty and inequality, documenting that unequal distributions of growth, in combination with high initial levels of inequality, have impaired the potential poverty-reducing effect of growth (NISR 2006).Footnote ⁷ In various places, different layers of disaggregation are included (i.e. gender, urban/rural, provinces). While this ‘evaluative’ initiative is laudable (identifying not only ‘levels of outcomes’ but also ‘changes in outcomes’), there are still major shortcomings when it comes to the analytical and evaluative quality of the exercise. The analysis does not consider confounding factors to arrive at some measure of ‘impact’, it hardly explores the underlying reasons for varied performance, and it does not distinguish between implementation failures and conceptual flaws in policies.

The gap between data collection and analysis is certainly not unique to Rwanda (CGD 2006), nor is the emphasis on ‘monitoring’ at the expense of ‘evaluation’ (Holvoet & Renard Reference Holvoet and Renard2007; World Bank 2002: 20). In spite of the fact that the MINECOFIN 2005 Cabinet Paper on M&E lends more importance to the evaluative function, the earlier emphasis on monitoring in Rwanda's 2002 PRSP persists. As in other countries, the impression is often given that monitoring somehow also constitutes evaluation. We acknowledge that monitoring is of the utmost importance, and even that more energy needs to be devoted to it than to evaluation, but there remains a persistent need for the latter. One of the consequences of favouring monitoring over evaluation is that the focus is much more on performance stocktaking (‘Were the targets met?’), rather than on performance analysis (‘Why were the targets not met?’), or on questioning the usefulness of the targets themselves (and/or of the interventions designed to meet them).

In line with the above-mentioned efforts in data collection and monitoring, Rwanda is also improving its indicators and targets. From the Interim PRSP to the full PRSP and the Annual Progress Report, the list of indicators has become more refined. In the context of PRSPs, Performance Assessment Frameworks (PAFs) and new aid modalities in general, there is an evolution towards a greater focus on a limited set of ‘aggregate’ indicators and targets. We do not argue against concise lists of indicators, but a unique focus on ‘aggregates’ easily hides distortions at disaggregated levels (Ravallion Reference Ravallion2006, Reference Ravallion, Jenkins and Micklewright2007). This evolution is particularly relevant in the context of countries such as Rwanda, where concerns are raised over increasing levels of inequality and potentially exclusionary poverty reduction policies and outcomes (Evans et al. Reference Evans, Piron, Curran and Driscoll2006). According to Prennushi et al. (Reference Prennushi, Rubio G. and Subbarao2001: 111), ‘the decision on the level of disaggregation of indicators is as important as the choice of indicators itself’. Particularly relevant in the Rwandan case are existing, but often concealed, horizontal inequalitiesFootnote ⁸ and how the government is handling them. While these tend to go unnoticed in the PRSP's focus on the ‘aggregate picture’, Brown and Stewart (Reference Brown and Stewart2007) identify at least three instrumental reasons for their importance: horizontal inequalities can negatively impact poverty reduction targets and economic efficiency; they can be a source of violent conflict. The move towards the ‘aggregate level’ clearly suits the GoR (e.g. it is virtually taboo to refer to ethnic differences),Footnote ⁹ as does the narrow technocratic vision on M&E in the Paris Declaration. In what follows, we broaden the scope of the discussion from the methodological and technical to the institutional and systemic.

… AND FROM AN INSTITUTIONAL AND SYSTEMIC PERSPECTIVE

While institutional and systemic issues have generally been neglected in M&E, under the new aid modalities a gradual recognition of their importance has started to emerge, at least within some donor agencies (Bedi et al. Reference Bedi, Coudouel, Cox, Goldstein and Thornton2006; GTZ/BMZ 2004; Kusek & Rist Reference Kusek and Rist2004). So far, however, this has not been translated into changes in donor assessment exercises or practices on the ground (in Rwanda or elsewhere). In what follows, we will try to fill this gap by focusing on: (i) the institutional set-up of Rwanda's M&E system, including issues of oversight, independence, horizontal and vertical integration; (ii) the role of non-state actors (parliament, the Auditor General's Office, research institutes, national and international NGOs, etc.) as sources of M&E supply and demand; and (iii) donors' M&E behaviour in Rwanda.

BELOW THE SURFACE: THE DENIAL OF POLITICS IN M&E UNDER THE NEW AID MODALITIES

While differing M&E observations (and conclusions) from a narrow methodological and technical perspective on one hand and from a broader institutional and systemic view on the other may appear surprising, these can, to a large extent, be attributed to one of the most serious flaws in the new aid paradigm: the denial of ‘politics’. ‘The conception of the PRS approach as being based upon an unwarranted faith in a technocratic, depoliticised mode of governance’ (Craig & Porter Reference Craig and Porter2003) is increasingly acknowledged in academic aid literature (Armon Reference Armon2007; Booth Reference Booth2005; Driscoll & Evans Reference Driscoll and Evans2005). This suggestion particularly holds when it comes to ‘monitoring and evaluation’, where donors and recipients both adopt a skewed technocratic approach, vigorously neglecting the fact that M&E takes place within a socio-political setting that can heavily impinge upon it.

The fact that politics are part and parcel of M&E has long been acknowledged in the context of project and programme evaluation (Palumbo Reference Palumbo1987; Weiss Reference Weiss1970, Reference Weiss and Palumbo1987). First, evaluations are supposed to feed into decision making and reports necessarily enter the political arena.Footnote ²⁰ Second, evaluations implicitly make political statements about the legitimacy, utility and appropriateness of projects and/or programmes. Third, since projects and/or programmes are creations of political decisions that remain subject to pressure(s) during implementation, the issue of politics is inevitable during evaluation (Weiss Reference Weiss1970). While the relationship between politics and evaluation is potentially damaging, not all political aspects of evaluation need to be understood negatively. For example, for utilisation-focused evaluations, usage is the driving force. Consequently the evaluator is encouraged to tie the evaluation into the specific political context, and the needs of the users (Chelimsky Reference Chelimsky1987a, Reference Chelimsky and Palumbo1987b; Patton Reference Patton and Palumbo1987; Weiss Reference Weiss1999).

With the broadening of programme scope, as indicated by Weiss (Reference Weiss1970), the politics of evaluation will become more pervasive and reach the national level. This is also true when M&E concerns interventions at the sectoral and national levels, and stakeholders and interests are multiplied. In this respect, Killick (Reference Killick2004: 5) refers to the often made suggestion that ‘the potential seriousness of the tensions among evidence on the one hand and preferences of politicians and incentive structures to which they are responding has been magnified by the trend within the new aid agenda for aid modalities to become more “macro”-based’.

Moreover, the new aid paradigm rightly puts the national government in the driver's seat, also when it comes to M&E. One can assume therefore that national politics and the political context in which stakeholders, both national and international, operate will have an influence on various dimensions of M&E. While some recent contributions have pointed to the importance of taking a broader perspective on M&E (Bedi et al. Reference Bedi, Coudouel, Cox, Goldstein and Thornton2006; Kusek & Rist Reference Kusek and Rist2004), there has so far been no research that explicitly explores the linkages between ‘politics’ and M&E in the context of the new aid modalities. Though it is beyond the scope of this article to fill the gap, we point to some elements that might be worth further exploration. As in previous sections, we illustrate with Rwanda case study material, but the general discussion is applicable to all countries where the new aid modalities are applied.

One concept from political theory that might prove useful in structuring future research on the linkage between ‘politics’ and M&E is the ‘Political Opportunity Structure’ (POS), which refers to the political context in which stakeholders operate. Initially the POS was considered a static framework of institutional variables, including stable aspects of government structure such as the administrative structures and/or institutional entities that could influence the position and power of stakeholders in society (Tilly Reference Tilly1978). Later, POS theory evolved towards a more dynamic model, giving recognition to the influence of more volatile aspects such as national policies, turnover of power and shifts in electoral outcomes (McAdam Reference McAdam1999). The POS of a country is considered to be in continuous evolution, with the evolution of both stable and volatile aspects depending very much upon the interests and power of the political elite. Changing or influencing the POS is not, however, the sole sanctuary of this elite. By defending and/or furthering their own interests, various stakeholders, including donors and CSOs, may influence the POS (Meyer & Staggenborg Reference Meyer and Staggenborg1996). Of course, the ‘action radius’ of non-state actors will be circumscribed by government organisations and structures.

In terms of M&E, it is obvious that different stakeholders may have their own interests, and the extent to which they are able to make their interests heard largely depends upon the POS. At the very least, there may be serious tensions between the political elites and ‘M&E’. M&E is about ‘learning’ and ‘accountability’. While, ideally, a government that is committed to inclusive development and poverty reduction will be eager to learn in order to improve its policies and outcomes, it would be naive to expect that political leaders and bureaucrats are always eager to learn from evidence. When less is known about what works and what does not, political leaders increase their room for manoeuvre (CGD 2006). Investment in evaluative exercises and ‘knowledge’ can even become counterproductive when, in cases like Rwanda, substantial amounts of general budget support can be generated without strong ‘poverty-reducing credentials’. As Pritchett (Reference Pritchett2002: 268) puts it, ‘if a program can already generate sufficient support to be adequately funded then knowledge is a danger’. Tensions between political elites and M&E are particularly obvious when it comes to the ‘accountability’ function of M&E. When policies are not implemented or targets not met, this may have implications for the politicians who were engaged to deliver them. As Booth (Reference Booth2005: 5) points out, ‘The easy notion that donors only have to back off from demanding accountability for aid funds in order for domestic accountability to flower in its place is unrealistic and unhistorical.’ Authoritarian regimes in particular are unlikely to stimulate the accountability aspect of M&E as this may directly jeopardise their power position: ‘… the more dramatic the power asymmetries among actors – that is the more power the political élites possess relative to the power of citizens and their political opposition and other actors – the weaker the incentive is for such an élite to transform established M&E programmes into an instrument for more public sector accountability’ (Gordillo & Andersson Reference Gordillo and Andersson2004: 309).

The actions undertaken by governments to curtail critical analysis and insight may range from overt forms of repression and human rights abuse (e.g. no/limited freedom of speech, arbitrary detentions, etc.) to more covert forms. Institutional autonomy can be impeded; parliamentary power can be curtailed; autonomous organisations (such as the Auditor General's Office) can be captured or infiltrated by another government agency (co-optation), etc. At times, governmental interference can be very subtle. For example, overcentralised functioning can be used to delay and suffocate initiative, especially when coupled with the monitoring of independent actors through central registration, compulsory reporting, etc.Footnote ²¹

While there are dramatic differences in donors' assessments of Rwanda's governance situation, mostly between budgetary and non-budgetary donors, there is no disagreement that ‘voice and accountability’ is, indeed, a problem in Rwanda. Particularly illuminating in this regard is the latest update of the governance indicators of Kaufmann, Kraay and Mastruzzi (Reference Kaufmann, Kraay and Mastruzzi2007), which point to sharply differentiated rankings for Rwanda on ‘control of corruption’ (55·8 percentile rank) and ‘voice and accountability’ (14·4 percentile rank). Complementary to these indices are more qualitative contributions from other sources (Jordaan Reference Jordaan2006; Reyntjens Reference Reyntjens2004), which illustrate that several government actions and incidents have led to a very restrained political climate and a serious curtailment of freedom of association and action. In Rwanda, 2003 was particularly pernicious for the freedom of speech. In that year the opposition party, Mouvement Démocratique Républicain (MDR), was banned and several CSOs (including the leading human rights organisation Liprodhor) were bashed in a parliamentary report. The run-up period to the constitutional referendum and the local elections further narrowed room for criticism and freedom of speech. Add to this the very restrictive law on CSOs that stipulates that ‘associations have to apply yearly for a renewed recognition; that the Minister of Justice may suspend the activities of an organisation when she/he discretionally judges them a threat to the law, public order or good moral standards of the country’ (Law N°20/2000), then the self-censoring behaviour of CSOs (and international NGOs) starts to make sense. The least one can say is that Rwanda's ‘straight-jacket-like’ POS is not encouraging the development of an independent M&E demand side. Importantly, this hinders the operation of PRSP logic and its related paradigm that need independent M&E as sources for evidence-based learning and accountability.

In heavily aid-dependent countries, such as Rwanda, the POS is co-determined by the international (official) donor community, which also has its interests in partner countries and in the entire development enterprise. Not only does the GoR want to convince contributors and bystanders that it is doing well, donors have a similar stake in presenting ‘success’ to their constituencies (Marriage Reference Marriage2006). Budgetary support donors, in particular, cannot afford to be too openly critical of the national M&E systems that constitute a growing (input) source for their own reporting, without by the same token undermining the credibility of the latter. From that point of view, the not-too-critical approach is understandable. The lack of an open and critical approach can sometimes be justified on the condition that alternative more diplomatic (and hopefully effective) forms of pressure will pay off. We concur that politically sensitive matters should generally not be dealt with aggressively and in the open. Yet, when policy dialogue is chosen as the instrument, its effectiveness should be followed up. One could even argue that in states with non-enabling POS, where accountability is almost by definition problematic, more emphasis could be put on learning rather than accountability (Gordillo & Andersson Reference Gordillo and Andersson2004). However, the temporary downscaling of accountability can only be justified if learning is truly taken seriously by all involved. Currently, however, donors focus overwhelmingly on ‘procedures’ and ‘processes’. They also have a tendency to neglect matters of ‘substance’, even if various sources have raised substantial concerns over potentially exclusionary policies and outcomes on the ground. We acknowledge that this often involves sensitive and complex issues, but allowing a government to practically declare such issues ‘off limits’ for serious M&E is unlikely to contribute to evidence-based poverty reduction and inclusive poverty reduction policies. In fact, ignoring politically sensitive issues in evaluations and/or reviews can fundamentally affect the process to the point that it not only becomes oblivious of political flaws, but is also technically flawed and/or defective.

***

The new aid paradigm both reconfirms and redefines the importance of monitoring and evaluation. The 2005 Paris Declaration sets out an ambitious M&E reform agenda for both recipients and donors. Recipients are expected to elaborate an effective national M&E system which is critical for the functioning of key components such as ‘results-orientation’, ‘iterative and evidence-based policy making’, and ‘downward accountability’. At the same time, donors are asked to wind down their own parallel and independent systems, and move towards integrating into the national M&E system. So far, donors and recipients have adopted a fairly fragmentary approach towards M&E, emphasising input accountability, indicators and targets, data collection and monitoring of inputs and MDGs, largely neglecting evaluative analysis and, in particular, the broader more institutional and systemic issues. This generally observed phenomenon has similarly been evidenced with case study material from Rwanda. While Rwanda does score well on a narrow technocratic level, there are much more serious shortcomings at the broader systemic and institutional levels.

We have argued that these striking differences boil down to one of the most serious flaws in the new aid paradigm: the denial of ‘politics’. This is particularly valid and critical when it comes to M&E. While the influence of ‘politics’ in M&E has long been acknowledged in projects and programmes, it seems to be disregarded when moving to the national and sectoral levels (just when interests and stakeholders are multiplied). Whereas a narrow technocratic vision may seem ‘politically neutral’ and the only one that is feasible, it may at the same time undermine M&E's functions of ‘accountability’ and ‘feedback and learning’, and even worsen ‘political constraints’. It potentially further lowers public voice and accountability, strengthens government's M&E monopoly, and hides and reinforces flaws in policies (which are particularly pernicious when they perpetuate existing exclusionary practices). It can eventually lead to the ineffective implementation of the key principles of the new aid approach and so undermine its very success.