Bureaucrats, and employees more broadly, often decry the “red tape,” controls, and reporting requirements their organizations place on them. I investigate whether these controls are beneficial or counterproductive for International Development Organizations (IDOs, for example, the World Bank and US Agency for International Development) that deliver foreign aid. There is a real trade-off in allowing field-worker judgment to guide foreign aid project implementation; following Aghion and Tirole, letting field workers (agents) take more initiative requires circumscribing managerial (principal) control.Footnote 1 In this paper, I compare the benefits of greater agent autonomy to its costs. Is more reliance on the perceptions and judgment of field agents associated with better foreign aid project performance, or do greater top-down controls and agents’ orientation toward measureable targets yield better results?
To explore these questions I examine variation in the constraints placed by political authorizing environment on IDOs, arguing that politically constrained organizational autonomy induces tight control of field agents by an organization's managers. When IDO managers need to report measurable success to legislators and executive boards, they need to manage field staff in such a way to generate numbers. Where agent autonomy is important, a reform-minded politician's desire to improve aid project performance by requiring measurable short-term results might undermine the success of the very aid projects the politician wishes to see perform well. Accounting for results and succeeding in actually delivering results are sometimes in tension with one another.
This does not mean top-down controls are necessarily counterproductive. Where rules and targets are well aligned with an organization's objectives, tight control may improve performance. However, tight control also reduces agent flexibility and risks orienting agents toward meeting targets at the expense of delivering on broader organizational goals. The costs of control need to be weighed against its benefits. This paper uses variation in recipient-country environments as a source of exogenous variation in the net effects of tight principal control. Some recipient-country environments are more unstable, and so there are greater returns to the agent flexibility and use of judgment that tight principal control precludes.
IDOs are well suited for exploring how environments and organizational control practices jointly affect organizational success. Individual IDOs work in a wide range of country environments across a range of tasks from road construction to anticorruption efforts, with limited ability to exit contexts or tasks in response to poor performance. I provide novel evidence connecting political authorizing environment constraint and agency performance via management practice. The empirical findings strongly suggest that more politically constrained IDOs do indeed differentially engage in management practices that increasingly undermine their own performance as environments become more unpredictable.
Examinations of IDO effectiveness, and international organization behavior more broadly, have recently taken what Gulrajani calls the “bureaucratic turn.”Footnote 2 This work contributes to that turn by focusing on the underexplored actor in principal-agent models of IO bureaucracy, the agent. As Hawkins and Jacoby put it, “In spite of the growing sophistication of the principal-agent literature, it still contains a remarkably thin view of agent behavior.”Footnote 3 Connecting the study of IOs to the rich literature on delegation and autonomy,Footnote 4 I describe the costs and benefits of principal control for IDOs.
After discussing the costs and benefits of principal versus agent control, I discuss how environmental unpredictability affects the balance of costs and benefits. I then argue why we should expect IDOs with insecure political authorizing environments to more tightly control field staff, even where such control is unlikely to augur for greater organizational success. Next I turn to operationalizing key variables and formalizing the hypothesis. Quantitative analysis then brings to bear an original cross-IDO data set composed of over 9,000 unique development projects, allowing us to test whether the impact of constrained autonomy is as hypothesized in the world's largest cross-organizational database to incorporate development outcomes.Footnote 5 The quantitative empirics are then complemented by qualitative interview data illustrating the central mechanism the paper theorizes.
Theory
How Principal Control Can Undermine Organizational Success
Principal-agent models have long wrestled with the reality that agents have asymmetric information—access to local knowledge that distant principals lack. Conventional applications of principal-agent models tend to focus on the risk this asymmetric information poses.Footnote 6 In a recent piece entitled “Why Organizations Fail,” two distinguished scholars argue that “incentive problems arise due to the presence of asymmetric information or imperfect commitment, which lead agents to act according to their own biases or preferences rather than in the interest of the organization.”Footnote 7
Attempts at strengthening principal control often take the form of costly monitoring technology. Principals can also induce agents to do what principals want by tying compensation, promotion, etc. to outcomes the principal can observe. However, these attempts at control have costs as well. First, controls may induce agents to focus on meeting formal requirements rather than the service they are meant to deliver. Second, controls might preclude agent initiative and productive use of the asymmetric information to which agents uniquely have access.
What principal control may unproductively induce
Given the difficulty of directly observing agent action, IDOs’ primary tool of agent control is setting performance targets and requiring reporting against them.Footnote 8 A recent Organisation for Economic Co-operation and Development (OECD) review of the US Agency from International Development (USAID) finds USAID uses “approximately 200 standard indicators (recently reduced from 500), and many more custom indicators” in their monitoring and evaluation of projects.Footnote 9 These indicators orient agent action, thus acting as a de facto management tool regardless of whether their intent was in fact to centralize control with the principal. Targets can orient field staff toward clear results and hold staff accountable if targets are not reached.
However, target-setting may also induce distortions, for example, inducing agents to focus on producing what can be measured and reported upon.Footnote 10 Because of their difficult monitoring environment, IDOs often measure short-term outputs to proxy longer-term outcomes; agents may achieve these outputs but without actually forwarding the IDOs’ goals.Footnote 11 As Kerr put it over forty years ago, there is potential for IDOs to engage in “the folly of rewarding A while hoping for B.”Footnote 12
What principal control may unproductively preclude
The danger to principals of agent asymmetric information and hidden action are well explored in applications of the principal-agent model to international organizations. While less commonly explored in conventional applications of principal-agent models, there have been scholars who conceive of agents’ private information as valuable for good organizational performance. Most directly, Lisa Martin has theorized that IMF staff members have private information about borrowers’ contexts that are important for IMF loan performance.Footnote 13
Like IMF loan performance, foreign aid project success also depends on asymmetric information held by agents. Attempts at control inevitably produce rules, targets, and other accountability measures that purposefully constrain agents. But the very constraint that precludes bad behavior by agents may also unintentionally preclude behaviors that are in service of the organization's mission. Sometimes good organizational performance depends on the gathering and use of asymmetric information, such as local contextual knowledge. When there are important things agents can know and their supervisors cannot (asymmetric information), Aghion and Tirole argue it is critical that agents have not just formal but “real” authority.Footnote 14 This means agents are not just given formal ability to make judgments but also that the organizational incentives they face encourage the use of their judgment. Aghion and Tirole argue that agents who do not have an incentive to gather asymmetric information will not do so, framing this as the trade-off between agent initiative and principal control.Footnote 15
Asymmetric information can also include soft information that a skilled observer might use to inform his or her decisions—it cannot be proven or “cannot be directly verified by anyone other than the agent who produces it.”Footnote 16 Tight principal control means agents will not gather asymmetric information, including soft information. Organizations are therefore left with a poorer knowledge base.
Principal control can also impede organizational response to changing contexts. Putting more control in the hands of agents empowers actors who are better placed to rapidly respond when flexibility and adaptation is needed, while simultaneously reducing the control mechanisms (review procedures, approval processes, etc.) that might impede rapid response. Flexibility is complementary, but distinct, from the asymmetric (soft) information channel; flexibility is in greater demand when contexts change more rapidly, whereas the direct returns to soft information may persist irrespective of the rate of environmental change.
The Benefits of Principal Control
These benefits of agent autonomy must be balanced against their costs. Putting more control in the hands of field agents also means those agents will find it easier to engage in a range of actions, including those that may be illegal or undesired.Footnote 17 Agents might be “captured” in their time away from headquarters, maximizing private benefits or simply implementing their own plans even when those plans do not serve organizational best interest.Footnote 18 Agents will be more susceptible to capture with less principal control. In addition, agent judgment can simply be wrong even when it is well intentioned. An IDO that gives agents more control will have more to fear from fallible agent judgments.
Principal control also produces more standardized behavior. By shifting control to agents, an organization may allow more scope for bias and prejudice.Footnote 19 Where standardization is critical to good outcomes—the organizational equivalent of baking a cake, where following a precise recipe is likely to yield the best results—less principal control will likely induce variation that will be detrimental to organizational performance.Footnote 20
There are potential costs to organizations in giving agents more autonomy and thus relying on their initiative, just as there are costs to tighter principal control and less autonomy. Aghion and Tirole frame this as the tension between principal control and agent initiative.Footnote 21 Environmental unpredictability plays an important role in determining how IDOs ought to best resolve the tension between principal control and agent initiative in a given context.
Environmental Unpredictability, Principal Control, and IDO Success
Whether more or less principal control augurs for greater success in delivering foreign aid depends on a number of situational factors; one critical systematic source of variation is environmental unpredictability. Rapidly changing contexts both increase the chances that targets will induce distortion and raise the value of what controls can preclude. More unpredictable environments require more flexibility and more use of asymmetric (soft) information.
As IDO project implementation occurs, there are many things that can affect how interventions ought best to proceed. Some changes are foreseeable, and thus a smart project plan could account for these contingencies. However, there are frequently what then-US Secretary of Defense Donald Rumsfeld once referred to as “unknown unknowns.”Footnote 22
Some “unknown unknowns” are unforeseeable when a project commences but nonetheless predictable at some time before the risk occurs. When a hypothetical project to provide youth vocational skills in collaboration with the Ministry of Youth and Sports begins, the recipient country's political environment may appear stable. However a year into implementation, the minister falls out of favor with the prime minister and is likely to lose his job; the current minister's successor is likely to marginalize a program closely associated with her predecessor. A wise and well-informed IDO field agent, foreseeing this possibility, might begin to include more career civil servants in the project's steering committee and consult the minister himself less. Such a decision requires the agent's freedom of action and use of asymmetric soft information. An IDO with tight principal control will have more poorly informed agents who would in any case be less able to act on their own unverifiable judgments of changing context to respond strategically to changing circumstances.Footnote 23
Environments vary with regard to legibility—the extent to which they can be understood from a distance.Footnote 24 More unpredictable environments are also likely to be less legible. In the context of international development this might be understood as the correlation of de jure structures with de facto reality. Formal structures and hierarchy vary with regard to whether they are good indicators, for example, of whose approval is needed in practice to ensure a project will proceed. The greater the gap between structures and reality, the greater the returns to soft information and thus agent autonomy. In less legible environments it will be hard for anyone other than field agents to make judgments about how to proceed in designing and implementing projects.
Deviations from Equilibrium: Heterogeneous Political Authorizing Environments
The discussion thus far perhaps begs the question of why IDOs, and indeed all organizations, would not simply adapt principal control appropriately to differences in the environment. Just as field operatives report to IDO headquarters, IDOs themselves are agents reporting to authorizing environments, the collection of actors to whom organizations are accountable (for example, their political principals). IDOs respond to the shadow of their authorizing environments. By “shadow” I mean the threat of possible future authorizer sanction, which in turn affects management's actions and degree of principal control.Footnote 25 For public organizations political authorizing environments are critical gatekeepers to resources, controlling the funding, mandate, and ultimately survival of public agencies.Footnote 26
Different IDOs have very different relationships with their authorizing environments. The expected probability of sanctions—for example, failure or reputation-damaging cases of corruption and fraud—varies. In other words, the “length” of an authorizing environment's shadow varies; some IDOs worry (to a much greater degree than others) about how authorizers will view their performance. Exploring the reasons for authorizing environment differences are beyond the scope of this work, which will largely take authorizing environments as given. I focus instead on the consequences of differential authorizing environment insecurity.
Insecure agencies will take fewer risks than they otherwise would.Footnote 27 If an organization needs to meet measures in the short term, for example, to receive continued funding, the organization may not take the risks necessary to achieve long-term ends—an organization's “risk appetite” may be inefficiently constrained. Insecure agencies are less likely to take smart risks, where the expected probability-weighted value of benefits exceeds costs.
Insecure agencies are also much more likely to be concerned with reporting success to authorizers. The greater the pressure to report organizational results, the greater the need for senior management to manage via measurement and target setting inside an organization to generate the data which can then be reported to authorizers.Footnote 28 Target setting does more than simply add an additional reporting step to agents’ workload; when pressure is put upon these measures for control purposes, measures change what agents and organizations actually do. While this is true of all kinds of measures, there are particularly large reasons to worry when management by measurement is employed for legitimacy-seeking reasons.
From the perspective of an agency in need of justifying itself, one attractive feature of measurement and reporting is measurement's role in making the organization's activities seem legitimate.Footnote 29 In the public sector measurement has increasingly become critical to justifying continued funding and building legitimacy as part of a broader discourse on accountability and control; the spread of performance measurement is often linked with legitimacy seeking.Footnote 30 The reason for measurement is then to appear successful; where appearing successful and actually accomplishing the organization's objectives are in tension, accomplishment is likely to be sacrificed in favor of appearing successful. By creating metrics and meeting targets, even when those targets are not well linked to ultimate organizational goals, organizations can appear to be performing well to political authorizers.Footnote 31
Insecure agencies are likely to engage in greater principal control at the expense of agent initiative even where that principal control might undermine the success of interventions. This is both because principal control is likely to better generate standardized data that can be used for legitimacy-seeking purposes and because tight control limits opportunities for agent malfeasance or bad action that might serve as a reputational risk for agencies.
Hypotheses and Operationalization
This work conceptualizes letting go of principal control and thus giving agents greater autonomy as a second-best strategy to employ when it is less bad than the distortions and constraints of top-down control. In some contexts, tight principal control is clearly superior to relying on fallible agent judgment. In other contexts the gains of being able to respond more flexibly and better utilize asymmetric (soft) information are superior to distortionary tight principal control. IDOs with greater political authorizing constraint will be less likely to give up principal control when appropriate as environments become more unpredictable. More constrained IDOs will thus be less able to cope with unpredictability than will their less-constrained peers. Thus
IDOs with more stable political authorizing environments will see less of a decline in performance in response to increasing environmental unpredictability than their more constrained peers.
The claim is not, then, that tight principal control is always inferior; nor is it that agent initiative allows IDOs to improve their absolute level of performance as environments become less predictable. It is simply that less principal control and greater agent initiative will be more helpful in more unpredictable contexts. This is because the costs imposed by principal control will go up as unpredictability rises, as will the benefits of relying on agent initiative and judgment. Insecure political authorizing environments will preclude IDOs from giving up principal control when otherwise appropriate.
Operationalizing Success
In implementing their work, IDOs structure their activities through projects. Projects are discrete, time- and place-bound activities implemented after careful planning and preparation. These projects can vary widely in location, sector, and purpose. World Bank projects approved by the World Bank's board in April 2017 include projects in the Dominican Republic focused on the quality of educational statistics, in Bosnia on public health behavior, in Benin on enhancing agricultural productivity, and in India on state-level urban development.Footnote 32 The empirics that follow employ a novel data set consisting of over 9,000 unique projects in 140 countries carried out by nine donor agencies from 1994 to 2013, the product of many months of labor. This data set is unique in including project performance data for multiple IDOs.Footnote 33 More detail on the data collection, cleaning, and coding process can be found in the online appendix.
Figure 1 shows the distribution of projects across countries, demonstrating the wide range of countries in which projects occur.
Overview of projects in data set
Project success ratings are assigned on a six-point Likert-type scale, with 6 referring to “highly satisfactory” projects and 1 as “highly unsatisfactory” projects.Footnote 34 The underlying construct that different IDOs employ for measuring the success of projects is relatively consistent, with an OECD-wide standard in place. A given project's rating is intended to incorporate a project's relevance, effectiveness, efficiency, sustainability, and impact.Footnote 35 Holistic success ratings are variously calculated by IDO staff, external evaluation departments, or independent evaluators.
This is, of course, a less than fully precise standard for what constitutes success. Success can be defined differently for different IDOs, or in different sectors. Fixed effects by IDO, sector, and recipient country partially help control for these potential sources of bias. Poor data quality and evaluation bias are potential threats to validity which I discuss later and treat in robustness checks in the appendix. It is also possible for the data to be accurate in the sense of correctly reflecting an organization's assessment, but for that assessment to bear little connection to the actual performance of the project. To the extent possible, I have also attempted to validate these evaluations by turning to primary documentation. The appendix describes this archival work, which broadly supports the conclusion that organizational assessments mapped “real” success and failure in the projects examined.
IDO Autonomy and Constraint: Authorizing Environment Insecurity and Propensity to Give Up Principal Control
In 2005 IDOs and recipient countries came together to agree to the Paris Declaration, a set of principles for achieving more effective aid tied to measurable targets.Footnote 36 Follow-up Paris Declaration monitoring surveys focused on various elements of aid delivery. The monitoring surveys asked both donors and recipient countries for reports on their own and each other's practices (i.e., recipient countries also reported on donor behavior).
From the quantitative indicators that formed part of the monitoring reports I construct IDO proxies for “propensity to devolve control” and “authorizing environment insecurity.”Footnote 37 The appendix describes the construction of these scales in substantial detail. The devolution propensity and authorizing environment constraint measures are reasonably well correlated (.41). I take the simple average of the indicators to form a simple scale of autonomy ranging from 0 to 1, coded so that higher scores on the scale represent lower levels of political authorizing insecurity and higher IDO propensity to devolve control.Footnote 38 The overall scale has a Cronbach's alpha of .825.Footnote 39 This provides reasonable confidence that these measures and the two subscales map the same essential facts regarding IDOs and thus provide suggestive evidence for my conjecture that political constraints do in fact trickle down to IDOs’ management practices.
A principal components analysis suggests this simple average is a more intuitive solution that will yield similar results to formal use of principal components; in any case results are robust to using a principal components approach. The appendix presents the relevant technical information (for example, eigenvector scree plots and component loading tables), as well as robustness checks employing principal components.
Given the critical role that measurement of politically constrained autonomy plays for the empirical strategy, I validated the Paris Declaration scale with more direct measurement. I conducted a small-scale direct field survey of aid experts—individuals who have substantial development experience or whose jobs bring them into contact with a wide variety of donors. The appendix contains a fuller explanation of this field survey measure.
Both the field survey measure and the Paris Declaration measure of IDO autonomy are time invariant. In employing a time-invariant measure of IDO autonomy I do not mean to imply that IDO autonomy does not, in fact, vary across time—it certainly does, as agencies’ relationships with their political authorizing environments change. While data limitations preclude modeling this source of variation, I attempt to control for these dynamics to the extent possible. I employ year-by-IDO fixed effects as a robustness check in Table 3, which will absorb any changes in autonomy for a given IDO (by absorbing any IDO-specific changes in performance dynamics where they differ from the general pattern). The results are robust to using any of the (differently timed) waves of the Paris Declaration monitoring reports (and thus using only the most recent, or least recent, wave or waves), as well as employing the (even more recent) direct field survey measure of autonomy. This field survey is well correlated with the Paris Declaration-derived politically constrained autonomy scale (.73), providing both an additional level of confidence in the accuracy of the Paris Declaration-based measure and suggestive evidence that IDO autonomy has not changed so greatly within IDO over the period of the data to make the time-invariant measure uninformative.
Environmental Unpredictability
I operationalize unpredictability by focusing on differential state fragility. Predictability and fragility are often linked explicitly in development practice, with practitioners speaking about the difficult and unpredictable nature of fragile state environments.Footnote 40 Fragility is in some sense the likelihood that the current equilibrium will break down or change rapidly. As the World Bank puts it, fragile states are “more unstable and unpredictable” than their less fragile peers.Footnote 41 The focus of this work is not on fragile states as a class or on those at the very extreme of the state-fragility measure. The theory here is intended to apply to the entire range of state fragility, and thus comparisons will be made across the entire universe of developing countries.
Environmental unpredictability is measured via the Polity IV State Fragility Index (SFI).Footnote 42 This index incorporates security, governance, economic development, and social development measures and has two subscales: effectiveness and legitimacy. The two subscales are highly correlated (.66) and Cronbach's alpha (.78) suggests that they map the same underlying construct.Footnote 43 The SFI varies at the country-year level, with every country holding an annual SFI score from 1994 to present.
Quantitative Results: Politically Constrained Autonomy and Project Success
This section lays out the primary findings then addresses potential econometric concerns. Before turning to the analysis, Table 1 displays summary statistics for key variables.
Summary statistics of key variables
The appendix provides additional summary statistics by IDO regarding the key dependent variable project success. A key weakness of these data is the modest number of IDOs in the sample. Throughout the analysis I take care to ensure this small “second-level N” is not leading to spurious conclusions. In particular, I employ quite simple and straightforward econometric models to minimize the chance that these models are “overfit,” with results driven by the relative lack of variation in outcome data compared to the number of explanatory variables.
The model for project i in recipient country j implemented by IDO k generalizes to
$$\eqalign{ \hbox{Project}\,\hbox{Succes}\hbox{s}_{i{\rm,} j{\rm,} k} =\; &\beta _1{\ast}\hbox{Environmental}\,\hbox{Unpredictabilit}\hbox{y}_j + \beta _2{\ast}\hbox{Environmental}\, \cr & \hbox{Unpredictabilit}\hbox{y}_j{\ast}\hbox{IDO}\,\hbox{Autonom}\hbox{y}_k + \beta _3{\ast}\hbox{Control}\hbox{s}_i \cr & + \hbox{Fixed}\,\hbox{Effect}\hbox{s}_j + \hbox{Fixed}\,\hbox{Effect}\hbox{s}_k + { \varepsilon} _i.} $$The key empirical prediction is that the coefficient on β 2 will be positive and statistically significant; that Project Successi,j,k will increasingly benefit from greater IDO Autonomyk as Environmental Unpredictabilityj rises.
One key shortcoming of the dependent variable, project success, is that it is not amenable to direct interorganizational comparisons; there is no reason to believe that one IDO's rating of “4” is in fact more successful than another IDO's rating of “3.” Any (constant) systematic differences among IDO evaluation criteria or measurement standards are addressed in two ways: by including IDO k fixed effects in the models (thus generating results that leverage intra-IDO comparisons across projects) and by normalizing project ratings using IDO-specific z-scores where fixed effects are not employed. As noted earlier, the measure of IDO politically constrained autonomy varies at the IDO k level and is time invariant. This means that the measure is collinear to IDO k fixed effects. As a result quantitative analysis cannot directly compare IDOs’ performance—it cannot say that, for example, KfW projects were more successful than IFAD in country X while IFAD projects were more successful than KfW in country Y. The interaction of IDO autonomy and a given country's level of environmental predictability does vary at the j,k level. The interaction term can thus still be informative about how within-IDO performance varies over recipient country and across time, though the absence of an autonomy base term precludes a direct comparison of two different IDOs’ project success in a given country-year. Some models also use recipient country j fixed effects, thus ensuring any fixed country-specific features are not driving results.
In a literal sense, using IDO fixed effects removes the mean of the dependent variable—project success—for each IDO. This also means we need not trust that projects are as successful as donors say they are to believe the results of this model. Table 1 indicates than the average project scores a 4.3 on a six-point scale. It seems possible, even highly likely, that the average project is not in fact a clear success—that these ratings are biased upward. This will not bias the results so long as for a given IDO higher numbers are still associated with greater success; so long as a project scoring a 6 is more successful than a 4, a 4 more than a 2, etc. By de-meaning project success via fixed effects we also avoid spurious conclusions about absolute levels of successfulness.
The quantitative analysis instead focuses on the differential performance of IDOs with varying levels of politically constrained autonomy in interaction with other explanatory variables. This takes advantage of the fact that a rating of 4 given by KfW means a project was more successful than a project assigned a 3 by KfW, while a 2 given by IFAD means a project was less successful than one given a 3 by IFAD. It is possible, then, for the quantitative analysis to yield conclusions of the type “KfW projects are more successful in country X than country Y, while IFAD projects are more successful in country Y than country X.” In this way inter-IDO comparisons can be made by comparing intra-IDO variation in project success.
To adjust for the possibility that project success may be correlated within a given recipient country, the main analyses report standard errors clustered at the recipient country level. It is also possible that project success is correlated within IDOs (even with fixed effects, errors may be correlated if the assignment of independent variables are clustered and there are heterogeneous treatment effects). Appendix Tables A10 and A11 suggest that clustering by IDO (or, when practicable, double-clustering on IDO and recipient country) does not alter the substantive findings.Footnote 44
Politically Constrained Autonomy and Environmental Unpredictability
There is a robust and statistically significant negative relationship between environmental unpredictability and overall project success. Table 2 reports the core findings. Environmental unpredictability is associated with less-successful project evaluations for IDOs, on average. The key explanatory variable, the interaction of IDO autonomy with environmental unpredictability, indicates that autonomy mediates the effect of environmental unpredictability on project success. While all IDOs see a decline in project success as environmental predictability falls, for more autonomous IDOs this decline is much less steep.
IDO autonomy mediates the relationship between environmental predictability and IDO project success
Notes: While all IDOs see performance decline as environments become less predictable, more autonomous (less constrained) IDOs have substantially smaller declines. Ordinary least squares (OLS) regression. Standard errors in parentheses, clustered by recipient country; *p < .10; **p < .05; ***p < .01.
All models include IDO fixed effects. The model's comparison is being made within each IDO's projects, comparing whether a given IDO—for example, the Asian Development Bank—sees more successful projects on average in more or less unpredictable environments (as measured by the State Fragility Index). Models 3 and 4 in Table 2 incorporate recipient country fixed effects, thus focusing only on changes within recipient countries over time. Models 5 and 6 incorporate sector fixed effects, controlling for sectors at the most fine-grained level available, the 222 unique five-digit OECD Development Assistance Committee Creditor Reporting System (CRS) purpose sectors. Findings are robust to focusing on differences in performance within sectors as well.
If project success ratings were simply arbitrarily assigned, we would expect no relationship between project success and environmental unpredictability. If in harder-to-monitor unpredictable environments all projects were declared more successful, we would expect environmental unpredictability to be associated with higher success ratings. But instead here we see the relationship theory, and arguably intuition, would predict: consistent with my theory, project success falls as environments become more unpredictable.
The key empirical prediction regards the interaction between IDO autonomy and environmental unpredictability. This interaction term is robustly positive and statistically significant, suggesting that autonomy does indeed play an important role in allowing an IDO to respond to greater environmental unpredictability. Once again this result holds when focusing on within-sector or within-recipient country data.
Figure 2 draws from model 1 of Table 2 to graphically represent differential performance by level of politically constrained autonomy. Note that the y-intercepts, and thus the relative level of the two lines in Figure 2, do not contain useful information. The direct effect of autonomy is absorbed by IDO fixed effects, making the vertical positions of the two lines arbitrary. What is informative is the differential slopes of the two lines—the differential success of IDOs of varying levels of autonomy in response to varying levels of environmental unpredictability.
Returns to autonomy in countries of differential predictability
All IDOs perform better in more predictable, stable contexts than they do in less predictable environments. More autonomous (less constrained) IDOs see a much smaller decline in performance than their less-autonomous (more constrained) peers as unpredictability rises. While an IDO with the lowest observed level of autonomy is predicted to have over half a point (.5) of difference between its performance in a state like Armenia (SFI = 7 in 2014, or one standard deviation more stable than the mean) and its performance in a state like Nigeria (SFI = 17 in 2014, or one standard deviation below the mean), an IDO with the highest observed level of autonomy is predicted to have about .06 of a point or one tenth as much performance differential.Footnote 45
Table 3 adds a series of fixed effects to the main findings. Inclusion of time fixed effects (either yearly or in five-year periods) does nothing to diminish the association between autonomy and recipient unpredictability. The result remains robust to including time*IDO fixed effects and time*recipient fixed effects. These results should allay any concerns that the primary results are driven by heterogeneous IDO project performance over time or by heterogeneous entry of IDOs into and out of recipient countries over time.
Expanding fixed effects for robustness
Notes: Controlling for time, or time interacted with recipients or IDOs, does little to change the main effects. Standard errors in parentheses, clustered by recipient country; *p < .10; **p < .05; ***p < .01.
Robustness
The appendix outlines a series of robustness checks that speak to the validity of the autonomy measure, as well as the sensitivity of the analysis to how the interaction term is modeled. It also identifies outliers, quirks in outcome variance, and a number of other potential threats to validity. Two primary concerns will be addressed here: the quality of the evaluations that form the core of the analysis and differential selection of IDOs into sectors or environments.
Evaluation Bias
These data rely on evaluations of project success made by the agencies themselves. One might worry that an agency with a fragile relationship with its political authorizing environment, in addition to being less autonomous, would have a greater incentive to self-evaluate projects to have been successes. This is not a threat to validity, inasmuch as a consistent bias would be absorbed by the IDO fixed effect. Of greater concern would be bias that moves with the interaction of autonomy and environmental predictability. If, for example, more autonomous IDOs give their agents more leeway in self-evaluations, which those agents differentially exercise to a greater degree as environmental unpredictability rises, this would be a threat to the validity of the main findings.
The involvement of independent evaluation units provides suggestive insight into this problem because independent evaluation units should not have the same degree of incentive as agents themselves to give favorable evaluations. Table 4 controls for the type of evaluation, that is, whether the data source is an internal review by project staff, a review conducted by an IDO's own independent evaluation unit, or a review conducted by an externally contracted evaluator.
Controlling for evaluation type
Notes: The third, omitted, category is externally contracted evaluators. Models 2 and 4 take an IDO-specific z-score of the dependent variable to allow the IDO fixed effect to be dropped, and thus the base term of IDO Autonomy to be added to the models. Standard errors in parentheses, clustered by recipient country; *p < .10; **p < .05; ***p < .01.
The relationship between autonomy and environmental unpredictability remains unchanged, suggesting that differential evaluation bias by different IDOs is not driving the results.
Selection
One natural concern might be that controlling for recipient and sector fixed effects does not account for the fact that different IDOs may make decisions about what projects to pursue in light of where projects would be more successful. Perhaps more autonomous IDOs engage in greater strategic selection of recipient countries and sectors, placing themselves in a better position to succeed. While this selection effect would be a channel from autonomy to differential IDO project success, it would mean more autonomous IDOs were not in fact more successful than their less autonomous peers in actually delivering projects in more unpredictable environments relative to their own performance in more predictable environments.
To explore selection I construct a parallel data set with the number of observations from each IDO in each country in each sector in each year. If differential IDO autonomy is working via selection, we should see differential presence or absence of projects by level of environmental unpredictability. Table 5 replicates Table 2’s regression model, but substitutes the number of projects completed in each IDO-country-sector-year as the dependent variable.Footnote 46 There are over 900,000 unique IDO-country-sector-years, allowing quite a bit of precision in this selection estimate.
IDO project selection
Notes: IDOs of different levels of autonomy do not differentially select into more or less unpredictable recipient country environments. Standard errors in parentheses, clustered by recipient country; *p < .05; **p < .01; ***p < .001.
Table 5 finds no selection along the main dimension of inquiry, the interaction between environmental unpredictability and IDO autonomy. This suggests that IDO selection of sectors and/or countries is not a systematic problem for this analysis.
The appendix provides a range of additional tests. To partially summarize, it does not seem to be quirks of measurement or subtle features of the construction of any of the key measures that are driving results. Using the survey measure of IDO autonomy or a principal components approach does not alter findings. The appendix also presents the IDO-by-IDO statistics on the relationship between environmental unpredictability and project success, explores variance in project success and how this varies with level of environmental unpredictability and IDO autonomy, and presents IDO-by-IDO summary statistics.
Qualitative Illustrations: Comparing USAID and DFID Authorizing Environments and Their Impact
To further investigate the relationship between political authorizing environments, IDO autonomy, environmental unpredictability, and project success I conducted eight case studies examining US Agency for International Development (USAID) and UK Department for International Development (DFID) projects in Liberia and South Africa.Footnote 47 These case studies allow a direct comparison of IDO performance, going beyond the intra-agency comparisons to which the quantitative analysis is limited. While I cannot do justice to the richness of the case study data here, I believe these data can help illustrate the mechanisms I theorized. This section discusses differences in USAID and DFID authorizing environments, then turns to illustrating the differences in level of principal control engaged in by USAID and DFID in a representative case in which both agencies were pursuing a similar goal (building municipal government capacity) in South Africa.
Authorizing Environments
USAID and DFID are IDOs with a clear difference in authorizing environment insecurity and constraint. An agency's formal status and its level of formal independence have long been thought of as important features of agency independence and insulation from political oversight.Footnote 48 DFID is a separate ministry run by a cabinet-level minister, while USAID is subordinate to a cabinet secretary, reporting to the US Secretary of State. DFID has power and access that USAID does not, a sign of the relative importance and power of DFID vis-à-vis USAID.
DFID has stable budgets and strong parliamentary support for foreign aid. USAID's budget is quite unstable, with no long-term budgetary commitments and a much lower level of funding as a share of government spending. USAID's budget, unlike DFID's, involves heavy use of “earmarks” that pre-specify what funds must be used for. By one Congressional Research Service estimate, earmarks made up almost 75 percent of USAID's budget.Footnote 49 While tight control cannot be measured merely by the number of words devoted to legislation, it is suggestive of the difference that the US foreign assistance act runs over 300 printed pages; comparable UK legislation runs fewer than forty pages.Footnote 50
USAID and DFID are by their own admission very different organizations when it comes to risk taking. USAID describes itself as having a “conservative risk appetite;” by contrast DFID describes itself as having “a relatively high risk appetite, and [DFID] is often willing to tolerate high levels of risk where there are substantial potential benefits.”Footnote 51 USAID also stands out regarding flexibility and the use of measurement. A recent OECD peer review of USAID—essentially a report written by other IDOs regarding USAID's systems—finds that USAID's need for authorization from Washington constrains its operating flexibility.Footnote 52
The IDO autonomy score for each IDO supports the view that USAID and DFID's differences in authorizing environment do indeed lead to different behavior. USAID has a score of .36, thirtieth among the thirty-three IDOs for whom Paris Declaration monitoring surveys allow the calculation of scores. DFID, on the other hand, has a score of .80, second among the thirty-three IDOs.Footnote 53
Politics and authorizing environments are much more salient for those discussing USAID than DFID government interventions. This is suggestive evidence of greater relative organizational focus on and preoccupation with its authorizing environment for USAID. In the interview data from South Africa, the word congress appears thirteen times; parliament is mentioned only once, and by way of contrasting DFID with USAID.
USAID's belief that Congress does not trust the organization and fear of being on the proverbial chopping block came up frequently in interviews. This insecurity was evoked with regard to USAID's hiring practices which limit USAID's ability to hire full-time staff and thus the use of contractors in project supervisory roles.Footnote 54 It was used to explain budget unpredictability and the constant need to fight for funding.Footnote 55 In perhaps the most vivid depiction, one senior official described USAID as “under siege” from Congress, saying “When you [USAID] have a hostile Congress and an ineffectual president—which was the case pretty much since … the last time USAID had any true swagger was under Reagan—it has been an agency under siege for, I guess it would be going on for over thirty years now.”Footnote 56
The need for reporting looms large for USAID. As one staffer of a development contractor implementing a USAID project put it, “USAID wanted reports. USAID pushed management and management pushed us.”Footnote 57 An individual with experience at a number of development contractors described this as the pressure from USAID to “document more than do,” suggesting that such trends leave projects with “some really beautiful reports” to please funders and authorizers but less impact on the ground than might have occurred if attention instead focused on implementation.Footnote 58 A former senior manager of a USAID project summarized a widely echoed view in saying “USAID's focus was around meeting numbers as opposed to the impact.”Footnote 59
Key to the Aghion and Tirole model is the insight that if agents cannot use asymmetric information there is no point in gathering it. One USAID interviewee suggested that very few of her colleagues ever learned much about the world just beyond the embassy walls because there was no way to make use of that information.Footnote 60 As one USAID official put it, the effect of the restrictions and constraint from above is to “make you cautious.”Footnote 61 Insecurity breeds conservatism—the need to ensure that any action taken can be defended.
The Impact of Authorizing Environment Constraint: Differential Principal Control, and Project Success, in South Africa
DFID and USAID's projects in South African municipal governance in the mid-2000s illustrate how legitimacy-seeking reporting induced by political authorizing environments can affect IDO projects in practice in a relatively predictable environment (and thus serve as a “hard case” for my theory). DFID's Consolidated Municipal Transformation ProgrammeFootnote 62 and USAID's Local Governance Support Program, Phase 2Footnote 63 both aimed to help municipalities efficiently and effectively deliver services.Footnote 64 Both focused on making local government more effective via capacity building by transferring knowledge to municipal staff. This capacity building was of both a management and a financial nature; both projects aimed to improve municipal accounting and billing systems and municipal debt management. How USAID and DFID delivered their interventions was quite different, however, as was their management, reporting, and design processes.
USAID settled on an initial model that delivered monitorable and measurable training to municipalities.Footnote 65 On a pre-arranged day, a trainer would arrive and hold a session, often in a conference room, for part or all of the day on the pre-arranged topic. Many municipalities were served by the project and many training sessions were delivered. Following the trainings, trainers verified that trainings had occurred and tracked how many individuals were trained.Footnote 66
DFID's project shared USAID's broad focus on improving municipal functioning via skills transfer and systems building. Contractors implemented DFID's project, as they did USAID's. But unlike USAID's project, DFID's project worked primarily by embedding advisers in local municipalities. Advisers resided in the municipalities for extended periods of time to build skills and systems on an ongoing basis. DFID's advisers were ultimately in charge of project direction; they set the specific goals against which they were reporting. As one implementer put it, DFID's reporting was “more content rich; it was not a numbers game.”Footnote 67
DFID and USAID both had reporting requirements for their respective projects. However DFID did not rely primarily on externally verifiable data in reporting, unlike USAID. DFID effectively put resident advisers and their soft information-laden judgments in control, something DFID not only condoned but actually explicitly designed into the project. The “price” of this greater degree of agent initiative was a lesser degree of principal control.
Meeting targets clearly served as a control in USAID's municipal governance project. Michelle Layte, the head of project implementation toward the end of the project, said indicators were chosen “because it was easier to count … but the numbers didn't tell about the impact.” Layte went on to say that, while USAID had been better earlier in the project, “it was more a number chasing towards the end especially because we needed to reach our target.”Footnote 68 Another interviewee described implementing the USAID project as “a numbers game … [USAID would say] we want the numbers, we want information.”Footnote 69
One USAID implementer described a clear sense inside the project-implementation team that the trainings were failing.Footnote 70 The correlation between measures and ultimate outcomes broke down in USAID's municipal governance project. The training numbers weren't fabricated; trainings were occurring and individuals were attending. One USAID actor described this as counting “bums on seats.”Footnote 71 These targets were simply disconnected from the actual broader purpose they had been designed to serve. These measures may have had little connection to impact, but they certainly affected implementation. Target-setting constrained the behavior of field agents and their managers, precluding alteration of the project.
There is very little evidence that training under the USAID project was effective. This was the view not just of observers but also of those who actually worked to implement the project.Footnote 72 As one team member put it, “I don't think [training under the project] contributed much … because you go there, you don't have any authority over the people that you're training, so if they don't cooperate you cannot say anything. You go there sometimes, they tell you that ‘we have other priorities,’ ‘we don't have time now’—those kinds of things.”Footnote 73
DFID's project was by no means an overwhelming success. That said, it was substantially more successful than USAID's was. Being full-time resident for the long term (two to three years), DFID project advisers were often—though not always—able to find a way to positively influence municipal systems. Both beneficiaries and project staff reported that advisers achieved some shifts in municipal practices.Footnote 74 Multiple actors noted the permanent status of advisers in the municipality prevented the program from being “sidelined” in the way USAID's project seems to have been.Footnote 75
The South African municipal governance case allows us to see what different levels of IDO autonomy look like in practice. USAID and DFID implemented programs with quite similar goals. They did so through rather similar contracting structures. But DFID's project exhibited far greater flexibility and use of agent initiative. Measurement and reporting via pre-specified targets played a substantial role in USAID's intervention but little in DFID's. USAID was more rule-bound, with substantial process controls and an orientation toward satisfying bureaucratic requirements.Footnote 76 DFID, by contrast, placed resident advisers in municipalities, and designed its project in a manner less tractable to control from above. DFID's project created reporting requirements that did not rely on externally verifiable and observable information. DFID's project was more successful than USAID's, a success clearly linked to the differences in how the projects were implemented and managed.
Conclusion
Environmental unpredictability has a negative effect on IDO project success. However, less politically constrained IDOs see systematically lower performance declines in more unpredictable contexts than do their more-constrained peers. The South Africa case study comparison provides suggestive evidence that what is true intra-organizationally is also true when comparing across organizations: constraints induced by political authorizing environment insecurity sometimes undermine comparative project success.
Variation in political authorizing environments has quite substantial potential impacts on development outcomes and consequently on developmental trajectories and conflicts. Effective delivery for a range of IDO projects is for some but not all IDOs precluded by political authorizing environments and the measurement and control systems to which they give rise. Constraints on agents that flow from an understandable, even laudable, desire to demonstrate results and accountability to politicians and citizens can undermine IDO performance.
In some instances output measurement and reporting may well improve organizational performance; when working in relatively predictable environments this may be the superior strategy. However in less-predictable environments, this reporting and tight principal control crowds out the organization's ability to serve its ends. The more unpredictable the environment, the more important it is for power and decision making to sit with field agents.
From public schools to multinational firms, many organizations struggle with Aghion and Tirole's tension between principal control and agent initiative. Philippe Aghion himself, in collaboration with a number of illustrious coauthors, has recently applied his model to private firms during the Great Recession.Footnote 77 Using data from eleven OECD countries, they find that private firms with more local-plant-manager control outperform more centralized firms in the sectors hardest hit by the crisis. As they put it, “higher turbulence benefits decentralized firms because the value of local information and urgent action increases.”Footnote 78 The usefulness of agent initiative and the ability to gather and use asymmetric (soft) information are far from an IDO-only, or even public sector, phenomenon.
My analysis suggests limits to the range of where external monitoring—the workhorse solution of applications of principal-agent theory to public bureaucracy—may indeed be a workable solution. For some tasks, in some environments, it is not just that the monitoring is costly—the monitoring itself may have deleterious effects. If indeed it is true that tight oversight is detrimental to performance in some circumstances, public accountability as conventionally conceived may sometimes come at the expense of desired performance outcomes. Reporting may be a façade, with reporting requirements inducing agents to produce numbers at the expense of actually forwarding the broader goals of their organizations.
IDOs operate in difficult contexts, and attempt to do difficult things. They are, perhaps unsurprisingly, often unsuccessful. In some of the domains where foreign aid has the potential to make the most difference, for example, in unpredictable fragile states, politically induced constraints on IDO autonomy make project success even less likely. This paper's findings suggest not only that we could do more to improve aid delivery, but that the move toward measurement and control in foreign aid in recent years may in some cases actually be hindering progress. The drive for measurement and quantitative results is usually framed around efficacy and value for money. If this encourages political authorizers to constrain IDOs’ ability to engage in more flexible, autonomous operational strategies, well-intentioned authorizers may end up accomplishing precisely the opposite of what they intend.
Supplementary Material
Supplementary material for this article is available at <https://dataverse.harvard.edu/dataverse/PPD>.
