In recent years, scholars and analysts have increasingly produced scientific studies of public policy, including many randomised controlled trials (RCTs), designed to precisely measure the impact of policy design. In this article, we refer to these studies collectively as impact evidence. Prominent research centres like J-PAL and funding entities like 3ie have now collectively produced thousands of publicly available studies about policy impact, often with a focus on developing countries. Impact evidence, if used in the public policy-making process, has been shown to increase the effectiveness of public policies in various fields including medicine, education, agriculture, technology and others (Slavin Reference Slavin2008; Banks Reference Banks2009). Consequently, powerful states, international organisations and funding agencies have pressed developing country governments to better use impact evidence in their policymaking.
Despite these efforts, government officials seldom employ impact evidence in designing policy (Head Reference Head2015, Reference Head2016), and a considerable literature has developed to promote evidence-based policymaking and investigate the reasons for its absence. Without high-quality evidence, governments and development NGOs may design programmes that are less effective or efficient than they could be. Such programmes might produce fewer positive outcomes than other potential programmes, be implemented in suboptimal ways, focus on populations that are less likely to benefit or consume too many resources when cheaper solutions are available. For example, after conducting multiple RCTs on educational programmes in Africa, prominent economists found that policies ranged in cost-effectiveness from $3.25 to $6,000 per person per year of increased educational attainment (Banerjee and Duflo Reference Banerjee and Duflo2009, 153).
Considerable evidence exists that lack of interest is not the primary obstacle to incorporating impact evidence into policy processes. In a recent global survey of more than 3,000 government officials in developing countries, 90% of them said that “there should be a larger number of links between academic and policy communities” (Maliniak et al. Reference Maliniak, Peterson and Tierney2012, 67–69). In fact, a survey of policy makers showed that only 4.7% of government officials thought academics should not be involved in the policy-making process (Avey and Desch Reference Avey and Desch2014). Despite these preferences and the increasing amount of relevant research, 76% of government officials said that the gap between academia and policymaking has not improved in the last 20 to 30 years (Maliniak et al. Reference Maliniak, Peterson and Tierney2012, 67–69). This continued failure to incorporate impact evidence into policy making could be explained by the many obstacles: accessibility, partisanship, emotion, corruption, laziness, bureaucratic inertia, policy path dependence (Head Reference Head2010) and low capacity or poorly prepared civil servants (Rajkumar and Swaroop Reference Rajkumar and Swaroop2008). These obstacles frequently lead government officials to utilise ideology, emotion, intuition and unique personal experiences rather than empirical evidence when making policy decisions (Banks Reference Banks2009).
How can politicians and bureaucratic leaders improve this situation? Many previous studies have focused on aggregate-level organisational and cultural factors that inhibit the use of high-quality policy impact evidence (Weiss Reference Weiss1999; Mayne and Rist Reference Mayne and Rist2006; Head Reference Head2010; Oliver et al. Reference Oliver, Lorenc and Innvær2014; Cairney et al. Reference Cairney, Oliver and Wellstead2016; Allard et al. Reference Allard, Wiegand, Schlecht, Rupa Datta, Goerge and Weignensberg2018). While such influences are undoubtedly important, individuals do not react uniformly to organisational or cultural factors, even within the same department or office.
Hence, we focus on variation at the individual level in response to administrative accountability efforts. As Christensen and Laegreid (Reference Christensen and Laegreid2015, 209) define it, “administrative accountability is related to a person’s position in a hierarchy whereby a superior calls a subordinate to account for his or her performance of delegated duties.” Administrative accountability is one important type of accountability that is theoretically tied to improving bureaucratic performance, though this relationship is the subject of extensive debates (Dubnick Reference Dubnick2005; Bovens et al. Reference Bovens, Schillemans and ‘T Hart2008; Brandsma and Schillemans Reference Brandsma and Schillemans2013; Laegrid Reference Laegreid2014; Christensen and Laegreid Reference Christensen and Laegreid2015). We contribute to this debate by investigating whether administrative accountability is tied to higher use of impact evidence in both a self-reported survey and in observed use of a website we created with easy access to impact evidence.
Theoretically, we draw on principal-agent approaches (Waterman and Meier Reference Waterman and Meier1998) to investigate whether enhancing bureaucratic accountability might prompt greater use of impact evidence. Previous research within this framework predicts that principals (high-level government officials, for example) can use certain tools of screening, monitoring, autonomy and sanctions to increase desired output by individual bureaucrats (agents). While government officials are often public-minded and want to do what is best for the people they serve, they are also subject to multiple human limitations that lead them to shirk, slack or lack the expertise to use impact evidence. If principal-agent control mechanisms can influence bureaucrats’ use of impact evidence, we should find that bureaucrats who are held administratively accountable through monitoring, selection or sanctioning processes or are (somewhat paradoxically) given more autonomy engage with evidence more. Of course, a host of other factors may also influence engagement with high-quality evidence and we control for as many of these as possible below. Given the general importance of managerial tools in bureaucratic behaviour and the absence of much prior work on the relationship between those tools and the use of impact evidence, this study can help flesh out scholarly understanding of this issue.
We examine individual-level responses to these accountability tools to identify which ones might work best, on average. We operationalise higher levels of screening by considering those whose job placement required passing a civil service exam. Monitoring is operationalised by whether bureaucrats report their managers emphasise the use of impact evidence. Higher autonomy is associated with self-reported individual independence. Finally, positive sanctions are measured using income.
We use an original survey of about 300 unelected government officials in Peru and India to examine the effect of these administrative accountability mechanisms. While much of the previous research on utilisation of impact evidence has focused on bureaucrats in wealthy countries, this study focuses instead on bureaucrats in developing countries. In face-to-face interviews, officials self-reported information about their own experience with different accountability tools and standard demographic characteristics. We measure engagement with policy impact studies in two different ways. First, the officials self-reported their level of engagement, including levels of interest and prior use. Second, following the survey we invited the officials to use a website we created (impactevidence.org) with summaries of more than 400 policy impact studies. The website summaries are written in accessible language, available in English or Spanish, searchable by topic and include easily interpretable graphs and summaries of the policy impact. We analyse data on practitioners’ website usage during the eight months following their survey participation.
Our findings suggest that positive sanctions, as measured by income, are consistently correlated with previous engagement with impact evidence as well as with actual website use. Management emphasis on academic research is positively correlated with interest in, but not reported or measured use of, impact evidence. Civil service exams and workplace autonomy are positively correlated with self-reported past use of impact evidence, but not website use. These results reinforce the point that promoting the use of impact evidence is a difficult task and requires using mechanisms that select and incentivise individuals to do so. While some have argued that lack of accessibility explains bureaucrats’ infrequent use of impact evidence (Oliver et al. Reference Oliver, Lorenc and Innvær2014; Cherney et al., Reference Cherney, Head, Povey, Ferguson and Boreham2015), our evidence indicates that making such evidence available does not mean it will be used. However, some administrative accountability tools may be helpful in promoting the use of impact evidence.
Theoretical framework
A large number of studies have examined obstacles to the uptake of impact evidence by government officials, primarily focusing on institutional structures and organisational characteristics. Obstacles include the complexity of the policy-making process (Weiss Reference Weiss1999), bargaining amid entrenched interests (Head Reference Head2010), management systems that do not mainstream impact evidence (Mayne and Rist Reference Mayne and Rist2006), low analytic capacity (Newman et al. Reference Newman, Cherney and Head2017; Allard et al. Reference Allard, Wiegand, Schlecht, Rupa Datta, Goerge and Weignensberg2018) and insufficient institutional valuing of research (Cherney et al. Reference Cherney, Head, Povey, Ferguson and Boreham2015).
While institutions and organisational cultures are undoubtedly important, at the end of the day, individual government officials make the choice whether to use impact evidence or not. Institutional rules do not guarantee compliance and cultures exert uneven influence on behaviours. Hence, it is helpful to know more about individual behaviour in order to understand how these broad forces manifest in individual action, particularly the use of impact evidence. One recent survey of over 2,000 Australian public servants recognised the importance of individual-level surveys (Newman et al. Reference Newman, Cherney and Head2015, Reference Newman, Cherney and Head2017). However, it examined a limited range of personal characteristics that focused on education level, previous employment, time employed, self-reported analytical skills, gender and age. It did not focus on management tools as we do here. The authors found that postgraduate education and previous work in a university or nonprofit were the main factors correlated with the self-reported use of academic research (Newman et al. Reference Newman, Cherney and Head2017, 170).
To fill the gap between bureaucratic structure and individual characteristics, we examine how individuals’ choice to use impact evidence might change in response to specific administrative accountability mechanisms. Politicians and scholars broadly praise accountability, but the jury is still out on whether accountability can actually improve bureaucratic performance (Laegreid and Verhoest Reference Laegreid and Verhoest2010; Brandsma and Schillemans Reference Brandsma and Schillemans2013). Because bureaucratic performance is such a complex phenonemenon responding to many inputs and because so many forms of accountability are possible (Christensen and Laegreid Reference Christensen and Laegreid2015, 209–212), the relationship deserves close study.
Consistent with this literature, we approach the question of whether and how administrative accountability mechanisms can improve individual bureaucrats’ engagement with impact evidence as an important empirical question. Of the many forms and approaches to accountability that exist (Bovens et al. Reference Bovens, Schillemans and ‘T Hart2008; Brandsma and Schillemans Reference Brandsma and Schillemans2013), we focus on those commonly employed within bureaucratic processes rather than other sorts of mechanisms such as broader stakeholder and political accountability. We lay out arguments below for why we expect accountability to have some effect and thus contribute both to scholarly understanding of the determinants of impact evidence use and to the broader literature on accountability and management techniques in shaping bureaucratic performance.
To better understand the potential impact of administrative accountability, we focus on mechanisms that are articulated in the literature on principal-agent approaches to accountability. This literature has identified five tools that higher level officials whose job is to exercise accountability (principals) can use as accountability mechanisms for lower level officials (agents): screening/selection; monitoring/reporting; rules/autonomy; sanctions and institutional checks and balances (Miller Reference Miller2005). Managers could apply these tools to promote the use of impact evidence. Within a bureaucracy, however, there is variation in how managers implement these tools and how they are understood by specific bureaucrats. To capture this variation, we utilise individual-level survey and website usage to identify associations between bureaucrats’ exposure to these tools and their use of impact evidence. We explore four of these tools, leaving aside institutional checks and balances.
Accountability methods often begin with screening job candidates through tools including prehiring interviews, evaluations and selection processes. Some studies have found that screening programmes for government employment improve bureaucratic outcomes. Kauppi and Raajj (Reference Kauppi and van Raaij2014) examined the comparative effectiveness of screening and monitoring and found that screening reduced noncompliance in bureaucratic procurement processes more effectively than monitoring. Likewise, Dahlstrom et al. (Reference Dahlstrom, Lapuente and Teorell2012) found that meritocratic recruitment reduces corruption among government officials to a much greater extent than other control mechansims.
While selection mechanisms for bureaucratic employment are not necessarily designed to select those more likely to use impact evidence in policy making, high-level analytical and quantitative skills are required to read, understand and apply impact evidence appropriately in the policy process. Some types of selection processes that favour these skills should increase use of impact evidence. Each country has different screening mechanisms, but civil service exams are a commonly used screening mechanism to ensure appropriate capacity among government officials (Sundell Reference Sundell2014). Such exams can help identify and select individuals with better education and understanding of principles that are central to employing impact evidence. While many countries, including Peru and India, have meritocratic recruitment including civil service exams, requirements are implemented unevenly (Grindle Reference Grindle2010), leading to important individual-level variation in the selection mechanisms to which different bureaucrats are subject.
Hypothesis 1: Government employees subject to more rigorous and meritocratic screening processes will use impact evidence more.
Monitoring government workers’ performance after they have been hired provides a tool for aligning the interests of principals and agents (Miller Reference Miller2005). Bureaucratic managers engaging in monitoring can focus either on internal compliance with methods and procedures or accountability for outputs (Christensen and Laegreid Reference Christensen and Laegreid2015, 212). New Public Management (NPM) reforms implemented in the past 30 years tend to favour a focus on outputs, but many scholars argue that managers are too often still focused on internal process compliance (Christensen and Laegreid Reference Christensen and Laegreid2015). With respect to the use of impact evidence within a bureaucracy, it is more straightforward to monitor whether employees actually use impact evidence during the policy-making process by examining their justifications for programme planning than it is to infer use of such evidence in the final programme design and results. Still, the low reported use of impact evidence among government officials (Head Reference Head2016) raises questions about the extent to which supervisors emphasise such evidence or the extent to which those efforts might matter.
To the extent that supervisors emphasise the use of evidence, individual bureaucrats reap both intangible and tangible benefits from complying with expectations expressed by their superiors. They respond to such monitoring expectations because (1) such expectations increase their beliefs about how to make their work more meaningful; (2) they enjoy the intrinsic rewards of meeting expectations or (3) because sanctioning may occur directly or indirectly. Failure to comply with an expectation may affect career and promotion prospects or budgetary support for their agency, even if there is no direct sanction for failure to comply. If public servants respond to monitoring signals, one would expect to find higher reported use of and interest in impact evidence by those who believe that their superiors promote academic research.
Hypothesis 2: Government employees subject to higher levels of monitoring on academic research will use impact evidence more.
Theoretically, the role of autonomy in promoting desired bureaucratic outcomes is complicated. Fukuyama suggests that “low-income countries [should] reduce bureaucratic autonomy while high income ones [should] seek to increase it” (Reference Fukuyama2013, 347). Low-income countries should rely on highly systematic bureaucracies, in this view, because superiors with more education can then mandate more effective policy processes to be implemented by less-educated subordinates. If subordinates have more autonomy, they are likely to make less effective choices. As both India and Peru fit this low-income profile, Fukuyama’s ideas suggest more effective policies are produced through less autonomy for public servants. In this view, lower autonomy should also result in more use of impact evidence because, consistent with the last hypothesis, superiors can simply mandate and monitor its use.
Other scholars, however, have linked worker autonomy to desirable outcomes because autonomy increases intrinsic motivation and creativity (Zhang and Bartol Reference Zhang and Bartol2010). Increasing bureaucratic-process autonomy has been at the heart of NPM reforms, which embrace the view that providing process autonomy while requiring improved outcomes will improve government performance (Christensen and Laegrid Reference Christensen and Laegreid2015). These reforms emphasise that too much focus on accountability may actually decrease performance because government officials focus on measurements associated with accountability rather than on actual improvements (Christensen and Laegreid Reference Christensen and Laegreid2015, 212–213).
More broadly, autonomy is an innate psychological need that, when satisfied, produces more self-motivation (Ryan and Deci Reference Ryan and Deci2000). A large literature has examined the role of psychological empowerment on performance at both team and individual levels (Maynard et al. Reference Travis, Gilson and Mathieu2012). Workplace autonomy is a fundamental aspect of psychological empowerment. Workers who feel empowered are associated with a variety of positive outcomes related to substantive and financial performance (Maynard et al. Reference Travis, Gilson and Mathieu2012). Autonomy is self-reported as one of the most important factors in determining job satisfaction and motivation among civil servants in India (Government of India 2010, 47–49). Rasul and Rogger (Reference Rasul and Rogger2016) find that higher levels of autonomy produce higher rates of public service delivery by bureaucracies in Nigeria. It seems plausible greater bureaucratic autonomy permits more scope for bureaucrats to incorporate innovation and new ideas, which increases opportunities and incentives for consulting impact evidence.
Because theoretical arguments about the role of autonomy cut both ways, we test competing hypotheses.
Hypothesis 3a: Government employees with less workplace autonomy will use impact evidence more.
Hypothesis 3b: Government employees with more workplace autonomy will use impact evidence more.
Additionally, managers may use positive or negative sanctions to incentivise certain behaviour in public servants. The most obvious and easily observable sanction is income. It is also a management tool that is visible and salient to bureaucrats, as it is fundamentally important to their overall well-being. Previous research has found a positive correlation between income/prospects of career progression and better policy outcomes (Haque and Sahay Reference Haque and Sahay1996; Evans and Rauch Reference Evans and Rauch1999; Van Rijckeghem and Weder Reference Van Rijckeghem and Weder2001; An and Kweon Reference An and Kweon2017). In theory, more competent and capable public officials should receive higher salaries.Footnote 1 Competence and capability should correlate with the use of impact evidence, which is expected to improve policy analyses and implementation, produce more efficient outcomes, generate new ideas and eliminate inferior ideas. Managers seeking to promote the use of impact evidence may explicitly tie salary increases to better analyses and clarify that high-quality evidence helps improve analyses.
Hypothesis 4: Government employees who have been positively sanctioned with higher salaries will use impact evidence more.
Research design
We investigate the use of impact evidence in developing countries because even small policy improvements could potentially yield large results. Peru and India were selected because are large, important developing countries that have had less success than some, but more than others at creating an effective bureacracy. Practically, they are countries where language barriers were surmountable for this study. Many civil servants in India speak reasonable English and both the survey and the website were translated into Spanish and administered by Spanish-speaking surveyors in Peru. Importantly, Peru and India are near the global average in level of democracy and government efficacy (both of which are important inputs into the quality of bureaucratic outputs), meaning they potentially provide insights into other developing countries. In particular, Peru ranked at the 49th percentile and India at the 57th percentile in the Government Effectiveness index of the World Bank’s Worldwide Governance Indicators.Footnote 2 This index tracks the sorts of issues we focus on in this article, such as the quality of the civil service, the degree of its independence from political pressures and the quality of policy formulation and implementation.
In India and Peru (and many other countries), their professional bureaucracies are institutionally designed to meet challenges (in particular the principal-agent problems) that afflict all bureaucracies and to address problems that were historically endemic in other types of bureaucratic arrangements (i.e. patronage systems). Professional bureaucratic organisations tend to have similar selection mechanisms (like a civil service exam), protections from discretionary hiring, seniority promotion and fixed retirement ages (Bekke et al. Reference Bekke, Toonen and Perry1996; Van derMeer et al. Reference Van der Meer, Raadschelders and Toonen2015; Bertrand et al. Reference Bertrand, Burgess, Chawla and Xu2019). Due to international diffusion, many bureaucracies tend to resemble each other, at least with respect to these types of formal institutional arrangements. Where similar institutions and professional practices exist, we might expect similarities in the effect on use of impact evidence to exist as well. Thus, we present pooled data from the two countries in our primary analysis.
In spite of these important similarities, obvious differences exist between Peru and India. Most notably, per capita GDP in Peru is around 80th in the world (about $7,000) while it is 140th in the world in India (about $2,000). While both are democracies, they have very different political, social and economic systems as well. We are well aware that the similarities induced by similar bureaucratic structure may be superficial or interact with the different national or organisational cultures. Thus, we also present results for each country separately (in the Appendix, Supplementary material) and discuss how any differences comport with the countries’ distinct features. Because of differences, similar results between the countries gain credibility as potentially generalisable. Different results between them, on the other hand, provide informative patterns for future research.
The sampling frame of government officials for this study was created from officials’ contact information provided by government websites for various offices and ministries in India and Peru. Officials were included in the sampling frame if they were (1) located in the country’s capital city; (2) an unelected government employee and (3) connected to the national rather than a state or local government
Teams of American and local researchers contacted officials by email, phone or in-person visits. Researchers reached out to all officials in the sampling frame at least once. However, second and third attempts focused more on officials with job titles suggesting that they have influence on policy decisions (such as directors of research or executives within government ministries).
Officials were invited to receive training on our impact evidence website. We include a brief sample of the nature of that website in Figures 1 and 2. Studies summarised on the website come from six respected research organisations that focus on policy-oriented RCTs such as the International Initiative for Impact Evaluation (3ie). They cover a very wide range of topics that we aggregate in the following categories (listed in order of the most studies): health, education, business, communities and families, agriculture, governance and infrastructure. The website summarises each study including study population demographics, countries, sectors, main insights and specific findings for up to six outcomes.
Screen shot of website home page.
Screen shot of website project page.
At the training, the researchers administered a survey that collected data on demographics, bureaucratic position and roles, experience with the principal control mechanisms discussed above and interest in and prior use of impact evidence. Data collection in both countries took place in summer 2017. We successfully surveyed 322 government officials in Peru and India. The survey data show that the trainings and survey primarily reached the website’s target audience. While our sample frame did not perfectly select only those involved in the policy process, most respondents fit that category, 74% of our survey respondents selected policy work when provided a list of tasks associated with their jobs.
Survey respondents also indicated which government sectors they had worked at in the past 10 years. Figure 3 shows the percentage of survey respondents that work in each government sector.Footnote 3 The sectors in which the survey respondents work tend to overlap the distribution of studies represented on the website. Nearly a quarter of respondents indicated that they work or had worked in the government health sector, which is also the most represented topic on the website.
Percent of survey respondents by sector of government.
Moreover, survey results demonstrate that our sample does not contain a disproportionately high number of impact evidence users. In fact, 40% of the sample reported not using impact evidence in their jobs at all during the previous six months. This number is similar to a recent survey sample of government officials in Australia, where 39% reported not using that sort of research in the past year (Newman et al. Reference Newman, Cherney and Head2017, 165). About 17% of the sample said they had never used impact evidence in their jobs. A strong proportion of the sample, 92%, said they believed that using impact evidence would improve the quality of their work. This is similar to the previous finding that 90% of government officials in developing countries agree that there should exist more links between policymakers and impact evidence (Maliniak et al. Reference Maliniak, Peterson and Tierney2012, 67–69).
Our respondents thus reasonably represent a viable population of government officials who might have some interest in and use for impact evidence. We cannot of course generalise to all government officials in both countries, as our sample was not random. At the same time, some government officials are simply not interested in policy impact evaluations, either because they have no relevant job responsibilities or because they preemptively refuse to consider it. This group is not of much interest to those attempting to promote better use of impact evidence in policy making. Other important surveys similar to this one have faced the same issues but still generated important insights (Newman et al. Reference Newman, Cherney and Head2015, 27; Reference Newman, Cherney and Head2017, 161–162). As Newman et al. (Reference Newman, Cherney and Head2017, 163) put it, a survey of this sort “serves as the best way to obtain information about a profession that is heavily guarded, has an institutionalised protection of anonymity and is generally opaque to outside investigation.”
For the self-reported outcomes, the survey asked nine questions about officials’ engagement with policy-impact evaluation studies. We used factor analysis to identify these questions and create one overall index and two subindices that separate out self-reported interest in and previous use of impact evidence.Footnote 4 Each of the questions is weighted equally in the created indices. Responses to each question were standardised on a scale from 0 to 1 to allow for comparison across indices and simple interpretation of substantive effects.
The second outcome measure is behavioural. During the survey, each respondent was provided with information on and a demonstration of how to use the website. Over the ensuing 8 months, we tracked respondent website registration and usage, as approved by an Institutional Review Board. For the behavioural outcome, we consider an index of website usage, with selection correction conditional on whether a respondent chose to register on the website. The website usage index is a weighted composite measure of the times the respondent logged into the website, total time spent on the website, number of unique studies the respondent used, number of questions clicked on, number of searches made, number of clicks on the “about us” icon, number of clicks on the “about the website” icon, number of article links opened, number of policy links opened and number of clicks on the browse-by-sectors icons. Table A1 (in the Appendix, Supplementary material) presents the summary statistics for these dependent variables.
Results: principal control mechanisms and impact evidence engagement indices (pooled sample)
Notes: Standard errors in parentheses. *** p < 0.01, ** p < 0.05, * p < 0.1. Fixed effects models for each of the three main indices with all controls included.
To operationalise screening mechanisms, we use a binomial variable indicating whether respondents reported taking a civil service exam. Approximately 43% (46% in India and 41% in Peru) of the sample took an exam to obtain their current position. To test the effects of monitoring, our survey included a question asking the respondent: “What sources do you think your superiors want you to use in decisionmaking?” The respondents ordered seven different options according to their perceived importance. Forty percent of all respondents said their supervisors would have ranked impact evidence as the most important decision-making factor. To measure autonomy, we use self-reported assessments. These self-assessments are measured on an increasing four-point scale. The mean value of autonomy is 3.1 (corresponding to “some autonomy”) in the pooled sample. To measure the effects of sanctions, we use respondents’ income. Income is measured as six income brackets. Income brackets for the responses were determined by looking at official government tax brackets and the Latin American Public Opinion Project (2017).
Control variables include a variety of personal and job characteristics that might be expected to influence bureaucrats’ interest in or use of impact evidence in policy making. These characteristics include gender, education (both level and field of study), age, years working for the government, the number of job tasks included in their current role, whether their job includes programme and policy-related tasks, how many ministries they have worked in and a four-point scale of traditional social values. Measures of gender are binary and equal to one for females. Overall, about 35% of the sample is female although only 17% of the India sample is female while nearly 45% of respondents in Peru are female.
Statistical analyses for the survey-based dependent variables employ OLS regressions. We use a Heckman selection model for the behavioural outcome in order to account for possible differences between those who chose to register for the website (for whom we have website usage data) and those who did not register. We use country fixed effects in the models using pooled data from each country. Alternative model specifications control for the sector of government respondents work in and the specific job tasks held (see Appendix Tables A4–A6 in Supplementary material).
Results and discussion
Table 1 shows the results of linear regressions with country fixed effects for each of the survey-based indices used as dependent variables. Column 1 reports the use index; column 2 the interest index and column 3 reports a master engagement index, which combines nine survey questions regarding engagement with impact evidence. Tables A2 and A3 in Supplementary material show the results for India and Peru separately.
Screening: civil service exams
For the master engagement index, individuals who took a civil service exam prior to hiring score about 5% higher than other government officials (Figure 4). Taking a civil service exam also correlates with 10% higher reported prior use of impact evidence, but not with higher interest in those studies. Examining the country samples separately, we find that this result appears to be driven by the respondents from Peru. The effect of having taken the civil service exam in India is insignificant on all three indices, with coefficients close to zero.
Civil service exam takers report more engagement with impact evidence than noncivil service exam takers (pooled sample).
Of course, as these data only reflect information about those who chose to participate in this study, it may be that self-selection into the study represents a more direct selection process on valuing the use of impact evidence in policy making than taking a civil service exam. If study participation is itself a proxy for the type of selection that the civil service exam would theoretically provide, this may underestimate the effects of the exam as a screening mechanism.
Monitoring: supervisor emphasis
Government officials who report higher levels of supervisor emphasis on use of academic studies are more likely to report engagement with impact evidence in our master index (Table 1). However, as with the screening mechanism, the effect of this variable differs in the two more specific indices. The importance that supervisors place on using policy impact evaluations makes a significant difference in their employees’ interest in those evaluations. In fact, employees who say their boss rates impact evidence as the most important factor to use in decision-making ranks more than 10% higher on the interest index than someone who says their boss rates impact evidence as last in importance (Figure 5). This effect exists in spite of the self-selected nature of our sample. However, a supervisor’s emphasis on academic research does not correspond to higher previous use of impact evaluations. By examining the two countries separately (Tables A2 and A3 in Supplementary material), we see that the influence of supervisor emphasis in India affects both the interest index and the master index, while in Peru it significantly affects only the master index. This may indicate different levels of deference to supervisors in the two countries. Hypothesis 2 thus receives mixed support
As supervisors rank academic research as more important, employees show more interest in using impact evidence (pooled sample).
According to these results, supervisor focus on impact evidence positively influences the employee’s reported interest in such reports. Obviously, there are different possible interpretations of this correlation. Those who feel their supervisors value something are likely to report valuing it as well even if they do not actively carry out their supervisor’s wishes. It is also possible that government employees who are more interested in impact evidence may assume their supervisors are emphasising it as well. The lack of effect of supervisor emphasis on prior use of impact evidence in either country indicates that even where people believe that management wants impact evidence to be used, management techniques do not effectively reinforce that preference. This is perhaps not surprising given potential issues of accessibility (management would be unreasonable to sanction failure to use something that most officials cannot realistically access).
These results suggest both the importance and limitations of a monitoring mechanism. Although government officials acknowledge greater importance of using impact evidence when their supervisors stress it, officials still do not use that evidence more often.
Rules: workplace autonomy
As predicted by Hypothesis 3b, Table 1 shows that greater autonomy has a statistically significant positive relation with overall engagement in impact evidence (p < 0.05). A move from least discretion to almost complete autonomy on the scale results in a positive shift of about 8% in the master index (Figure 6). Of the two more specific indices, autonomy is better correlated with use than with interest, though the result is only significant at the p < 0.1 level. Examining the two countries separately (Tables A2 and A3 in Supplemntary material) shows that this effect is driven by the respondents in India, where the effect of autonomy on the master and prior use indexes mirror those in the pooled sample. In Peru, autonomy has no effect on our measures. This effect of self-reported autonomy is not attributable to greater seniority, education, civil service position or management emphasis on impact evidence. Not only does our analysis consider the effects of autonomy while controlling for these other factors, but the data also reveal that there is no significant correlation between self-reported autonomy and any of these other variables.
Officials with more autonomy report more engagement with impact evidence (pooled sample).
These findings suggest that more workplace autonomy provides greater scope for trying new and different methods of research for public policies and empowers individuals to do so, even in the context of developing countries. As agents receive more freedom to devise their own public policy plans, they expand their vision and use nontraditional sources of information, such as impact evidence. We deem it less likely, but possible, that reverse causation is at play, and people most engaged in impact evidence are granted more autonomy.
Positive sanctions: income
As predicted by Hypothesis 4, higher income provides positive sanctioning or incentives to use impact evidence (Figure 7). In the pooled sample, those reporting higher income score higher in both the interest and the master indices. In India, they also are (weakly) higher on the prior use index. In Peru alone, however, income is not significantly correlated with use, interest or the master index. We speculate that this difference in the effects of income in the different national analyses may be due to the more severe truncation of the variation in the income data in Peru. About 70% of the sample in Peru indicated the highest income category, versus only 30% in India. This suggests that government officials are better paid relative to the national income distribution in Peru than in India; it also means that we are measuring similar income levels where there is actually variation, reducing the measured effect of our income variable. We believe that income incentives (as positive sanctions) should theoretically prompt better bureaucratic performance (including use of impact evidence). Of course, if bureaucrats are already well paid relative to the national population in Peru, it may be politically or economically too costly to use income as a sanction to encourage use of impact evidence in settings like Peru.
Officials with higher income report more engagement with impact evidence (pooled sample).
Control variables
Our control variables include several interesting correlations. Those with greater seniority (more years in government) in the bureaucracy tend to score lower on the use and master indices in the pooled sample and lower on the use index in Peru. Seniority has no significant effect in the Indian sample alone. These effects exist net of the effect of age, which is also included as a control in these models. This may indicate that in Peru, more senior officials are less likely to face meaningful sanctions or a need to strive for promotion. Interestingly, similar to Teodoro’s (Reference Teodoro2009) finding in the United States (US) that bureaucrats recruited from other departments are more entrepreneurial, our results show that those who have worked in more government sectors during the past decade score higher on both the use and master indices. These effects exist in both countries and in the pooled sample. In the pooled sample, those who hold more traditional social values score lower on the use and master index.
However, perhaps the most notable result for the control variables is the effect of gender. Across models and dependent variables in the survey results, female bureaucrats consistently rank higher on engagement with impact evidence in the pooled sample and in India. In Peru, they rank higher only on the index of prior use of impact evidence. While we did not hypothesise about these findings, they are worthy of some discussion and investigation that could lay a foundation for future research. Principal-agent theory and accountability approaches tend to view accountability mechanisms as broadly applicable. However, it is possible that women in our sample are responding differently to accountability mechanisms than men. For example, if women are more likely to be overqualified for their jobs due to biases in selection processes or discrimination (Kirai and Elegwa Reference Kirai and Elegwa2012; Mohr Reference Mohr2014), they may engage with impact evidence to a greater degree than men even when they are not subject to accountability mechanisms (selection, monitoring, autonomy and sanctioning). Women may be more responsive to monitoring mechanisms (especially if they are in the form of informal expectations) than men because they frequently must put extra effort into demonstrating their competence and performance (Lyness and Heilman Reference Lyness and Heilman2006; Gorman and Kmec Reference Gorman and Kmec2007). In terms of autonomy, there is evidence that women are socially conditioned to disproportionately take on emotional labour and make the workplace run smoothly for everyone (Guy and Newman Reference Guy and Newman2004). If so, greater autonomy for women may lead them to take on tasks other than engaging with impact evidence.
Given the lack of concrete prior research on this topic, we considered the possibilities of gendered responses to accountability mechanisms in an exploratory fashion by interacting gender with the those mechanisms. The results are shown in Appendix Table A7, Supplementary material. The results show that the effects of the accountability mechanisms for male bureaucrats (the uninteracted coefficients) remain similar to those reported in Table 1. Additionally, women and men have no significantly different responses to accountability mechanisms except for autonomy. With increasing autonomy, female bureaucrats significantly decrease their engagement with impact evidence for all three indexes. However, increased autonomy still increases use, interest and overall engagement with impact evidence for male bureaucrats. Since female bureaucrats have a higher baseline engagement with impact evidence, men with greater workplace autonomy become as proactive as women in engaging with impact evidence, but at low levels of autonomy, women do so to a greater extent (see Appendix Figure A1 in Supplementary material). Given their already higher engagement with impact evidence, women may prioritise other things when they have more autonomy.
Website use
We now turn to testing our hypotheses by examining website usage as a dependent variable. We utilise an weighted index comprising the ten measures capturing how much participants used the website as described above. Each is a cumulative measure of usage over a period of 8 months after the training, normalised to the range from 0 to 1. Around 40% of survey respondents never registered for the website. In India, only 19% of respondents did not register for the website. In Peru, fully 50% did not register.
Web usage data are only observed for those who chose to register for the website. Those who chose to register are self-selected, and such selection is unlikely to be as-if random. Thus, statistical adjustments are needed to avoid reporting biased results. A Heckman selection model (Heckman Reference Heckman1979) is employed to correct for the selection bias that exists in the observed data on website usage. This procedure estimates the odds of a respondent registering for the website in a first-stage equation. Those estimates are then incorporated into the equation estimating the effects of our accountability variables on website usage.
To identify the model, an indicator of whether the bureaucrat has a graduate degree is utilised. Theoretically, those with more academic training would be expected to register at higher rates due to greater familiarity with the kind of studies presented on the website. However, among those that did register for the website, those with graduate degrees are not expected to differ in their overall website usage. Analyses of a registration dummy variable and the website usage index confirm that those with graduate degrees are significantly more likely to have registered, but do not differ from other registrants in their website usage.Footnote 5
First, do any of the measures of accountability mechanisms (civil service exam, supervisor emphasis, autonomy or income) used in the analyses above have any effect on website usage? Column 1 of Table 2 shows the model using pooled data and graduate education as the selection variable. Columns 2 and 3 of Table 2 show the results from India and Peru, respectively. Table 2 shows that most of the principal-agent variables do not influence website usage. The lone exception is income. Those with higher income use the website more in the pooled sample and in both countries individually. Even in Peru, where income did not significantly correlate with higher prior use or interest in impact evidence, better paid bureaucrats used the website at a higher level. This indicates that either the selection into higher income or the positive sanctioning that higher income represents does lead to greater use of impact evidence, at least in the situation where accessibility obstacles have been reduced, as with this website.
Results: principal control mechanisms and website usage (pooled sample, Heckman selection model, selection: graduate degree)
Speculatively, the general absence of correlation regarding accountability mechanisms and website usage is perhaps explained by the fact that such mechanisms need to be targeted for particular tasks. Well-paid officials probably already have the skills, interests and personal characteristics to utilise the website. Others might need more specific incentives in the form of particular assigned goals and monitoring from superiors that may also include the autonomy to spend the time to peruse and make use of such a resource. It is unlikely, however, that supervisors would change their standard operating procedures quickly to incorporate a newly introduced and largely unknown resource such as our website.
Given the interesting results on the control for gender in the previous section, we reexamined the website usage index with gender interacted with accountability mechanisms (see Appendix Table A9 in Supplementary material). Unlike in the survey data, female bureaucrats have a significantly lower level of website usage than do males, both in the pooled sample and in Peru (Table A9 in Supplemntary material, Columns 1 and 3). In the pooled sample, the gender interactions do not change the results. In India, the interaction with gender weakens the effect of income. In Peru, website usage by female bureaucrats increases in response to income more strongly than among male bureaucrats (p < 0.10) (the effect of income is also significant for men). Additionally in Peru, supervisor emphasis has no effect on men’s website usage, but significantly increases female bureaucrats’ usage of the website. This provides some insight that, in some contexts, women might be more receptive to accountability mechanisms when given similar access to impact evidence.
Conclusion
Many states, international organisations, practitioners and scholars strongly promote the use of rigorous research in policy making, yet surveys show that government officials do not regularly draw on impact evidence. Efforts to integrate such evidence might be especially important in developing countries, which have many needs and large potential for change. How can governments improve their performance in this regard?
Understanding the utility of administrative accountability mechanisms, especially those identified by principal-agent theorists, provides insights into this question, although it is far from the only possible approach. Accountability mechanisms such as screening, monitoring, autonomy and sanctions offer possible tools. High-level government managers can act as principals to better screen applicants through civil service exams for analytical skills important to using policy impact evaluations. Managers can also monitor the use of rigorous research by requiring subordinates to regularly report on their use of impact evidence. Additionally, they can grant greater autonomy to officials who might otherwise feel constrained by the many factors that have traditionally dominated policy-making procedures in their country and use positive inducements like salary to reward the use of impact evidence.
Survey evidence in this article suggests that screening, monitoring, autonomy and positive sanctions are indeed associated with engagement with impact evidence, though with many qualifications and nuances. Income is the most consistently correlated with our outcomes, as it predicted overall engagement and interest in impact evidence as well as actual use of our website. The relationship is very strong in India, but not in Peru. Our income question in Peru, however, failed to capture much variation, with the strong majority of officials reporting income in the highest category. Screening is associated with overall impact evidence engagement in the pooled sample and correlates more with use than with interest, with especially strong results in Peru. Monitoring is also strongly associated with overall engagement in the pooled sample, though it correlates better with interest than with use. The relationship is more consistent in India, but also occurs in Peru. Autonomy is positively correlated with overall engagement in impact evidence. The relationship holds up in India and is positively signed but insignificant in Peru.
Accessibility can also be an important issue for government officials. Policy impact evaluations are often difficult to find, located behind paywalls, filled with jargon, written in English and targeted at scholars rather than practitioners. The barriers to entry into impact evidence for public officials in developing countries are quite high. The study reported in this article included a strong attempt to reduce those barriers by building a free website that summarises the results of policy impact evaluations in easily interpretable graphs and concise, policy-oriented statements. The site is also in Spanish to ease accessibility issues further for Latin American officials. Many government officials responded positively to this tool by requesting a meeting to demonstrate the website and by registering to use it.
Still, it would be naïve to think that better accessibility alone will resolve the useability gap for impact evidence. As many have argued, access alone does not ensure that government officials will use or even look at that evidence. Other research has identified a wide variety of serious obstacles to the use of such evidence. Results reported here suggest that some of those obstacles might be overcome using administrative accountability tools (particularly positive sanctions), but that task will not be easy.
Acknowledgements
The authors acknowledge the helpful comments of members of the Global Politics Lab at Brigham Young University.
Funding
The authors thank the Department of Political Science and the College of Family, Home, and Social Sciences at Brigham Young University, AidData, and USAID for funding used in this project.
Data Availability Statement
Replication materials are available in the Journal of Public Policy Dataverse at https://doi.org/10.7910/DVN/I15POM.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S0143814X21000015
