There is nothing fundamental to disagree with in the scientific argument presented in Al-Ubaydli et al.'s ‘How Can Experiments Play a Greater Role in Public Policy’. Scholars have been arguing along similar lines for many years now (e.g., Miguel et al., Reference Miguel, Camerer, Casey, Cohen, Esterling, Gerber and Glennerster2014; Lin & Green, Reference Lin and Green2016). Al-Ubaydli et al.'s contribution is to place these recommendations for improvements to the conduct of trials into a more integrated framework, which reduces the problems of scaling up and strengthens the link between science and implemented public policies. So, all good then? Well, maybe. To complement the framework developed by economists, political scientists and scholars of the policy process can draw attention to the political factors that constrain the use of evidence by public agencies and explain why reforms to research practice might not achieve their intended effects. In fact, by increasing the costs of doing research, there could be unintended negative consequences behind the programme of Al-Ubaydli et al., which might reduce the chance of more robust forms of evidence emerging. This statement of limits to knowledge utilization is not intended to imply that policymakers and researchers should not adopt these proposals, but all researchers need to be aware of the presence of politics in what they do and understand how policymakers typically respond to research findings. It should be stressed that this comment piece only focuses on the implications for policymakers. The proposals Al-Ubaydli et al. make for academic rewards and practice, such as publishing null findings, are not controversial and should be heartily supported by anyone wanting better science.

At bottom, there is an assumption in this article (and in many others) that there is a form of market failure at work with the diffusion of results from trials and other robust forms of evidence, which explains the relatively lower than optimal levels of utilization. By thinking about the lack of the use of evidence in this way, it is a natural move to recommend corrections to the information-generating process. As a result, a better-functioning market would allow for a greater influence of trials and a greater use of evidence. Add a beneficent and rational policymaker as commissioner and it is reasonable to expect an upswing in the use of trials and better inferences made from them by the removal of barriers to trading efficiently. The problem is that such signals in the market for information are received by actors who have essentially political objectives, especially the politicians who have interests in re-election and advancing in their careers. Bureaucrats too are political animals whose interests are tied to those of the politicians who usually command them and who have an interest in the preservation of their bureaus and in promoting their own careers. The other characteristic of these principals is that their time horizons tend to be short: they are tied to election terms and to natural cycles of career progression. Of course, bureaucrats and politicians have their own values, which include a belief in serving the greater good and being effective policymakers, which can enhance their reputations, but they also need to balance and trade off multiple goals against each other. In Policy, Office, or Votes? How Political Parties in Western Europe Make Hard Decisions, Müller and Strøm (Reference Müller and Strøm1999) set these out as office-seeking, policy-seeking and vote-seeking objectives of parties and political leaders. Evidence needs to be interpreted within the assessment of these trade-offs, and arguably only the policy-seeking politician is going to be interested in developing trials; even then, the policymaker is going to use evidence that more closely follows ideologically pre-set objectives. Al-Ubaydli et al. believe they are “recognizing the various incentives of the actors in the scientific market of knowledge creation” and encouraging “scholars to place themselves in the shoes of the policymakers whom they are trying to influence,” but this assumes that policymakers’ actions are themselves best represented by an economic model of decision-making and where improving aggregate welfare is the only goal.

Politicians and bureaucrats operate with the same heuristics and biases as the citizens they seek to influence, which means their time horizons may be inconsistent, and they find it hard to weigh up evidence by full ranking of all objectives against their utility (Sheffer et al., Reference Sheffer, Loewen, Soroka, Walgrave and Sheafer2018). In addition, the political environment is fast-moving, needing evidence to be produced quickly and in a form that can help achieve quick decision-making. The claim that information from evaluation needs to be understood in terms of the broader constraints of the policy process is very conventional in studies of the utilization of evidence. It goes back to the early 1970s when scholars reflected on the expansion of research associated with the growth of US government interventions at that time and where there emerged considerable realism about which research ideas gain influence and which do not (Weiss, Reference Weiss1977, Reference Weiss1979, Reference Weiss1998).

Much of the policy process operates in areas that politicians are less worried about in terms of fulfilling their core objectives. Trials have recently proved their mettle in these fields, especially over procedures for service delivery and implementation, which persuaded policymakers to commission randomized evaluations more frequently (Halpern, Reference Halpern2016; John, Reference John2018), as evidenced by the success of the UK government's What Works Trial Advice Panel (TAP). Even with service evaluations, the advocates of trials and nudges still have to be very careful – maybe even show some cunning – when selecting which interventions to test, and they need to be aware of the preferences and time horizons of bureaucrats and politicians. They need to ensure successes that have tangible benefits and avoid political controversy. Just demonstrating the superiority of knowledge will not be enough.

The proposals set out by Al-Ubaydli et al., if implemented, are not going to make it more likely that higher standards of evidence produced will be listened to. Recent scholarly work on the adoption of policies focuses on the importance of the ease of translation of ideas and recommendations, which are about targeting the presentation and simplifying findings (Flinders, Reference Flinders2013; Wood, Reference Wood2014). Clearer exposition is related to the idea that official time is limited and there is not much appetite for technical and jargon-heavy materials. The presentation of research findings needs to be attentive to the strong likelihood that recommended policies will not be adopted, but also aware that a researcher can get lucky if in the right place at the right time when a sympathetic official or politician is in place (Kingdon, Reference Kingdon1984). The presentation of research material from trials needs to be alert to the non-linear processes within the policy process as well as the influence of the climate of ideas (John, Reference John2013). Figure 1 in Al-Ubaydli et al.'s article, entitled ‘The Knowledge Creation Market’, seems like a massive oversimplification of decision-making from the perspective of the student of the policy process, even as a stylized model.

Rather than assume a market for information working on its own or with reforms, it might be better to let skilled advocates and entrepreneurs carry on presenting shiny new things to policymakers, as these practices increase the chances of policymakers trying things out experimentally rather than falling back on weaker research methods of using observational data or their gut instincts, or taking up ideas cooked up by political advisers and think-tank staff. They are going to look at simplified summaries and promising early results because they are presented in a way that is attractive. More technical presentation of the information and requiring more procedures to follow could reproduce the long-running divide between policy and research, which remains as entrenched as ever in many jurisdictions, such as the UK. The great thing about behavioural public policy is that it has to a certain extent managed to get over this divide by showing pathways to change illustrated with randomized controlled trials. In the end, even if the process has been faulty, it has got policymakers more used to thinking about testing policy with trials, even if the results can be disappointing when rolled out, as the example at the start of the paper by Al-Ubaydli et al. makes clear.

Some of their proposals, if implemented, may increase the costs of commissioning and implementing trials. The skilled advocate knows that at an early phase of diffusion those costs need to be minimized so that the decision to adopt a trial amounts to common sense and a slight adjustment to standard operating procedures by the agency. Proposals such as insisting on the registration of government trials, increasing the number of replications before evidence is accepted, insisting on the presence of the researcher on the implementation team and continuous measurement might discourage an agency from adopting a trial as its preferred method. It may also increase the number of veto players to a trial and introduce hassle for other partners in the policy process, including other units within the bureaucracy, such as information technology divisions that provide this information. It is hard enough anyway to do trials; putting even more obstacles in the way of busy policymakers reduces the chances of them ever commissioning or implementing them. Most people working in government agencies know it is an uphill struggle to get a trial off the ground. The challenge is to ensure that the implementer respects randomization, that the treatment is faithfully implemented, that subjects are not lost during the implementation of the trials, that one-sided attrition is minimized, that the outcome data are collected correctly and that contamination is reduced. The research world is full of trials that failed (Karlan & Appel, Reference Karlan and Appel2016), and the reason for this is usually a lack of faithfulness to the basic design principles (John, Reference John2017). Given this precarious existence, the proposals here might even reduce the level of research utilization by increasing the costs of gathering evidence from trials so that policymakers see them as too risky and unlikely to yield short-run benefits. The strategy should be to persuade policymakers to use trials and not to be fearful of them. There is also a risk of sounding patronizing to policymakers by telling them to do things, like with Al-Ubaydli et al.'s Proposals #7 and #8.

For the advocate of trials, there has to be a dual strategy: one is increasing the strength of the science behind trials and their reporting and interpretation; the other is creating the conditions whereby policymakers will be more comfortable using them. In the long term, there may be more consensus behind the proposals in Al-Ubaydli et al.'s piece, but for now, it is important to recognize that the market for research information needs to work alongside the market for political information. When the two markets join more closely together, there is greater likelihood that these scaling-up proposals will be adopted and work as effectively as of course they should.