SELECTION BIAS IN ON-FARM TRIALS AND THE IMPLICATIONS FOR TECHNOLOGY SCALING

Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019)’s comment contests the socially embedded nature of AR4D and the ways in which this can result in unintended biases in the design and implementation of on-farm trials. As a consequence, they do not engage with the ramifications of selection bias for scaling. Their comment ironically provides further evidence of how the social organisation of AR4D project implementation can produce selection biases. For instance, Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019) emphasise that previous experience with extending CA in southern Africa had shown ‘. . .[t]he need to concentrate initially on innovative and progressive farmers, rather than on “average” or random farmers’. They also acknowledge selection bias emanating from their interactions with farmer communities, which had sometimes selected ‘influential farmers [among themselves] rather than only representative farmers’ for hosting the on-farm trials. Yet, at the same time, Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019) assume that holding community meetings ensures the selection of representative farmers: ‘. . .the community was unlikely to select farmers they thought were outliers of who did not represent their conditions’. In addition to the apparent contradictions between these statements, we consider this assumption to be highly problematic.

There is a considerable literature on the social dynamics of group meetings in development contexts that emphasises the effects of power, authority, gender and the presence of outsiders–such as researchers, extension agents and/or development professionals–on the outcomes of participatory processes (Cooke and Kothari, Reference Cooke and Kothari2001; Leach et al., Reference Leach, Mearns and Scoones1999; Mosse, Reference Mosse1994). In addition, studies of smallholder technology adoption–as well as the Bangladesh case in our paper–show that farmers’ interest to participate in AR4D projects cannot be reduced to an interest in the (testing of) technologies alone (Andersson and D'Souza, Reference Andersson and D'Souza2014; Kiptot et al., Reference Kiptot, Hebinck, Franzel and Richards2007, Pircher et al., Reference Pircher, Almekinders and Kamanga2013). Agronomists thus need to be acutely aware of the social environment in which they conduct their on-farm experiments. Organising on-farm trials in a participatory manner, as Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019) suggest they did, is likely to introduce different kinds of selection bias, rather than guarantee against it.

We fully acknowledge the challenges inherent to conducting rigorous experiments in partnership with farmers and other stakeholders, while also assuring that technologies can be adapted to local conditions. There are no easy solutions here. Yet, to collaborate with innovative, progressive and influential farmers when evaluating and adapting technologies underscores the very dilemma that our paper seeks to address: that the selected farmers may not be subject to the same resource constraints as ‘average’ or randomly selected farmers. Similarly, Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019)’s use of one standardised rate of fertilisation ‘across the plots and comparisons within a community to facilitate farmer comparisons’ is another of example of how the organisation of on-farm trials may–unintentionally–reduce the recommendation domain of a technology (CIMMYT, 1988) and, hence, its scalability.

The use of one standardised fertilisation rate in on-farm trials is, of course, sound experimental design that can facilitate the isolation of treatment effects and the above-mentioned ‘farmer comparisons’. Yet, as this fertiliser rate is often higher than what most African smallholder farmers can afford or are willing to apply outside on-farm experimental fields–as Cheesman et al. (Reference Cheesman, Andersson and Frossard2017) show for farmers participating in one of the AR4D projects discussed in our paper–this experimental design may also result in misleading interpretations of the technology's performance outside the experimental conditions. It may be, as Vanlauwe et al. (Reference Vanlauwe, Wendt, Giller, Corbeels, Gerard and Nolte2014) have argued for the case of CA, that fertiliser use is required for the technologies to have positive effects on African smallholders’ crop productivity. Hence, it is not merely the social environment in which the on-farm trials are conducted; the application of the experimental design itself may introduce selection bias that inadvertently affects the technology's scalability.

These kind of implementation modalities can influence the performance evaluation of a technology despite it being tested ‘on-farm’, as the farmers participating in the on-farm trials may not represent the concerns, interests or abilities of the broader farmer population (as exemplified in our paper's Bangladesh case study), thereby reducing the wider applicability of trial results. Consequently, initially promising results from on-farm trials may be misleading when technologies are moved from research into extension and out-scaling. Such methodological problems have ramifications, both for up-scaling (institutionalisation) and the out-scaling of technologies among smallholder farmers.

ON-FARM ‘DEMO-TRIALS’ AS ‘INFECTION POINTS’, BUT NOT FOR SCALING?

Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019) suggest that the southern Africa AR4D projects did not aim to scale-out CA technologies and practices, but rather, that they were intended to develop a scalable model, which would aid the ‘facilitation of the widespread adoption of CA ’. While it is indeed true that not all projects discussed in our paper had the same emphasis on the out-scaling of technologies, it is the inherent tension in the use of on-farm trials ‘as a means to understand and foster technology adoption’ (de Roo et al. Reference de Roo, Andersson and Krupnik2019) that we sought to analyse. This tension, which affects both up- and out-scaling, is clearly discernible in these AR4D projects.

A first indication of this tension can be found in the framing and naming of projects. The reader may be forgiven when thinking that projects with titles such as: ‘Facilitating the widespread adoption of Conservation Agriculture. . .’ and ‘Facilitating the adoption of Conservation Agriculture by resource-poor smallholder farmers. . .’ have nothing to do with technology out-scaling. Yet, as a strategy document on these projects suggests, collaborating public and NGO partners could apparently view on-farm trials as demonstration sites for technology scaling as they ‘have often spaced the plots out more than we would have liked, given normal extension practices of widespread blanket recommendations’ (Neely et al. Reference Neely, Dixon and Wall2008). Project activities such as farmer field days and study visits comprise another indication of the blurred boundaries between research and out-scaling in these projects’ use of on-farm trials. For example, Neely et al. (Reference Neely, Dixon and Wall2008) suggest that farmer study visits serve both learning among farmers, and the ‘[s]caling out of the technologies to other sites and communities’. As Wall et al. (Reference Wall, Thierfelder, Nyagumbo, Rusinamhodzi and Mupangwa2019)’s comment, communications on the AR4D projects we referred to in the southern Africa case study also evidence a multi-purpose perspective on the use of on-farm trials. One project even has very explicit out-scaling targets:

It is this kind of double-edged framing, and the differential way it is being interpreted and communicated by project partners, farmers and implementing agronomists, that reveals the intricacies of combining research and scaling activities in an impact-oriented development context. Our paper aimed to highlight the tensions involved when scientists work on the continuum between research and development. It showed the difficulties involved in maintaining experimental control and collecting quality data from farmers managing trials in the context of projects that aim to have both scientific and development impact. Acknowledging that such tensions exist is, in our view, a first and necessary step towards a reconceptualisation of the role on-farm experiments in scaling.