The publication of Nudge (Thaler & Sunstein, Reference Thaler and Sunstein2008) ushered in an era in which governments and businesses started increasingly depending on the results of evidence to make better product, program and process design decisions (OECD, 2017). Over a decade after the publication of Nudge, Soman and Yeung (Reference Soman and Yeungin press) argue that while the relevance, acceptability and methods of behavioral science in organizations are now not questioned, two challenges for the next decade of applied behavioral scientists remain. The first challenge relates to creating organizations that truly understand the nature of behavioral science and embed it deeply in their structures and methods. The second challenge is one of scaling – how do we ensure that the results of small-scale experiments and pilots successfully scale into interventions that reliably improve consumer welfare?
In their lead article, Al-Ubaydli and colleagues acknowledge the increasing role of the experimental method to inform large-scale public policy programs and welfare initiatives, and they provide a list of 12 proposals that will advance the ability of policymakers to scale the results of pilots. We applaud their thoughtful analysis and agree that without a better science of using science, the promise of behavioral science in policymaking and end-user welfare will not be fully realized. We also agree that the ‘voltage drop’ – the drop in effectiveness of a scaled-up intervention relative to the original pilot (Kilbourne et al., Reference Kilbourne, Neumann, Pincus, Bauer and Stall2007) – presents a significant scaling challenge. Voltage drops occur because: (1) researchers often make imprecise and exaggerated inferences based on the original pilot; and (2) the original pilot might not be representative of either the larger population or the situation in which the intervention will be scaled up.
While our opinions are largely similar, we diverge and extend their thinking in two ways. First, unlike the model they draw heavily on (Al-Ubaydli et al., Reference Al-Ubaydli, List and Suskind2019), we do not necessarily believe that government entities and researchers are completely independent actors, but instead make a call for a different kind of evidence (in situ testing) created by collaborations between the two entities. Second, we believe that the lack of representativeness of the situation can not only result in a voltage drop, but might also result in backfiring effects and produce the opposite result of what they were intended to do.
We identify three specific scaling challenges that are worthy of further investigation. First, one of the biggest themes from the research on behavioral science and judgment and decision-making is the notion of context dependence (Tversky & Simonson, Reference Tversky and Simonson1993). We now know that a large number of elements in the context (e.g., the presence of irrelevant options, medium of the message, time of the day, information framing and ambient factors, to name a few) influence choices (for overviews, see Soman, Reference Soman2015; Thaler, Reference Thaler2015). A rich literature of preference reversals has shown, for example, that consumers’ preferences might reverse with seemingly irrelevant changes in the context (Lichtenstein & Slovic, Reference Lichtenstein and Slovic2006). For policymakers, the implication should be clear – the fact that a particular intervention worked well in one particular context should not guarantee its success in a different context.
Second, like Al-Ubaydli et al., we agree that the lack of representativeness of pilot study participants poses a challenge. One area of analysis that is relatively uncommon in this literature is heterogeneous treatment effects. While it is important to understand the efficacy of a treatment among a representative set of participants, we posit that it is equally important to understand the heterogeneity in treatment effect. As small-scale research studies may focus on the set of people for whom the treatment effects are likely to be the largest, a goal for large-scale replications can be to figure out the effect sizes for different groups of the population. If done successfully, this may lead to successful use of treatments that may not scale well to the whole population, but can be very effective on a subgroup of that population. By identifying the boundaries of the effectiveness of an intervention and understanding the heterogeneous effects, policymakers and researchers can target interventions to the appropriate population. Heterogeneous treatment effects imply that successful policies do not have to be one-size-fits-all.
Third, there is often a temptation on the part of policymakers to adopt a kitchen sink approach in which multiple interventions that have been successfully tested independently are deployed simultaneously in the naive belief of improving efficacy. Unfortunately, these multiple interventions can interact in complex and unpredictable ways, which might result in backfiring.
We next provide three examples to illustrate some of these scaling challenges. We then speculate on how these scaling challenges might play a role if we were to scale up the results from a field experiment that one of us was involved in (Hossain & List, Reference Hossain and List2012). We will then conclude with a discussion and offer additional propositions.
Example 1: credit card reminders in South Korea
Research shows that consumers tend to spend more when using a credit card rather than paying with cash or by check (Feinberg, Reference Feinberg1986; Prelec & Simester, Reference Prelec and Simester2001; Soman, Reference Soman2001). One reason for the credit card premium is the fact that continued use of the credit card weakens memory of past expenses. Consequently, giving people feedback (reminders) on how much they have spent on prior expenses will improve mental tracking and mitigate spending differences (Soman, Reference Soman2001).
In response to growing concerns about credit card debt, the government in South Korea (in 2010) mandated that credit card providers introduce a text messaging service that would remind consumers about transactions that they had recently incurred. In addition to prevent fraudulent use of credit cards, the policy was designed to “improve rational behavior in credit card spending” (Lee & Yoon, Reference Lee and Yoon2011, translated from Korean). In general, the expectation was that an introduction of the credit card spending text alert system would help people control their credit card spending and hence would reduce credit card balances.
An evaluation showed that the policy had the intended outcome for only about 12% of the population – the heavy spenders. For the remaining 88%, the policy actually resulted in an increase in spending (Kim et al., Reference Kim, Yoon, Choi and Soman2020). This backfiring effect might be due to an important difference in the manner in which the reminder was delivered in the pilot studies (where it was available on the same screen as that where the spending decisions were made) versus in the scaled-up intervention (where it appeared on a mobile device). This created a degree of ‘digital dependency’(Sparrow et al., Reference Sparrow, Liu and Wegner2011) by which consumers believed that they could easily access their past spending if they needed to, thereby reducing the motivation to mentally track it.
Example 2: backfiring effect of a retirement pension contribution intervention in Mexico
Mexico is facing an elderly poverty crisis (Fertig et al., Reference Fertig, Fishbane and Lefkowiz2018). Salaried employees in Mexico make a 6.5% mandatory contribution to their pensions, while projections show that they need to make an additional 5% through voluntary contributions in order to retire comfortably. Unfortunately, the voluntary contribution rate in Mexico is abysmally low.
In working with the pension authority in Mexico, Shah et al. (Reference Shah, Osborne, Lefkowitz, Fishbane and Soman2020) redesigned the quarterly pension statement that every salaried employee in Mexico receives. Prior research shows that simplifying communication and making it engaging will increase the likelihood that recipients of that communication will consume the information and act on it (Bhargava & Manoli, Reference Bhargava and Manoli2015). Accordingly, the redesigned quarterly statement was significantly more engaging than the original, and it differed in two ways. First, it provided a simplified visual illustration (in the form of a categorical thermometer) showing whether the recipient's current stock of pension savings was inadequate for retirement. Second, the statement included one of multiple behavioral interventions (gain versus loss framing, a wallet cutout to increase implementation intentions, an appeal that made the family's welfare salient and a fresh-start intervention encouraging recipients to start saving after a particular temporal landmark) that had been shown to be successful elsewhere. In a large-scale field trial with members of two pension funds, the intervention was a success with one of the funds (it increased the contribution incidence, the contribution amounts and the contribution frequency), but backfired in the other fund. Why did this happen?
Backfiring happened because of a specific design feature unique to the Mexican pension system. In particular, Mexicans needed to first choose a pension fund and then make contribution decisions. The quarterly pension statements displayed the performance of each fund in a tabular form. By making these statements more engaging, the researchers increased the attention that was being paid to the table. If the fund was high performing, then the engaging statement ended up improving voluntary contributions because it increased the motivation to save. If, on the other hand, the recipient had chosen a low-performing fund, then the improved engagement of the statement actually resulted in demotivation because it highlighted attention to the low performance of the fund.
Example 3: heterogeneity in responses to text message reminders
In ongoing work with pension contributions in Mexico, Shah et al. (Reference Shah, Osborne, Lefkowitz, Fishbane and Soman2019) compared a control condition with a condition in which recipients received the redesigned quarterly statement. In a third condition, they additionally received a text message with a variety of call-to-action messages. Text reminders have been used successfully to convert intention to action (Karlan et al., Reference Karlan, McConnell, Mullainathan and Zinman2016).
The results of the field trial showed that compared to the control, a condition in which recipients received a text message emphasizing their family's financial security in addition to the redesigned statement increased contribution rates significantly. However, do both men and women care equally about families? What about people who do not have families? Does age matter? Did it matter whether the recipient of the text message was an existing contributor or was a new contributor? Using the machine learning technique of causal tree and causal forest (Athey & Imbens, Reference Athey and Imbens2016), the researchers identified heterogeneous treatment effects across the scaled-up population. Their results were unsurprising, and they yet worked to show exactly for which sub-segments in the population the intervention was particularly successful and where it was not. For instance, they found that the family security text messaging increased contribution rates for people who were aged between 28 and 42. It did not have a significant effect for people above the age of 43, and it had a backfiring effect for people under the age of 27.
Knowing exactly where the family security text message (as well as each of the other messages that were tested) was most effective could now allow them to target the sub-segments of the population properly and customize the content of the text messages to different sub-segments in the population accordingly. Successful scaling, in this case, would embrace heterogeneity by using a family of interventions, each targeted to segments where it might be most effective.
Potential scaling challenges
We next illustrate how some of these scaling challenges might play a role if the intervention from a landmark paper (Hossain & List, Reference Hossain and List2012) were to be scaled. This paper used one of the most robust findings from behavioral science – the framing of monetary outcomes as gains or losses. In a field study in a high-tech manufacturing facility, the authors manipulated the manner in which a productivity bonus was presented to workers. In particular, the same bonus was presented to a random subset of workers as a potential gain and to another subset as a potential loss. The authors found that both the gain and loss framing of incentives increased productivity compared to a control condition in which there were no incentives. Incentives clearly worked! More importantly, framing the incentive as a potential loss led to a small but significant increase in productivity over presenting it as a potential gain.Footnote 1
While they found that loss aversion is manifested in a small treatment effect, it is quite important that this small increase in productivity happened in a real-work setting. If this small gain in productivity can be scaled up to large populations of workers, this would indeed have a significant impact on welfare. The question remains as to whether scaling up happens at exactly the same rate or whether there will be a voltage drop. Is it possible that the intervention might backfire?
It is noteworthy that Hossain and List did not find any significant framing effect for individual workers. The sign of the impact fluctuated between positive and negative across econometric specifications. If this intervention is scaled up with individual workers, it may in fact backfire. Moreover, in their study, the intervention was implemented for only about a month. While the impact was evident throughout the experiment, a negative framing of incentives might be less likely to be successful if the intervention is permanent or much longer in duration.
The effect of loss aversion on productivity depends on the interaction of the framing and the underlying incentive scheme (Hong et al., Reference Hong, Hossain and List2015). Hence, one needs to be careful in choosing the economic part of the intervention while scaling up to ensure that it does not backfire. Moreover, the effectiveness of loss framing may also depend on the size of the economic incentives. In Hossain and List, the size of the bonus was above 20% of any of the workers’ base salaries. The treatment effect for smaller-sized incentives may be insignificant or opposite in direction, as Hoffman and Thommes (Reference Hoffman and Thommes2020) found for individual workers.
Why do we have scaling challenges?
Our views of why scaling challenges exist are similar to those of Al-Ubaydli et al., and they are grounded in organizational realities. In particular, we would like to highlight three organizational reasons for this, as well as one reason to do with the nature of evidence.
We believe that there are three organizational reasons that drive some of the scaling challenges. The first reason is solution-mindedness. Unlike scientists, governments are typically under enormous time pressures to solve problems. Second, as the applications of our field have spread, organizations increasingly rely on non-specialists to design and deploy behavioral interventions. In an effort to help non-specialists, a number of heuristic frameworks (e.g., Easy, Attractive, Social, Timely (EAST); Service et al., Reference Service, Hallsworth, Halpern, Algate, Gallagher, Nguyen, Ruda and Sanders2014) have been developed. Frameworks are elegant in that they allow a non-specialist to try to design interventions based on the learnings of others. However, as Soman (Reference Soman2017) argues, frameworks can often be counterproductive because they change the process of intervention design from one that begins with a careful audit of the context to a heuristic-type checklist-based approach. Third, most organizations that we have worked with operate in silos of capability. In the context of some of the work described here, the behavioral scientists typically are located in a different department from the data scientists and the design teams. Some of the scaling challenges will be best addressed if we have large and diverse teams working on these projects. However, organizational realities often make it difficult to achieve this result. Collectively, and perhaps most importantly, the cost of experimentation (including the costs of collecting and analyzing data, as well as organizational costs such as fear of failure, time commitment, organizational buy-in) in most organizations is high; therefore, they are more likely to embrace off-the-shelf results (Feng et al., Reference Feng, Kim and Somanin press).
We also note that the nature of evidence in the behavioral sciences is significantly different from evidence in other sciences. Consider fields such as physics or medicine. As an example, objects released from a height will fall. They will do so irrespective of who releases them, the height of release, their color and whether there are other objects being released at the same time or not. This is because the theory of gravity has a large bracket of context surrounding it, and how the theory works in foreseeable but uncommon contexts is also well understood. For example, releasing the same object on the surface of the moon (or in outer space) will not result in the same sort of drop (there will be a ‘voltage drop’). Nevertheless, the theory of gravity clearly predicts how the ball will drop, as we properly understand the difference between the two contexts. Unfortunately, unlike these sciences, the field of behavioral science has very narrow brackets of context. Therefore, it is important to emphasize that the documented robust demonstration of an effect in a particular context with a particular type of end user at a particular point in time does not guarantee that it will successfully replicate across those brackets of context.
Therefore, unlike the natural or medical sciences, where a rigorous meta-analysis of available evidence is typically enough to help predict what would happen in a scaled-up scenario, behavioral interventions often call for a sequential approach to evidence. A meta-analysis is helpful to serve as a starting point for developing interventions. These interventions would then probably need to be tested in low-cost and quick-win environments such as laboratory studies or online panels. The successful interventions in these domains might then be scaled into pilot studies – and then into larger pilots – before being then enshrined in policy. However, it is of utmost importance that these pilots are done in the context in which the intervention is going to be scaled up (in situ evidence). In situ testing in South Korea should have flagged the potential backfiring effect of reminders at a stage where it could have been corrected. Finally, scaling up must be accompanied by continuous testing, learning and adapting (Haynes et al., Reference Haynes, Service, Goldacre and Torgerson2013) in order to detect and respond to potential backfiring due to context changes, due to interactions between different interventions and due to heterogeneity. For this to happen, we need to push down the cost of experimentation in organizations.
We close with a few additional proposals:
Proposal A: Rather than viewing researchers purely as producers and governments purely as consumers of research, we should strive to co-create evidence.
Proposal B: Researchers should be incentivized to articulate all of the dimensions of the context (or features of the situation) under which their documented effects hold. They should further be incentivized to test for the effect under different contexts.
Proposal C: When in situ testing is not possible, we should document all of the dimensions of the context of the scaled-up situation that might be different from the testing phase and invite the original researcher to speculate on whether these differences will change the results.
Proposal D: We should test for heterogeneous treatment effects of interventions and use a family of appropriately targeted interventions to scale solutions.
Proposal E: We should encourage researchers and governments to create project teams of behavioral scientists, designers and data scientists to collectively design, test, scale and monitor interventions.
Proposal F: We should strive to reduce the cost of experimentation within organizations.
Disclaimer
Both authors are fellows at the Behavioural Economics in Action at Rotman (BEAR) research center and acknowledge support from the center.
