1. Introduction
Scientists make their mark by disagreeing with one another, so when they agree about something, it’s worth taking note. Across the many disciplines that study human morality—psychology, anthropology, economics, biology—there is widespread consensus about the function of morality. Humans have diverse interests and goals, and each person’s success at achieving their goals depends on decisions made by others. We must, then, figure out individually and collectively how to adjudicate our competing interests. Morality helps us solve this problem (Baumard et al., Reference Baumard, André and Sperber2013; Bicchieri, Reference Bicchieri2005; Binmore, Reference Binmore2005; Curry, Reference Curry2016; Gauthier, Reference Gauthier1987; J. Greene, Reference Greene2014; Haidt, Reference Haidt2007; Rawls, Reference Rawls1971; Tomasello, Reference Tomasello2009).
Considerable effort has been devoted to defining, from a rational perspective, how people should negotiate such “interdependent choices”. A key, recurrent idea is that people should settle into the kinds of arrangements for mutual benefit that they would have agreed to in an idealized bargaining process (e.g. Binmore, Reference Binmore2005; Gauthier, Reference Gauthier1987; Nash, Reference Nash1950). And, indeed, the agreements we make with one another are central to our moral lives. We make promises, draw up contracts, and “talk it out” when our interests conflict. These agreements create obligations, and breaking them is morally problematic. Thus, one might naturally assume that our moral psychology would be structured around finding and adhering to mutually beneficial agreements.
Yet, while there is a large literature on the psychological mechanisms that people use to make moral judgments, relatively little of it is framed in terms of finding mutually beneficial agreements (but for important exceptions see: Baumard et al., Reference Baumard, André and Sperber2013; Everett et al., Reference Everett, Pizarro and Crockett2016; Kohlberg, Reference Kohlberg1969; Levine, Kleiman-Weiner, et al., Reference Levine, Kleiman-Weiner, Schulz, Tenenbaum and Cushman2020; Piaget, Reference Piaget1932; Tomasello, Reference Tomasello2020). The field has focused instead on prohibited actions (i.e. “deontological” processes; whether established by emotions, heuristics, or rules (e.g. Baron & Ritov, Reference Baron and Ritov2004; Cushman et al., Reference Cushman, Young and Hauser2006; Haidt et al., Reference Haidt, Koller and Dias1993; Nichols, Reference Nichols2021; Nichols & Mallon, Reference Nichols and Mallon2006)), utility-based evaluations of outcomes and welfare (i.e. “consequentialist” processes)(e.g. FeldmanHall et al., Reference FeldmanHall, Dalgleish, Evans, Navrady, Tedeschi and Mobbs2016; Hsu et al., Reference Hsu, Anen and Quartz2008; Lockwood et al., Reference Lockwood, Klein-Flügge, Abdurahman and Crockett2020; Williams, Reference Williams1968), and the psychological conflicts that arise when prohibitions and welfare concerns clash (e.g. Crockett, Reference Crockett2013; Cushman, Reference Cushman2013; J. Greene, Reference Greene2014).
We aim to fill this gap, offering a theory of psychological contractualism. Starting with the premise that the ultimate function of morality is to guide interdependent choice, we argue that the proximate mechanisms of our moral psychology can be usefully organized around the concept of agreement—the creation and application of “contracts”, formal or informal, between people.
What would contractualist psychological mechanisms look like? The ideal way of striking mutually beneficial agreements might be to get everyone in a room together to talk through the best way to proceed—the kind of setting sometimes envisioned by game theory or moral philosophy. But explicit negotiation is usually impractical. When we lack the time, information, computational resources, and social capital for ad hoc negotiation, our view posits that we fall back on cognitively efficient shortcuts that approximate this ideal. These heuristics are the heart of our account. We propose a resource-rational contractualist psychology—one designed to trade off the cost of cognitive and social effort against the gains of mutually beneficial agreement.
Specifically, our view describes two axes of heuristic approximation of the contractualist ideal. The first involves the terms of the agreement. Rather than negotiating over what should be done in a single instance, we instead consider abstractions, or standards, that can be reused in future circumstances. The second axis of approximation involves the process by which agreement is reached. Rather than actually getting the affected parties together to bargain, we could anticipate how the negotiation would proceed using mental simulation.
This resource-rational approach has two important payoffs. First, we show how two well-understood parts of our moral psychology are, in fact, particularly useful ways of approximating ad hoc negotiation. One implements a form of utility maximization over weighted outcomes. Another generalizes broad constraints on allowable actions. Thus, two of the best studied elements of our moral psychology—those often called “utilitarian” and “deontological” in the moral psychology literature (Conway & Gawronski, Reference Conway and Gawronski2013; Cushman et al., Reference Cushman, Young, Greene and Doris2010; Everett & Kahane, Reference Everett and Kahane2020; J. D. Greene et al., Reference Greene, Sommerville, Nystrom, Darley and Cohen2001; Holyoak & Powell, Reference Holyoak and Powell2016)—can be explained, in part, as resource-rational approximations to mutually beneficial agreements.
The second payoff of this view is that it explains the flexibility of our moral minds. Previous theories of moral psychology have largely been silent on how moral standards get established, updated, overridden, and revised. A resourcerational contractualist account suggests that these phenomena can be explained as renegotiations prompted by changes in the world, or by the discovery of new and better ways of achieving mutual benefit.
We call this a psychological “triple theory” of moral cognition, with a nod to Derek Parfit’s normative theory of the same name (Parfit, Reference Parfit2011). Like Parfit, we propose that rule-based, utility-based, and agreement-based approaches to moral decision-making are much more intimately related than they appear at first. Our theory departs from Parfit’s, however, in many respects. Chief among them is that we offer resource-rational contractualism as a powerful framework that helps explain both the connections and clashes between the three philosophical approaches, but not as an all-encompassing theory reconciling every facet of our moral minds. In this paper we define the predictions and potential of the resource-rational contractualist approach—and argue that many rules and welfare-based processes of moral judgment can be encompassed within it—but we also define its limitations and borders.
In sum, psychological contractualism offers a computational framework that is grounded in a rational analysis of the function of morality, that relates diverse mechanistic elements of our moral mind, and that offers clues about how they might interact.
2. The Problem of Interdependent Choice
In order to understand the structure of any cognitive system it helps to identify its function—the problem that it is designed to solve (Marr, Reference Marr1982; Tinbergen, Reference Tinbergen1963). Across a broad range of fields—biology, anthropology, economics, political theory, and psychology—there is broad agreement that a principle function of morality is to help guide the choices we make that affect each other, given our diverse interests (André et al., Reference André, Debove, Fitouchi and Baumard2022; Baumard et al., Reference Baumard, André and Sperber2013; Binmore, Reference Binmore2005; Curry, Reference Curry2016; Gauthier, Reference Gauthier1987; J. Greene, Reference Greene2014; Ostrom, Reference Ostrom1990; Rawls, Reference Rawls1971). After all, as humans, we are interested in pursuing our own goals based on our subjective interests. Footnote 1 Meanwhile, everyone else around us is trying to do the same. People’s interests often conflict—what one person prefers may be dispreferred by others. Yet, often, each person could benefit from the help of others—there are things we can achieve together that no one can achieve alone.
An extensive formal literature in game theory attempts to characterize this problem of “interdependent choice” from a rational perspective, asking how idealized agents would act when the outcomes of their actions are partially determined by others’ actions (Bacharach, Reference Bacharach2018; Binmore, Reference Binmore1994; Gintis, Reference Gintis2014). This work establishes a normative standard for interdependent rational choice, and it also provides a template for organizing descriptive theories of human thought and behavior. Although humans are clearly not fully rational creatures, idealized rational models of choice can be a useful way to organize a scientific inquiry into the actual psychological mechanisms used by ordinary people (Chater & Oaksford, Reference Chater and Oaksford1999; Marr, Reference Marr1982).
Two ideas arising from this literature are central to our argument: That interdependent rational agents will choose arrangements of “mutual benefit”, and that they can discover these by processes of bargaining and agreement. After first describing the role of these ideas in idealized models of interdependent choice (addressing the function of moral judgment), we then use them to motivate our key claim (addressing its structure): Much of our actual moral psychology arises from resource-rational cognitive mechanisms for reaching and adjusting agreements—including many situations where the agreements or agreement processes may be not be visible or only implicit.
2.1. Mutual benefit: What would rational agents agree to?
People engage in a wide variety of relationships (friendships, couples, corporations, etc.), make many types of deals, promises, and agreements, and insist on many and varied moral standards regarding how they are treated and how they must treat others. We argue that these can all be viewed as ways to achieve mutual benefit: Arrangements that tend to make each party better off than they would be otherwise. But, can we be more specific about what “mutual benefit” amounts to, and which particular arrangements we should expect?
2.1.1. The bargaining problem
Our strategy is to begin with a simple, well-studied, and analytically tractable case, and then to consider how its core lessons can be applied to the more complex settings which are our primary concern.
The bargaining problem is an economic game that formalizes the “mutually beneficial” arrangement that would be reached when two agents can create something of value together, and then must decide how to divide their profit (Nash, Reference Nash1950). For instance, maybe two friends need to share a cab and divide the bill; two spouses need to divide childcare and housework; an employer and an employee need to divide company profits; or, a seller and a buyer need to agree on the price of a car. In each case, there are many different specific arrangements that would leave both parties better off than they would be if they failed to come to an agreement. But which one should they choose?
In cases like these, a widely used criterion is that rational agents will agree on a division that maximizes the product of the utility gains between the players, known as the Nash bargaining solution (Nash, Reference Nash1950)Footnote 2 . When two players are identically situated, and the goods are equally valuable to them both, the Nash bargaining solution corresponds to an equal division of resources. But the players might be asymmetrically situated such that a deadlock between them is much more harmful to one of them than the other—for instance, one might go hungry while the other scarcely notices. A player with poor “outside options” if the bargain fails is said to have weak bargaining power, and the Nash Bargaining Solution predicts that they will obtain less. This is a familiar concept in labor disputes: management will obtain a better offer when they can easily hire new workers; workers will obtain a better offer if they can easily find other jobs. Meanwhile, the players might also benefit asymmetrically from the goods (e.g., a smaller person can be fed with fewer calories, while a larger person requires more to achieve the same benefit), and here the Nash bargaining solution allocates more resources to the person who requires more to achieve the same gain.
This simple, well-studied example captures the dynamics at the core of many moral situations. It also furnishes an simple example of what we mean by “mutual benefit”: The specific agreement that rational agents would come to, leaving both better off. Indeed, the logic of bargaining theory has already been used to explain the structure of human morality in simple two-person situations. Experimental work shows that when people work together to divide resources they come up with arrangements well predicted by the Nash bargaining solution, and they consider these arrangements to be fair (Binmore et al., Reference Binmore, Swierzbinski, Hsu and Proulx1993; Le Pargneux & Cushman, Reference Le Pargneux and Cushman2023; Mallucci et al., Reference Mallucci, Wu and Cui2019; Rustichini & Villeval, Reference Rustichini and Villeval2014).
Obviously, most of the interdependent choices that we face deviate from the stylized assumptions of this simple case and therefore go beyond the simplified assumptions within which the Nash Bargaining Solution applies. Real bargaining is a dynamic, time-consuming process, and this can affect the bargaining solution (Binmore et al., Reference Binmore, Rubinstein and Wolinsky1986; Rubinstein, Reference Rubinstein1982). Real bargaining often involves more than two players (Harsanyi, Reference Harsanyi1963). And, we often encounter interdependent choice problems in which literal bargaining and enforceable agreements cannot be taken for granted (Misyak & Chater, Reference Misyak and Chater2014). As these various simplifying assumptions are relaxed, the specific agreements that people reach are sometimes predicted to deviate from the precise Nash bargaining solution. Nevertheless, what is perhaps more remarkable is how often its animating principles are preserved—that is, that we can understand each party’s behavior as an attempt to achieve a specific division of resources that both parties would agree to, leaving both parties better off than they would be otherwise, and taking into account the relative advantages and disadvantages of their negotiating positions.
To illustrate this in a bit more detail, consider one particular extension of the Nash bargaining solution of special relevance to our proposal. In the classic Nash bargaining problem the parties are dividing goods, but this logic has been generalized to the case of bargaining over the norms or rules that govern conduct (André et al., Reference André, Debove, Fitouchi and Baumard2022; Baumard et al., Reference Baumard, André and Sperber2013; Binmore, Reference Binmore2005). Rules and norms are powerful tools for shaping behavior—indeed, when enforced, they can stabilize an astonishing variety of different resolutions to conflicts of interest, including both cooperative and non-cooperative ones (Bicchieri, Reference Bicchieri2005; Boyd & Richerson, Reference Boyd and Richerson1992; Ostrom, Reference Ostrom1990). We know from both field and laboratory studies that people often establish rules and norms through a process of discussion, negotiation, and agreement (Ostrom et al., Reference Ostrom, Walker and Gardner1992). A classic example are the informal norms and agreements that fishermen create to govern the valuable, limited resource of their fisheries (Acheson, Reference Acheson1988; Ostrom, Reference Ostrom1990). In fisheries—and across human societies more generally—we tend to observe rules and norms that stabilize cooperative behavior leading to mutually beneficial outcomes (Bowles & Gintis, Reference Bowles and Gintis2011; Curry, Reference Curry2016; Haidt, Reference Haidt2007; Tomasello, Reference Tomasello2009), although not always (Herrmann et al., Reference Herrmann, Thoni and Gachter2008). This is not surprising because, when bargaining over the contents of rules and norms, the key concepts of the bargaining game apply: Rational agents are expected mutually agree upon norms that generate resources distributions of mutual benefit—in some cases, ones that conform quite precisely to the Nash bargaining solution (Binmore, Reference Binmore2014).
2.1.2. Mutual benefit versus aggregate benefit
It is important to point out that mutually beneficial outcomes predicted by the theory of bargaining diverge from the “welfare maximizing” outcome favored by utilitarian moral theories, such as maximum aggregate benefit.
Indeed, rational agents would often not agree to divisions that maximize aggregate benefit. For instance, suppose that 10 tokens are to be divided between two people, and each token is worth $1 to the first person and $2 to the second person. Aggregate benefit is maximized by giving all tokens to the second player (this results in aggregate benefit of $20, the maximum sum possible)Footnote 3 . But we would not expect the first player to agree to this, since it provides them with no benefit at all. Bargaining theory instead predicts a mutually-beneficial division—the Nash Bargaining Solution, for instance, dictates that the players receive 5 tokens each (resulting in $5 for the first player and $10 for the second, which yields the maximum product possible). Although aggregate welfare is lower (the sum: $15), this is likely a better match to ordinary peoples’ moral intuitions about a fair split.
To choose another example, suppose a rich baker has money they don’t need and bread they wouldn’t enjoy much, while a poor customer has money they need dearly and would benefit a lot from the bread. The agreement they would likely strike—i.e., the predicted bargaining solution—is not for the baker to give away the bread for free (and perhaps some money, too, for good measure!), even though this would maximize aggregate benefit (the raw sum of their utilities). Rather, the agreement would likely involve the customer paying at least something for the bread—a solution that achieves mutual benefit.Footnote 4
2.2. Finding mutual benefit
How do people arrive at mutually beneficial arrangements? While bargaining theory can describe, explain, and predict solutions to the problem of interdependent choice, it is largely silent on the actual processes that bring those solutions about.
Existing work tends to focus on the evolutionary dynamics that lead to mutual benefit. Evolutionary models show that adaptive dynamics favor (although do not always guarantee) cooperative Nash bargaining solutions in many cases of conflicting interest (J. M. Alexander, Reference Alexander2000; André et al., Reference André, Debove, Fitouchi and Baumard2022; Bruner, Reference Bruner2021; Skyrms, Reference Skyrms2014), and competitive equilibria predicted by game theory in others (Nowak, Reference Nowak2006; Smith & Price, Reference Smith and Price1973). In these models, this arises as the outcome of a typical “blind” evolutionary process (whether biological or cultural), not due to any reasoning or explicit bargaining on the part of the agents themselves. Similarly, there is considerable theorizing about how selection among candidate human norms might occur by “blind” biological or (more often) cultural evolutionary processes (Binmore, Reference Binmore2014; Boyd & Richerson, Reference Boyd and Richerson1990; Henrich, Reference Henrich2004). These sorts of evolutionary accounts are aligned with our emphasis on the logic of contractualism, but are beyond the scope of the psychological account we provide.
Our focus is different. Blind evolutionary processes can be powerful, but they take a long time and their results—the policies, structures or mechanisms they construct—are rigid. But the human moral world changes all the time—there is new information, new opportunities, and new interdependent choice structures to navigate and we have to figure out how to act in light of these constantly changing circumstances. Cognitive processes—in contrast to evolutionarily-endowed intuitions—can adapt quickly and flexibly to a changing world. The human mind is designed to solve an open-ended space of new problems and tasks. We argue that the cognitive processes of human morality respond to and exhibit this flexibility, which derives from agreement-based processes; people approximate bargaining solutions by explicitly or implicitly negotiating with each other, both over specific resource distributions but also over rules and norms. It is the cognitive processes of negotiation that allow us to design agreements that “make sense” to both parties (rather than being the product of blind variation and selection).
Our main purpose is this article, then, is twofold. First, we argue that agreement-based methods are in fact a cornerstone of human moral cognition, and we chart a framework for identifying and characterizing them at a mechanistic level. Second, we argue that many of the ways we rely on agreement-based methods involve cognitively efficient heuristics. These “resource-rational” versions of contractualism explain large and important parts of our moral psychology that have not previously been understood as grounded in the logic of agreement and mutual benefit.
2.3. Limitations and bounds of the bargaining framework
We argue that a wide range of moral judgments and behaviors can be organized around resource-rational contractualist principles. But not all of them can, and it is important to understand the limits of the theory.
First, the moral judgments we attempt to explain all fall within the domain of interdependent choice—cases that involve the interacting utilities of multiple agents. However, there are some moral judgments that fall outside this scope. For instance, sometimes people condemn “victimless” crimes (such as consensual incest (Haidt et al., Reference Haidt, Koller and Dias1993) or the eating of certain foods (Levine, Rottman, et al., Reference Levine, Rottman, Davis, O’Neill, Stich and Machery2020)) as morally wrong, which our account does not explain. Moreover, a virtue-based perspective on moral judgment posits that it is morally good to pursue certain ”self-regarding” virtues like fortitude, prudence, and courage (Taylor & Wolfram, Reference Taylor and Wolfram1968), which need not have anything to do with other people. Our account likewise does not explain these virtues, unless they are framed as having down-stream impacts on other agents (Taylor & Wolfram, Reference Taylor and Wolfram1968, as indeed, they sometimes are) (though see §6.1).
Even some cases of interdependent choice are outside the scope of our proposal because they bypass the cognitive processes of negotiation, such as the sense of moral duty to provide for one’s children. This intuition, and others like it, likely originates in “blind” adaptive processes rather than agreement-based reasoning. As discussed in § 2.2, evolutionary processes can lead to behaviors or instincts with an underlying contractualist logic, but are nevertheless outside the scope of our view because they bypass the cognitive processes underlying individual learning and reasoning.
Conversely, bargaining theory makes some predictions that many people find to be unjust or immoral. Consider a case where a person agrees to their own exploitation, such as a worker who accepts a very low wage and very poor working conditions because they truly have no better option. The most straightforward application of an agreement-based view would hold that there is nothing morally wrong with this arrangement—after all, everyone is better off than they would be otherwise. Indeed, an agreement favoring the advantaged party is specifically predicted by the Nash bargaining solution. Yet, many people hold that there is something morally wrong with this arrangement. The origin of our preference for equal, universal human rights, regardless of a person’s bargaining position and power, is something that contractualist theories often regard as outside the scope of what emerges from agreement (Gauthier, Reference Gauthier1987).Footnote 5
Meanwhile, many people probably also hold a roughly “libertarian” view of the exploitation case—namely, that any social arrangement is morally acceptable as long as it is mutually agreed upon (or feel the force of the viewpoint even if balanced against competing egalitarian viewpoints). This part of our moral psychology is well explained by our theory. Finally, while there is also an extensive literature purporting to show the pervasiveness of egalitarian preferences around resource distributions (Starmans et al., Reference Starmans, Sheskin and Bloom2017, for a review, see), Starmans and colleagues note that these typically match recipients on a variety of important attributes such as effort, ability, and desert (Starmans et al., Reference Starmans, Sheskin and Bloom2017). Indeed, when a bargaining situation is perfectly symmetric, the Nash bargaining solution is egalitarian (as pointed out by André et al., Reference André, Debove, Fitouchi and Baumard2022). When effort, ability, and desert are manipulated or observed, however, people tend to prefer unequal distributions —just as bargaining theory would predict if those attributes were to be indicative of bargaining power.
Following past work (André et al., Reference André, Debove, Fitouchi and Baumard2022; Melkonyan et al., Reference Melkonyan, Zeitoun and Chater2017), we use the Nash Bargaining Solution to illustrate the broad principles arising from agreement-based methods. But many other bargaining solutions have been proposed in the literature in game theory; and the general process of bargaining and negotiation is likely to depend on many factors not typically considered in formal models of bargaining (e.g., including reputation, historical precedent, salience, and many more). Irrespective of the detailed account, the bargaining process has two important features: (1) that parties can walk away freely from an interaction without interference, and (2) that once an agreement is made, neither party can unilaterally back out. These conditions are common, for instance, in economic markets regulated by states—nobody can be compelled to enter a contract but, once a contract is entered, it is backed by the force of law. But there are cases in which a person may be compelled to participate in social interactions that they would not have chosen to enter into in the first place (e.g., cases of coercion); and/or there is no credible enforcement of agreements (either by the parties themselves or an external agent). It remains for future work to determine whether, and how, principles of agreement in moral psychology apply in such cases (see, for example, (Hoffman et al., Reference Hoffman, Yoeli, Navarrete, Shackelford and Hansen2016)).
Finally, in this paper we largely restrict our analysis to the cognitive mechanisms people use to making decisions involving a few others, or small groups. We provide little discussion of inter-group relations, or of society’s formal, largescale institutions (though see the discussion of this point in § 6.3). Particularly in the case of some institutions, it is natural to think that agreement-based principles might have substantial explanatory power. After all, institutions often have formal mechanisms for finding and enforcing agreements. What is less clear, however, is whether these mechanisms are meaningfully grounded in a contractualist moral psychology—one which, to the extent that it is evolved, presumably evolved in smaller and less institutionally formalized settings. These psychological mechanisms are our primary focus.
3. Mechanisms of Moral Cognition
Much prior work in moral psychology takes inspiration from philosophical theories, but builds almost exclusively on two different approaches in moral philosophy: deontology (focusing on restrictions on actions) and consequentialism (focusing on calculating outcomes). In contrast, our proposal draws in part on the contractualist tradition in moral philosophy. Contrasting these three approaches provides a helpful framework for organizing the current state of the field.
3.1. Contractualism and its psychological counterparts
Philosophical contractualism is a normative theory: It attempts to explain how we should think and act. Footnote 6 At the broadest level, philosophical contractualism posits that human agreements, institutions, and rules ought to be modeled on what rational agents should agree to in order to achieve mutual benefit. This takes many different particular forms: an idealized dialog between people attempting to adjudicate their differences (Habermas, Reference Habermas1996); the identification of behaviors that could not be “reasonably rejected” by anyone (Scanlon, Reference Scanlon1998); the behaviors of rational agents acting in concert (Gauthier, Reference Gauthier1987); the choices we would make behind a “veil of ignorance” (Rawls, Reference Rawls1971); or, those we could rationally will as a “universal law” governing the conduct of all (Kant, Reference Kant1785; O’Neill, Reference O’Neill and Ellis2012). What is common among all is the idea that we must find arrangements to which all relevant parties would agree.
As we have already seen, actual human moral judgments, norms, institutions, and rules often show broad alignment with these models. It is natural to assume, then, that our moral psychology would be organized at least in part around agreement-based methods. Yet, surprisingly little psychological research has characterized the precise mechanisms by which people agree on what is right and wrong.
Piaget’s (Piaget, Reference Piaget1932) and Kohlberg’s (Kohlberg, Reference Kohlberg1969) theories of moral development are early and important exceptions, having proposed that certain stages of a child’s moral understanding were organized around convention and agreement. One important line of work in child development has elaborated on these ideas quite directly (Killen, Reference Killen1995; Killen & Turiel, Reference Killen and Turiel1991). Others share its more general focus on the role of mutual agreement in children’s understanding of how moral norms are established or updated (Tomasello, Reference Tomasello2020; Zhao & Kushnir, Reference Zhao and Kushnir2018).
Outside of the developmental literature, agreement-based descriptions of the moral mind have received less attention, although with some notable exceptions. Everett and colleagues have suggested that contractualist notions (in particular, a “respect for persons and the honoring of social contracts”) can explain our responses in certain moral dilemmas (Everett et al., Reference Everett, Pizarro and Crockett2016), and Levine and colleagues have described mechanisms through which this may occur (Levine, Kleiman-Weiner, et al., Reference Levine, Kleiman-Weiner, Schulz, Tenenbaum and Cushman2020; Levine et al., Reference Levine, Kleiman-Weiner, Chater, Cushman and Tenenbaum2024). Sell and colleagues’ recalibrational theory of anger views anger as a mechanism that people use to bargain for better treatment with those they interact with (Sell et al., Reference Sell, Sznycer, Al-Shawaf, Lim, Krauss, Feldman, Rascanu, Sugiyama, Cosmides and Tooby2017). Most ambitiously, André and colleagues have argued that many of our moral judgments can be explained via an evolutionary contractualist approach (André et al., Reference André, Debove, Fitouchi and Baumard2022). Specifically, they propose that evolutionary processes naturally lead our moral judgments to approximate generalized Nash bargaining solutions, providing an adaptive rationale for our intuitions about distributive justice, ownership, authority, our responses to moral dilemmas, special obligations towards kin, and even the moralization of supposedly harmless wrongs. This description of the ultimate function of our moral psychology leaves open questions about its proximate-level psychological implementation.
Nonetheless, the most notable thing about the psychological literature on contractualism is its sparsity. This stands in stark contrast to its two main philosophical rivals: Consequentialism and deontology.
3.2. Consequentialism and its psychological counterparts
Consequentialist theories posit that the moral properties of an action depends exclusively on the “state of affairs” (consequences) that it brings about. Often, the relevant consequence is something like welfareFootnote 7 , and the claim is often that it should be impartially maximized in aggregate. “Welfare maximization” also plays a central role in many theories of moral psychology. For instance, many computational accounts of moral decision-making are structured around a backbone of expected value maximization, where value is often defined largely in terms of one’s own and others’ welfare. These models have been frequently and successfully applied in studies of the neural basis of moral decision-making (FeldmanHall et al., Reference FeldmanHall, Dalgleish, Evans, Navrady, Tedeschi and Mobbs2016; Hsu et al., Reference Hsu, Anen and Quartz2008; Lockwood et al., Reference Lockwood, Klein-Flügge, Abdurahman and Crockett2020; Williams, Reference Williams1968), which often show striking overlap with the neural basis of self-interested expected value maximization in ordinary, non-social decision making (KleimanWeiner et al., Reference Kleiman-Weiner, Gerstenberg, Levine and Tenenbaum2015; Shenhav & Greene, Reference Shenhav and Greene2010). Also, a rich and productive literature shows that when people face welfare tradeoff dilemmas, such as the trolley problem, a major (although not exclusive) contributor to their moral judgments is a mechanism that makes characteristically consequentialist choices (Cushman et al., Reference Cushman, Young and Hauser2006; J. Greene, Reference Greene2014). In other words, it decides whether to sacrifice one life in order to save five by “doing the math” and favoring the welfare of five over one. We refer to this mechanism of moral judgment as “welfare-based:” it determines what to do by considering the welfare consequences to various people given a model linking actions to outcomes. Unlike many versions of philosophical consequentialism, however, most psychological models of this kind assume that different peoples’ welfares are weighted differently—i.e., that we care more about some people (ourselves, family, friends, etc.) than others.
3.3. Deontology and its psychological counterparts
Deontological theories are concerned not with outcomes but with whether actions conform to moral rules, rights, and duties (L. Alexander & Moore, Reference Alexander, Moore and Zalta2021). Deontological theories stand in contrast to consequentialiast theories in positing that some actions are morally impermissible regardless of the consequences that they bring about (L. Alexander & Moore, Reference Alexander, Moore and Zalta2021). A wealth of psychological evidence shows that people’s moral judgments are indeed responsive to intrinsic properties of actions, and not just their consequences. For example, judgments of moral cases seem to be affected by whether the case involves a commission or an omission (Baron & Ritov, Reference Baron and Ritov2004; Cushman et al., Reference Cushman, Young and Hauser2006; Petrinovich & O’Neill, Reference Petrinovich and O’Neill1996), the relationship of the victim to the actor (Kleiman-Weiner et al., Reference Kleiman-Weiner, Gerstenberg, Levine and Tenenbaum2015; Petrinovich & O’Neill, Reference Petrinovich and O’Neill1996), whether or not the harm was intended or accidental (Barrett & Saxe, Reference Barrett and Saxe2021; Cushman et al., Reference Cushman, Sheketoff, Wharton and Carey2013; Piaget, Reference Piaget1932), whether the harm was a means or a side-effect (Cushman et al., Reference Cushman, Young and Hauser2006; J. Greene, Reference Greene2014; Levine & Leslie, Reference Levine and Leslie2020; Mikhail, Reference Mikhail2011), whether the harm was brought about through bodily contact (Cushman et al., Reference Cushman, Young and Hauser2006; J. Greene, Reference Greene2014; Pellizzoni et al., Reference Pellizzoni, Siegal and Surian2010), whether the harm is caused “directly” or indirectly (Royzman & Baron, Reference Royzman and Baron2002), to name just a few. Some theories argue that this arises from explicit representations of abstract categories such as harm, knowledge, and intention, although perhaps outside of conscious awareness (Cushman, Reference Cushman2015; Darley & Shultz, Reference Darley and Shultz1990; Mikhail, Reference Mikhail2011; Nichols, Reference Nichols2021). Other accounts emphasize the role for moral emotions (J. D. Greene et al., Reference Greene, Sommerville, Nystrom, Darley and Cohen2001; Haidt, Reference Haidt2001)) or other mechanisms of value representation (Crockett, Reference Crockett2013; Cushman, Reference Cushman2013) in triggering the condemnation of certain types and properties of actions. What these views share is that moral judgment does not depend on consideration of the welfare consequences of actions, but on properties of actions themselves. Whether or not these properties are explicitly represented in rule-like form, they dictate which categories of actions are permissible, obligatory, or forbidden. We refer to this broad and rather diverse class of moral mechanisms—those concerned with properties of action, rather than the outcomes of those actions—as “action-based” mechanisms.
3.4. Conflict and consilience between mechanisms of moral judgment
How can we best understand the relationship between these parts of our moral minds: the consequentialist (welfare-based), deontological (action-based), and contractualist (agreement-based)?
Generally, philosophers think of these normative theories as rivals. Much philosophical work is organized around finding cases that reveal the conflicts between them—where different philosophical views render different judgments—to help adjudicate which theory should be preferred. Similarly, psychologists have seen different theories as standing in opposition, and have exhaustively explored how moral dilemmas reveal cognitive conflict between welfare-based and action-based mechanisms (Conway & Gawronski, Reference Conway and Gawronski2013; Crockett et al., Reference Crockett, Clark, Hauser and Robbins2010; J. D. Greene et al., Reference Greene, Nystrom, Engell, Darley and Cohen2004; Koenigs et al., Reference Koenigs, Young, Adolphs, Tranel, Cushman, Hauser and Damasio2007).
However, there has also been some effort to understand how welfare-based and action-based mechanisms of moral judgment are related. It has been influentially argued that action-based rules are cognitively cheap approximations of welfare maximization (Baron, Reference Baron1994; Crockett, Reference Crockett2013; Cushman, Reference Cushman2013; J. Greene, Reference Greene2014). But two things are striking about this attempt to relate welfare-based and action-based mechanisms. First, it proposes that the ultimate function of our moral psychology is to maximize aggregate welfare. As we have already argued, however, rational models of interdependent choice predict not welfare maximization, but instead bargaining solutions. Second, this proposal provides no natural account of the relationship between agreement-based methods and action- or welfare-based ones.
We draw inspiration from a proposal in the philosophy literature that unifies all three. Derek Parfit famously pointed out that contractualism, deontology, and consequentialism agree on moral judgments far more often than they disagree (Parfit, Reference Parfit2011). This suggests that they may be quite intimately related.
He proposes that contractualism—i.e., agreement between people—provides a basis for accepting or rejecting rules that govern action, and does so on the basis of their welfare consequences. His key insight—one which holds great promise for descriptive theories of moral psychology—is that contractualism provides the ultimate justification for moral standards, but that these standards may be framed in deontological or consequential terms. By analogy, we propose that many action-based or welfare-based judgments can be explained as resource rational approximations of agreement-based ones. We turn next to consider how this works.
4. Resource-rational Contractualism
How would a contractualist moral psychology work, in practice? Agreement-seeking in its idealized form is often infeasible in the real world. Thus, while we might sometimes observe thoughts and behaviors that are close to that ideal, what we will find more often are cognitively and socially expedient heuristics designed to approximate it. Our strategy is to divide the idealized version of contractualism into two elements: the process of agreement (how we find it), and the terms of the agreement (what it covers) (Fig. 1). We then consider potential heuristic approximations for each (Fig. 2).
The idealized form of contractualist moral decision-making involves a particular process (getting everyone together to negotiate) and particular terms (an agreed upon solution to the specific instance under discussion). Both the process and the terms have resource-rational approximations that trade off accuracy against effort. We can replace the process with an anticipation of what an in-person negotiation would yield via either model-based or model-free strategies. Likewise, we can replace the terms with those that are more abstract than those dealing with particular instances, such as welfare-based and action-based standards.
Two principle axes of resource-rationality: abstraction and anticipation. The ideal is to actually negotiate for mutual benefit with all relevant parties over each specific instance (top left cell). Given that this is resource intensive, we can instead bargain over more abstract welfare or action standards (rows) or anticipate bargaining solutions through model-based or model-free methods (columns).
4.1. The process of agreement: Resource-rational anticipation
The contractualist ideal is actual bargaining. But this imposes both cognitive and social demands, requiring multiple people to sit down and think hard together.
As a heuristic, rather than engaging in actual bargaining, we can instead anticipate the likely outcome of the bargaining process. One way to do this is “model-based”: We mentally simulate the bargaining process to guess its hypothetical outcome. Put simply, we imagine what another person would agree to if we were talking things over with them. Another way is “model-free”: We generalize from past agreements, as relevant precedent. Our use of the terms “model-based” and “model-free” in this context draws an intentional analogy to theories of reinforcement learning and decision-making (i.e., sequential value-guided decision-making), where these terms first emerged. In that context, as here, model-based methods rely on inference (via planning, simulation, or other means) in the moment using a generative causal model—in this case, a model of bargaining. Model-free methods draw up previously cached (or “amortized”) model-based computations, by generalizing from them, or by other methods such as statistical generalization, analogy, or other means.
In summary, we propose a spectrum of possibilities from the costly but exact process of actual bargaining, through a less costly mental model of bargaining, ultimately to the least costly possibility of generalizing directly to decisions from past agreements.
4.2. The terms of agreement: Resource-rational abstraction
In the idealized case, the terms of any agreement will be precise and narrow—agents agree to the specific behaviors that attain mutual benefit—but the agreement would narrowly cover their conduct only in the present circumstances, since no future circumstances will ever be completely identical. As a less costly alternative, however, people may strike more abstract bargains—ones that apply to many cases, preventing the need for constant re-negotiation.
We consider two possible forms of abstraction, motivated by the kinds of decision-making mechanisms with which we know the human mind is well-equipped for purposes of individual decision making (Dolan & Dayan, Reference Dolan and Dayan2013).
First, people often solve decision-problems by the logic of expected value maximization. When planning a day, for instance, one considers a range of possible actions, along with one’s goals and preferences, and then chooses the specific actions that maximize expected value. Since humans are good at this kind of computation, we might expect that they would use it to solve the problem of interdependent choice by defining a single utility function over all the agents concerned and then maximizing it. In this case, the nature of the utility function—which utilities, and especially whose utilities, it favors—would constitute the terms of the bargain.
This is an approximation of the ideal computation of mutual benefit because it substitutes a weighted welfare maximization problem for a bargaining problem. Nevertheless, it preserves the essential characteristics of agreement-based methods because the weightings applied during welfare maximization are subject to negotiation. We call the result of negotiating those weightings, a “welfare-based” standard.
Moreover, as a general rule for choice, maximizing overall choice will typically benefit most people, most of the time; indeed, when aggregating over a sequence of many choices, the results of maximizing overall benefit, and ignoring distributional concerns, is likely to yield benefit to most people. Overall welfare maximization provides a “rising tide” that will raise most, if not all, boats.
A second common and even more computationally efficient method of making decisions is to define standards directly over actions in context—that is, to assign actions intrinsic value, or an intrinsic probability of selection, contingent on a class of situations (Konda & Tsitsiklis, Reference Konda and Tsitsiklis1999; Watkins & Dayan, Reference Watkins and Dayan1992). Representations of this sort—which we sometimes call rules, habits, or customs—can often make individual decision-making very straightforward. While each situation is unique, they can often usefully be organized into broad classes, so that once we have figured out the best way to act in one situation (whether by learning from experience, or by model-based planning to maximize utility), we can then generalize that guidance to all instances of the class without the hard computational work of learning or thinking through each new situation from scratch. After one encounter with a hot stove, for example, we may simply encode and operate a rule to keep our distance in the future.
The same is true in the context of interdependent choice: multiple agents or groups can strike agreements about how to act jointly that are designed to achieve mutual benefit across broad classes of situations. Since people are good at implementing rules like this in the case of individual decision-making, we might expect them to characterize an important part of psychological contractualism as well. We call agreements of this sort “action standards”.
We next consider each of these two forms of abstraction in more detail, describing the specific predictions rendered when construing welfare-based and action-based standards as resource rational approximations of agreement.
4.2.1. Welfare-based standards
Sometimes people bargain over the relative weighting of their welfare—a standard governing how each values the other (Adams, Reference Adams and Berkowitz1965; Delton & Robertson, Reference Delton and Robertson2016; Tooby et al., Reference Tooby, Cosmides, Sell, Lieberman and Sznycer2008). In certain contemporary institutionalized settings, negotiation over welfare standards is explicit; one example is the distribution of limited resources through public health agencies such as the World Health Organization or the UK’s National Institute for Health and Care Excellence (Hirth et al., Reference Hirth, Chernew, Miller, Fendrick and Weissert2000; Owen & Fischer, Reference Owen and Fischer2019). For the most part, however, people don’t have debates along the lines of “You seem to value me at half of the value you place on yourself, but I feel like it should be more like three quarters.” Nevertheless, real-world social processes involve implicit negotiation over welfare trade-offs. For instance, there is a rich research program which proposes that certain kinds of social emotions—anger, jealousy, resentment, gratitude, remorse, etc.—are actually a vehicle for just this kind of welfare-based bargaining to occur (Adams, Reference Adams and Berkowitz1965). Anger can be viewed as an attempt for a wronged person to express that someone else is not taking their welfare into account as much as they would like. And, remorse can be viewed as the “recalibration” of a welfare standard in response to justifiable anger (Chang et al., Reference Chang, Smith, Dufwenberg and Sanfey2011; Sell et al., Reference Sell, Sznycer, Al-Shawaf, Lim, Krauss, Feldman, Rascanu, Sugiyama, Cosmides and Tooby2017). For instance, when a child fusses about whether they got as much juice as their sister, or an employee bristles when the boss celebrates a colleague’s birthday but ignores his, their anger may principally reflect not the literal resources at question, but the implied relative valuation—and, indeed, that anger may act as a bid for revaluation. Because we know that people often make decisions by taking others’ welfare interests into account—but to varying degrees—it makes good sense that we would have implicit yet powerful mechanisms for negotiating and renegotiating how each others’ welfares ought to be weighted.
Once welfare standards are established, people can then make interdependent decisions by selecting the action that maximizes the agreed weighted aggregate welfare for some set of relevant parties. For instance, given that we value ourselves greatly, our immediate family highly, certain friends and acquaintances slightly less, even strangers to some degree, etc., we can then decide how big a gift or favor to give, how scarce resources should be divided, and on whom various burdens can be imposed (Marshall et al., Reference Marshall, Wynn and Bloom2020, Reference Marshall, Gollwitzer, Mermin-Bunnell, Shinomiya, Retelsdorf and Bloom2022; McManus et al., Reference McManus, Kleiman-Weiner and Young2020); and we anticipate that others will judge our behaviour using similar weightings. A wealth of research demonstrates that people do, in fact, often navigate interdependent choice problems in precisely this way (FeldmanHall et al., Reference FeldmanHall, Dalgleish, Evans, Navrady, Tedeschi and Mobbs2016; Hsu et al., Reference Hsu, Anen and Quartz2008; Lockwood et al., Reference Lockwood, Klein-Flügge, Abdurahman and Crockett2020; Williams, Reference Williams1968, e.g.).
It is useful to bargain over welfare standards when we will interact with the same person many times, but in varied situations. For instance, perhaps we are in a long-term relationship with a friend, coworker, or romantic partner. In any of these relationships we might encounter unusual circumstances that involve interdependent choice. For instance, will I pick up their dry cleaning across town, despite my urgent work deadline, knowing that they are on the phone dealing with the fallout from their sister getting arrested? Footnote 8 It would be possible to engage in ad hoc negotiation in a case like this, but is likely to be more efficient to fall back on a general sense of the relative costs and benefits and the weighting of our interpersonal interests. Thus, when we participate in enduring relationships that elicit very diverse interdependent choices, it may be rational to agree (presumably implicitly) on welfare standards.
Any plausible theory of welfare-based moral standards requires certain situational and role-specific constraints on their application. When a kindergarten teacher is at work, for instance, he will value each of his students quite highly, while thinking relatively little about the welfare of his elderly parents. While visiting his parents over the holidays the reverse is true. In each context his behavior can be partially understood as the product of welfare-based reasoning—a balancing of the interests and needs of children against each other, or parents against himself, etc. But one cannot perfectly capture his behavior cross-situationally according to a consistent set of welfare standards: A sacrifice he makes for a needy student in the classroom does not imply an equivalent sacrifice for the same student over holidays. This situational sensitivity is implicit in the tug-and-pull of interpersonal negotiation over welfare standards. An employee who complains that her boss should value her more just means to say she should be more valued at work, not that the boss should devote more attention to her home life.
These variations make sense if we see welfare as approximating the results of agreements: the teacher has “signed up” to care for his class in a specific context; a boss is primarily responsible for the welfare of her employees at work, and so on. In contrast, these variations are relatively harder to capture on a standard consequentialist view of welfare standards, according to which the valuation we place on another’s welfare should apply equally across situations. This becomes particularly apparent if we consider the norm of evenly splitting windfall resources: the “fair” 50/50 split. This norm is shared across many cultures (Henrich et al., Reference Henrich, Boyd, Bowles, Camerer, Fehr, Gintis, McElreath, Alvard, Barr and Ensminger2005, Reference Henrich, Ensminger, McElreath, Barr, Barrett, Bolyanatz, Cardenas, Gurven, Gwako and Henrich2010, although not all, see), and dictates how people behave even in one-shot anonymous interactions. It is obviously not the case, however, that in these cultures everybody weights others’ welfare equally with their own. Rather, the norm applies to a narrow class of situations in which windfall profits are divided.
This analysis invites a new perspective on welfare-based moral judgment. It has been argued that one can view such welfare-based mechanism as basically consequentialist; and that consequentialism is also the rational standard that other mechanisms of moral judgment (such as action-based constraints) are designed to approximate. According to the present proposal, however, welfare-based mechanisms of moral judgment are best understood as a heuristic approach to approximating the ad hoc agreements that we would reach under idealized circumstances: the psychologically relevant rational standard is not utilitarian but consequentialist.
4.2.2. Action-based standards
People can also bargain over action-based standards such as rules and norms. This is obviously true at an institutional level, where explicit bargaining over rules, norms, laws, and policies often happens. Debate and negotiation in legislative bodies, for instance, concerns what rules should be adopted to govern a community such as a school, a place of work, organization, city, region or an entire country. But, action standards are also negotiated informally as well. Extensive fieldwork from the past half century—specifically looking at how groups self-organize to manage “common pool resources”—reveals that bargaining over rules happens via a predictable processes (Acheson, Reference Acheson1988; Ostrom, Reference Ostrom2000). These are generally most successful when the directly affected parties negotiate their own solutions (rather than solutions being imposed by outside authorities), when their solutions impose enforced constraints on behavior, when resources are shared or distributed across all those subject to regulation, and when each person’s near-term costs are outweighed by their understanding of the long-term benefits of the system (Ostrom et al., Reference Ostrom, Burger, Field, Norgaard and Policansky1999). In other words, the standards established by the group help regulate each individual’s behavior in order to achieve an outcome that leaves nobody worse off than they would have been in the absence of an agreement (Ostrom, Reference Ostrom2000). Similarly, children use the agreement and negotiation to establish the rules of games and other social interactions by middle childhood (Piaget, Reference Piaget1932), drawing in part on their ability to form joint intentions and plans (Tomasello, Reference Tomasello2020).
Once established, rules and norms tell us how to act, or introduces constraints on action, across a wide range of relevantly similar circumstances. Thus, we have: “No non-resident street parking after 8pm on weekdays”; “I’ll pick up the kids for gymnastics every Wednesday afternoon this fall”; “Good restaurant service should be rewarded with a 20% tip”. Naturally, bargaining over action standards will be efficient when sufficiently similar circumstances recur. Rather than bargaining anew, and repeatedly arriving at very similar bargaining solutions, we can establish a standard—a sort of moral heuristic—that can be quickly retrieved and applied, and which efficiently gets us close enough to the optimal answer on each new occasion.
Action standards need to strike a delicate balance between accuracy and generality. Consider the rule: “Don’t steal money from the woman wearing a floral dress, sitting in a cafe on the corner of Kirkland and Washington Streets on May 7”. Such an over-specific rule may always be accurate, but no efficiency is gained by establishing this rule because the circumstance is unlikely to arise more than once. On the other hand, the (wildly) over-general rule “Don’t do anything!” is universally applicable, but errs in every case where something really should be done. Rules are useful when the costs of their inevitable errors are outweighed by the benefits of frequent, cognitively cheap application (Hare et al., Reference Hare, Hare, Hare and Hare1981; Sunstein & Ullmann-Margalit, Reference Sunstein and Ullmann-Margalit1999). Finding this balance is, in essence, a problem of resource rationality.
Explicit action standards, such as rules, also have benefits for communication and coordination. Communication is enhanced when rules are formulated in simple terms over observable features. They also help groups coordinate actions for mutual benefit, such as choosing between multiple equilibria (e.g. driving on one side of the road or backing the same candidate) (Anderson, Reference Anderson2001; Gauthier, Reference Gauthier1987; Harsanyi, Reference Harsanyi1977). Action standards also enable coordinated action because whether a rule is being followed is typically more easily observed than, for instance, how much welfare various individuals obtain, or how an actor weights the welfare of others. Thus, action based standards help us determine who to trust, praise, or blame with greater certainty than welfare based standards— and also afford greater confidence that others will arrive at the same conclusion (DeScioli & Kurzban, Reference DeScioli and Kurzban2013).
Not only can action-based moral standards be efficiently reused, they can also be culturally transmitted in the form of rules, norms, laws, etc. Collectively, this set of standards is an important part of our cultural inheritance—one that each new generation of children imbibes partly through explicit guidance and instruction from others and partially through observation and inference (Bandura & Walters, Reference Bandura and Walters1977; Cialdini & Trost, Reference Cialdini, Trost and Gilbert1998; Henrich & Muthukrishna, Reference Henrich and Muthukrishna2021; Nichols, Reference Nichols2021; Stegall et al., Reference Stegall, Mikhail, Nichols and Kushnir2023).
Thus, just as with welfare standards, action standards can be best understood as a resource-rational approximation of the ad hoc agreements we would reach if we bargained over every unique situation that arises.
4.3. The mechanisms of resource rational cognition
Having described the different abstractions that can be employed during bargaining, we now turn to the specific heuristic cognitive mechanism we can use to anticipate the outcome of bargaining processes (Fig. 3).
Relationship of each mechanism to the others and to moral judgment. Contractualist methods of moral judgment (all those pictured) are highlighted in blue. Processes that have been described as “consequentialist” (i.e., concerned with welfare outcomes) are highlighted in red and those described as “deontological” (i.e., concerned with action constraints) in yellow. Yellow and red boxes sit on top of the blue one, indicating that “deontological” and “consequentialist” processes are versions of contractualist ones. The arrows indicate how the processes relate to one another and to moral judgment. Moral judgment can be rendered directly by ad hoc negotiation, by any of the model-based bargaining processes, or by the application of model-free bargaining approximations (solid lines). Bargaining over standards either in person (left-most column) or through mental simulation (middle column) can lead to those standards being cached for future re-use (right-most column) (dotted lines). The section numbers in the figure refer to the discussion of each process in the text. 2007). In each case, if you were to act (taking their food, or performing a medical intervention) in a way that you would not have expected the affected person to consent to, this would be a gross moral violation.
4.3.1. Model-based contractualist cognition
One way to anticipate bargaining outcomes through private mental processes is to employ a cognitive model of bargaining. We consider several accounts of this kind, which focus on model-based anticipation of bargains over ad hoc circumstances, over welfare-based standards, and over action-based standards.
4.3.1.1 Model-based bargaining over individual instances: Virtual bargaining
It is quite intuitive to think that in some situations we decide whether an action is OK by asking whether the party affected by that action would agree to it. For instance, if you are hungry and you know your co-worker keeps granola bars in her desk, can you simply take one? We might answer this question by asking, “what would she answer if I could ask her?” This same idea plays a much weightier role in the law surrounding emergency medical care. A doctor who encounters an unconscious victim of a car collision, for instance, is permitted to perform life-saving procedures because of the presumption that the patient would consent to that treatment if he were able to (Easton et al.,
This method of making moral judgments can be addressed using a process of “virtual bargaining”—using a cognitive model of bargaining processes to coordinate behavior and resolve ad hoc any conflicts of interest. Footnote 9 Virtual bargaining has been developed primarily outside the moral domain, to understand the reasoning that underpins how people are able flexibly to coordinate on communicative and social conventions “in the moment” (Chater et al., Reference Chater, Zeitoun and Melkonyan2022; Misyak & Chater, Reference Misyak and Chater2014; Misyak et al., Reference Misyak, Melkonyan, Zeitoun and Chater2014). But the same mechanism applies naturally to moral judgments.
The virtual bargaining approach contrasts with standard game-theoretic reasoning, in which agents do not think about possible bargains, but attempt to choose between Nash equilibria: strategically stable combinations of actions such that no person can benefit by unilaterally changing their choice of action. Virtual bargaining allows people to find mutually beneficial agreements that are not Nash equilibria; and also provides a mechanism for choosing between the multiple possible bargains (including many Nash equilibria) that arise in any but the very simplest interactions.
In order to distinguish these hypotheses, social interactions can be formalized as a multiplayer game. Consistent with the predictions of virtual bargaining, in game-based experiments people often choose non-Nash equilibrium bargains, where these are most mutually beneficial; and they also select between the many possible Nash equilibria (and other options) by coordinating on the option with greatest mutual benefits.Footnote 10 That is, people choose actions characterized by the equilibrium that would be mutually agreed upon, according to the theory of bargaining—even in the absence of any actual negotiation. Related work shows that this model predicts not only their behavior, but also their moral judgments (Le Pargneux et al., Reference Le Pargneux, Chater and Zeitoun2023).
The virtual bargaining view makes predictions not only about peoples’ behaviors, but also about their moral judgments. Confirming this prediction, Levine and colleagues (Levine et al., Reference Levine, Kleiman-Weiner, Chater, Cushman and Tenenbaum2024; Trujillo Jiménez et al., Reference Trujillo Jiménez, Zhang, Zhi-Xuan, Tenenbaum and Levine2025) presented participants with a hypothetical situation in which somebody is able to earn a large sum of money by painting their neighbor’s house blue while their neighbor cannot be contacted. Participants’ moral judgments are best modeled by an account of virtual bargaining—which assumes that accepting the money is permissible if a sizeable enough portion is given to the neighbor (such that the neighbor would have agreed to it in a negotiation)—but not by rule-based or welfare-based accounts.
The theory of virtual bargaining also makes the clear developmental prediction that children’s behavior and moral judgments will change as they gain the ability to reason about bargaining and agreement. Some current data, while circumstantial, shows that signatures of virtual bargaining emerges by the preschool years. For instance, young children find it morally permissible to cause a small harm to someone to prevent a greater one to that same person (Jambon & Smetana, Reference Jambon and Smetana2014; Levine & Leslie, Reference Levine and Leslie2020)—perhaps because the wouldbe victim would agree to it. Moreover, when a child collaborates on a task with another person and then receives a greater reward than their partner, they will often share the excess reward (Hamann et al., Reference Hamann, Warneken, Greenberg and Tomasello2011)—potentially because they realize that the collaborator’s contribution was contingent on a tacit agreement that the reward would be shared. Moreover, there are contexts in which children treat implicit commitments as having the binding power of explicit ones(Kachel & Tomasello, Reference Kachel and Tomasello2019). This suggests that, by a young age, children are able to infer the agreements that would be made if they could discuss the situation at hand, and feel that they are bound by those agreements. It remains to be seen whether these patterns of judgment emerge only as the child becomes able to reason about agreement, and this stands out as an important area for further study.
4.3.1.2 Model-based bargaining over welfare standards: Implied valuation
When trying to anticipate others’ reactions to our behavior, we often consider not only whether they would agree to the particular act in question, but also whether they would accept the implied value we assign to their welfare (Adams, Reference Adams and Berkowitz1965). Put another way, when we make choices, we often consider whether those choices would make others feel as if we value them sufficiently. And this make sense, given that people do in fact interpret social behavior not just in light of its immediate, local consequences, but also in terms of the way it signals a more global valuation—how much they are cared for in general (Adams, Reference Adams and Berkowitz1965; McManus et al., Reference McManus, Kleiman-Weiner and Young2020; Radkani & Saxe, Reference Radkani and Saxe2023; Shaw, Reference Shaw2013; Uhlmann et al., Reference Uhlmann, Pizarro and Diermeier2015). This, for instance, is a natural way to understand “recalibrational” emotions like anger, gratitude, and so on (Sell et al., Reference Sell, Sznycer, Al-Shawaf, Lim, Krauss, Feldman, Rascanu, Sugiyama, Cosmides and Tooby2017).
This view predicts that people have and use cognitive models of how others would infer their welfare-based standards and react to them. By contrast, it might be conjectured instead that even if people use welfare-based standards to choose their actions and make moral judgments, they may fail to anticipate how others would respond to or attempt to renegotiate those standards.
There is, though, already some evidence that people explicitly model the anticipated (i.e. model-based) reactions of a social partner (Houlihan et al., Reference Houlihan, Kleiman-Weiner, Hewitt, Tenenbaum and Saxe2023; Railton, Reference Railton2017). This is consistent with a large literature on impression management (Leary & Kowalski, Reference Leary and Kowalski1990). It is also consistent with experimental evidence that when people allocate resources between themselves and another, they are attuned to their social partner’s perception of how the resources are divided (Houlihan et al., Reference Houlihan, Kleiman-Weiner, Hewitt, Tenenbaum and Saxe2023), even when those perceptions are false (van Baar et al., Reference van Baar, Chang and Sanfey2019). (Such attunement to false beliefs is a hallmark of model-based reasoning about another’s mental states). This same literature suggests that in anticipating those reactions, we often focus on the implied degree to which we generally care about the other person’s welfare versus our own (rather than the specific bargain we would strike over a single act).
4.3.1.3 Model-based bargaining over norms and rules: Universalization
Finally, there are circumstances in which we use model-based reasoning to determine what norms or rules people would agree to. When considering whether to litter, vote, speed, recycle, etc., we often feel as if we should adopt the pattern of behavior that the relevant community would prefer to be universally adopted. This is often called universalization (Levine et al., Reference Levine, Kleiman-Weiner, Schulz, Tenenbaum and Cushman2020a). Put simply, it asks: “what if everybody did that?”
Universalization appears, in one form or another, in many contractualist views in moral philosophy (Gauthier, Reference Gauthier1987; Habermas, Reference Habermas1990; O’Neill, Reference O’Neill and Ellis2012; Rawls, Reference Rawls1971; Scanlon, Reference Scanlon1998). A variety of theoretical traditions point out that suitably enforced norms play a crucial role in allowing people to solve social dilemmas, such as the protection and use of public goods (Bicchieri, Reference Bicchieri2005; Henrich, Reference Henrich2004; Ostrom, Reference Ostrom2000; Tomasello, Reference Tomasello2009). Norms allow people to transform any pattern of behavior into a stable equilibrium via enforcement (Boyd & Richerson, Reference Boyd and Richerson1992). A key question, then, is which norm will be chosen. The logic of universalization allows people to anticipate the answer to this question, finding equilibria that generate mutual benefit, and to set their own behavioral standards accordingly, through a cognitive model of collective agreement over rules (Binmore, Reference Binmore2014).
Consistent with these predictions, many people spontaneously render moral judgments consistent with the logic of universalization (Kwon et al., Reference Kwon, Tenenbaum and Levine2022, Reference Kwon, Tan, Tenenbaum and Levine2023; Levine, Kleiman-Weiner, et al., Reference Levine, Kleiman-Weiner, Schulz, Tenenbaum and Cushman2020). Existing work shows how universalization guides the judgment of individual acts, however. Our view suggests that universalization should naturally be used not only as a method of telling us which acts to perform by considering what would happen if rule were in force, but also as a method of determining which rules to adoptFootnote 11 . After all, its very logic asks us to consider the question: What if this action were adopted as a universal rule?
Testing this prediction is an important area for future work. One of the important implications of a rule-based model of universalization is that it may help us to understand the application of egalitarian rules to even cases where one party holds a bargaining advantage. Viewed through the lens of ad hoc bargaining, the egalitarian solution is not predicted by a contractualist account. Viewed through the lens of a society-wide bargain over suitable standards to cover a wide variety of circumstances—ones where individuals are sometimes advantaged and other times disadvantaged, or may not know what position they will hold—egalitarian rules may be easier to explain (see also §6).
4.3.2. Model-free contractualist cognition
A still more frugal approach to anticipating bargaining outcomes is to extrapolate from previous bargains. This can be described as a “model-free” method of moral judgment in the sense that it does not rely on a cognitive model of bargaining, but instead retrieves a stored solution. It offers a natural account of how people make moral judgments based on precedent, welfare maximization, and action-based moral standards such as rules and norms.
4.3.2.1. Precedent: Judgment by generalization
Perhaps the simplest model-free approach to moral decision-making is to apply precedents from previous agreements to govern sufficiently similar novel cases. For instance, suppose that a student shows up to class 5 minutes late on the first day and the professor says nothing, but when they do the same on the second day the professor gives them no credit for class participation that day. Here, the student might feel as if the professor’s initial reaction established an implicit precedent, and that the departure from this precedent is unfair.
Reasoning from precedent is one example of moral “generalization”: A process by which one moves from established moral agreements of a narrower scope to assumed moral agreements of a wider scope (Stegall et al., Reference Stegall, Mikhail, Nichols and Kushnir2023, see also). Its cognitive efficiency is obtained by relying on a simple, intuitive sense that a past situation is sufficiently similar to the present one.
Consistent with the predictions of this view, precedent plays an important role in moral judgment. Indeed, there is evidence of this both in informal moral commitments (Chen & Saxe, Reference Chen and Saxe2023; Graeber, Reference Graeber2012; Theriault et al., Reference Theriault, Young and Barrett2021) and in formal structures like the law, where judicial decisions are strongly guided by precedent (Daston, Reference Daston2022; Schauer, Reference Schauer1987).
4.3.2.2. Model-free welfare-based judgment
When people have agreed on welfare standards—how much a person should care for themselves versus others—this supports a particularly efficient form of decision-making in future situations. One can simply apply these without any need for actual or imagined renegotiation, by engaging in a weighted cost-benefit analysis. Thus, for instance, a family might have previously come to the understanding that desserts are divided equally among children. When they are presented with a new conundrum—a fancy cake, where one child want lots of frosting, another the corner piece, and the third a piece with frosting rose—the parents need not conduct a messy negotiation, or even imagine one, but instead merely determine which gerrymandered cuts maximize each child’s preferences maximally and equally.
Or, imagine a doctor who witnesses a car accident and has the opportunity to care for its victims. What degree of concern for the victims’ welfare is incumbent upon the doctor? They might analogize from other situations where the duties of care have already been established. Is this like their obligation to a patient in the emergency room? Or more like their obligation (or lack there of) to any stranger they meet? Both of these possibilities embody generalization from existing welfare standards to a novel, unanticipated case.
Crucially, in each of these cases we imagine that the decision-maker does not mentally renegotiate the welfare-based standard (a “model-based” approach), but instead generalizes from relevantly similar ones already negotiated (a “modelfree” approach). This makes it less computationally demanding than modelbased approaches.
Still, engaging in welfare-based reasoning will usually be more cognitively demanding than to simply retrieve a simple rule or heuristic (J. Greene, Reference Greene2014). This is because it does require model-based planning of a different sort: Deriving the welfare consequences of ones behavior from a generative model linking actions to outcomes. In other words, this method avoids having to model negotiation, but retains modeling the effects of one’s action on others. Concretely, one can say “Adopting my standard approach, I ought to satisfy each child’s wishes equally” and proceed to considering how to slice the cake, without first asking, “if they bargained over welfare standards, what standard would these children agree to?”
Since many other theories posit that people engage in welfare-based reasoning during moral judgment, what are the distinctive predictions of the contractualist view? The key question is how the “weights” on welfare are determined: From past agreements (real, or imagined), or exclusively from other sources, such as a rational commitment to broadly utilitarian preferences (J. Greene, Reference Greene2008; Kahane et al., Reference Kahane, Everett, Earp, Farias and Savulescu2015), or from blind adaptive processes of biological or cultural evolution (Kleiman-Weiner et al., Reference Kleiman-Weiner, Saxe and Tenenbaum2017)?
The contractualist view helps to explain why those weightings are highly parochial (i.e., showing strong preferences for some individuals’ welfare over others), because of the disparate social benefits and bargaining power of different individuals we interact with (Chen & Saxe, Reference Chen and Saxe2023; Le Pargneux & Cushman, Reference Le Pargneux and Cushman2023; O’Connor, Reference O’Connor2019). Thus, for instance, we might feel stronger moral obligations towards those with whom relationships are especially beneficial to us, or those who have more available alternative social relationships rather than fewer. These are features that a standard analysis of utilitarian reasoning, in terms of something approximating philosophical consequentialism (J. Greene, Reference Greene2008), struggles to explain (although parochialism depending on biologial relatedness is much easier to explain from the standpoint of blind adaptation). The contractualist framework also explains why our welfare-based reasoning is highly situational (i.e., the way we value others’ welfare depends greatly on context and social role, such as a doctor who cares for her patients on shift but not off), since the agreements we reach are often situationally constrained, and generalizations from past to present agreements will depend on the apparent situational similarity. This feature of welfare-based reasoning is harder to explain either in terms of folk-utilitarianism or biological adaptive processes, although it could arise from precedent-based cultural evolution.
4.3.2.3. Model-free action-based judgment
People also often make moral judgments based on action standards: by attending to rules, norms, etc., that render certain classes of actions permissible or impermissible. In the present framework, this approach achieves maximum computational savings, since it is neither necessary to model the bargaining process, nor is it necessary to model the consequences of one’s actions on others’ welfare.
We have already raised a few of the ways that rules can be established. One is actual negotiation, whether formal or informal. A second is through modelbased methods, such as virtual bargaining or universalization. (These methods are not model-free, of course, but once they establish a rule it can be deployed in a model-free manner subsequently.) Action standards can also be established by generalizing from sufficiently similar precedents. For instance, if you are aware of the rules that apply at one swimming pool, you might readily generalize the same rules to other swimming pools.
When rules take the form of laws or norms, they can also be socially learned. Many formal rules, such as laws, are explicitly taught. Informal moral norms can also be inferred by observational social learning—i.e., seeing how people around you act (Goldstein & Cialdini, Reference Goldstein, Cialdini and Pratkanis2011; Nichols, Reference Nichols2021). In this case the bargaining processes that establishes the standard occurs in one set of individuals, and then the resulting rule is acquired and deployed in another.
Many prior discussions of moral rules agree that that they are cognitively efficient heuristics, but propose that they are designed to approximate a utilitarian ideal (Baron, Reference Baron1994; J. Greene, Reference Greene2014; Sunstein & Ullmann-Margalit, Reference Sunstein and Ullmann-Margalit1999). In contrast, we propose that moral rules are often heuristics designed to approximate a contractualist ideal of instance-based decision-making—i.e., rules are supposed to guide us towards the acts we would have agreed to in negotiation. This aligns with the utilitarian ideal in one specific way: its concern for outcomes. After all, the idealized bargain derives the quality of acts from the quality of the outcomes that they will tend to produce. However, it diverges from the utilitarian ideal in that, rather than maximizing aggregate benefit, bargaining solutions aim for mutual benefit—the outcomes predicted by gametheoretic bargaining solutions.
Once we look closely at actual moral rules, we see the fingerprints of the contractualist ideal everywhere. Rules governing property rights, for instance, can be elegantly explained through the lens of attempting to bring about a bargaining solution (Smith & Price, Reference Smith and Price1973). Formal analysis shows that self-interested agents bargaining over resources will tend to settle into an equilibrium where each fights hard to maintain one’s current resources, while acting deferentially towards others’ comparable claims. It is harder to explain how strict adherence to absolute property rights bring about overall utility maximization, however; on the contrary, it is often apparent that taking things from one person and giving them to another would maximize welfare, at least in certain specific cases. The institution of promising also finds a natural fit with a contractualist function. Promising someone something creates a moral obligation to carry through. But why? A welfare-maximization model would suggest that we should ignore a promise if the harm to one party is offset by the benefit to another. But, as fits our intuitions, a contractualist approach holds that a promise, like other agreements, must be kept unless both parties agree to break it (or would do so, where actual negotiation is not possible).
5. Moral flexibility
It is tempting to suppose that people trust and rely on moral standards because of their sturdy and stable appearance. But upon closer inspection our moral standards are neither sturdy nor stable. People constantly revise moral rules, craft exceptions to them, and generalize them to new situations. (In this sense our morals are like airplanes—what naive passengers call a jet, seasoned pilots joke, is just a collection of spare parts flying in close formation). Situations constantly arise in which the moral standards that we’ve assembled conflict, cannot be clearly applied, or manifestly fail to achieve their original purpose. In these cases we are forced to flexibly reassemble new morals from spare parts on the fly.
Current theories of moral psychology fall short of explaining how, when, and why this occurs. The resource-rational contractualist account, however, makes clear predictions about how, when, and why people will exhibit moral flexibility. Specifically, it predicts that moral flexibility arises from renegotiation, whether actual or virtual, and that it will occur when the benefits of finding a better solution outweigh the efficiency gains of sticking with a heuristic approach (such as a moral rule) which produces results we would be unlikely to agree to if negotiating explicitly.
5.1. The flexibility of moral standards
Moral standards are seldom absolute and unchanging. Consider the months following the emergence of the COVID virus in 2020. Early on, for many people, the normative landscape was characterized by relatively broad standards: “Socially isolate completely”; “Wear a mask at all times”, etc. Over time, however, people began to confront situations that seemed to warrant limited exceptions to these standards: Perhaps it would be acceptable to have social interactions with a limited, defined “pod” of close others; perhaps it would be acceptable to go maskless outside, when maintaining sufficient distance; perhaps one could remove a mask briefly to take a sip of water on an airplane; etc.
In doing so, we suggest, people likely drew on the entire set of cognitive mechanisms described above. After all, even though each mechanism approximates the ideal standard of achieving mutual benefit in instance-specific ways, each approximation operates differently, leading to different patterns of judgment. We propose that people revise their moral standards when they become dissatisfied with one pattern and seek another. The key question, then, is how we determine when one set of standards is failing, and how to find a better alternative.
5.1.1. When do we negotiate and renegotiate?
Broadly speaking, renegotiation can be divided into two types. First, we may need to renegotiate agreements because the world has changed, so that our past agreements and standards are out of date. The early days of the COVID pandemic are an excellent example. Just as occurred during the pandemic, we might often expect that our initial attempts at renegotiation will depend on especially wide-scope abstractions—a highly heuristic, rough “first-pass” at achieving mutual benefit.
During moments of relative stability, however, renegotiation can serve a distinct purpose: To refine the precise contours of moral judgment to better achieve mutual benefit in the cases that our broad agreements do not handle well. The later days of the COVID pandemic, and the myriad particular rules, standards and exceptions that occurred, exemplify this variety of moral flexibility.
A contractualist framework predicts that several factors will tend to encourage more effortful approaches to moral judgment.
First—as we have already suggested—the more unique a situation is, the greater the likelihood that generalizing from past agreements (whether based on precedent, action standards, or welfare standards) will not adequately approximate the outcome of ad hoc negotiation. Unique situations might also introduce sufficient uncertainty about others’ welfare interests and beliefs that mentally simulating those interests will be unreliable. Unique situations, therefore, will tend to justify more cognitively or socially costly methods.
Second, the higher the stakes of a moral decision, the greater the cost of a sub-optimal choice. Thus, like unique situations, high-stakes situations should favor those mechanisms of moral judgment that achieve greater accuracy at the cost of effort.
Third, people may be motivated to question existing standards when they suspect that they, personally, would profit from instance-based renegotiation. For instance, suppose that two colleagues are vying for a new and desirable office space that has become vacant. If either suspects that they would have superior leverage in ad hoc negotiation (but would be disfavored under existing standards, such as a rule based on seniority), they might favor the cognitive and social costs of renegotiation.
5.2. Inverting a model of agreement
So far, we have discussed how people can use a generative model of agreement in a “forward” direction—i.e., moving from an understanding of others’ utilities towards a conclusion about the standards that they would agree to. A key insight of contemporary cognitive science, however, is that we can use generative models to support inference on unobserved variables (Chater et al., Reference Chater, Oaksford, Hahn and Heit2010; Griffiths et al., Reference Griffiths, Chater and Tenenbaum2024; Tenenbaum et al., Reference Tenenbaum, Griffiths and Kemp2006, Reference Tenenbaum, Kemp, Griffiths and Goodman2011). For instance, we can use a generative model of others’ minds (a “theory of mind”) to infer their unobserved mental states, or we can use a generative model of physics to infer unobserved forces such as gravity or the wind (Baker et al., Reference Baker, Jara-Ettinger, Saxe and Tenenbaum2017; Battaglia et al., Reference Battaglia, Hamrick and Tenenbaum2013; Kleiman-Weiner et al., Reference Kleiman-Weiner, Gerstenberg, Levine and Tenenbaum2015; Ullman et al., Reference Ullman, Baker, Macindoe, Evans, Goodman and Tenenbaum2009).
Similarly, our view predicts that people should be able to use a generative model of agreement to infer unobserved variables that are relevant to moral decision-making: The utilities of the agents we are interacting with, and the functions of the standards that currently govern our conduct. In other words, we can ask, “what motives best explain the agreements we see around us” in cases where the motives themselves are not perfectly known (Fig. 4).
In addition to (A) using one’s knowledge of agents’ preferences and environmental constraints to model the mutually beneficial agreements they would come to, one can also (B) invert this model to infer agents’ preferences and environmental constraints from the agreements they have reached, including the welfare and action standards that they employ.
5.2.1. Inferences about the environment and the utilities of agents
Imagine, for a moment, that you are a traveler to a small foreign village. You learn that in this village people are allowed to take as much water as they want from a nearby well, but each person is allowed to take only one bucket of water each week from a distant well, unless they are pregnant or sick, in which case they can take a bucket each day. From this information alone, you can reasonably infer certain things about the environment and peoples’ preferences in the village. It would be reasonable to infer that (1) the water from nearby well is plentiful; (2) the water from the distant well is a limited resource; and (3) the water from the distant well is more valuable than the water from the nearby well, and perhaps (4) it is less prone to pathogens, in particular. These sorts of inferences come naturally to us, but it is worth reflecting on why. Presumably it is because we have an understanding of the generative process by which standards are constructed: Namely, they are constructed when people come to agreement on binding constraints in order to achieve mutual benefit. Thus, we can ask ourselves, “which facts about the environment and people’s preferences would best explain why people agreed to these standards?”
Although its details are fanciful, the basic structure of this case is ubiquitous. We acquire many moral standards by social learning: We are either taught the standards, or we infer them by observation (Nichols, Reference Nichols2021). In either case, the standards become apparent to us often without any discussion of the underlying principles that establish or justify them. A young child might observe, for instance, that she is permitted to slurp soup in front of her disgusted sister, but she is absolutely forbidden from doing so in front of her impassive grandmother. From this she can infer that, despite her grandmothers impassivity, she really doesn’t like slurping noises. She can also infer that, in her family, grandma’s preferences about social niceties count for much more than her sister’s. Testing this prediction of our view is a critical avenue for future work.
Our view also predicts that performing these inferences should help us to better exercise moral flexibility in the future. Having inferred “most people don’t like slurping noises” and “grandma’s preferences count for a lot” one can draw better conclusions about whether to slurp soup in front of the rabbi, or prank grandma with a whoopee cushion. It can also help you identify permissible exceptions to the standard—for instance, it informs you that, when at grandma’s house, she can tell you, “it’s alright to slurp your soup”, but your sister cannot.
5.2.2. Inferences about the function of standards
Many standards have specific functions that can be stated with greater precision than just “achieve mutual benefit”. These specific functions allow us to substitute local, tractable problems for global, intractable ones. For instance, in many cultures there is a strong norm that one must wait in line for goods and services, such as coffee in a shop. Its function is to achieve a fair and predictable distribution of costs (waiting times) and benefits (coffee) across customers. Optimizing the rules of a line to balance costs and benefits is much easier than trying to figure out what arrangement leads to mutual benefit in a global, all-things-considered sense (Awad et al., Reference Awad, Levine, Loreggia, Mattei, Rahwan, Rossi, Talamadupula, Tenenbaum and Kleiman-Weiner2022; Kwon et al., Reference Kwon, Tenenbaum and Levine2022).
Crucially for our present purposes, our theory predicts that people should be able to infer those functions and, when they do, use them to guide their reasoning about exceptions and other forms of moral flexibility. Examples of this are everywhere in daily life. Returning to the example of lines in coffee shops, people are willing to consider certain exceptions to this norm. For instance, if you are given the wrong coffee, perhaps you can cut in line to replace it. There are other exceptions people will not grant, however: Just being “in a big hurry” does not permit you to skip to the head of the line. Why? This discrepancy is hard to explain in terms of “mutual benefit”, since skipping ahead might benefit both the hurried customer and the customer with the wrong order equally. It is easier to explain, however, in terms of the specific function of the line: To distribute costs (waiting) and benefits (coffee) equally. The hurried customer seeks to obtain a benefit without paying any cost, creating inequity, while the customer with the wrong coffee seeks to obtain the proper benefit for a cost they have already paid.
How do we discover the specific function of a moral standard? We propose that this can be accomplished by inverting a generative model of the process by which the standard was adopted. For instance, we know that it is wrong to charge customers twice as much for water on a hot day, scoot down the empty shoulder of a highway in a traffic jam, or scoop all the pennies out of the change jar by the checkout counter; yet, many people who know these things have never been explicitly told what function each standard serves. Still, we can make educated guesses by asking ourselves how they are designed to achieve mutual benefit in some local sense. Store-owners and clients mutually profit when prices are predictable; one person can scoot down a highway, but if everyone tried it nobody would be better off and emergency vehicles wouldn’t be able to get through, etc.
5.3. Summary
We have proposed that moral judgments can be rendered by a variety of processes that occupy distinct points on a continuum governed by principles of resource rationality. Although the representations at each level are distinct, nevertheless information can be exchanged among them. This can occur in two dimensions. One can use the bargaining process, or a cognitive model of it, to move “forwards” from a representation of specific agents, their environment, and their utilities towards cached standards. Or, one can invert a cognitive model of the bargaining process in order to infer properties of agents, their environment, their utilities, and so forth, based on the cached standards and specific judgments that one observes. This is an application of the general principle of “representational exchange” (Cushman, Reference Cushman2020; Vélez et al., Reference Vélez, Wu and Cushman2022) to the moral domain.
6. Contractualist frontiers
What, in principle, can we bargain over? The trichotomy of instance-based, welfare-based and action standards provides an appealing organizational scheme because it aligns with major areas of philosophical and psychological inquiry: contractualism, consequentialism, and deontology. It is by no means, however, an exhaustive set. Several other potential targets of negotiation suggest promising avenues for further study.
Bargaining over virtues. People do not just evaluate acts, consequences, and agreements, but also other people. Put another way, social agents make decisions knowing that others are keeping track of their reputation as a cooperative partner (Andre et al, Reference André, Debove, Fitouchi and Baumard2022). In philosophy this is studied under the rubric of virtue ethics (see Hursthouse & Pettigrove (Reference Hursthouse, Pettigrove, Zalta and Nodelman2022) for an overview), and in psychology there is also a rich tradition of thinking about person- or character-based moral judgment (Merritt et al., Reference Merritt, Doris, Harman and Doris2010; Miller, Reference Miller2014). A potential topic of negotiation, then, is which virtues count and how much. For instance, what counts as being a good cooperator (Andre et al., Reference André, Debove, Fitouchi and Baumard2022)? Should a good cooperator be fiercely loyal to those in their in-group, or generous to all regardless of group status (Enke, Reference Enke2019; Graham et al., Reference Graham, Nosek, Haidt, Iyer, Koleva and Ditto2011)? Similarly, how should a person (e.g., in a public position) balance the virtues of impartiality and integrity against family loyalty or kindness to a deserving individual (Dungan et al., Reference Dungan, Waytz and Young2015; McManus et al., Reference McManus, Kleiman-Weiner and Young2020)? The definition and weighting of various virtues may be negotiated and renegotiated across cultures.
Bargaining over beliefs. People disagree not just about values, but also about facts. In cases of factual disagreement, we use argument and persuasion to attempt to change each others’ minds. Now, of course, we can’t trade beliefs as we can goods—e.g., one person cannot give up one belief if the other agrees to relinquish another. But we often do negotiate a common understanding of a situation for the purposes of future negotiation. This is one function of courts: they decide whether a person will be treated as guilty or innocent henceforth by the state and society at large, largely overriding any personal beliefs. That is, we collectively agree that we should treat the person as if innocent or guilty, irrespective our own opinions (of course, we might challenge the court’s decision). Similarly, when a person is “declared the winner” of a literary competition, after intense negotiation in a judging panel, there is a collective agreement concerning the result. To a limited extent, companies, professions, and governments also have “received views” that dictate how their members must act and decide, independent of their personal opinions.
There are limits to this analogy. One may not be able to directly apply game theoretic concepts like a bargaining solution or equilibrium analysis—concepts defined over the payoff matrices of multiplayer games—to the dialectical process (Rubinstein, Reference Rubinstein2000, but see). And, whereas there is no particular challenge in agreeing to any ad hoc exchange of goods or services, there is a distinctive challenge to agreeing to believe something. Thus, one may not be able to simply say, “Fine, since you insist, I will simply go ahead and believe that the sky is green”, and then make good on the promise. But one might be able to say, when negotiating an insurance claim “OK, let’s work on the basis that all crops were wiped out by the flood” even if one doubts this is quite correct; or in a police investigation we might implicitly agree to work on the assumption that a victim of theft legitimately owned the now-stolen artworks, whatever one’s secret suspicions. The potential for “epistemic” rather than “deontic” negotiation deserves further investigation.
Bargaining in social groups and institutions. This paper has largely been concerned with how contractualism explains the moral cognition of dyadic and small-group interactions. However, there are several reasons to think that contractualism can potentially also explain aspects of how people navigate larger social groups and institutions and how those groups interact with one another. First, formal agreement-based processes are ubiquitous in democratic governance, from legislatures determining the policies that should guide conduct in a society to juries arguing to reach verdicts in trials. Second, private institutions also implement formal agreement-based methods for self-governance (Hadfield & Weingast, Reference Hadfield and Weingast2014). Third, intergroup relations may be characterized by similar bargaining principles as are interpersonal relations. Recent work emphasizes that conflict and cooperation between social groups are dynamic, with shifting alliances over time (Cikara, Reference Cikara2021). Theories of cooperative games offer a powerful tool to predict and explain coalitional behavior (Ray, Reference Ray2007). Yet, while it seems likely that the logic of agreement can contribute much to any analysis of institutions and social groups, it remains an open question when, and to what extent, its value here arises from a contractualist moral psychology operating in individual minds. By analogy, some aspects of economic activity are very usefully analyzed through the lens of individual cognitive mechanisms for value-guided decision-making (e.g., consumer behavior), while other aspects are less so (e.g., the monetary policy of the Federal Reserve). Presumably there are also some cases in which inter-group and institutional decision-making is usefully understood through the lens of our individual cognitive mechanisms for resource-rational contractualism, and other cases in which it is not.
The boundaries of a bargain. A contractualist approach to moral judgment immediately raises the question “Who is in on the bargain?” It is difficult to specify exactly how this boundary should be drawn. For instance, when parents decide who will pick up the kids, are they the only ones who get a say—or do the kids? When one country sets its carbon emissions policies, do other countries get a say? And so forth. It remains to future work to explore how people resolve this problem.
Moreover, we sometimes feel moral commitments to people beyond what a contractualist approach would seem to demand. We might feel obligations to strangers who we will never interact with or need to reach agreement with— distant victims of a crisis, for instance. And, we might feel obligations to provide people with more than they would have bargained for—to share resources equitably, for instance, even when we hold an advantaged position. How should we make sense of these cases?
Some cases of this kind may simply lay beyond the scope of our account. There are clearly some elements of our moral psychology that are best explained by principles other than contractualism. One example is our moral commitment to our own children. The most obvious explanation is grounded in the logic of kin selection, not rational agreement. After all, who has less “bargaining power” than an infant? (See also, §2.3.)
On the other hand, there are other ways our account might be extended to cover certain cases of this kind. First, if people generalize agreements (whether ad hoc, welfare-based, or rule-based) to novel contexts as a resource-rational heuristic, a natural consequence is that they may overextend moral concern to others where it is not warranted by the logic of bargaining. (Other times, of course, they may undershoot.) For example, a shop owner may treat a one-time out-of-town patron with the same courtesy he extends to repeat customers because he generalizes from the welfare standard that typically applies to a usual circumstance. Second, there may be important reputational benefits to overextending favorable treatment to those who cannot bargain for it, in order to signal one’s favorable qualities as a social partner to others who can. This logic has been very successfully developed elsewhere (André et al., Reference André, Debove, Fitouchi and Baumard2022; Barclay & Willer, Reference Barclay and Willer2007). It may help to explain why contemporary large-scale societies, in which there is a robust market for productive positive-sum social relationships via market transactions, are characterized by especially high levels of moral concern for socially distant others (Enke, Reference Enke2019; Henrich et al., Reference Henrich, Ensminger, McElreath, Barr, Barrett, Bolyanatz, Cardenas, Gurven, Gwako and Henrich2010).
Finally, following the contractualist tradition in political philosophy, it is possible that such feelings of obligation can partially be explained as emerging from an agreement that is made under certain normative conditions. Rawls, for instance, famously suggested that moral principles arise from negotiations that take place behind “a veil of ignorance”, a state that obscures each person’s specific circumstances from themselves (Rawls, Reference Rawls1971). Future work should investigate the potential role of veil-of-ignorance reasoning in a contractualist moral psychology (Huang et al., Reference Huang, Greene and Bazerman2019, see also Weidinger et al., Reference Weidinger, McKee, Everett, Huang, Zhu, Chadwick, Summerfield and Gabriel2023).
7. Conclusion
Tucked into the pages of his magnum opus, and penned in twilight of his life, Parfit offers an arresting image (Parfit, Reference Parfit2011). Like mountaineers, moral philosophers labor to ever higher positions of insight. Each asserts that their own path is best. But it is hard to see any standard by which they could judge one path better than another. From this discord, however, Parfit offers a hopeful view. Perhaps the climbers’ paths share one essential feature: They all point up the same mountain. As each path is pursued to its logical conclusion, Parfit suggests, it will be discovered that they all converge at a common summit. From this he draws a lesson for moral philosophy: That the surest way forward for each moral theory is to discover its points of convergence with the others, rather than to dispute their discrepancies.
Parfit’s vision was inspired by the fragmentary and conflicting state of moral philosophy, but the discord he describes applies equally to the prevailing picture of our moral psychology. According to this prevailing view, the moral mind is composed of fragmented and warring mechanisms vying for control over our actions and decisions. Perhaps, though, there a more abstract vantage point from which the many distinct and discordant capacities for moral judgment can be viewed as paths towards the same goal. In this paper we have suggested that there is.
There is wide-spread agreement across numerous fields that the goal of morality is to help us get along with each other. Humans face the challenge of interdependent choice: each person must figure out how to act given all the ways that their actions are intertwined with those of others. We aim for solutions to these problems that achieve mutual benefit. The most accurate way to find mutual benefit will often be actual negotiation between the affected parties, where all relevant information is discussed on a case-by-case basis. But humans have limitations—our time, effort, and computational power needs to be used wisely. We therefore draw on our species’ unique strengths to offset these constraints to make rational use of our limited resources.
Human moral judgment is governed by the logic of resource rationality along two principle axes. First, we not only bargain over one case at a time, but also over general standards that will apply in the future. Bargains can therefore concern how to manage welfare trade-offs, or which actions are permissible or impermissible. Second, bargaining is not limited to actual face-to-face negotiation. Instead, we can imagine and extend the outcomes of negotiations in a model-based or model-free way.
We can now see how the seemingly fragmentary moral mind is, in fact, elegantly unified. Distinct moral mechanisms can be seen as resource-rational approximations of bargaining solutions to multi-agent decision problems. Thus, morality reflects the ways in which humans are both sophisticated and limited. We have tremendous powers to understand others’ minds, make good decisions, build world models, infer latent structure from sparse data, and craft useful abstractions and rules. Yet our computational resources are limited. Thus, the restrictions we accept on our behavior in order to navigate the social world reflect not only the kind of world that we live, but also the kind of minds that we have.
Acknowledgements
The authors gratefully acknowledge the feedback they received on the ideas presented in this paper from the following individuals: Jean-Baptist André, Nicolas Baumard, Yejin Choi, Joshua Cohen, Molly Crockett, Léo Fitouchi, Ori Friedman, Josh Greene, Gillian Hadfield, Max Kleiman-Weiner, Chris Kutz, Arthur Le Pargneux, John Mikhail, Jonathan Phillips, Iyad Rahwan, Rebecca Saxe, Walter Sinnott-Armstrong, Tan Zhi-Xuan. Likewise, the following groups provided invaluable feedback over the years in which these ideas were developed: Evolution and Social Cognition group in the Department of Cognitive Sciences at ENS - Paris, the Behavioural Science Group at the Warwick Business School, the Moral Psychology Research Lab at Harvard, the Computational Cognitive Science Lab at MIT, and the Social Heckling and Manuscript Editing (SHaME) writing group (East-coast chapter). Murphy was a constant source of encouragement during the writing of this manuscript. Finally, the authors thank Gottlob Frege for his continual inspiration.
Funding
This project was supported by grants from the Templeton World Charity Foundation and a US Army Research Office MURI grant (W911NF-19-1-0217) to J.B.T and by grant N00014-22-1-2205 from the Office of Naval Research and grant 61061 from the John Templeton Foundation to F.C. N.C. gratefully acknowledge support from the European Union/UKRI under Horizon Europe Programme Grant Agreement no. 101120763 – TANGO. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. For the purpose of open access, the author has applied a Creative Commons Attribution (CC-BY) licence to any Author Accepted Manuscript version arising from this submission.
Competing Interests
The authors declare none.
Target article
Resource-rational contractualism: A triple theory of moral cognition
Related commentaries (27)
Altruism and normative bargaining
Animals, future generations, and resource-rational contractualism
Contractarian partiality, contractualist impartiality, and the question of falsifiability
Contractualist moral cognition needs a legalese-of-thought
Culturally-evolved joint programs as a bridge between social norms and in-the-moment reasoning
Delegation to legitimate authority as a resource rational mechanism
Fair contracts for artificial intelligence?
Institutions as cached computation for resource-rational negotiation
Justice, egocentrism, and resource-rational contractualism
Minds, morality, and the red hand rule: facilitating resource-rational agreement through policy
Moral cognition is contractualist, but does not work by simulating a bargaining process
Moral decision-making entails negotiation over the psychological mechanisms underlying decisions
Moral flexibility without mutual benefits: From change to disagreement
Morality is more than agreement: Negotiating justice, rights, and social inequalities
Navigating moral cognition under uncertainty
Non-negotiable problems for a negotiation framework of morality
Outsourcing moral cognition: Delegation, diffusion of responsibility, and coalitional exclusion
Rational contractualist solutions are not perceived as moral
Reasonable social cognition
Resource-rational contractualism can also explain personal moral judgments
The challenge of moral hierarchy in resource-rational contractualism
The difference between street parking and slavery
The missing moral content in contractarian models
The third axis: partner choice
The veil and the deal: Bargaining between case-specific solutions and unknown rules
What is morality for? Contractualism versus global consequentialism and expected choiceworthiness
When consent isn’t enough: explaining third-party disapproval of consensual decisions within the contractualist framework
Author response
Agreements and disagreements with resource-rational contractualism