Levine et al. argue that human moral cognition can be understood as resource-rational contractualism: moral reasoning arises from cognitively efficient heuristics approximating an idealized bargaining process in pairs and small groups. One limitation of resource-rational contractualism, however, is the lack of attention to a broader institutional environment, which may explicitly assist with the resource limitations of computationally finite agents – humans or machines. Society-level institutions like community norms and societal laws accelerate computation by functioning as pre-computed and enforceable heuristics to dramatically increase the predictability of reasonable expectations held by contractual partners. Institutions evolved through processes of design (e.g., social contracts), mutation (e.g., imperfect copying), cross-over (e.g., institutional mixing), and selection (e.g., community competition) (Padgett & Powell, Reference Padgett and Powell2012). They are consistent with the resource-rational vision insofar as they function as computational shortcuts: they encode, repurpose, and generalize societal agreements reached historically through extensive, resource-intensive negotiations. Once established, such norms and laws eliminate the need for repeated bargaining over similar circumstances, significantly conserving cognitive resources. However, precisely because institutions represent cached computations, they may become rigid, path-dependent, or insensitive to changing societal interests – diverging from the idealized, dynamically negotiated solutions emphasized by resource-rational contractualism. This may further create novel opportunities to negotiate workarounds that compensate for legal limitations. We argue that higher-order institutions represent a natural extension to the “resource rational” vision promoted by the authors, but rather than heuristics held by individual agents, codes, rules, and laws form the basis of shared computations pre-computed in prior negotiations made accessible to the collective.

Consider, for a moment, a simulation of resource-rational contractualism using large language model (LLM)-based “agents” in place of negotiating persons. LLMs have increasingly been deployed as chatbots, schooled with human interests and values, and employed as computational proxies for simulating moral cognition, potentially enabling in silico implementation of the virtual bargaining processes described by Levine et al. (Dillion et al., Reference Dillion, Mondal, Tandon and Gray2025; Ji et al., Reference Ji, Chen, Jin, Xu, Hua and Zhang2024). While LLM agents natively manifest and can be steered or fine-tuned to exhibit specified morals, they naturally face many limitations relative to the contractual ideal. Obviously, LLMs lack authentic stakes or genuine self-interests, key ingredients in moral negotiations. Their incentive structure, centered on predicting plausible continuations of textual patterns (McCoy et al., Reference McCoy, Yao, Friedman, Hardy and Griffiths2024), obviously differs from authentic human bargaining. Nevertheless, they are being increasingly trained and reinforced to present reasonable interests. Like humans, LLMs suffer from cognitive resource constraints (e.g., finite context windows, inference costs). And while LLMs fail to engage in some of the social processes central to norm acquisition, they do demonstrate social learning and a number of cognitive biases and heuristics exhibited by human agents (Piao et al., Reference Piao, Lu, Gao, Xu, Santos, Li and Evans2025; Sarkar et al., Reference Sarkar, Muresanu, Blair, Sharma, Trivedi and Hadfield2024; Savarimuthu, Ranathunga, & Cranefield, Reference Savarimuthu, Ranathunga and Cranefield2024; Xie et al., Reference Xie, Chen, Jia, Ye, Lai, Shu, Gu, Bibi, Hu, Jurgens, Evans, Torr, Ghanem and Li2024). Humans internalize norms dynamically through repeated interactions, shared histories, and reciprocal social learning, and LLMs manifest comparable learning capacities. Recent work has explored the degree to which LLMs exhibit “theory of mind” where they implicitly model the needs and expectations of other agents (Kosinski, Reference Kosinski2023; Li et al., Reference Li, Chong, Stepputtis, Campbell, Hughes, Lewis and Sycara2023), capacities required to engage in iterative, multi-step bargaining reasoning. In short, while LLMs’ current capabilities remain insufficient for robustly simulating authentic contractualist reasoning as envisaged by Levine et al., the resource-rational approach suggests a series of benchmarks against which AI agents should perform effectively before being allowed to negotiate and enter contracts with legal authority (Almeida et al., Reference Almeida, Nunes, Engelmann, Wiegmann and de Araújo2023; Dillion et al., Reference Dillion, Mondal, Tandon and Gray2025).

Nevertheless, the task of plausible LLM negotiation becomes dramatically easier – the high-dimensional design space of possible negotiated outcomes collapses – by simply “constitution” training such models on the laws, values, and norms, they may assume other agents will honor (Bai et al., Reference Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, Goldie, Mirhoseini, McKinnon, Chen, Olsson, Olah, Hernandez, Drain, Ganguli, Li, Tran-Johnson, Perez and Kaplan2022). This would reduce the reasonable expectation of adverse behavior by their negotiation partners and decrease the “degrees of freedom” associated with their negotiation (Lai et al., Reference Lai, Potter, Kim, Zhuang, Song and Evans2024). Shared institutions function as public goods to increase the predictability of shared norms. But they provide more than this. Laws and customs not only specify general proscriptions (e.g., “thou shalt not…”), but may also precompute penalties that improve the prediction of negotiated outcomes and enable generalization to other avenues of possible agreement.

This gedanken experiment highlights how psychological heuristics should be paired with collective heuristics in the form of institutions, norms, and laws in order to achieve the resource-rational vision of the authors by reducing the design space of plausibly computable negotiated outcomes.