2.1. Dissonance and consonance

Rationalization is usually explained by positing a desire for consonance between thought and action. The simplest model of this kind, balance theory (Heider 1958/Reference Heider2013), posits that people want to achieve harmony among their attitudes. Thus, if you like your spouse and your spouse likes guacamole, you will tend to acquire a taste for guacamole (or, in theory, a distaste for your spouse). Unfortunately, this model's generality is also its Achilles’ heel: It is easy to come up with compelling counterexamples and hard for the theory to explain them away. As Festinger (Reference Festinger, Harmon-Jones and Mills1999) quipped, “I like chicken, chickens like chicken food, and I don't like chicken food.”

Festinger’s (Reference Festinger1962) and Festinger et al.’s (Reference Festinger, Riecken and Schacter1956) own theory of rationalization, cognitive dissonance, was therefore more specific. It posits that only certain sets of beliefs, desires, and actions can occupy states of consonance or dissonance. Although Festinger did not describe the principle of rational action as such, he grasped it intuitively:Footnote ² People tend to act in a way that maximizes their desires, consistent with their beliefs. Sets of beliefs, desires, and actions that fit this specific principle are consonant; sets that do not are dissonant.Footnote ³ Crucially, Festinger also proposed that the state of dissonance is psychologically aversive, motivating people to achieve consonance. If you have already acted (e.g., shooting a man), of course, it is too late to adjust your action. Instead, consonance must be achieved by adjusting your beliefs or desires (e.g., deciding he must have been a threat). This is the essence of rationalization.

Cognitive dissonance is the best-known psychological theory of rationalization; indeed, it is among the best-known psychological theories of anything. Its two main pillars enjoy strong empirical support. First, people change their preferences and beliefs to match actions they have already performed. Specifically, they adopt the beliefs and desires that would have made their past action rational. A classic method that produces such effects, the free choice paradigm (Brehm Reference Brehm1956), is still widely used (e.g., Sharot et al. Reference Sharot, De Martino and Dolan2009; Reference Sharot, Velasquez and Dolan2010; Vinckier et al. Reference Vinckier, Rigoux, Kurniawan, Hu, Bourgeois-Gironde, Daunizeau and Pessiglione2019). People are given a choice between two things, such as a toaster and a radio, that they value roughly equally. The act of choosing one of these things causes them to value that thing more and the other thing less. This occurs from an early age and in non-human primates (Egan et al. Reference Egan, Santos and Bloom2007), even when people cannot remember what they chose (Lieberman et al. Reference Lieberman, Ochsner, Gilbert and Schacter2001), and even when the experimenter tricks them into believing they chose something they didn't (Sharot et al. Reference Sharot, Velasquez and Dolan2010). People's justifications for such choices can be detailed and elaborate (Johansson et al. Reference Johansson, Hall, Sikström and Olsson2005). Second, experiments confirm that dissonance is psychologically aversive: People say so (Elliot & Devine Reference Elliot and Devine1994), and it is also revealed by convert measures such as affect misattribution (Losch & Cacioppo Reference Losch and Cacioppo1990; Zanna & Cooper Reference Zanna and Cooper1974), psychophysiology (Losch & Cacioppo Reference Losch and Cacioppo1990; Harmon-Jones et al. Reference Harmon-Jones, Brehm, Greenberg, Simon and Nelson1996), and functional neuroimaging (van Veen et al. Reference van Veen, Krug, Schooler and Carter2009). In sum, the theory is well known for a good reason.

Yet, while the theory of cognitive dissonance describes the psychological mechanisms involved in rationalization, it does not offer an ultimate explanation – an answer to the question, “Why did it evolve?” It is akin to a theory that explains why we eat by saying, “We are motivated by hunger.” Although true and important, such a theory is incomplete: It does not explain why the hunger motive evolved (i.e., because food provides the raw material for metabolism, and that therefore organisms that possess the hunger motive tend to outcompete those that lack it).

Festinger was merely the first in a long line of theorists to explain rationalization while eliding its adaptive function (Harmon-Jones & Mills Reference Harmon-Jones and Mills1999): Aronson (Reference Aronson, Abelson, Aronson, McGuire, Newcomb, Rosenberg and Tannenbaum1968) proposed that dissonance arises most powerfully when actions are incongruent with a person's self-concept; Steele (Reference Steele1988; Steele et al. Reference Steele, Spencer and Lynch1993) proposed that dissonance is aversive because people feel that apparently irrational actions threaten their self-image or self-worth; Beggan (Reference Beggan1992) extended this concept to the objects people possess, even when not freely chosen. There are still other possibilities: Perhaps dissonance minimizes post-decisional regret.Footnote ⁴ Each of these proposals elaborates on Festinger's mechanistic account, but they do not offer an ultimate, adaptive explanation for rationalization.

Indeed, among the classic approaches, the descendants of Heider's balance theory come closest to an adaptive rationale. Balance among beliefs and attitudes can be formalized as a form of logical consistency or constraint satisfaction (Read & Marcus-Newhall Reference Read and Marcus-Newhall1993; Shultz & Lepper Reference Shultz and Lepper1999; Thagard Reference Thagard1989). If a person represents that Socrates is a man, and that all men are mortal, but that Socrates is immortal, something must give. A well-designed system will repair such inconsistencies in a manner that makes its representations more accurate (Ackley et al. Reference Ackley, Hinton and Sejnowski1985; see also a review by Gawronski et al. Reference Gawronski, Rydell, De Houwer, Brannon, Ye, Vervliet, Hu and Olson2018).Footnote ⁵ This idea of network repair or cognitive consistency can be fruitfully applied to networks of interrelated beliefs or desires (Cushman & Paul, Reference Cushman, Paul, Doris and Vargasin press).

But how could it be applied to classic cases of rationalization, in which a person revises their beliefs and desires to match their own past action? Suppose that you desire cookies and believe them to be in the kitchen, but go to the living room. Clearly, one adaptive response to this imbalance is to correct your action: To go to the kitchen. This applies the principle of rational action. But now consider the outcome of rationalization: You either decide you didn't want cookies in the first place or convince yourself that they are actually in the living room. Although this achieves coherence of a kind, it certainly does not improve your desires or beliefs. Rather, it takes take a clear error of reasoning and then multiplies it, infecting thought with a pathology of choice.

Rationalization would indeed be counterproductive if our actions were only produced by sound reasons or outright errors. In this case, dissonance would only arise in cases of error – after all, sound reasoning cannot produce actions that violate principles of rationality. And, if rationalization only applied to errors, then it would propagate those errors from action back to our beliefs and desires – a counterproductive result.

Crucially, however, our behavior is influenced by sources other than reason and error. Whether innately, through habit learning, or through cultural learning, our behavior is influenced by processes that are adaptive and yet non-rational. Because these processes share the same ultimate purpose of reasoning – fitness maximization – the beliefs and desires that support reasoning can be improved by learning from the behavioral influence of non-rational sources.

2.2. Self-perception

Bem’s (Reference Bem1967) theory of self-perception provides quite a different explanation for rationalization. It denies the two mechanistic pillars of cognitive dissonance: First, that dissonance is psychologically aversive and, second, that underlying desires or beliefs actually change. Rather, it posits that people only change their perceived beliefs and desires.

Bem's original statement of this theory was heavily influenced by behaviorism in two ways. First, he assumed that we have no direct introspective access to the mechanisms that produce our behaviors. Second, he claimed that we receive strong reinforcement from others when we can explain our behavior in terms of mental states, and consequently, we often construct such mentalistic explanations. Desiring mentalistic explanations for our behavior, but lacking introspective access, we resort to our best guess: post hoc rationalization.

Although intended as an alternative to cognitive dissonance, self-perception theory stands in its own right as a powerful statement: People are unaware of the causes of their behavior and often attempt to infer these causes by observing their actions. These basic claims are supported by a wealth of experimental research (Devine Reference Devine1989; Gazzaniga Reference Gazzaniga1967; Greenwald & Banaji Reference Greenwald and Banaji1995; Haidt Reference Haidt2001; Miller & Buckhout Reference Miller and Buckhout1962/1973; Neisser 1967/Reference Neisser2014; Nisbett & Wilson Reference Nisbett and Wilson1977; Wilson Reference Wilson2004). Bem's (Reference Bem1967) statement of this theory may have been influenced by seminal work on emotion misattribution (Schachter & Singer Reference Schachter and Singer1962) and confabulation in split brain patients (Gazzaniga Reference Gazzaniga1967). Nisbett and Wilson (Reference Nisbett and Wilson1977) later condensed these varied insights into three core claims: People are often unaware of the causes of their behavior; self-reports of the causes of behavior are generated by folk-causal theories; and, therefore, when they correctly report the cause of their behavior, it is usually the result of inference, not introspection.

In sum, there is undeniably something right about the theory of self-perception. Yet, when aimed at the topic of rationalization, it misses two key empirical marks. First, dissonance induces an aversive psychological state that motivates rationalization. Second – and of greatest importance – rationalization changes people's actual beliefs and desires, not just their self-perception. For instance, in the free choice paradigm, objects are not just reported to have higher value after being chosen (or lower after rejection), but they are also actually chosen more often in the future (or less often, after rejected).

Like Festinger, Bem did not address what ultimate adaptive purpose might be fulfilled by cognitive dissonance. Instead, he focused on its proximate psychological motivation: The reinforcement of social partners who demand mentalistic explanations of our behaviors. Neither did he squarely address the issue of whether self-perception is usually accurate or inaccurate. In the grip of behaviorism, Bem may have regarded this as beside the point – he was likely skeptical that our behavior relies on structured mental representations at all. Rather, the essence of Bem's claim was simply that we attribute mental states to ourselves by the same processes that we attribute mental states to others.

The present account builds on important insights of self-perception theories (Bem's, and those that followed it), but with two crucial modifications. First, it posits that the function of self “perception” is not merely to satisfy our own curiosity or that of our peers. It also constructs new beliefs and desires based on information implicit in other adaptive control mechanisms. Thus, “perception” is a misnomer: Rationalization is designed not to accurately infer unconscious mental states, but to construct new ones; it is not a discovery, but a fiction. Second, we can, should, and do actually adjust our beliefs and desires to match this fiction. This is adaptive because reasoning and non-rational processes are ultimately trying to maximize the same goal: biological fitness. In other words, rationalization is a fiction, but a decidedly useful one. Mixed right, it can be nourishing to drink your own Kool-Aid.

2.3. Responsibility avoidance and impression management

Finally, some theories posit that rationalization is designed not so much to inform others as to persuade them, casting your behavior (or other information) in favorable and possibly deceptive light (Tedeschi et al. Reference Tedeschi, Schlenker and Bonoma1971; von Hippel & Trivers Reference Von Hippel and Trivers2011). When risking blame, for instance, we may profess benign motives or faultless naiveté, even at the expense of the truth.

Mercier and Sperber (Reference Mercier and Sperber2011) go so far as to claim that reasoning itself is principally adapted to the problem of changing others’ minds, and therefore they interpret rationalization as an adaptive solution to the problem of winning arguments (see also Haidt Reference Haidt2001; Tetlock Reference Tetlock2002). Indeed, on their view, reasoning itself is mostly an instance of rationalization. Its goal is to present information to another person in a manner that compels them, by logic or intuition, to accept your conclusion (see also Haidt Reference Haidt2001).

At first blush, such theories seem to explain only why we express rationalization to others, but not why we adjust our own beliefs or desires (Tedeschi et al. Reference Tedeschi, Schlenker and Bonoma1971). Yet, it is also plausible that “true believers” are better deceivers. In other words, the best way to convince others that you shot an (innocent) man for good reason might be to first convince yourself of his guilt (Trivers Reference Trivers2000; von Hippel & Trivers Reference Von Hippel and Trivers2011).

This family of theories likely explains a part of the function of rationalization. If the present account also explains a part, then it is a complementary but largely independent explanation.

3.1. Rational action as planning

Before considering how instincts, norms, and habits can improve reasoning, we must have a clearer image of how reasoning itself works. Reasoning, sometimes called planning, chooses actions by expected value maximization (Fig. 1a). A simplified model of planning has three parts. First, there is a mechanism for learning a causal model of the world, one's beliefs. This model allows you to predict what is likely to occur in different situations, depending in part on your own actions. Second, there is a mechanism that assigns intrinsic value to certain outcomes, one's desires (also sometimes described as reinforcement or reward). Third, there is a mechanism that chooses actions by maximizing the satisfaction of your desires, given your beliefs. Our next goal is to understand how a system designed this way could extract useful information from instincts, norms, and habits.

3.2. Instincts

Instincts are innate influences on behavior, designed by natural selection, that bias certain actions to be performed in the presence of certain stimuli.Footnote ⁶ For instance, humans instinctively drink when they are thirsty, flee from threats or fight them, reject likely pathogens, fall in love with other humans, and so on. Many of these examples involve very abstract actions (flee) or stimuli (threat). Instincts need not be low-level or concrete, or grounded in a single, well-defined neural mechanism. Rather, instinct often describes a very abstract kind of innate mental organization. Its key property is just that the relationship between the stimuli and the actions is innate and direct. For instance, the perception of a threat may directly bias action toward flight. This is what makes instincts different from planning, which would instead require a computation like “fleeing avoids threats, threats might harm me, and I don't like being harmed.”

Because instincts are shaped by natural selection, they tend to increase our biological fitness. Similarly, rational planning is designed to increase biological fitness. This is an important part of why rationalization makes sense: It extracts information from one adaptive system (instinct) and makes it available to another (rational planning). If a person's instinct is incongruent with her beliefs and desires, adjusting those beliefs and desires to match her action may ultimately improve them.

For instance, suppose that a person instinctively recoils from snakes. This instinct is adaptive because many snakes are venomous, but she happens to be unaware of this. Rationalizing her instinct (i.e., attempting to explain her act of recoiling in terms of beliefs and desires), she might adopt the belief that snakes are dangerous. Similarly, she could rationalize her behavior by adopting a general desire to be far from snakes, and this is a useful desire. In either case, the outcomes of her future reasoning are improved.

3.3. Rationalization as a form of rational inference

Even if rationalization could possibly generate true beliefs and useful desires, what guarantees that it would do so typically? The specifics of the snake case are suspiciously convenient; this naïf might have concluded instead that snakes breathe fire, shoot crossbows, or dredge up hurtful memories of adolescence. Such beliefs would explain one's instinctive recoiling, but they are false. Or she might have adopted the desire to avoid all animals or all things that are long and straight. What processes could ensure that the rationalized beliefs and desires are, in fact, useful ones?

The construction of new beliefs and desires should presumably be structured as a form of rational inference. In Bayesian terms, the posterior beliefs about snakes (“snakes bite” vs. “snakes shoot crossbows”) should be sensitive not just to the likelihood of an action (recoiling) given a percept (snake) and candidate beliefs (e.g., “snakes bite” vs. “snakes shoot crossbows”), but also the prior probability of the candidate beliefs, including their compatibility with other beliefs (e.g., animals can't use crossbows; many long, straight things are perfectly safe).

Indeed, these pieces of information may be integrated in a rational manner, according to Bayes’ rule. This form of inference has been well characterized in models of inverse planning (Baker et al. Reference Baker, Saxe and Tenenbaum2009; Ng & Russell Reference Ng and Russell2000). This brings into focus an important dimension of the claim that rationalization is rational – it is not just biologically adaptive, but it may also approximate a well-understood form of rational inference. Importantly, however, the approximation of a rational inference (at Marr's computational level) may be quite cognitively simple (at Marr's algorithmic level). These relationships, between rational inference and the actual mechanisms of rationalization, are discussed more fully in section 4.

3.4. Norms

Human psychology is influenced not just by the biological inheritance of natural selection but also by a vast cultural inheritance. And just as biological natural selection ensures that instincts will typically be adaptive, cultural selection ensures that norms will typically be adaptive (Boyd et al. Reference Boyd, Richerson and Henrich2011), although maladaptations may arise in each case. Our cultural inheritance takes many forms: concepts (π), artifacts (knives), beliefs (the earth is round), desires (money), norms (drive right, pass left), and much more. The specific form of norms is often transmitted by social conformity. These operate analogously to instincts: just as instincts are innate biases on action, norm conformity may be defined as a set of socially learned biases on action.

As with instincts, norms may be very abstract. For instance, there are cultural norms of cooperation and fairness that generalize over many diverse features of specific cases. Norms may also be redundant with other kinds of cultural influence. For instance, somebody might comply with the Jewish laws of kashrut because (1) they wish to get along with their religious peers, or (2) they believe that God asks this of them, or (3) it just feels like the right thing to do. These are, in fact, independent and redundant elements of cultural learning. According to our restrictive definition of norms, only the third influence is sufficiently direct to count as a norm. The first two – a desire to get along and a belief about God – instead influence her behavior indirectly, by reasoning.

Given this homology between instincts and norms, the very same logic that makes instinct a useful target of rationalization therefore also applies to norms. When a person performs a behavior due to cultural influences, she may often be able to extract useful beliefs and desires by rationalizing her action.

Among the indigenous people of Fiji, for instance, it is taboo to eat certain kinds of seafood when pregnant or nursing (Henrich & Henrich Reference Henrich and Henrich2010). Most of the taboo seafoods are toxic and pose special risks to fetuses and infants, but the people of Fiji do not have precise knowledge of this – indeed, the taboo extends to several closely related seafoods that are actually harmless. Rather, most mothers avoid eating these foods simply because of the norm. Rationalization, however, might lead a mother to the correct belief that these fish are dangerous. Then, by reasoning, she may generalize other useful conclusions: don't even touch these fish, don't let your baby eat them, don't feed them to sick people, and so forth.

3.5. A “useful fiction”: How theory of mind supports social learning

Because norms are commonly transmitted via observation and imitation (Boyd et al. Reference Boyd, Richerson and Henrich2011; Cialdini & Goldstein Reference Cialdini and Goldstein2004), the benefits of rationalization one's own behavior can also be obtained by rationalizing others’ behaviors. Put more simply, there is a deep homology between rationalization and theory of mind (Fig. 3; see also Bem Reference Bem1967). This motivates a brief but important detour to consider the relationship between theory of mind, self-perception, and rationalization.

Figure 3.

Two parallel pathways of social learning: one characterized by theory of mind and the other characterized by rationalization.

Consider again the seafood taboos of Fiji. A mother might conclude that taboo seafood (suppose it is shellfish) is dangerous to infants by either of two paths: one via rationalization and another via theory of mind. Following the first path, she first complies with the norm itself, avoiding the shellfish simply because others do. Next, observing her own behavior, she rationalizes that shellfish must be dangerous. Following the second pathway, she instead first attempts to understand the behavior of her social partners. This act of mental state inference, or theory of mind, really just amounts to rationalization of others’ behavior. She concludes that her social partners must believe that shellfish are dangerous. Then, assuming that they know something she doesn't, she adopts this belief herself.

Each of these paths involves a crucial step in which a belief is extracted from an action. In the first path it is the observer's own action, so we call it rationalization; in the second path it is another person's action, so we call it theory of mind. (This connection, of course, originates with self-perception theory; Bem Reference Bem1967).

This clarifies two important but distinct functions of theory of mind. Many past treatments assume that theory of mind is designed to infer the true causes of another person's behavior (Baron-Cohen Reference Baron-Cohen1995; Dennett Reference Dennett1987; Gopnik et al. Reference Gopnik, Meltzoff and Bryant1997). Likewise, self-perception theory might be construed as an attempt to accurately infer the true cause of one's own behavior (although Bem himself was agnostic on this point). But we have emphasized an alternative function of rationalization: to construct representations that are implied by behavior (one's own, or another person's). This process of construction need not result in a perfectly accurate representation of the causes of behavior in order to be useful (a point made by Dennett [Reference Dennett1987] in introducing the intentional stanceFootnote ⁷). To the contrary, rationalization can extract useful beliefs and desires from the influence of non-rational systems – systems whose influence on behavior had nothing to do with those beliefs and desires. Rationalization is, in this sense, a useful fiction. It takes the form of inference, but with a very different function.

This same function can also apply to theory of mind. Theory of mind may often involve useful fictions, in which we ascribe inaccurate causes to others’ behavior – goal-directed plans, based on beliefs and desires – even when those behaviors were produced by non-rational processes.Footnote ⁸ Although inaccurate, such ascriptions could still extract useful information for us: true beliefs and adaptive desires.

This perspective has at least one attractive feature: Useful or not, theory of mind seems to involve a great deal of fiction. Despite widespread consensus that human behavior is not exclusively rational, nearly all studies of mental state inference posit a folk theory of rational action. Despite more than 40 years of study, there is virtually no research on folk theories of instinct, habit, reflex, and the like – in other words, a theory-of-the-rest-of-our-minds. Moreover, what little research exists suggests that people interpret others’ actions as the product of goal-directed reasoning far more than it actually is the cause (Gershman et al. Reference Gershman, Gerstenberg, Baker and Cushman2016). Similarly, experimental demonstrations of automatic behavior are often surprising to lay audiences, while experimental demonstrations of rational behavior are not. On the useful fiction model, this is because theory of mind is not only designed to infer the true causes of a person's behavior, but also to extract useful beliefs and desires from their behavior even when it was caused by non-rational processes. It is, therefore, biased to perceive all behavior as rational, even though much behavior is not.

3.6. Habits

Habits are a third major non-rational influence on behavior. Habits are learned stimulus-response mappings, often reinforced by reward and punishment (reviewed in Dolan & Dayan Reference Dolan and Dayan2013). For instance, a person might habitually flip the lights on when they walk into a room because it is typically useful (i.e., rewarding). Each time that the behavior is performed and rewarded, habit learning strengthens the mapping from stimulus to response. As a result, executing habitual action requires little cognitive effort. A person does not have to consider desires (“I need light”) and beliefs (“switches cause light”) to derive the value of performing an action; rather, the behavior is habitized based on its value in the past. But, for the same reason, habitual control can be inflexible. You might habitually switch the lights on even though you are walking into the room of a sleeping baby and want it to be dark. Rational planning, in contrast, can flexibly adjust to new or unusual circumstances.

Despite these differences, there are two key similarities between habit and reasoning: Both involve learning from direct experience, and both are sensitive to the same rewards. These similarities make it challenging to explain how the rationalization of habitual action could provide new information to a system of reasoning. The challenge has two parts: how to improve beliefs and how to improve desires.

3.7. Hybrid control

Although it is convenient to act as if certain actions are wholly under habitual control, others wholly instinctual, and so on, this is a caricature. Even the simplest targets of rationalization studied in the laboratory – the choice of a toaster over a radio, for instance – are not the product of pure instinct, habit, or norm compliance. Rather, they involve at least some degree of conscious, deliberative planning (“let's see, what could I do with a new toaster?”). More generally, it is disputed whether systems of habit, instinct, or norm adherence could be cleanly severed from reasoning at all (Dayan Reference Dayan2012; Graybiel Reference Graybiel2008; Kool et al. Reference Kool, Cushman, Gershman, Morris, Bornstein and Shenhav2018).

Yet, while reasoning often contributes to choice, it rarely operates alone. Rather, most behavior is the result of some form of approximate planning – an elaborate background of automatic and non-rational processes that construct a restricted and tractable decision space in which limited rational planning can effectively guide behavior (Cushman & Morris Reference Cushman and Morris2015; Dayan Reference Dayan2012; Gigerenzer & Selten Reference Gigerenzer and Selten2002; Huys et al. Reference Huys, Lally, Faulkner, Eshel, Seifritz, Gershman, Dayan and Roiser2015; Keramati et al. Reference Keramati, Smittenaar, Dolan and Dayan2016; Kool et al. Reference Kool, Cushman, Gershman, Morris, Bornstein and Shenhav2018). Instinct guides our minds away from rationally deliberating the possibility of marrying our siblings (Lieberman et al. Reference Lieberman, Tooby and Cosmides2007); habit guides our minds away from the possibility of making coffee by putting bread in the toaster (Morris & Cushman Reference Morris and Cushman2017); norms guide our minds away from the possibility of catching a ride to the airport by stealing a car (Phillips & Cushman Reference Phillips and Cushman2017). Non-rational processes also structure tractable planning by identifying valuable end states (Keramati et al. Reference Keramati, Smittenaar, Dolan and Dayan2016) or goals (Cushman & Morris Reference Cushman and Morris2015). A person may seek revenge instinctually, and yet plot his revenge by reasoning; he may seek cocaine habitually, and yet plan to get cocaine by reasoning; he may seek to divide his resources fairly due to blind norm adherence, but then reason carefully about how the fairest division could be accomplished.

Thus, even when our behavior is jointly determined by the influence of rational and non-rational processes, there is an opportunity for rationalization to extract useful information from the influence of non-rational processes and translate these into a form useful to the rational system.

3.8. Summary: Rationalization is rational

Instincts, norms, and habits shape our behavior in adaptive ways, but not by rational planning based on beliefs or desires. Still, these influences are adaptive: We instinctively recoil from precipices because they are dangerous; norm-based food taboos reflect real toxins, and habitually flipping a light switch is usually a good idea. Rationalization, then, is a useful fiction: When we observe our own behavior, we infer the beliefs and desires that would have been most likely to have caused that behavior, as if it had been an exclusive product of reasoning. Then we adopt those beliefs and desires. This is adaptive because, on average, the new beliefs are true and the new desires promote fitness. For the same reason, theory of mind might often entail a useful fiction as well: By assuming that others’ behaviors are rational (when they are merely adaptive), we can extract useful information. Crucially, whether rationalizing our own action or that of others, the process of inferring information from behavior is structured as a rational inference (Baker et al. Reference Baker, Saxe and Tenenbaum2009).

In sum, rationalization exchanges representations of Do this! for representations of the type Believe this! or Desire that! This is a kind of representational exchange: It extracts information implicit in the representations of non-rational systems and transforms it into the format useful to the rational system. The final section expands this view of representational exchange, showing how rationalization is one example of a much broader class of cognitive operations that facilitate the flow of information among distinct systems of behavioral control.

4.1. The structure and function of representational exchange

During rationalization, information flows from non-rational systems to rational ones. Could information flow in the opposite direction – from reason to other adaptive systems, or among the other systems themselves (Fig. 4)? Several examples come to mind. During habitization, choices that were effortful (i.e., rationally planned) become automatic (i.e., habitual). During norm internalization, actions that we observed others perform shape our intrinsic preferences. Although traditionally these processes have been considered unrelated, they are all forms of representational exchange: the sharing of information between distinct mechanisms of behavioral control.

Figure 4.

(a) Rationalization is one example of the more general process of (b) representational exchange.

Representational exchange is useful because distinct mechanisms of behavioral control have different ways of representing information and guiding action, each with unique advantages and disadvantages. For instance, habits enable rapid, computationally frugal decision making that is occasionally suboptimal, whereas planning attains greater optimality at the expense of time and effort. Representational exchange allows us to keep thought efficient – that is, to attain the most important opportunities for flexibility and generalization, subject to the resource constraint of a limited cognitive capacity. In this manner it fosters “resource-rational” cognition (Griffiths et al. Reference Griffiths, Lieder and Goodman2015), improving the overall welfare of the organism.

Consider a simple example. For most people, computing 26 + 52 takes moment of thought, while 25 + 25 comes easily to mind. This reflects two different cognitive organizations. One system encodes a procedure for addition and requires effort to derive specific sums (e.g., 26 and 52). Another system encodes a precompiled set of sums – roughly, a table in which one looks up the entry “25 + 25” and retrieves “50.” The first requires computation; the second merely requires retrieval.

Why do we have two such systems, and why are certain sums represented one way and other sums another? On the one hand, knowing the rules of addition is useful because it compresses an infinitely large mapping of inputs to outputs (i.e., arbitrary sets of numbers to their sums) via a compact rule. This requires far less memory than, for instance, storing a table of precomputed sums. Although effort is required to compute each sum, this is a worthwhile trade-off as compared with the storage demands of the tabular representation and the learning demands of acquiring it.

On the other hand, certain sums must be computed far more often than others. If you are a cashier who makes change every day, then you do store at least a small table of common sums: “nickel + 2 dimes = quarter,” “4 quarters = dollar,” and so on. These sums are required so frequently that it would be inefficient to compute them anew each time. Instead, it is worth storing a small cache of common sums for ready and quick retrieval.

Any resource-rational cognitive system must find efficient ways to represent information, managing the competing demands of computational effort, memory, accuracy, and flexibility (Batchelder & Alexander Reference Batchelder and Alexander2012; Griffiths et al. Reference Griffiths, Lieder and Goodman2015). We must choose whether to represent procedures or merely their outputs (Gershman et al. Reference Gershman, Markman and Otto2014; Sutton Reference Sutton1991). We must choose when to represent specifics and when, instead, to fall back on generalities (O'Donnell Reference O'Donnell2015). We must choose when to be exact and when to satisfied with an approximation (Daw et al. Reference Daw, Niv and Dayan2005; Gigerenzer & Selten Reference Gigerenzer and Selten2002; Kahneman Reference Kahneman2011b; Kool et al. Reference Kool, Gershman and Cushman2017).

It must be rare that we have attained the optimal balance at any given time – and just as rare that the optimal balance could ever be permanent. Rather, as we learn and change – and as our circumstances and the world around us change – there is a continual demand to adjust the format of the representations that guide our action. This requires mechanisms for representational exchange.

Some forms of representational exchange will pack information into compressed forms, storing outputs, abstractions, and heuristics in place of procedures, specifics, and computations. Other will perform the reverse operation, unpacking information by inferring the more detailed and precise information implicit in outputs, abstractions, and rules. Viewed from this perspective, rationalization is a particular kind of unpacking that occurs in the specific context of choice behavior: Specifically, it unpacks behavior into beliefs and desires. It belongs to a broader family of cognitive operations that facilitate representational exchange – not just from non-rational systems to rational ones, but also among the many systems that contribute to decision making.

Representational exchange can be situated within a broader taxonomy of operations demanded by a successful multisystem cognitive architecture:

1. 1. Control. What are the several mechanisms that guide our behavior? What representations and computations do they rely on, and what are the distinctive advantages and disadvantages of each? Much prior research addresses these questions (reviewed in Dolan & Dayan Reference Dolan and Dayan2013; Kahneman Reference Kahneman2011b; Sloman Reference Sloman1996; Squire Reference Squire2004).

2. 2. Metacontrol. Which system, or weighted combination, guides our behavior at any given time? In other words, from moment to moment, how do we decide whether to act habitually, rationally, by instinct, or another way? A growing body of contemporary research addresses this question (e.g., Kool et al. Reference Kool, Gershman and Cushman2017; Daw et al. Reference Daw, Niv and Dayan2005; Griffiths et al. Reference Griffiths, Lieder and Goodman2015; Shenhav et al. Reference Shenhav, Botvinick and Cohen2013).

3. 3. Exchange. What mechanisms enable the exchange of information between systems (Gershman et al. Reference Gershman, Markman and Otto2014; Lombrozo Reference Lombrozo2017)? For instance, how can a behavior formerly produced by reasoning become habitual, and how can a behavior that was formerly habitual influence subsequent reasoning? This is our present focus.

4. 4. Exchange control. How do we decide what, and when, to exchange? Assuming that representational exchange can be beneficial in the long run, but also carries immediate costs, how is cost-benefit analysis performed? And, when systems embody conflicting information, which system gets prioritized? These are important issues for further development, but they are not pursued here.

The next few sections present an account of representational exchange somewhat more formally, drawing connections to current computational models of decision making used in psychology, neuroscience, and computer science. Although the main focus is on representational exchange among decision-making systems, it is clear that the concept applies beyond the domain of decision making, and some examples are noted at the end of this article.

4.2. The purpose of representational exchange

The purpose of representational exchange is to make an organism more biologically fit by making its decision making more efficient. Efficiency is a balance of accuracy and effort. Thus, sometimes we increase efficiency by making more accurate decisions; other times, by making decisions faster or with fewer cognitive resources. Efficiency can be optimized by sharing information across decision-making systems in order to give an individual an array of options: more controlled and accurate thought, or more rapid and automatic thought, depending on the circumstances.

To describe representational exchange in more detail it is useful to use some formal concepts and notations. These should highlight useful themes for those who are already familiar with them, but without frustrating those who are not. The purpose is not to offer a formal model of representational exchange, which is well beyond the scope of this article. Rather, it is to establish points of contact with formal models of control and metacontrol developed elsewhere.

We envision an organism's life as a kind of Markov decision process. This means that the organism experiences certain states of the world, and in each of these states, it performs some actions. These actions help to determine the next states it experiences, all of which influence its biological fitness. Any individual's mind can thus be characterized by the probabilistic mapping from states to actions, or policy. Colloquially, a policy says: “Here is the thing to do in any given situation” (or “the several things you might do and their associated probabilities”). From the standpoint of natural selection, some optimal policy exists that maximizes expected biological fitness. Nobody actually has an optimal policy, but it is a useful ideal to consider: the total set of instructions for life that maximize your chances of biologically fit children. The closer an organism gets to this ideal, the more fit it is.

Figure 5.

An idealized model of behavioral control.

As a simplifying assumption, suppose that instinct, norm compliance, habit, and planning each dictate their own specific policy to an organism. In other words, instincts would provide you with one set of instructions; habits with another set of instructions, and so on. These are different mechanisms of behavioral control. In a perfect world every one of these policies would be identical; specifically, they would all encode the optimal policy. In reality, however, different systems are likely to do better or worse in different cases – that is, to recommend more or less fitness-maximizing actions in different states. Metacontrol is the problem of deciding how to allocate control to one policy or another in any given situation, or how to blend them.

The goal of representational exchange is to improve the individual policies of each system by transferring information between them. This can improve the overall efficiency of decision making by allowing optimal-but-effortful thought when appropriate, and suboptimal-but-easy thought when appropriate.

4.3. Advantages and disadvantages of control mechanisms

Before asking how these different influences on our behavior might exchange information, greater precision is required on two points: the different formats in which information is represented, and the relative advantages and disadvantages of each format. Briefly addressing these issues will put us in a better position to understand how and why information might be exchanged between representational formats.

4.4. Varieties of representational exchange

Having reviewed the representational format of several different influences on our behavior and the advantages and disadvantages of each, we can now consider several mechanisms of representational exchange in greater detail.

4.4.1. Rationalization as inverse reinforcement learning

We have already seen that rationalization translates observed actions into useful beliefs and desires. Our next goal is to re- describe this idea both more formally and more abstractly, revealing useful connections to several literatures.

Natural selection and instrumental learning share a common structure: Both are trying to maximize some objective (fitness or reward) by shaping the actions we take in the environments we encounter. An interesting property of rationalization is that it can use the common notion of objective to turn one kind of objective (fitness) into another (reward). To see this more clearly, we will begin by representing both processes (natural selection and instrumental learning) identically, as a function:

$$f\,( {\underbrace {objective,environment}_{inputs}} ) = {\underbrace{\,policy}_{output}} $$

Later it will be useful to consider the differences between natural selection and instrumental learning. First, however, having defined this function, consider what happens if we flip its direction, swapping inputs and outputs while preserving their mappings. This is the inverse function:

$$f^{-1}({\underbrace{\,policy}_{input}}) = {\underbrace {objective,environment}_{outputs}} $$

This inverse function describes rationalization in very abstract terms: you input a sample from the policy (i.e., a set of actions in a state), and you output information about the environment and the objective that the agent is trying to maximize. In more ordinary terms, the inverse function could observe a person's behavior (actions) and, on this basis, draw inferences about how the world is (beliefs), and what is valuable (desires). In short, it acts like Jason Bourne.

This basic idea has been widely explored in computer science, where it is often called inverse reinforcement learning (IRL) or inverse optimal control (Ng & Russell Reference Ng and Russell2000). Whereas a typical problem that artificial intelligence (AI) is designed to solve is choosing actions given an objective (i.e., reinforcement learning or optimal control), in some settings it is desirable to solve the inverse problem: inferring an objective from a set of observed actions. This goal often arises in social (or multiagent) settings. For instance, if a programmer wishes for a machine to learn by observing humans or to predict human behavior, then the programmer might design the machine to try infer the set of rewards that best explains the human's behavior. Once a set of likely rewards has been observed, the AI can copy human performance by maximizing those rewards itself, or it can predict human behavior by computing which actions would be reward-maximizing for the human.

IRL is easier said than done. In practice, it is often accomplished by some approximation of Bayesian inference. To see how this works, note that we can consider reinforcement learning itself as a probabilistic generative model. That is, given some specification of reward and an environment (composed of many states, actions, and the transition probabilities between them), reinforcement learning algorithms generate a probability distribution over actions – that is, the policy:

$$P\lpar {action{\rm \mid} reward,environment, state} \rpar $$

What IRL seeks, however, is the opposite: the probability of different rewards and environments given an observed action in a given state. This can be computed by inverting the generative model according to Bayes’ rule (for brevity, action, state, reward, and environment are now represented by their first letters):

$$P\lpar {r,e\,{\rm \mid} \,a,s} \rpar \propto P\lpar {a\,{\rm \mid}\, s,r,e} \rpar P\lpar {r,e\,{\rm \mid}\, s} \rpar $$

The leftmost term states what we want: inferences about rewards and environments generated by the observation of what a person does, a, in some state, s. This is proportional to two things we have: the principle of rational action, which derives a policy by reinforcement learning (i.e., P(a | s, r, e)), and a prior distribution over rewards and environments. Thus, we can guess how the world is, and what is valuable, by inferring the beliefs and desires that would render observed actions rational given our theory of mind.

Notably, a variety of the same Bayesian inversion is essential to computational models of mental state inference (Baker et al. Reference Baker, Saxe and Tenenbaum2009). This could be formalized in the language of a Markov decision process (states, actions, rewards, etc.), but it is more natural to formalize it in the ordinary language of folk psychology (beliefs and desires). The homology between these formalizations is, however, apparent. We begin with a generative model that predicts action on the basis of an agent's beliefs (i.e., its perception of its current state as well as general beliefs about its environment) and desires (i.e., rewards):

$$P\lpar {action{\rm \mid} desires,\; beliefs} \rpar $$

This is given by the principle of rational action. Often, however, our goal is to infer unknown beliefs and desires by observing actions. In this case, we may invert the generative model according to Bayes’ rule (again, variables are represented by their first letters):

$$P (d,b\; \vert \; a)\; \propto P (a\; {\vert} \; d,\; b)\; P\lpar {d,\; b} \rpar $$

Thus, we derive a guess about somebody's beliefs and desires given the actions we have seen them perform, P(d, b | a), from capacity to predict their actions based on beliefs and desires by the principle of rational action, P(a | d, b), and a prior distribution over beliefs and desires P(d, b). These examples illustrate that IRL and theory of mind are, in essence, the same.

What happens if we extend the logic of these computations to a setting where there are multiple forms of behavioral control: not just reasoning, but also instinct, habit, norms, and so forth? Crucially, the basic machinery of IRL can work even when an observer assumes a different cognitive architecture than the actor is actually employing. For instance, the actor could be operating with an innate and unchanging policy derived from some process of natural selection (i.e., instincts), in which case its objective is to maximize fitness; nevertheless, the observer could attempt to infer an objective function in terms of rewards stated within the reinforcement learning framework (as well as the structure of their common environment). This could be a useful fiction if the observer is designed as a reinforcement learning agent, and if the fitness objective of the actor is relevant to the reinforcement learning problem. Thus, we shall now stop referring to fitness and desires in common terms as objectives and instead represent the crucial difference between them: One is a property of the world, and another is a mental state. This divide is real, and yet it is bridged by the act of rationalization.

Consider, for instance, an organism rationalizing an instinct. Natural selection has shaped our instinctual policy not according to beliefs and desires, but according to actual facts about the world and biological fitness:

$$f ( {\underbrace {\,fitness,environment}_{\,properties \;of \;world}}) = policy$$

where natural selection defines the objective in terms of fitness. Yet, during rationalization, the inverse function computed is:

$$f^{-1}\lpar {\,policy} \rpar = {\underbrace {desires,beliefs}_{mental\; representations}} $$

Put in plain words, whereas fitness and actual environmental conditions shaped our instincts, rationalization extracts desires/rewards (a mental representation of an objective function that roughly correspond to the objective of fitness) and beliefs (a mental representation of the environment that roughly correspond to actual environmental conditions). Inferences about the causes of our actions become a bridge that translates properties of the world into mental representations of those properties. This makes sense because, roughly speaking, the ultimate function of belief is to represent true properties of the world, and the ultimate function of desire is to represent the fitness consequences of these properties.

In sum, rationalization approximates a form of rational inference and thus can be understood as a variety of IRL at Marr's computational level – its function is to extract useful information from observed actions. This does not imply, however, that rationalization always involves Bayesian inference at a mechanistic level. In some cases, it may, but in other cases relatively simple cognitive processes, akin to those identified by Heider and Festinger, may approximate the rational inferences described above.

As we shall see next, a benefit of construing the present theory of rationalization in terms of these more formal concepts, and at Marr's computational level, is that it makes apparent the relationship between rationalization other forms of representational exchange.

4.4.2. Habitization: Cached value and cached policy

During rationalization, information is extracted to improve reasoning; we next consider several ways in which analogous processes can extract information to improve habitual action.

According to several theories, habits can be understood as cached representations of instrumental value – that is, the expected value of actions, in terms of long-run reward. Plausibly, then, useful habits can be constructed by extracting information about value from the actions selected by other systems. Here, again, we envision this as a process of IRL, but this time the goal is to derive a value function that can be cached for habitual action:

$$P\lpar {v\,{\rm \mid}\, a,s} \rpar \propto P\lpar {a\,{\rm \mid}\, s,v} \rpar P\lpar {v\,{\rm \mid}\, s} \rpar $$

Thus, while certain instincts, or norms, may not themselves depend on any representation of value, still we may update our cached value representations (i.e., habits) by inferring the values that are consistent with the actions performed.

In contrast to such value-based models of habit, however, some alternative theories posit that habits depend upon cached policy – direct stimulus/response mappings, with no representation of value. On this view, habits are chunks of policy “stamped in” through mere repetition (Dezfouli & Balleine Reference Dezfouli and Balleine2012; Reference Dezfouli and Balleine2013; Dezfouli et al. Reference Dezfouli, Lingawi and Balleine2014). For instance, by repeatedly tying our shoes in a particular manner, a specific sequence of actions is chunked into an easily retrieved bundle. Such cached policy representations introduce a new target for representational exchange. When a person's action is determined by instinct, planning, or even value-based habitual action, this action may be cached directly as a policy weight – its informational content thus exchanged into a new format particularly suited to efficient online execution.

Thus, while value- and policy-based theories of habits have been viewed as competitors, they may instead be viewed as complementary representations within a unified scheme (Fig. 6). The hallmark of an instrumental system (whether habit or planning) is that rewards shape values, and values shape policy. Planning makes maximal demands on online computation – it must derive value representations from basic representations of reward and the environment, and then derive a policy from those values. Value-based habitual control (value caching) requires substantially less online computation – it can derive a policy from a cached value representation. Value-free habitual control (policy caching) makes the minimal demands – it simply enacts the stored policy. Thus, value- and policy-based theories of habit need not be considered as rivals, but as distinct points on a common spectrum.

Figure 6.

The representational hierarchy of instrumental learning and decision making. Information can be stored at any level: Cached policy, cached value, or beliefs and rewards.

4.5. Beyond decision making: Other forms of representational exchange

Representational exchange is useful not just for decision-making processes, but in many other areas of cognition as well. The examples of computing “25 + 25” versus “26 + 52,” for instance, do not really belong to the same general category of decision making as, say, planning a trip to the grocery store, tying one's shoes, or leaping away from a snake. Still, we have relatively computationally cheap, precompiled knowledge of common sums (such as 25 + 25) and relatively more computationally intensive methods of deriving uncommon sums (such as 26 + 52). And there are circumstances in which it will be optimal to exchange information between these formats.

Representational exchange has been well explored in at least one domain that isn't principally about decision making: thought experiments and other forms of imaginative learning (Lombrozo Reference Lombrozo2017). These are cases in which an individual has some kind of intuitive knowledge of a phenomenon (e.g., the behavior of a physical system) and uses this intuitive knowledge as a basis for improving their explicit theory of that phenomenon (e.g., a new theory of physics, such as gravity or relativity). This may appear to be a form of alchemy, conjuring gold from iron filings. In reality, it is more akin to rationalization. First, a generative process is trained by experience; for instance, the visual system might learn to anticipate the motion of physical bodies under various conditions. Next, an individual inspects the information that the generative process makes explicit (e.g., an intuition about how physical objects will interact) and uses that information to draw rational inferences about information that is merely implicit (e.g., the laws of motion). Just as rationalization extracts structured information from precompiled value representations, a thought experiment about physics extracts structured information from precompiled predictive perceptual representations.

The resulting information – a theory of physics, for instance – is far more computationally expensive to use under many conditions. (In other words, it is harder to determine how an apple will fall from a tree by applying Newton's laws than by relying on precompiled predictive perceptual representations). But this explicit theory is also far more flexible, allowing us to solve problems for which we have no adequate precompiled predictive perceptual representations such as landing an astronaut on the moon.

Although rationalization, theory of mind, and thought experiments might ordinarily be considered very distinct phenomena, they share both structural and functional similarities – an unpacking of implicit information compressed into a narrow format. Noticing these similarities may help us to develop an abstract framework for understanding how, why, and when people engage in representational exchanges of any kind (see Batchelder & Alexander Reference Batchelder and Alexander2012).

Thought experiments are common in philosophy, and they are sometimes used in the process of achieving reflective equilibrium. This is a canonical case of representational exchange. When seeking reflective equilibrium, one contrasts intuitive judgments about particular cases with principled rules or reasons that govern those cases, and then seeks the minimal modifications to both that bring them into alignment (Daniels Reference Daniels and Zalta2003). It is commonly used during moral reasoning: Individuals seek to bring their intuitive judgments of particular cases into alignment with a more general normative theory, and this involves revision to both the particular judgments and to the theory itself.

Reflective equilibrium is attractive to philosophers who want to take intuition seriously without giving it absolute priority. They emphasize that intuition is the result of adaptive processes, which could include habit learning, natural selection, and cultural evolution, among others (e.g., Railton Reference Railton2014). This echoes a basic argument offered here: The policy recommendations of non-rational systems can be used to improve reasoning, at least from the standpoint of fitness maximization.

Yet, skeptics have reasonably countered that even if intuition tends toward adaptive outcomes, it is at best a heuristic approximation (e.g., Greene Reference Greene2014). Why, then, ought intuition ever be favored over reasoning? If a person has devoted appropriate mental effort to reasoning, then what superior policy recommendation could arise from the heuristic representations of non-rational systems?

One simple reply is that a person's beliefs might be wrong. But this is a shallow response; presumably, we are often quite confident in our reasoning, and yet it still conflicts with our intuitions. A deeper reply is that rationalizing non-rational adaptive processes can improve intrinsic rewards maximized by reasoning. By itself, reasoning has no method for questioning or improving its own goals; it is, in Hume's (Reference Hume1739) words, “a slave to the passions.” Even a system of reasoning possessed of perfect causal knowledge and unconstrained by computational resource might be improved by information from intuition. Of course, improvement in this case means more biologically fit; whether this is the kind of improvement that philosophers want is a question for philosophers.

This highlights the relationship between rationalization and the naturalistic fallacy (that is, deriving what “ought to be” from what “is”). In a specific setting – adopting new desires for instrumental learning from observations of one's own or others’ actions – we have considered an adaptive rationale for this inference. Of course, it is not always the case that people do the best (i.e., most fit) thing; on average, however, there is something important to be learned about what one ought to do simply by observing the actions that people, including yourself, actually perform.

4.6. Summary

More than a century of psychological research shows that our behavior is a product of multiple systems. Given that these each have relative advantages and disadvantages, there is good reason to exchange information across representational formats, gradually optimizing the manner in which information is represented. This perspective highlights the common function of many different processes that convert information between diverse representational formats and across individuals (Fig. 7).

Figure 7.

Varieties of representational exchange.

A theory of representational exchange suggests several important topics for further development. First, assuming that representational exchange has a computational cost, how is it allocated efficiently? In other words, is there a mechanism of exchange control that regulates when and where representational exchanges of different types occur? Second, when information conflicts arise between systems during representational exchange, how are they adjudicated? One obvious prediction is that an individual will not adopt new beliefs or desires when the action produced by a non-rational system can easily be explained away as a mistake. For instance, if someone reflexively jumps away from a fake rubber snake, they might not adopt the belief that rubber snakes are dangerous. Rather, they might conclude that this was a misfiring of a non-rational system, and discount it as a source of information. It remains to be seen how this inference can be formalized.