The rational agent from game theory to bounded rationality

The transition from neoclassical economic theory to rational choice theory has been described as an instantiation of a “neoliberal social contract, or Prisoner’s Dilemma model of governance, [that] displaces normative bargaining predicated on the no-harm principle [of classical liberalism].”Footnote ⁶ In fact, while Vilfredo Pareto’s criterion, grounded in neoclassical theory, held that trade can potentially benefit someone without making anyone worse off, the rational choice or decision theory developed in the postwar period maintains as its central pillar the effort of each agent to strive to achieve the maximum expected outcome, in constant competition with every other.Footnote ⁷ As Philip Mirowski points out, rational choice theory has its roots in the mathematical treatment of the utility-maximization problem by John von Neumann and Oskar Morgenstern in the book Theory of Games and Economic Behavior (1944), the foundational text of game theory.Footnote ⁸ Von Neumann, a prominent figure in different scientific fields—ranging from economics to computer science—engaged in the task of “establish[ing] satisfactorily … that the typical problems of economic behavior become strictly identical with the mathematical notions of suitable games of strategy.”Footnote ⁹ That is, his aim was to provide a comprehensive mathematical treatment of the utility-maximization process typical of the rational agent’s economic behavior described by the neoclassical school. While in the course of the book von Neumann and Morgenstern also investigate multiplayer games, whose analysis is intended to bring about a more thorough understanding of the interaction among multiple economic actors, the only case they manage to solve completely is the two-person zero-sum game, in which the sum of the overall payoffs equals zero. Here, the utility-maximization problem posed by neoclassical economics receives a mathematical formalization, and minimax is established as its optimization criterion.Footnote ¹⁰ The focus is no longer the improvement of individual economic conditions toward a Pareto optimum, but rather the maximization of rewards in a competition over limited resources, where the gain of one player invariably corresponds to the loss of the opponent.

In the subsequent reinterpretations of game theory, different elements emerged over time, in the continuous attempt to harmonize the neoclassical idea of individual utility maximization with what Mirowski defines as the emergence of a “cyborg science” in the field of economics—namely, an approach that “construe[s] markets … as formal automata,” shifting “from the previous neoclassical field definition of economics as ‘the allocation of scarce resources to given ends’ to the cyborg definition of the economy as a giant information processor.”Footnote ¹¹ Following the transformation identified by Mirowski, the concept of the rational agent also underwent, during the 1950s, a major fork from the field of economics, due to its incorporation into the theoretical framework of AI. This bifurcation was mainly driven by the work of Herbert Simon, one of the foremost critics of game theory and arguably the single most important figure involved in the computational turn of the rational agent. Simon was a major presence in both economics and computer science, making significant contributions to the study of decision-making in organizations and its connection to AI. Unlike other economists who sought to maintain the newly formalized rational agent of game theory as an entity consistent with the neoclassical economic man—a coherent whole seeking utility maximization through perfect calculation—Simon acknowledged that the concept was untenable within a solid theoretical framework, albeit useful for modeling rational agents specialized in narrow operative tasks and organized in hierarchical structures. The fundamental tension that characterizes his intellectual trajectory is described by Hunter Crowther-Heyck as an attempt to mediate between the “sciences of choice”—economics and the study of free rational decision—and the “sciences of control,” such as sociology and social psychology, concerned with the conditions that actually determine the available choices.Footnote ¹²

It was through the concept of bounded rationality that Simon sought to reconcile these two poles, respectively embodied in the perfectly rational economic man and the satisficing administrative man. Indeed, he stated that “the concept of ‘economic man’ (and, I might add, of his brother ‘administrative man’) is in need of fairly drastic revision,” explaining that “the task is to replace the global rationality of economic man with a kind of rational behavior that is compatible with the access to information and the computational capacities that are actually possessed by organisms, including man.”Footnote ¹³ He suggested elsewhere that the kind of organism capable of making a rational choice is not limited to human beings, but also includes other kinds of “organisms,” most notably computers.Footnote ¹⁴ Considering computers as the proper rational agents to be modeled, in fact, immediately solves the paradox of the impossibility of rational behavior, substituting the inconsistent economic man with an abstract entity that makes decisions upon limited knowledge and ability.

However, when this theoretical model was actually implemented by Simon and his colleagues into the Logic Theory Machine, a program written for the JOHNNIAC computer at RAND, what shaped the resulting system were more the affordances of the machine than the bounds of human cognition, as Stephanie Dick has shown.Footnote ¹⁵ In her analysis, Dick recalls the critique of the logician Hao Wang, who noted that bounded rationality had displaced the ideal of the perfectly rational economic agent only to install a different one—that of human reasoning as rule following.Footnote ¹⁶ What Wang emphasizes is the distance between the model and the human practices it is supposed to describe, implicitly highlighting that bounded rationality is a performative, rather than a descriptive, model. This is a crucial point for the present paper, which traces the source of Simon’s model of rationality back to the organizational structures he had previously analyzed in his work on administrative behavior.Footnote ¹⁷ Simon attributed the limits of individual rationality to the limited information-processing capacity of human beings, concluding that “the rational individual is, and must be, an organized and institutionalized individual.”Footnote ¹⁸ It is, in other words, the organization that creates the premises upon which decisions are made, and rationality emerges as a property of the whole system, rather than of the individuals who form it. The resulting model of the mind can thus be interpreted as reproducing the structure of organizational decision-making, which it is in turn designed to optimize.

The purpose of bounded rationality, in this perspective, is not to simply codify a model of human behavior capable of consistently explaining our reasoning processes, but rather to create artificial agents and model their behavior according to the needs of organizations in the attainment of specific goals—that is, to create computers with purposes and capable of making plans. The concept of bounded rationality has, in turn, always entailed that of procedural rationality, later defined by Simon as the search for “efficient procedures for computing actual solutions to concrete decision problems.”Footnote ¹⁹ These two concepts—bounded and procedural rationality—set the terms in which Simon’s legacy within symbolic AI came to conceive of intelligent action as the search for efficient procedures under constraints of information and computational capacity.

The age of symbolic rational agents

Symbolic AI was the prevailing approach until the second AI winter in the late 1980s.Footnote ²⁰ Within this paradigm, the concept of bounded rationality provided the framework through which effective decision-making processes could be implemented into AI systems, using techniques such as heuristic search, rule-based reasoning, and task decomposition into subroutines.

The most visible achievement of symbolic AI was the creation of expert systems, which embodied the very vision developed by Simon regarding the possibility of creating automated information-processing systems to support and guide human decision-making within organizational settings. This is particularly evident in the work of Edward Feigenbaum and Bruce Buchanan, leading theorists and developers of expert systems between the 1960s and the 1980s. The two main projects on which they worked during this period were DENDRAL and MYCIN. DENDRAL, developed from 1965, was the first expert system ever created, with the goal of inferring possible molecular structures from mass spectrometry data, using a knowledge base built by expert chemists and analyzed through an algorithm based on heuristic rules.Footnote ²¹ MYCIN, developed from 1972 at Stanford University, was created to help physicians diagnose bacterial infections and recommend suitable treatments by applying a rule-based knowledge structure derived from medical expertise.Footnote ²² This expertise was, in both cases, provided by the so-called domain experts, a key figure in the development of expert systems, whose deep understanding of a specific field—be it chemistry, medicine, or other specialized areas—was essential for building the model’s knowledge base. The domain experts provided the rules, heuristics, and decision-making strategies that were encoded into AI, ensuring that the system could emulate and support the decisions of a human specialist.

The development of early expert systems led to the first corporate adoption of AI as a technology for reasoning and decision-making. Feigenbaum and Buchanan recount how the termination of DENDRAL at Stanford led to its licensing to an unspecified company “specializing in software for chemical structure manipulation, synthesis planning, and literature searching,” which hired computational chemists “to do the technology transfer [to the industry], and more importantly for the company to guide and develop other projects within the firm.”Footnote ²³ In other words, the project was transformed during the 1980s into a fully commercially oriented enterprise. Similarly, later expert systems such as R1/XCON, developed at Carnegie Mellon University and deployed by the Digital Equipment Corporation from 1980 to automate the configuration of complex computer systems, and PROSPECTOR, developed by the Stanford Research Institute from the 1970s to support mineral exploration by integrating geological, geochemical, and geophysical data, were prominent examples of the attempt to transfer AI to industry in the following years.Footnote ²⁴

Feigenbaum and Buchanan, moreover, stressed that the development of expert systems created a paradigm shift within symbolic AI, moving the perspective from what they called “a generality dimension,”Footnote ²⁵ exemplified by the General Problem Solver—one of the first AI programs developed by Newell, Shaw, and Simon to apply formal reasoning across a wide range of abstract problems—to the more specialized, domain-focused approach typical of expert systems. While the transformation brought about by their work is evident, the deployment of this technology should nonetheless be understood as carrying forth Simon’s original vision of decision-making optimization within organizations. Expert systems were indeed, at the time, the main AI research program focused on the optimization of information processing within corporate structures.Footnote ²⁶

As Allen Newell later noted,

While the “not only” reference to the corporate employment of AI signals that this was already the most prominent application of this technology at the time, the “scientific problems” it identifies appear to stem largely from the deployment of AI systems in real-world commercial and industrial applications, including decision support, process optimization, and the automation of expert tasks. This shows how the development of AI was already deeply intertwined with organizational needs and practical applications, emphasizing efficiency and optimization over the modeling of human cognition.

Rational agents as cyborg workers

About thirty years after “A Behavioral Model of Rational Choice,” Simon published “Two Heads Are Better than One: The Collaboration between AI and OR.”Footnote ²⁸ In this paper, he argued that a structural bond between AI and operations research (OR)—the mathematical treatment of decision-making optimization within organizations—had existed at the inception of the two disciplines after World War II, and that the solutions to OR’s problems had initially been limited only by insufficient computational capacity. Indeed, Simon identifies the inherent “aspiration of AI to deal with all the aspects of managerial decision making that stretch beyond the limits of classical OR.”Footnote ²⁹ It follows that “management science must incorporate a still more general and powerful kit of tools that can be applied to … ill-structured, knowledge-rich, non-quantitative decision domains.”Footnote ³⁰ This approach, which we see today in a late developmental stage, could be traced back to Simon’s previously discussed studies on administrative behavior. The core idea is that control could be enforced by establishing prescriptive criteria for decision-making,Footnote ³¹ deployed both through the implementation of procedures designed to constrain human decisions and through the creation of autonomous artificial agents operating alongside human workers.

Another major figure of early AI, Marvin Minsky, developed in The Society of Mind—a book written in the same period that represents a comprehensive synthesis of his lifelong research—a theory of agents that appears consistent with that of Simon, as the former posits that simple mental processes organized in hierarchical structures are the fundamental units of the mind, just as the latter considers simple decision processes as the fundamental units of organizations. The intersection of the two perspectives is captured in the title of Minsky’s book, which suggests a structural equivalence between mind and society. Minsky explains that his conception of a society of mind constitutes a model of how the human mind actually works, being composed of a multitude of processes called agents, organized in hierarchical structures. These agents control other sets of subagents, which in turn can control additional layers, and so on. This means, for example, that an agent Add that seeks to add a block of wood to a tower is in turn composed of subagents Find, Get, Put. Footnote ³² Minsky is here describing the possibility of decomposing the process of putting the block on the tower into a chain of simpler subprocesses, which are in turn composed of—and thus in control of—more subordinate layers.Footnote ³³ The process of adding blocks to the tower, which could eventually be performed by a mechanical arm, is entirely analogous to the same action performed by a human being, who is composed of the same arrangement of agents. The mental activities conceptualized as agents, however, are not intelligent entities in themselves, but simple mechanical processes organized in stacks: intelligence only arises as an emergent feature generated by the interaction of the whole system.

Minsky goes on to affirm that the hierarchical organization of agents within machines is, in turn, equivalent to that of society. Indeed, there is no significant difference between the two, as “designing any society, be it human or mechanical, involves decisions like these: Which agents choose which others to do what jobs? Who will decide which jobs are done at all?”Footnote ³⁴ The society of human beings and the society of mind mirror each other, both being based on the principle of the division of labor. This is further explained shortly after, when he states that

Later, Minsky deepens the analogy between computers, minds, and organizations, stating that an agent is like a “bureaucrat [that] sees but a fraction of what happens underneath its place in the pyramid of information flow. The best subordinates are those that work most quietly. Indeed, that’s why we build administrative pyramids for jobs we don’t know how to do or don’t have time to do ourselves.”Footnote ³⁶ In other words, agents constitute the fundamental units of both the human mind and software applications, being analogous to workers and bureaucrats operating within the hierarchical structure of a real society.

Minsky’s arguments are crucial for understanding the historical foundations and developments of this research program. While his use of the concept of agent differs from that of Simon—since for Minsky it describes a subprocess within the proper rational agent—it nonetheless serves to reveal Simon’s epistemic construction with remarkable clarity. Indeed, Minsky explicitly acknowledges that it was Simon, together with his colleagues J. C. Shaw and Allen Newell, who first grasped the fundamental principles of autonomous information-processing machines.Footnote ³⁷ Yet, the connection with Simon’s work extends far beyond the surface similarity in the description of intramachinic processes. In conceptualizing the mind as equivalent to a society of workers and bureaucrats obedient to their superiors and rationally enclosed within an ordered hierarchy, Minsky inherits from Simon more than the mere idea that an autonomous agent consists of a series of subprocesses aimed at optimizing specific goals: he reformulates and elaborates Simon’s assumption that computational processes are intended to mirror the structure of human organizations. Intelligence, from his perspective, is a faculty that emerges only when individual agents are embedded within an organizational structure—when they are, in other words, socialized in an orderly and hierarchical fashion. This further points to the fact that it is the division of labor and its complex social organization that can be effectively reproduced within the architecture of an information-processing machine.

Minsky, interestingly, calls this kind of machine a Difference Engine, the same name as the famous machine designed in the 1820s by the mathematician and inventor Charles Babbage. The history of the original Difference Engine and its relationship to the subsequent development of AI is well documented elsewhere.Footnote ³⁸ It suffices here to note that Babbage’s machine was designed to partially automate human calculations for the production of mathematical and astronomical tables. Babbage himself claimed that computation as a technical field arose from the principle of the division of laborFootnote ³⁹—an idea that, as we have seen, also deeply permeates the perspectives of Simon and Minsky. Linked to the Difference Engine is, indeed, the so-called Babbage principle,Footnote ⁴⁰ which asserts that the division of labor enables the purchase of only the precise amount of skill and force required by each process. In the development of their frameworks, a central concept for both Simon and Minsky was, in fact, that of efficiency,Footnote ⁴¹ which would eventually be applied both to the assessment of workers’ and machines’ performance and to the effectiveness of the control exerted through the latter over the former.

Rational agents at the turn of the millennium

At the end of the twentieth century, interest in the concept of rational agent gained new momentum, with a few anthologies beginning to conceptualize in more rigorous terms what this entity is, where it came from, and what we ought to do with it, on the premise that “Artificial Intelligence is, as much as anything, about building rational agents.”Footnote ⁴² The quote highlights the definitive appropriation of the concept, now thoroughly instantiated in “agents whose actions make sense from the point of view of the information possessed by the agent and its goals (or the task for which it was designed).”Footnote ⁴³ This perspective also marked, in the interpretation of AI practitioners, a shift within the field itself: whereas previously “it was common to define AI as the computational study of ‘mental faculties’ or ‘intelligent systems’ [now,] instead, one can define AI as the problem of designing systems that do the right thing.”Footnote ⁴⁴

The idea of a rational agent that simply “does the right thing” incorporates Simon’s original proposal to develop computer programs capable of maximizing their efficiency within the constraints of their computational resources. Stuart Russell builds on Simon’s distinction between substantive and procedural rationality, “the difference between the question of what decision to make and the question of how to make it.”Footnote ⁴⁵ Rationality, in his framework, is a property of actions that constrains the selection process without necessarily specifying it, and his concept of bounded optimality—the capacity to generate maximally successful behavior given the available information and computational resources—allows different architectures to operate under the same criterion. Here, the old interpretation of the economic man and Simon’s revision of the concept converge: the whole framework is built around maximization, which, however, now operates under the very constraint Simon had introduced as a critique of the maximizing agent. Russell, indeed, regrets that the techniques developed in economics for decomposing utility into combinable attributes “have made few inroads into AI,” and the well-known textbook he co-authored with Peter Norvig devotes an entire chapter to utility theory and to the maximization of expected utility.Footnote ⁴⁶

The concept of rational agent thus came, at this point, to span different research traditions. Simon and Minsky belonged to the symbolic one, grounded in the principles of formal logic: a program like the General Problem Solver could address only precisely formalized problems by executing operations constrained by fixed logical rules, and the same principles were later applied to expert systems. With the increasing centrality of optimization-based methods, the definition of the rational agent no longer required this structure, as the target of computation ceased to be internal to the architecture and became “a value that the world assigned to it from outside”—a value that is, very often, supplied by the manual labor of labeling data.Footnote ⁴⁷ In recent years, specifically, it is the connectionist tradition that has become dominant, as reflected in the spread of NNs, which learn patterns directly from data without relying on explicit logical constraints or a pre-specified knowledge base. The capacity of large-scale models, such as large language models (LLMs), to generalize this “knowledge” to previously unseen information lies at the core of their success, since, unlike symbolic models, they are not confined to specific knowledge domains.

Certainly, some degree of conceptual continuity exists between the two traditions. The architecture of connectionist AI, where “information processing takes place through the interactions of a large number of simple processing elements called units,”Footnote ⁴⁸ still echoes the logic of the division of labor that informed the work of Simon and Minsky. Connectionists themselves insisted, to some extent, on this continuity: in the seminal book Parallel Distributed Processing, Hinton, McClelland, and Rumelhart state that “it would be wrong to view distributed representations as an alternative to representational schemes like semantic networks or production systems that have been found useful in cognitive psychology and artificial intelligence. It is more fruitful to view them as one way of implementing these more abstract schemes in parallel networks.”Footnote ⁴⁹ Yet, unlike the subprocesses of symbolic AI, these units do not correspond to identifiable subtasks.

NNs thus fit Russell’s definition of a rational agent while lacking the procedural approach that Simon identified as a central feature of AI, which emphasizes the ability to decompose a problem into a sequence of subproblems and solve it through a defined series of steps, implementing a computational process that can be explicitly divided into controllable stages and subgoals. This is hardly applicable to NNs, since procedurality requires both an explicit knowledge base and a set of controllable logical constraints to guide the inferential process. But is it truly the case that procedural rationality, a concept that can be understood as the operational arm of bounded rationality in creating actual rational agents, has entirely disappeared from connectionist AI?

Agents-in-the-loop

Although the creation of AI models may appear to be the prerogative of a small circle of specialists, it in fact relies on the labor of millions of people worldwide.Footnote ⁵⁰ The recent expansion of this workforce, which was considerably smaller just a few years ago,Footnote ⁵¹ is linked both to the commercial diffusion of AI and to the structural limitations of NNs described earlier—that is, their inability to reason under logical constraints. From this standpoint, the production process of NNs offers a different perspective on the paradigm shift from symbolic to connectionist AI, highlighting that procedural rationality has not disappeared, but has rather been displaced onto the human component of the computational network—the data workers.

Data work consists of the outsourcing of tasks used to train, evaluate, and fine-tune AI, either through online platforms or through third-party suppliers, often located in the Global South. This work, which is the subject of a growing critical literature,Footnote ⁵² primarily involves two macro-categories of tasks. The first category includes data labeling and production to train AI in specific acts of classification—for example, recognizing a bottle in an image. The second category comprises verification tasks, oriented toward assessing the quality of an output and providing feedback to the model or its developers—for instance, giving positive feedback to an LLM for correctly including information in its response to a prompt.

Clearly, data work contributes to the production of NNs by supporting practitioners’ activities, as it enables the training, evaluation, and fine-tuning of models developed through capital- and skill-intensive production processes. The development of NNs, however, also depends heavily on the repetitive and mechanical tasks of data preparation and output verification described above, which are generally outsourced to ensure maximum flexibility and scalability. The harsh characterization of data workers as “people-as-a-service,” proposed by Jeff Bezos in a speech at the Massachusetts Institute of Technology while presenting the first microwork platform, Amazon Mechanical Turk, is perhaps the clearest instantiation of an approach that reduces workers to task-oriented entities embedded within a flexible, algorithmically managed hierarchy.Footnote ⁵³ What is, however, more significant for the present analysis is that, in the same speech, Bezos further emphasized that the goal of data work is to “encode human intelligence inside a software application.”Footnote ⁵⁴ This statement offers a useful entry point for analyzing the development of NNs.

The model of intelligence that Simon and Minsky envisioned in the development of symbolic AI was that of symbolic reasoning—the ability to manipulate symbols to derive specific conclusions from given premises. As explained above, NNs lack this type of internal logical structure, since their operations are purely inductive, meaning they can only extract patterns from data. The absence of logical reasoning, however, is mitigated by the integration of data work into the production pipeline of these AI models to provide consistent examples for aligning and verifying NNs. The guidelines that data workers must follow reintroduce the principles of procedural rationality into this alignment process, structuring the production of specific outputs.

To clarify this point, I will use the example of LLMs, which are currently among the most widely recognized types of NNs. It is well known that LLMs can produce hallucinations, namely incorrect or entirely fabricated information generated in response to a prompt. This occurs because these models do not rely on a specific knowledge base to verify facts and are therefore liable to produce inaccurate outputs, particularly when asked to generalize from underrepresented data. Hallucinations are often mitigated by grounding LLMs’ outputs in external sources, such as Wikipedia, and their factual accuracy can then be verified by data workers against the same sources. Here is an example of how it works:

Data workers are provided with references—in this case, a few paragraphs from Wikipedia—and a set of guidelines that explain, step by step, the actions they must perform to verify whether a claim included in a response to a specific prompt is factually correct. In this process, the principles of procedural rationality are transposed onto human labor through the codification of the task, which essentially involves two aspects: (i) making an inference or verifying information against a specific knowledge base—the references provided for the task—and (ii) applying logical constraints—the guidelines—to perform this inference. The procedural use of logical rules to extract information from a knowledge base, characteristic of symbolic AI, is here redirected to govern the work of humans who evaluate an LLM, which lacks the inherent logical structure to carry out such operations. The same kind of procedurality organizes the feedback through which models are trained. In the annotation process designed by NVIDIA for the alignment of its version of Llama, for instance, data workers were given a prompt and two responses: they first had to rate each response on a five-point scale along five dimensions—helpfulness, correctness, coherence, complexity, and verbosity—then choose among seven predefined preference options, and finally justify their choice, applying the criteria in a fixed order of priority set by the guidelines.Footnote ⁵⁶ This reveals a fundamental paradox: while NNs are not fully controllable in their internal operations, the human labor that aligns them can instead be regulated and organized through fixed procedures to influence their behavior. This process also constitutes a reframing of the domain expert figure typical of symbolic AI, as the transfer of expertise from humans to AI is no longer solely dependent on individuals possessing domain-specific skills, but is instead distributed across a vast computational network of human and artificial agents performing narrow tasks. While in symbolic AI knowledge preceded and determined the model’s architecture, which was built out of the explicit codification of experts’ reasoning, in contemporary connectionist AI the order is reversed, as it is the architecture that makes the extraction and organization of knowledge possible. Human workers are no longer required to formalize the domain for the system, but are incorporated within a system that formalizes the domain through them.

Thus, we can see that in the production pipeline of contemporary AI models, it is the human being who is installed as the logical machine subjected to procedural rationality. Humans now constitute an integral part of these vast computational networks, within which they perform specific rationalized functions that enable the “intelligence” of the system to emerge, similarly to the organized individuals described by Simon and the agents envisioned by Minsky. In this hybrid human–machine computational arrangement, the hierarchy of agents becomes a hierarchy of computational nodes, where humans are reduced to simple information-processing functions that machines can directly call upon to resolve ambiguities. This, of course, does not imply that these machines have attained a sufficient degree of autonomy to exert direct control over humans, but rather that they embody a more structured, mediated, far-reaching instantiation of the power inherent in any relation of production.