2. Clear cases of agency and non-agency

In this section, I will describe two possible AI systems and argue that the first is an agent while the second is not. These two systems are not sequence models; my aim in describing them is to establish precedents that will be useful when I turn to sequence models in subsequent sections.

As an example of an agent, consider a neural network trained by model-free reinforcement learning (RL) to play the video game Pong, using pixel maps as inputs (more concretely, this could be Mnih et al.’s (Reference Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik, Antonoglou, King, Kumaran, Wierstra, Legg and Hassabis2015) deep Q-network). This system may be able to interact effectively with the Pong environment so as to achieve a high score. But beyond this, there are two deeper reasons to think of it as an agent.

First, model-free RL works in a way that justifies explanations of the system’s behaviour that describe it as pursuing a goal.Footnote ¹ In a value-based model-free RL algorithm such as Q-learning, systems maintain representations of the expected value of performing actions in given states of the environment. These representations are initialised randomly, then updated in the following way: when the system performs an action in a given state, the value for that state–action pair is updated in the direction of the difference between it and the reward received plus the current estimate of the value of the best action in the new state. This means that actions that lead directly to high reward come to be represented as valuable. But what’s more, it has the same effect on actions that lead indirectly to high reward. This is because, once the system has enough experience, it will represent that valuable actions—that might eventually lead to reward—are available from the states to which these actions lead.

When systems of this kind come to select actions, they typically choose those that they represent as having the highest values (although exceptions to this rule are needed to promote exploration). As we have just seen, they come to represent these actions as having high values because the actions are instrumentally conducive to states that yield high reward. So these systems work in such a way that when a type of action is conducive to a goal, this can cause them to take actions of the type—and most of their actions are caused in this way. This is excellent grounds to claim that these systems produce outputs in order to bring about goals.

Second, model-free RL appears to play a central role in human and animal agency. An extensive research programme uses the conceptual and computational framework of RL to explain how we learn and select actions (Niv Reference Niv2009; Dolan and Dayan Reference Dolan and Dayan2013). So the Pong-playing RL system works in a fundamentally similar way to paradigm agents. From now on, I will call it ‘the Pong agent’.

This is not to say that the Pong agent is uncontroversially an agent. Some researchers who approach agency from a biological perspective would deny that this system is an agent on the grounds that it lacks autopoietic autonomy (Barandiaran et al. Reference Barandiaran, Paolo and Rohde2009; Virenque and Mossio Reference Virenque and Mossio2024). But there are strong enough grounds for attributing agency to the Pong agent that it is reasonable to take this as a starting point for examining further examples.

As an example of a non-agent, consider a neural network trained by supervised learning to imitate the behaviour of the Pong agent—this could be similar to the systems trained to imitate human players using behaviour cloning by Kanervisto et al. (Reference Kanervisto, Pussinen and Hautamäki2020). Like the Pong agent, this network could take pixel maps as inputs and generate instructions to perform actions—to move the Pong paddle up or down—as outputs. However, it would be trained differently. Where the Pong agent generates its own training data by interacting with the environment and is updated using an RL algorithm that takes the game score as reward, this network would be trained to discriminate the states that are paired with different actions in a dataset generated by recording the Pong agent’s behaviour. It would not see the game score or the consequences of the ‘actions’ it chooses in training. Nonetheless, it is possible that this system could also interact with the Pong environment so as to achieve a high score.

This system cannot be an agent because it is too similar to discriminative models trained to classify images, which are paradigm examples of artifacts that have functions but do not pursue goals. The system in question is trained in the same way as the image classifier, by supervised learning on a dataset of input–output pairs. Furthermore, its task can be conceived as one of image classification; it classifies pixel maps according to the different actions the Pong agent takes in response—moving the paddle up or down, or leaving it in the same position. And it is likely to learn a fairly simple algorithm for doing this, since Pong is a simple game. The rule could be to always move the paddle up when the ball is above it and down when the ball is below it. We can imagine that in some circumstances the best way to accurately imitate expert agents is to understand the environment oneself and adopt and pursue the experts’ goals; if there are cases like this, training for imitation could conceivably promote agent-like cognition. But the present case is not of this kind.

It may be useful for some purposes to describe all AI systems that interact with environments in ongoing ways as ‘agents’ (Sumers et al. Reference Sumers, Yao, Narasimhan and Griffiths2023). But if agents are distinguished by how they learn and select actions, then, again, there are strong enough grounds to deny that this system is an agent that we can reasonably start from this claim.

3. Return-conditioned sequence models

As I have described, sequence models are models trained to predict the next item in a sequence. They are often based on the transformer architecture. Sequence models can be trained on sequences of interleaved observations and actions, allowing them to interact with environments in which those observations and actions are possible. The systems I want to focus on are return-conditioned sequence models, which have extra capacities because their training data also include returns-to-go.

Suppose we have records of a number of trajectories through an environment, including data about the rewards accrued at each time-step. These could be random walks or generated by an agent. Then we can organise this data so that each trajectory is made up of triples of the form [returns-to-go, observation, action], where ‘returns-to-go’ is the sum of the rewards subsequently accrued on that trajectory. We can now train a sequence model on this data and prompt it with our desired returns; the model will then tend to choose the actions that are most likely given that these returns will be achieved. Decision Transformer works in this way and was found to perform capably on a range of tasks including some Atari games (Chen et al. Reference Chen, Lu, Rajeswaran, Lee, Grover, Laskin, Abbeel, Srinivas and Mordatch2021).

A remarkable extension to this method is Algorithm Distillation (Laskin et al. Reference Laskin, Wang, Oh, Parisotto, Spencer, Steigerwald, Strouse, Hansen, Filos, Brooks, Gazeau, Sahni, Singh and Minh2022), in which trajectories collected from RL agents are used for training. These trajectories show improvements in the agents’ policies in accordance with their learning algorithms. When Laskin et al. trained a transformer with a sufficiently long context window on this data they found that it learned an RL algorithm, which allowed it to subsequently improve its own performance through interaction with the environment.

Algorithm Distillation models are agents because they learn and use algorithms for in-context RL. Using these algorithms, these models learn in the course of each forward pass which types of actions are rewarding and select outputs accordingly. Although they are trained to imitate other agents, they learn to do so by responding to reward in a similar way.

Whether Decision Transformer is an agent is much less clear. In favour of agency, one might argue that in training this model learns about the structure of the environment—how states, actions and rewards are related—and that this information allows it to select rewarding actions in deployment, when suitably prompted. It can derive a policy that allows it to navigate the environment successfully, achieving goals, from data about the environment that does not specify such a policy. But against agency, the same kind of argument that applied to the Pong-imitator system also applies to Decision Transformer. It is trained offline to predict the next token in a sequence, like an LLM or a behaviour-cloning model, and in this respect is like paradigm non-agents. Furthermore, in contrast with Algorithm Distillation, it does not learn to predict the next token by a recognisably agentic method.

4. The case against Decision Transformer

Because LLMs are sequence models and LLM-based agents are perceived to be central to the future of AI, researchers have considered the potential for sequence model agency. This work has identified characteristic failure modes to which sequence models such as Decision Transformer are susceptible when used as agents; these cast doubt on their status as agents.

The first failure mode is that sequence models are subject to auto-suggestive delusions (Ortega et al. Reference Ortega, Kunesch, Delétang, Genewein, Jordi Grau-Moya, Buchli, Degrave, Piot, Perolat, Everitt, Tallec, Parisotto, Erez, Chen, Reed, Hutter, de Freitas and Legg2021). For example, consider a sequence model trained on the behaviour of an expert stock trader, which does not include all of the information they use to make decisions. Suppose that the trader’s practice is to call their broker only for one purpose: to make trades of over a million dollars. Now a situation can arise in which the sequence model is highly uncertain about what to do, because how the trader would behave depends on information to which it does not have access. In this situation, it might call the broker—perhaps the context implies that this is the most likely of several possible actions. Having called the broker, however, the model will assign a very high probability that the next action will be a trade with a value of over a million dollars, so this is what it will do. It will see its own previous output, of calling the broker, as very strong evidence that this is what happens next. That is, it will treat its own action as providing evidence that it lacked when it selected that action.

The second failure mode is predictor-policy incoherence (Douglas et al. Reference Douglas, Karwowski, Bae, Draguns and Krakovna2024). This problem can arise because the training data will typically not reflect the model’s own capabilities, with the consequence that the action most likely to succeed in the training data may not be the best action for the model. For example, consider Douglas et al.’s ‘three cards game’ (figure 1).

Figure 1.

Three cards game. From Douglas et al. (Reference Douglas, Karwowski, Bae, Draguns and Krakovna2024).

In this game, the player chooses three cards from a pack, taking either a diamond or a heart each time. If they choose three diamonds they receive a guaranteed payment of one unit. If they choose any other set starting with a diamond they receive nothing, but if they choose any set starting with a heart they receive one unit with 50% probability. Now consider a return-conditioned sequence model trained on random walks through the three cards game, which is prompted to obtain a return of one unit. If this model chose a diamond at the first step, it would be guaranteed to succeed, because after this choosing further diamonds is the only action that ever leads to reward. However, this is not what the model will do. It will begin with the action that most often leads to a return of one unit in the training data, which is choosing a heart. Because the data is made up of random walks, beginning with a heart leads to reward 50% of the time, whereas beginning with a diamond only leads to reward 25% of the time. Despite having all the relevant information, the model fails to choose the option that guarantees success.

One possible interpretation of these weaknesses is that sequence models fail to distinguish between their own actions and other events. In the case of auto-suggestive delusions, sequence models behave as though they can learn about what to do from their own earlier actions. In the case of predictor-policy incoherence, sequence models behave as though their own subsequent actions will be similar to those in the training data, rather than being under their control. Ortega et al. argue that auto-suggestive delusions arise because sequence models fail to treat their own actions as uncaused causal interventions; the problem, they write, is that ‘one cannot learn from one’s own actions, but only from their effects’ (2021, 6).

This line of thought seems to adopt a characteristic commitment of causal decision theory: that one should think of one’s own actions differently from other events. In Newcomb’s problem, causalists may say that what one should choose (two-boxing) is different from what one should want to happen (one-boxing, and thus receiving the higher reward). Something similar can be said of the three-card game: the action the agent should choose is different from the one that appears to succeed most often. It may be that some causalists believe that one is not really an agent if one does not think this way.Footnote ²

However, this analysis is either over-intellectualised, in attributing mistaken reasoning to sequence models, or relies too heavily on talk of models behaving as though they make certain reasoning errors. It is unclear what treating one’s own actions as different from other events means outside the context of reflective deliberation.

A better analysis is indicated by Douglas et al.’s (Reference Douglas, Karwowski, Bae, Draguns and Krakovna2024) finding that both problems can be alleviated by fine-tuning sequence models on their own behaviour. This reduces auto-suggestive delusions because in self-generated data the agent’s actions no longer provide evidence that was not otherwise available, and reduces predictor-policy incoherence because in this data the observed actions are the agent’s own. Furthermore, this is how simple model-free RL agents trained online avoid these two failure modes. They learn from data about the environment that they generate through their own interactions, and thus learn to select the actions that work for them. In contrast, sequence models imitate the behaviour of others that have different knowledge or dispositions.

We could decisively resolve the question of whether Decision Transformer is an agent by adopting this as a criterion for agency. To be an agent, a system must learn about the environment in part through interacting with it. This condition enjoys intuitive support from the idea that a system that is set up to behave in a particular way prior to contact with the environment is not an agent. Agential autonomy, perhaps, consists in one’s actions depending on one’s own contact with the environment (Russell and Norvig Reference Russell and Norvig2022). On this view Decision Transformer would not qualify.

That said, objections to this view are possible. One possible objection is that we can imagine a system that interacts with an environment, using model-based planning to select actions, but is equipped with the model in advance. This system would not meet the proposed condition but would select actions in a way that is characteristic of agency (Butlin Reference Butlin2024).

5. Purist and pragmatic conceptions of agency

It is not obvious whether Decision Transformer is an agent. In favour of the view that it is an agent, one could argue that it learns about the environment and the distribution of rewards, then uses this knowledge to select rewarding actions, performing capably on some tasks. Against, one could argue that it learns wholly offline with a prediction objective, like a mere imitator, and that the fact that it doesn’t learn from its own interaction with the environment leads to characteristic failure modes. As an alternative to trying to settle this dispute, we could attempt to diagnose it: to identify a tension between ideas about agency that explains why this case is difficult.

A promising way to do this may be to distinguish between a conception of agency on which agents work in a characteristic way, and one on which agents solve a particular kind of problem, perhaps imperfectly.Footnote ³ We can call the former a purist conception of agency and the latter a pragmatic conception. In the present case, I am imagining that the purist conception would deny that Decision Transformer is an agent while the pragmatic conception would accept it. But that need not be the case. There are a wide range of possible purist theories of agency and a wide range of possible pragmatic theories.

For a purist, being an agent is a matter of using agentic methods for learning and action selection. Characterising these methods is the central objective of research on agency, assuming a purist perspective. The claim that agents must learn from their own interactions with an environment could form part of a purist theory. One would expect the conditions in a purist account of agency to be related to standards of practical rationality—to be an agent is to learn and make choices as agents should—although agency must be consistent with some imperfections.

The pragmatic conception, in contrast, sees agency as a matter of functioning to solve a particular kind of problem, and thus as a matter of having certain capabilities. An initial account might describe the problem that agents solve as achieving goals through interaction with an environment. Agents vary in the generality, flexibility and reliability with which they are able to do this, and in the sources of their limitations. But goal-directedness and interaction are plausible necessary conditions for agency that may be seen as features of the task rather than the system. Pragmatic theorists of agency may well wish to add further conditions to this account, in order to narrow down the range of tasks seen as characteristically agentic—since, for example, the task that thermostats perform involves achieving a goal through interaction. For pragmatic theorists, developing and defending an account of the capabilities required for agency may be the central research objective.

Thinking of Decision Transformer as an agent accords with the pragmatic conception because the researchers who created it saw themselves as studying a new kind of solution to an agency-related problem. They write that they ‘seek to study if generative trajectory modeling – i.e. modeling the joint distribution of the sequence of states, actions, and rewards – can serve as a replacement for conventional RL algorithms’ (Chen et al. Reference Chen, Lu, Rajeswaran, Lee, Grover, Laskin, Abbeel, Srinivas and Mordatch2021, 3, emphasis in original). They go on to describe potential practical advantages of sequence modelling over RL. However, there are versions of the pragmatic conception that would entail that Decision Transformer is not an agent. The requirement that agents learn about the environment in part through interacting with it could just as well feature in an account of the problem that agents solve as in a purist theory describing a class of algorithms.

Hong Yu Wong (Reference Wong2025) distinguishes between architectural and behavioural perspectives on agency. According to Wong, the architectural perspective is the default position of most action researchers, and involves ‘extrapolating and abstracting from details of the capacities and internal architecture of [paradigm agents] to arrive at accounts of agency and action’ (4). Meanwhile, the behavioural perspective ‘focuses on the interpretability of candidate systems as agents based on observing the system’s behaviour’ (5). One difference from my distinction is that I think of purists as focused on algorithms rather than architectures—but Wong also talks about ‘mechanisms’, which can cover both. The main difference, however, is that the pragmatic conception of agency need not be purely behavioural. The problem of agency can plausibly be seen as the problem of achieving behavioural performance on the basis of training data or methods that meet certain constraints. For instance, one could argue that behaviour cloning is not sufficient for agency because it requires possession of a satisfactory policy in advance, whereas return-conditioned sequence modelling may be sufficient, because it can be used to discover satisfactory policies. This is not a behavioural difference but a difference in the data on which the system is trained before it begins to interact with the environment.