The goal of neuroscience is to explain how the brain enables intelligent behaviour, while the goal of agent-based artificial intelligence (AI) is to build agents that behave intelligently. Neuroscience, Brette attests, has suffered from an exaggerated (and technically inaccurate) concern for the codes transmitted by particular parts of the brain. In AI, on the other hand, some of the most notable recent progress has been made not by deeply considering neural coding and its implications, but by focusing on higher-level principles from optimization, learning, and control.

Thanks to deep artificial networks trained via backpropagation, we now have artificial learning systems capable of impressive exhibitions of specific human-like skills, such as object recognition and language translation (e.g., He et al. Reference He, Zhang, Ren and Sun2016; Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, Guyon, Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett2017). In artificial, rather than biological, neural networks, we can more tractably characterize the relationship between a model's neural codes, behaviour, and its external “world.” AI researchers have full access to their models' input data distribution, can visualise weights and activations in any part of the network and even make causal interventions on them, and can quickly implement new models informed by any coding hypotheses they may have.

Nevertheless, in-depth analysis of a model's internal representations is of increasingly rare concern for getting these models to work. Consider AlphaGo, which is one of the more compelling recent breakthroughs in AI (Silver et al. Reference Silver, Huang, Maddison, Guez, Sifre, Van Den Driessche, Schrittwieser, Antonoglou, Panneershelvam, Lanctot, Dieleman, Grewe, Nham, Kalchbrenner, Sutskever, Lillicrap, Leach, Kavukcuoglu, Graepel and Hassabis2016). Researchers on this project precisely defined the model's goals, the dynamics underlying the model's interactions with its environment, how the model plans its actions, and how the model learns. Each of these components contributes to the model's success, and yet none of them fundamentally depends on considerations from neural coding.

This is not to say that we cannot usefully apply representational analyses to such agents post hoc, regardless of whether the representations satisfy Brette's criteria for neural codes (Barack & Jaegle Reference Barack and Jaegle2019). Indeed, since the earliest days of connectionism, researchers have been interested in the neural codes that emerge when a clearly specified learning algorithm is applied to a well-understood model trained to execute a particular task. A more recent and important collaboration between AI and neuroscience revealed insight into the conditions under which well-known codes can emerge: Grid cells can increasingly be understood as the product of particular optimization processes (Banino et al. Reference Banino, Barry, Uria, Blundell, Lillicrap, Mirowski, Pritzel, Chadwick, Degris, Modayil, Wayne, Soyer, Viola, Zhang, Goroshin, Rabinowitz, Pascanu, Beattie, Petersen, Sadik, Gaffney, King, Kavukcuoglu, Hassabis, Hadsell, Kumaran and Wayne2018; Cueva and Wei Reference Cueva and Wei2018). A key feature of these examples, however, is the central descriptive role given to the learning algorithms, architectures, and optimization objectives; neural coding was incidental, and in many cases the codes were not fundamental, privileged primitives on top of which the models were built (Marblestone et al. Reference Marblestone, Wayne and Kording2016).

If the broader aim of agent-based AI (Russell & Norvig Reference Russell and Norvig2016) is to produce a system that accomplishes short- and long-term goals in a complex, changing environment, then there may be a more pernicious problem to the neural coding framework than it simply being out of vogue in modern AI. How internal responses arrive from given stimuli – a goal that is implicit in the neural coding metaphor – may be logically insufficient for producing intelligent behaviour. In outlining the reasons why, we recall arguments that any system – artificial or biological – needs to exert control over its environment to achieve intelligent behaviour.

First, the observations with which an agent may compute do not exist as a prespecified data set, independently of the agent's actions in the world. Rather, it is precisely the decisions that the agent takes in that world that determine the sensory data from which it learns. Second, “[w]ithout an ongoing participation and perception of the world there is no meaning for an agent” (Brooks Reference Brooks1991a, p. 16). An agent participating in an external world that responds to its decisions learns useful, reliable, and meaningful interactions (Cisek Reference Cisek1999). It is these meaningful interactions that ground the agent's representations and allow them to be used for understanding and reasoning about its world. Therefore, insofar as neural coding is understood as a framework to help understand a system's internal stimulus-response patterns, it is a logically insufficient framework for designing AI because of its failure to engage with the agent-environment causal loop.

Given these considerations, what, then, can we say about neural coding's role in describing the brain? In neuroscience, we ultimately care about understanding how the brain enables intelligent behaviour. It is often argued that such an understanding cannot come from analyzing low-level, mechanistic details such as neural codes, because “[a] description of neural activity and connections is not synonymous with knowing what they are doing to cause behavior” (Krakauer et al. Reference Krakauer, Ghazanfar, Gomez-Marin, MacIver and Poeppel2017). For this level of understanding, we need high-level computational and algorithmic theories that embrace agent-environment interactions. The history of AI tells us that the most useful principles, and the richest theoretical insights, emerged from studying control, optimization, and learning processes rather than the particularities of representations or codes (Sutton Reference Sutton2019). A focus on inferring such processes using our increasing quantities of neural data, rather than characterizing neural codes for their own sake, may also be the most productive way of making progress on understanding intelligent behaviour in humans and animals.