We advocate an approach that grounds claims about neural codes in evidence about the brain and mind. Under this approach, the search for neural representations begins with an understanding of the task that an organism solves, in the spirit of rational analysis (Anderson Reference Anderson, Roediger and Craik1989). The next step is to propose computational models capable of solving this task. These models constitute candidate hypotheses for the representations and computations employed within the brain, allowing us to establish principled constraints on possible neural codes and give strict satisfaction conditions for their implementation.

This program portrays brains as representational and computational devices. Brette's arguments challenge this foundational idea. We address his arguments on three fronts: (1) the indeterminacy of claims about the neural code, (2) the ability of neural networks and dynamical systems to instantiate structured representations, and (3) the historical success of the search for representations in the brain.

First, Brette argues that many studies of neural codes fail to account for relevant contextual features in ecological behaviors. Brette criticizes studies of neural codes driving sound localization, for example, which neglect how contextual features such as sound amplitude interact with frequency. Brette draws the conclusion that neuroscientific evidence underconstrains the search for the contents of neural codes. But this search can be helpfully constrained by the computational theories introduced above. As accounts of ecological behavior, computational models both generate and constrain our hypotheses about the environmental features relevant for a given task. Such research puts us in a position to make educated guesses as to the environmental features that are likely to be represented in neural codes.

For example, computational models of vision and audition describe how ecological tasks (e.g., object recognition) are carried out via computational mechanisms. These models demonstrate both why representations with particular content are present in the mind and how they are used to produce behavior. After being empirically tested against observed organism behavior, their internal representations can be used to guide the search for neural codes (see, e.g., Rajalingham et al. Reference Rajalingham, Issa, Bashivan, Kar, Schmidt and DiCarlo2018; Młynarski & McDermott Reference Młynarski and McDermott2018). This constraint – that neural codes must implement the empirically validated representations of computational models – greatly reduces an otherwise open-ended and intractable search for neural representations.

Second, computational-level models can help us narrow down candidate neural representations, but such a search will be fruitful only if representations can actually be realized in neural circuitry. Brette argues that this cannot be done. We think that the evidence suggests otherwise: Both connectionist models and neural dynamical systems are capable of implementing structured representations.

Brette worries that neural codes composed of cell assemblies can encode only “objects or features to be related, but not the relations between them,” and points to connectionist models as evidence. But connectionist researchers have worked to address this issue and have adapted models to learn relational structure, since almost the inception of the field (McClelland Reference McClelland and Rogers2003; Smolensky & Legendre Reference Smolensky and Legendre2006). Indeed, modern neural networks continue to challenge and broaden our understanding of the kinds of relational structures they can learn to represent when designed and trained carefully: from abstract, hierarchical syntactic dependencies in natural language (Gulordava et al. Reference Gulordava, Bojanowski, Grave, Linzen and Baroni2018), to dynamics in intuitive physical systems (Fragkiadaki Reference Fragkiadaki, Agrawal, Levine and Malik2016; Chang Reference Chang, Ullman, Torralba and Tenenbaum2017), to complex relationships between objects in visual scenes (Hudson and Manning Reference Hudson and Manning2018; Johnson Reference Johnson, Hariharan, van der Maaten, Hoffman, Fei-Fei, Lawrence Zitnick and Girshick2017; Yi Reference Yi, Wu, Gan, Torralba, Kohli, Tenenbaum, Bengio, Wallach, Larochelle, Grauman, Cesa-Bianchi and Garnett2018). The need to represent complex structures can and should drive us to think creatively about how cell assemblies can instantiate them, rather than abandon the project because of arguments from inconceivability leveled by its detractors.

Brette also alleges that “dynamical systems cannot in general be mapped to algorithmic descriptions” (sect. 4.1, para. 1). But recent theoretical work offers paradigms using recurrent networks to do precisely that (Eliasmith and Anderson Reference Eliasmith and Anderson2004; Mastrogiuseppe and Ostojic Reference Mastrogiuseppe and Ostojic2018). Concurrently, additional work has suggested that it is possible to map latent states of behavior to dimensions of neural activation (Afraz and Jazayeri Reference Jazayeri and Afraz2017) and verified predictions about the encoding of more complex tasks in higher-dimensional spaces (Gao et al. Reference Gao, Trautmann, Yu, Santhanam, Ryu, Shenoy and Ganguli2017). Once we accept that population-level neural dynamics provides a substrate for representations, it makes sense to look for algorithmic transformations of those representations, that is, a code. All signs indicate that a major goal of neuroscience in the near future will be identifying context-invariant subspaces of neural activity that act as such a code (Saxena and Cunningham Reference Saxena and Cunningham2019).

Finally, this integrated approach – one that begins by understanding the task, proposes algorithms to compute it, and finds its representations in the brain – is exemplified by research on temporal difference learning. Temporal difference is a reinforcement learning algorithm that requires representing the current reward and the predictive value of the current state, and computing reward prediction error. Early research on the neural implementation of reinforcement learning showed that reward prediction error correlated with phasic dopamine signals (Montague Reference Montague, Dayan and Sejnowski1996; Schulz Reference Schultz, Dayan and Montague1997). Correlational evidence, as Brette is quick to point out, does not imply neural coding. But research on temporal difference learning in the brain has gone far beyond correlation. This single representational model has yielded continuous empirical successes: predicting neural responses, both quantitatively and qualitatively (Niv Reference Niv2009), causally manipulating neural responses and observing predicted behaviors (Steinberg Reference Steinberg, Keiflin, Boivin, Witten, Deisseroth and Janak2013), and discovering the mechanisms responsible for key symptoms of Parkinson's disease (Frank Reference Frank, Seeberger and O'Reilly2004). If such representations are not actually instantiated in the brain, this streak of results spanning more than two decades would be nothing short of a miracle.

We believe that principled constraints can be placed on the search for neural codes. These constraints come from integrating multiple levels of analysis, including an understanding of the task being solved, a hypothesis space of algorithms capable of solving it, and behavioral and neuroscientific evidence to decide between candidate hypotheses. We've highlighted work showing how structured representations can be implemented in connectionist and dynamical systems models. We've described one strikingly successful search for neural codes in the brain. Together, these successes suggest that Brette's skepticism is unfounded. This integrated search for neural codes remains the best framework for understanding the brain.