We agree that the target article is right in highlighting some of the sloppiness with which information theory is used in neuroscience, and that in some cases the term code is used to describe the relationship between neural activity and sensory input, when actually the term correlate would be more appropriate. We have previously argued that neuroscience all too often focuses on what an experimenter can decode from neural activity, rather than testing what (or whether) the brain might be able to decode from that activity (de-Wit et al. Reference de-Wit, Alexander, Ekroll and Wagemans2016). Rather than reflecting a fundamental problem with information theory, however (which always clearly acknowledged the importance of a receiver), we argue this actually reflects the way in which “coding” is used too loosely as a metaphor.
Information theory always required a careful consideration of the sender, channel, and receiver. We have previously argued that neuroscience is often too hasty in assuming that the “channel” used by the brain to convey information is the firing rate of neurons when, of course, there are many aspects of neural activity (synchrony, precise timing) which could also be used as the channel over which information is conveyed. Again, it is not the fault of information theory that there is insufficient consideration in neuroimaging regarding the channel by which information might be conveyed. This is particularly significant for strong claims regarding an “absence” of information when using fMRI, which is completely unable to detect information that might be conveyed by precise temporal coding.
One of the other weaknesses highlighted in the target article, namely, a lack of consideration of context effects, also does not reflect a fundamental problem with information theory, but rather a limited theoretical framework within visual neuroscience. We would argue that the Gestalt tradition offers a much clearer insight into some of the challenges faced in processing sensory input, but that modern neuroscience largely tries to side-step these challenges and simply focuses on correlations between “parts” of the input and neural activity. From a Gestalt perspective trying to understand how “parts” might be represented in the brain without also considering how those parts will form together into “wholes” was obviously going to be a limited enterprise from the start. Indeed, even seemingly simple “parts” like edges actually need to considered in context to understand what information neural signals might be conveying (Kogo & Wagemans Reference Kogo and Wagemans2013).
Finally, however, the target article does make a stronger more fundamental argument, that the coding metaphor is wrong because information theory can account only for the reference between information states and objective properties of the world. Here we also agree that perception is not simply a process of “re-presenting” the “objective” properties of the external world. This is a bigger challenge for information theory, but we would argue that this challenge always should have been at the heart of how information theory was used in neuroscience.
Indeed, the focus on understanding the “goals” of different information processing systems was clearly articulated as an important part of Marr's (Reference Marr1982b) levels of analysis, but is often neglected in modern neuroscience. Indeed, much neuroscience starts from an assumption that there is only one version of reality that can be derived from sensory input, and that the job of the visual system is simply to “optimally decode” that representation of reality. A biologically plausible theory of information processing also has to address the goals of the organism (as Marr made clear) to then think about what kinds of representations might be useful to achieve those goals. A frog's visual system may extract transient motion signals to provide information about food/flies; a stickleback fish may represent curved contrast boundaries with specific wavelengths of light to provide information about competing mates; a human visual system will combine different views of the same object to provide information that this is the same object over time. Thus, what different organisms do with sensory evidence will differ depending on their umwelt (von Uexküll Reference von Uexküll1992) and their evolutionary needs, but we would argue it is still useful to think about those perceptual systems as representing the information that is useful for (and subjective to) their goals.
The idea that different representations might be derived from the same message is not an entirely alien concept for information theory. The content of an encrypted message, for example, is dependent on the receiver having the right decryption key. Thus, decoding is always subjective not only to the content from the sender, but to the computations performed by the receiver. Different visual systems will most certainly decode different representations from the same input but, clearly framed in this way, we would argue that information theory is not necessarily the problem here; rather, the problem is many of the assumptions that surround its use.
Whether or not information theory will ultimately prove useful in explaining the activity of the brain is of course an empirical question, but we would argue that more progress could be made with information theory if the problem of perception were conceived more correctly. The target article is right in highlighting many of these problematic assumptions, and we agree we cannot look for neural codes in any sensible way without questioning these assumptions. But we would question whether the problem really lies with information theory, or the lack of a psychologically and biologically plausible account of the information processing challenges our brain has to solve. To paraphrase Mausfeld (Reference Mausfeld2003), if we don't put any substantive psychological theory into the use of information theory in neuroscience, then we can't expect any plausible psychological answers from it.
We agree that the target article is right in highlighting some of the sloppiness with which information theory is used in neuroscience, and that in some cases the term code is used to describe the relationship between neural activity and sensory input, when actually the term correlate would be more appropriate. We have previously argued that neuroscience all too often focuses on what an experimenter can decode from neural activity, rather than testing what (or whether) the brain might be able to decode from that activity (de-Wit et al. Reference de-Wit, Alexander, Ekroll and Wagemans2016). Rather than reflecting a fundamental problem with information theory, however (which always clearly acknowledged the importance of a receiver), we argue this actually reflects the way in which “coding” is used too loosely as a metaphor.
Information theory always required a careful consideration of the sender, channel, and receiver. We have previously argued that neuroscience is often too hasty in assuming that the “channel” used by the brain to convey information is the firing rate of neurons when, of course, there are many aspects of neural activity (synchrony, precise timing) which could also be used as the channel over which information is conveyed. Again, it is not the fault of information theory that there is insufficient consideration in neuroimaging regarding the channel by which information might be conveyed. This is particularly significant for strong claims regarding an “absence” of information when using fMRI, which is completely unable to detect information that might be conveyed by precise temporal coding.
One of the other weaknesses highlighted in the target article, namely, a lack of consideration of context effects, also does not reflect a fundamental problem with information theory, but rather a limited theoretical framework within visual neuroscience. We would argue that the Gestalt tradition offers a much clearer insight into some of the challenges faced in processing sensory input, but that modern neuroscience largely tries to side-step these challenges and simply focuses on correlations between “parts” of the input and neural activity. From a Gestalt perspective trying to understand how “parts” might be represented in the brain without also considering how those parts will form together into “wholes” was obviously going to be a limited enterprise from the start. Indeed, even seemingly simple “parts” like edges actually need to considered in context to understand what information neural signals might be conveying (Kogo & Wagemans Reference Kogo and Wagemans2013).
Finally, however, the target article does make a stronger more fundamental argument, that the coding metaphor is wrong because information theory can account only for the reference between information states and objective properties of the world. Here we also agree that perception is not simply a process of “re-presenting” the “objective” properties of the external world. This is a bigger challenge for information theory, but we would argue that this challenge always should have been at the heart of how information theory was used in neuroscience.
Indeed, the focus on understanding the “goals” of different information processing systems was clearly articulated as an important part of Marr's (Reference Marr1982b) levels of analysis, but is often neglected in modern neuroscience. Indeed, much neuroscience starts from an assumption that there is only one version of reality that can be derived from sensory input, and that the job of the visual system is simply to “optimally decode” that representation of reality. A biologically plausible theory of information processing also has to address the goals of the organism (as Marr made clear) to then think about what kinds of representations might be useful to achieve those goals. A frog's visual system may extract transient motion signals to provide information about food/flies; a stickleback fish may represent curved contrast boundaries with specific wavelengths of light to provide information about competing mates; a human visual system will combine different views of the same object to provide information that this is the same object over time. Thus, what different organisms do with sensory evidence will differ depending on their umwelt (von Uexküll Reference von Uexküll1992) and their evolutionary needs, but we would argue it is still useful to think about those perceptual systems as representing the information that is useful for (and subjective to) their goals.
The idea that different representations might be derived from the same message is not an entirely alien concept for information theory. The content of an encrypted message, for example, is dependent on the receiver having the right decryption key. Thus, decoding is always subjective not only to the content from the sender, but to the computations performed by the receiver. Different visual systems will most certainly decode different representations from the same input but, clearly framed in this way, we would argue that information theory is not necessarily the problem here; rather, the problem is many of the assumptions that surround its use.
Whether or not information theory will ultimately prove useful in explaining the activity of the brain is of course an empirical question, but we would argue that more progress could be made with information theory if the problem of perception were conceived more correctly. The target article is right in highlighting many of these problematic assumptions, and we agree we cannot look for neural codes in any sensible way without questioning these assumptions. But we would question whether the problem really lies with information theory, or the lack of a psychologically and biologically plausible account of the information processing challenges our brain has to solve. To paraphrase Mausfeld (Reference Mausfeld2003), if we don't put any substantive psychological theory into the use of information theory in neuroscience, then we can't expect any plausible psychological answers from it.