1. Introduction
The first decade of the twenty-first century has seen an extraordinary increasing number of publications reporting research into how the brain processes music from a neuroscientific perspective. A relatively new field of research, referred to as the neuroscience of music,Footnote 1 is actively thriving (Zatorre and Peretz Reference Zatorre and Peretz2001; Peretz and Zatorre Reference Peretz and Zatorre2003; Avanzini, Lopez, Koelsch and Majno Reference Avanzini, Lopez, Koelsch and Majno2005).
Music is an interesting topic for the neurosciencesFootnote 2 because it engages a number of neural mechanisms in concerted ways. It is an ideal domain for the study of brain connectivity: how different parts of the brain communicate, interact with one another and perform concurrent multitasking (Miranda and Overy Reference Miranda and Overy2009). The levels of engagement of those neural mechanisms are very much dependent on the intrinsic physiological properties of the brain and on learning from past experience. Moreover, specific characteristics of the auditory stimuli also determine how the brain processes them. In simpler words, different types of music engage the brain differently.
For musicians and developers of music technology, the prospect of being able to gain a better understanding of the musical brain is very exciting. Composers would certainly welcome the possibility of being able to predict more objectively the effect of particular musical configurations on their audiences. Furthermore, we believe that such understanding may lead to the development of new music technologies based on, or inspired by, the neurosciences. To some extent such developments have already started: a new area of investigation at the crossroads of the neurosciences, engineering and music, music neurotechnology, is on the rise (Miranda and Matthias Reference Miranda and Matthias2009; Miranda, Bull, Gueguen and Uroukov Reference Miranda, Bull, Gueguen and Uroukov2009).
However, despite its thriving research activity, the neuroscience of music still remains largely uncharted for musicians and unexplored by technologists. The literature is complex and highly specialist, making it difficult to disentangle for most of those who are not neuroscientists. Obviously, this literature, which is written by neuroscientists for neuroscientists, addresses issues concerning the brain and not music as such.
Conversely, the field of music remains largely uncharted for neuroscientists. Contemporary music developments, such electroacoustic music, remain anathema to the great majority of researchers into neuroscience of music. Lack of beat or metre, noise, non-melodic pitch sequences, and cacophony are well-accepted components of our contemporary musical vocabulary, and yet the great majority of the research focuses on regular rhythm, pitch, melody and tonality.
Notwithstanding its current narrow scope, research into the neuroscience of music is nevertheless unravelling a number of facts about music processing in the brain. These are likely to apply across the board, despite the existence of a plethora of musical genres and different philosophical takes on what music might or might not be. However, any attempt at putting the pieces of the musical brain jigsaw together is not a trivial task, and indeed numerous parts of the jigsaw are still missing. The objective of this paper is threefold.
Firstly, it endeavours to put together a few pieces of the musical-brain jigsaw, as an attempt to offer the readership of this journal a reasonable picture of how our brain listens to music – that is, organised sound– from a neuroscience perspective.
Secondly, as the paper introduces the technical aspects of the neural basis of listening, it unpacks a working hypothesis for the neural basis of mental imageries elicited by music, based on the notion that such imageries are by-products of the inherent abstracting and predicting properties of the brain. In this sense, such mental imageries are regarded as scaffolds for music perception. Our objective is not, however, to propose yet another definition of musical imagery (see for example Godoy and Jorgensen Reference Godoy and Jorgensen2001), but to identify neural mechanisms that may yield such phenomena. In this context, musical mental imagery may be a rather abstract feeling (e.g., ‘this sound feels heavy’), and it may involve associations with other sensory modalities, particularly in people who are prone to synaesthesiaFootnote 3 (e.g., lexical–gustatory synaesthesia whereby flavours are evoked by music).
Thirdly, the paper indicates how a better understanding of our musical brain may lead to the development of new music technologies, especially for electroacoustic music.
The paper is organised as follows. It begins with a review of the physiology of the auditory pathways, including the early auditory pathways and cortical auditory pathways. Then it discusses the notions of abstraction, representation, anticipation and organisation of auditory information, and their roles in the development of mental imageries elicited by music. Next, it presents a tentative functional map of the musical brain, followed by a speculative discussion on the future of electroacoustic music technology based on, or inspired by, the neurosciences.
2. Auditory Pathways
The auditory system may be divided into two pathways, or stages: early auditory pathways, which lead from the outer ear through a number of subcortical regions and terminate in the primary auditory cortex, and cortical auditory pathways, which pass out activation from the primary auditory cortex to a number of other cortical areas (figure 1). Whereas the early pathways are essentially the same for all types of sounds (i.e., in the sense that pretty much everything we hear passes through the same path), the cortical pathways largely depend on the nature of the auditory signal; that is, sounds may go through different cortical pathways depending on their specific characteristics. Also, it must be noted that early auditory pathways deal mostly with the analysis of single relatively short events, whilst cortical auditory pathways are primarily concerned with sequencing, grouping, making abstractions, building representations and a number of other activities associated with music. Cortical pathways enjoy greater plasticityFootnote 4 than early pathways and their functioning are subject to learning and acculturation.Footnote 5 However, the fact that sounds go through the same early pathways does not mean that everybody processes them in the same way at this stage. The brain differs slightly from person to person; therefore, degrees of functionality may differ slightly in the early auditory pathways.
The early auditory pathways begin at the ear and ends at the auditory cortex. 1 = auditory nerve, 2 = cochlear nuclei, 3 = superior olivary complex, 4 = lateral lemniscus, 5 = inferior colliculus, 6 = thalamus, 7 = transverse termporal gyri (primary auditory cortex). The schematic figure is combined with two fMRI scans indicating approximately where the various components are located, viewing from the sagittal (to the right of the figure) and coronal (to the top of the figure) planes.
2.1. Early auditory pathways
While the pathways are introduced below in terms of ascending pathways, the reader must bear in mind that there are also descending pathways, which trace identical routes back from the primary auditory cortex all the way to the ear. The specific functions of the descending pathways are still subject to much debate. It is generally agreed that they are involved in top-down processes modulating the early processing of incoming sounds on the basis of previous sounds (Schofield and Cant Reference Schofield and Cant1999). This notion is fundamental for the discussions on mental imageries elicited by music, which will be introduced later.Footnote 6
The early auditory pathways will be introduced below in terms of major points of functional convergence of neuronal structures, where significant auditory signal processing seems to take place: cochlear nuclei, lateral lemniscus, superior olivary complex, inferior colliculus and thalamus (figure 1).
2.1.1. Outer ear
Auditory signal processing in humans starts with the outer ear, which channels sounds towards the tympanic membrane, or eardrum. The tympanic membrane vibrates in response to air pressure changes and this vibration is relayed through the middle ear via three small bones: the malleus, the incus and the stapes. The last of these is connected to the oval window, which leads into the cochlea, which in turn is filled with fluid. The spiral-shaped cochlea forms the inner ear, and is responsible for transforming liquid motion into electrical neural signals. The movement of the liquid results in the movement of the basilar membrane and the subsequent stimulation of motion-sensitive hair cells. The basilar membrane gets wider towards the centre of the spiral, and also decreases in stiffness. These two factors give it the crucial property of tonotopicFootnote 7 frequency selectivity: hair cells at given locations will respond maximally to particular frequenciesFootnote 8 and do so in an ordered fashion, with the high frequencies stimulating cells near the outer edge of the spiral, and the low frequencies stimulating cells near the centre. Note, however, that this notion of tonotopy in the cochlea is not universally accepted. There are arguments suggesting that hair cell’s frequency selectivity is actually derived from the auditory cortex. A compromise here would be to consider that such tonotopy is due to a combination of number of factors, which may indeed include an important contribution from the auditory cortex; hence the importance of the descending pathways mentioned earlier.
Spiral ganglion cells synapse onto the hair cells, and due to a pattern of innervating nearby cells, also show tonotopic frequency selectivity. They fire action potentials when stimulated, and their firing rate is approximately linearly related to the intensity of the sound, hence this information is also encoded. Altogether, the ear takes differences in air pressure, transforms them first into liquid motion, and then into tonotopically organised neural signals that encode both frequency and intensity (i.e., loudness) information.
2.1.2. Cochlear nuclei
The axons of the spiral ganglion cells of the cochlea form the auditory nerve (1, in figure 1), and from here the signal moves from the ear to subcortical brain regions, more specifically to the brainstem. The auditory nerve terminates in the cochlear nuclei (2, in figure 1), which can be divided into one dorsal and two ventral regions. The ventral cochlear nuclei contain stellate cells, which are frequency-specific and encode intensity information within their firing rate, and bushy cells, which fire once at a stimulus onset, thus providing sound onset timing information. As there are also substantial connections between the cochlear nuclei of the two hemispheres (Shore, Godfrey, Helfert, Altschuler and Bledsoe Reference Shore, Godfrey, Helfert, Altschuler and Bledsoe1992), bushy cells also provide enough information to start to encode horizontal position information (since this is dependent on timing between the two ears). The dorsal cochlear nucleus contains fusiform cells,Footnote 9 which appear to be involved in vertical position encoding, and tuberculoventral cells, which are involved in identifying and suppressing the response of ventral bushy cells to echo sounds, thus allowing a very rapid discrimination between a source sound and its echo.
2.1.3. Lateral lemniscus
There are several pathways from the cochlear nuclei onwards. The dorsal acoustic stria is a collection of axons that represent one of the main pathways, leading from the dorsal cochlear nucleus to both the lateral lemniscus (in the pons: 4, in figure 1) and the inferior colliculus (in the midbrain: 5, in figure 1). Some of the axons from the cochlear nuclei terminate in the lateral lemniscus, but others use it as a relay station, passing their signal to other neurons that move on to the inferior colliculus. However, the majority simply pass right through the lateral lemniscus, projecting directly to the inferior colliculus. Like the cochlear nuclei, the lateral lemnisci in both hemispheres are also connected, allowing some form of binaural processing at this stage.
It is not entirely clear why most axons pass right through the lateral lemniscus, projecting directly to the inferior colliculus, whereas only a few connect to neurons in the lateral lemniscus, and others even terminate here. It is readily understood nevertheless that the brain begins to process binaural information very early on, and it seems that some sort of filtering is taking place here.
2.1.4. Superior olivary complex
In addition to the dorsal pathways (towards the upper side of the brain), the main ventral pathway (towards the lower side of the brain) projects first to the superior olivary complex (3, in figure 1) in both hemispheres via the trapezoid body (Bear, Connors and Paradiso Reference Bear, Connors and Paradiso2001). The medial superior olive (part of the olivary complex) appears to be involved in identifying source location through the use of timing information (which is more sensitive to low-frequency sounds), while the lateral superior olive (part of the olivary complex) appears to identify location by using sound intensity information instead (which is more sensitive to high-frequency sounds). Thus two different types of source location mechanisms are available at this early point in the auditory pathway. Basically, they sense that sounds should be louder in the ear next to the source. These combined with frequency information allows for rapid processing of localisation information.
2.1.5. Inferior colliculus
The inferior colliculus (5, in figure 1) is where the various early auditory pathways once again converge and it appears to be engaged in further processing of source location. The inferior colliculus also receives inputs from inferior colliculus in the opposite hemisphere. It is arranged in layers, and orthogonal to these layers the organisation seems to reflect the tonotopic map of the basilar membrane. Two additional important properties to note here are: (a) it receives somatosensory inputs and (b) it has much greater processing power than lower structures, having some 400,000 neurons, as opposed to about 34,000 in the superior olivary complex (Worden Reference Worden1971).
Perhaps as a result of these properties, the inferior colliculus function as some sort of processing bottleneck in the auditory system. Lower inputs converge upon on it, and higher inputs to the superior colliculus (which performs early integration with the visual system), to the reticular formation (involved in autonomic processing), to the cerebellum (important centre of motor activity) and to the thalamus are projected from it.
Interestingly, and of particular relevance to composers interested in using space as a compositional parameter, the processing of spatial localisation of sound occurs very early on in the auditory system.Footnote 10 The fact that the inferior colliculus sends some information bypassing the auditory cortex to vital areas of the brain – such as the superior colliculus (which performs early integration with the visualsystem), the reticular formation (involved in autonomic processing) and the cerebellum (involved in motor coordination) – indicates that the brain begins to use information on sound localisation very early indeed.
2.1.6. Thalamus
The thalamus (6, in figure 1), or more precisely, a part of the thalamus referred to as the medial geniculate nucleus (MGN), represents the next stage in the early auditory pathways. It is considered to be the gateway to the cerebral cortex. The thalamus is largely responsible for control of attention; for example, it enables us to focus on a particular instrument from all other sounds in an orchestral piece. It seems that its role is to direct information to the cortex or to suppress it. The MGN receives inputs from the inferior colliculus and from a variety of other areas, which include visual and somatosensory areas. It is here that sensory information from different modalities is combined (Hodges, Hairston and Burdette Reference Hodges, Hairston and Burdette2005) and where visual information, for example, can effectively modulate the auditory signal. Neurons in the MGN seem to be organised tonotopically and are frequency-specific (i.e., different groups of neurons respond to specific bands of frequencies) but at much broader tuning curves than the neurons earlier in the auditory pathway. Some neurons here also specifically encode sound intensity level.
2.2. Cortical auditory pathways
The MGN projects mostly onto the transverse temporal gyrus,Footnote 11 bilaterally (i.e., on both lateral sides of the brain), popularly known as the primary auditory cortex – hereafter referred to as A1 (7, in figure 1). Some of it also projects to other neighbouring regions, such as the secondary auditory cortex. As in preceding areas of the early auditory pathway, A1 is also tonotopically organised (figure 2).
The early cortical pathways end at the transverse temporal gyri, also known as the primary auditory cortex. The primary auditory cortex is tonotopically organised, reflecting the organisation of hair cells in the basilar membrane. In the auditory cortex, low frequencies are represented laterally, towards to the surface of the cortex, and high frequencies represented medially, towards to the centre of the brain. (The image of the brain was published in Nolte and Angevine (Reference Nolte and Angevine2007) and is reproduced with permission from Mosby/Elsevier.)
Located on the superior surface of the temporal lobe, A1 is a crucial neural area for auditory processing, playing a central role in a process that has been referred to as auditory Gestalten (Koelsch and Siebel Reference Koelsch and Siebel2005). It communicates with a number of areas, which contribute to the formation of such Gestalten; for instance, its connectivity with the cerebellum indicates influence of motor function in auditory processing, and vice versa.
A variety of functional cortical networks develop from A1. One important distinction that appears in the areas immediately surrounding A1 is between a ‘what’ pathway and a ‘where’ pathway,Footnote 12 with ‘what’ information being processed anterior to A1 on the superior temporal gyrus, and ‘where’ information being processed posterior to A1 (Arnott, Grady, Hevenor, Graham and Alain Reference Arnott, Grady, Hevenor, Graham and Alain2005).
The early auditory pathways end at A1, but auditory processing certainly does not. However, from here onwards, there is no longer a clear principal route for the auditory signal as we have seen in the early auditory pathways. Instead, which cortical areas are actively involved depend largely on the characteristics of the sounds in question. Clearly, the brain does not process music as a single monolithic entity. Rather, music is processed as a set of auditory characteristics (pitch, rhythm, spectrum, etc.), each of which may engage distinct neural structures, or music mental modules Footnote 13 (Peretz and Morais Reference Peretz and Morais1989; Besson and Schön Reference Besson and Schön2003; Peretz and Coltheart Reference Peretz and Coltheart2003).
The notion of cortical music mental modules (hereafter referred to as MMM) is more theoretical than strictly anatomical. A MMM is not necessarily a particular cluster of neurons that one could clearly pin down in the brain. Rather, it should be thought of as a collection of neural structures, which are likely to be spread out across the brain, that together tackle a particular class of problems. For instance, the MMM for processing pitch seems to involve (but not exclusively): the superior temporal gyrus, which includes A1 (especially on the right hemisphere) (Griffiths Reference Griffiths2003); the cerebellum (Parsons Reference Parsons2003, Reference Parsons2005); and the sensorimotor cortex in the parietal lobe (Halpern Reference Halpern2003). In contrast, the MMM for processing rhythm seems to involve (but not exclusively): the left inferior temporal gyrus, the bilateral middle temporal gyrus (Parsons Reference Parsons2003) and the right frontal operculum (Limb, Kemeny, Ortigoza, Rouhani and Braun Reference Limb, Kemeny, Ortigoza, Rouhani and Braun2006). At the Interdisciplinary Centre for Computer Music Research (ICCMR) in Plymouth, we detected evidence that the MMM for processing changes in tonal key may include A1, right insula and left anterior cingulated gyrus (Durrant, Hardoon, Miranda, Shawe-Taylor, Brechmann and Scheich Reference Durrant, Hardoon, Miranda, Shawe-Taylor, Brechmann and Scheich2007; Durrant, Hardoon, Brechmann, Shawe-Taylor, Miranda and Scheich Reference Durrant, Hardoon, Brechmann, Shawe-Taylor, Miranda and Scheich2009). Figure 3 shows a functional magnetic resonance imaging (fMRI) image where increase in blood flowFootnote 14 was detected in such areas in response to modulations of chord sequences around the circle of fifths. Obviously, some of these neural structures may participate in more than one MMM. For instance, Parsons (Reference Parsons2003) found that the cerebellum is also involved in processing rhythm. Obviously, the neural structures of an MMM are not necessarily dedicated to processing music only. They may overlap with neural structures for other functions such as body movement, speech and emotion, to cite but three. Table 1 lists examples of roles of the aforementioned neural structures in tasks other than music.
A functional magnetic resonance imaging (fMRI) brain scanning experiment with modulations around the circle of fifths detected increased blood flow in a number of neural structures, including bilateral activation of transverse temporal gyrus (1, 2), right insula (3) and left anterior cingulate gyrus (4). Top of the image is the front of the head. Thus, the image is reversed; that is, the left side of the brain is shown on the right side of the image, and vice versa. Talairach x, y, z coordinates are as follows: right transverse temporal gyrus = {51, −17, 10}, left transverse temporal gyrus = {−51, −18, 11}, right insula = {36, 17, 13} and left anterior cingulate gyrus = {−1, 41, 11}. Talairach coordinates are used to describe the location of brain structures on a generic model of the brain, independent from individual differences and size.
Examples of non-musical roles of neural structures engaged in processing music.
There have been a number of reports on a condition referred to as acquired amusia Footnote 15 (Peretz Reference Peretz2003), where selective impairments in music recognition abilities occurred after brain damage. Amusia can selectively impact pitch, interval, contour, rhythm, metre, timbre and emotional response (Brust Reference Brust2003), which further supports the existence of MMMs for different aspects of music rather than a single, unitary musical processing system in the brain. This should not necessarily imply that mental modules are readily available at birth. It is more likely that the cortex may to some extent begin as a general-purpose processor that becomes progressively more specific (Altenmüller Reference Altenmüller2003). There certainly are, however, regions of the cortex that are particularly relevant for musical processing from birth (for instance, A1), which, during development and in engagement with some form of musical activity, become progressively more reactive to music. For instance, ask a non-musician to close their eyes and imagine for a moment the sound of one note (any note) on an instrument, say a flute. Then, ask the same person to imagine the instrument itself. Ask if they can hear the note in their ‘mind’s ear’ as clearly as they can depict a flute in their ‘mind’s eye’. The answer would most probably be negative. But if you do the same exercise with a musician, then the answer would most probably be affirmative. Quoting Damasio (Reference Damasio2001: 59) ‘We come to this world with a brain equipped with a variety of preset circuits. Most of these have to do with life regulation; they are located in the diencephalon and brain stem and they regulate basic biological functions, which ensure our survival. After birth, most of the non-preset brain circuitry begins to be shaped by our own activities and experiences. The term “plasticity” refers to that reshaping. Because our encounters with the environment are unique, the brain circuitries are shaped somewhat differently in each of us.’ To a large extent, to understand in greater detail how the brain of a particular individual processes music, one would need to have access to their ‘learning biography’, some sort of ontomemetic memory (Gimenes and Miranda forthcoming), reflecting how particular neurological bonds developed as a result of particular experiences.
So far we have stressed that the nature of the musical stimuli and the plasticity of the cortex define the cortical auditory pathways and the extent to which certain MMMs may contribute to processing music. However, little has been said about another, perhaps even more important, detail: the type of the musical activity in question. Listening to music played through loudspeakers comfortably seated on a cosy sofa in the sitting room, and listening to a work-in-progress piece of electroacoustic music as it is being mixed in the studio are different activities, which would most certainly engage the brain differently. And so would sight-reading a Beethoven piano sonata and improvising in a jam session on the piano. The auditory brain mechanisms discussed so far are somewhat generally applicable to the great majority of musical activities. But from hereafter the discussion becomes increasingly less generalised. Therefore, unless otherwise stated, from now on the first scenario above is assumed: that is, listening to music played through loudspeakers.
3. Sequencing and Higher-order Structures
Obviously, the processing of a single sound is not music processing. Rather, music concerns sequences of sounds organised in some way; it is a time-based art form. Perhaps the most straightforward form of sound sequencing in music is the melody, which would normally involve sequencing sounds in function of their pitch.
Given that pitch and rhythm are important constituents of melody, the MMM for melody processing would certainly include neural structures for processing pitch and rhythm. These may be combined additively or interactively with respect to particular cognitive functions, such as phrase perception (Palmer and Krumhansl Reference Palmer and Krumhansl1987; Jusczyk and Krumhansl Reference Jusczyk and Krumhansl1993) or metre perception (Hannon, Snyder, Eerola and Krumhansl Reference Hannon, Snyder, Eerola and Krumhansl2004). In addition, melody processing involves representations that span over time. The neural structures of the MMM for processing melody therefore include not only those for pitch and rhythm, but also those involved in working memory and the processing of higher-order structure.
Griffiths (Reference Griffiths2003) reports findings that sequential pitch processing produces activity in the posterior superior temporal gyrus (planum temporale) and frontal opercula (inferior frontal gyrus), with greater activity in the right hemisphere. These regions are believed to be involved in the processing of sequences, and may play a role in passing information to regions of higher-order structure processing in the frontal lobe. Consistent with this, Griffiths also highlights networks of frontal-lobe activation quite distinct from those in the temporal lobe. This is in agreement with the facts that the frontal lobe is known to be involved in higher-order cognitive functions, whereas the temporal lobe, by contrast, seems to be involved in lower-level pitch and rhythm processing. Research by Peretz (Reference Peretz1990) and more recently by Stewart, Overath, Warren, Foxton and Griffiths (Reference Stewart, Overath, Warren, Foxton and Griffiths2008) suggests that higher-order structure processing does not necessarily take place as a consequence of detailed processing of its lower-level components. In the case of melody, some form of hierarchical cooperation seems to take place in the sense that the processing of contour may even precede the processing of the specific pitches.
4. Abstraction and Anticipation
Higher-order representation and processing in the brain involve a great deal of abstraction. The notion of abstraction is hard to pin down. We will attempt to shed light on this by means of a couple of examples.
The aforementioned suggestion that some higher-order structure processes the contour of a melody, while some lower-order structure processes specific pitches, is a good example to illustrate what we mean by abstraction; that is, assuming that we come to an understanding that the notion of a melodic contour is more abstract than the notion of a sequence of pitch values. Another example is the notions of beat and metre. The perception of rhythm is structured by beat and metre induction mechanisms. Our brain always tries to infer an underlying regular beat in a sequence of tones. Even in a sequence of absolutely uniform tones (i.e., same pitch, duration, loudness and timbre) the brain would infer a beat by ‘imposing’ a metric template on the perceived signal.Footnote 16 This phenomenon does not seem to be dependent on training or attention, which suggests that such a metric template is a high-level abstraction emerging from some low-level biological feature of the brain. Such mechanisms for abstracting higher-level musical structures in response to avalanches of lower-level auditory information pervade our brain when we listen to music.Footnote 17
Cortical neural processing of music is an incredibly complex affair, which is still not well understood. It is clear, however, that the brain employs hierarchical neural structuresFootnote 18 to process music (Griffiths, Büchel, Frackowiack and Patterson Reference Griffiths, Büchel, Frackowiack and Patterson1998; Patterson, Upperkamp, Johnsrude and Griffiths Reference Patterson, Upperkamp, Johnsrude and Griffiths2002; Stewart et al. Reference Stewart, Overath, Warren, Foxton and Griffiths2008), and these processes may not necessarily happen sequentially. The brain is a complex distributed processing system, with various structures operating concurrently and at different time scales, from short-term to long-term musical forms. For instance, whereas lower-level structures may take care of processing the pitches of a sound sequence, higher-level structures would take care of processing the melodic contour engendered by the pitches of those sounds. But these processes might not necessarily be bottom-up; higher-level structures in the brain may make estimations of how the contour should evolve and this may influence how lower-level structures process pitches.
The amount of information that flows in the brain is immense. Although this paper focuses on music cognition, we ought to not turn our back to the fact that the brain is in charge of running our entire body: it will be engaged in a number of other vital tasks while we listen to music. It is unlikely that the brain would process those tasks completely unconnected from each other. As we have briefly seen above, brain resources are shared, albeit it is not entirely clear how the brain manages simultaneous tasks whose resources overlap. This is a problem of consciousness.
A discussion on consciousness is beyond the scope of this paper. However, we would hardly need to plunge into such discussion to appreciate that the brain cannot afford the delay that it would take to wire from scratch billions of neurons for every leap of consciousness. We have evolved strategies to react to sensations as quickly as possible. Memory mechanisms of all sorts certainly optimise brain functioning, but it still takes time to retrieve memorised neurological configurations. Our reactions to most of the stimuli around us are delayed to some extent and music stimuli are not exceptions. One of the strategies that evolved in the brain to deal with huge amounts of information flow and minimise reaction delays is to make predictions, or anticipations.Footnote 19
Neuroscientists generally agree that the brain is often prepared in advance by the very first incoming signals, for how it will react prior to actually processing the whole lot of sensory information that is coming in. Concerning auditory processing, our soundscape is normally composed of several simultaneous sources. It is therefore important to keep track of sound sources by building representations to distinguish between the sounds streaming from the same source and the sounds originating from different sources. The brain needs to evaluate how well incoming sounds fit within the existing representations, because the arrival of a sound that cannot be deemed as a continuation of any of the previously registered streams indicates either the beginning of a new source or a change in the activity of an existing source. In order to do this, the brain needs to build predictive models, whose purpose is to estimate patterns in the incoming stimuli. These predictive models allow the brain to interact with the world efficiently.
5. An Impelling Force to Organise Sound
We have seen earlier that whenever we hear a sound sequence the brain separates out information relating to location, intensity, pitch, onset timing and so forth during its journey from the early auditory pathway to the superior temporal gyrus, which contains A1. The pitch of the melody is by and large processed in A1 and surrounding areas, while the rhythm is processed elsewhere, mostly in the temporal lobe (e.g., the bilateral middle temporal gyrus). Other information is processed elsewhere, and so on. Then, all these features are (re-)combined in the frontal lobe, where higher-order features, which may include information as to how the pitch and rhythm interact, are processed; this process is referred to as binding. This frontal-lobe activity may include dorsolateral frontal areas involved in working memory.
Quoting Ratey (Reference Ratey2001: 93), ‘We receive sound as an incoming mishmash of pressure waves – not just one vibration, but layer upon layer. … Part of our ability to make sense of all this is due to the fact that we develop models of what we expect to hear: phonemes, words, music. As we perceive sound, it either fulfils our expected models or surprises us.’
Our auditory system sports a rather sophisticated filtering system, which helps us to focus on the primary sounds we are listening out for – or paying attention to – as well as those that we would expect to experience. Much of this filtering happens at the early auditory pathways, notably at the thalamus, long before we are conscious of it. People with problems in the thalamus often hear things that others do not. In severe cases they become easily overwhelmed by the environment, which can be very frustrating for them; minor sounds that most people can ignore may become unbearable to people with an impaired thalamus (Ratey Reference Ratey2001). This illustrates well the reason we have neurons projecting from the brain to the ears as well as from the ears to the brain.
The brain actively detects patterns in auditory input; this is referred to as active hearing. As we listen to music, our brain will continuously seek for regularities in the incoming stimuli. A range of features, or combinations of them, defines these regularities and, as we have seen earlier, they are extracted at many different levels and timescales. Active hearing is not a flimsy concept: the brain really tries hard and it may make up something if necessary; for example, impose a metric template on a sequence of entirely uniform tones, as discussed earlier. Such metre is not in the signal; it is in the brain.
Active hearing would not be possible without the descending pathways and all other projections from higher-order cortical functions to lower-order functions. For instance, it is known that descending pathways from the superior olivary complex to the outer ear carry efferent signals to the ear, which affect the vibration of its basilar membrane. This can change the response of the cochlea; for example, modulating the gain of a sound in the affected cochlear region (Cooper and Guinan Reference Cooper and Guinan2003).
Building predictive models of the incoming sensory input through the extraction of regularities, towards emergent (and not so emergent) abstractions, is a fundamental aspect of cognition. By adapting to patterns in the world, the brain becomes more sensitive to stimuli that differ from those implied by the detected regularities. Such different signals excite the brain to refine its representations to more closely match the sensory experience. In this way, we construct models of the world, which are increasingly more specialised. Therefore, intrinsic innate processing strategies combined with evolving experience drive our impelling force to organise sound.
6. Towards a Functional map of the Musical Brain
Essentially, the brain is a predictive organ, which strives to find structure in sensory information (or to impose it). In order to do this efficiently, it needs to make abstractions to fuel the relentless processes of making internal representations of the world. Behind these processes there is an impelling force to organise sensory information, which is driven by the physiological nature of our brain and its own evolving internal representations, or models, of the world.
While we listen to music the auditory system breaks down sensory information into elementary units, which are processed at various parts of the brain. At various stages through the cortical auditory pathways, the brain processes this information according to our memories of previous experiences.
In figure 4 is an attempt at drawing a functional map of the musical brain. The shaded circle represents the auditory pathways; there are three concentric subareas delimited by dashed lines representing the outer ear (where hearing begins), the early auditory pathways and the cortical auditory pathways, respectively. The lines are dashed because the functional boundaries are hazy. Labelled squared boxes represent the various functions that were mentioned in this article, which are deemed important for the emergence of mental imageries elicited by music. The functions are placed over regions of the circle where they roughly take place, but they should be seen as drifting within their respective hierarchical bandwidths. The closer a function is to the outer border, the higher its order, and therefore the more abstract the information at stake. On the left side, the two dashed arrows represent the traffic of neural activation going in opposite directions, from ear to cortex (ascending) and from cortex to ear (descending). These arrows indicate that while the brain breaks down sensory information into elementary units for bottom-up processing, it also applies a number of higher-level cognitive frameworks for top-down processing. For instance, the brain makes a number of changes in the inner ear thorough the descending auditory pathways, in order to maximise its ability to focus our hearing to what we need or, perhaps more so, what our brain expects to hear. These cortex-to-ear pathways are central to our working hypothesis that mental imageries elicited by music are a by-product of the inherent abstracting and predicting properties of the brain; these pathways convey anticipations. They seem to act as scaffolds for music perception.
Tentative functional map of the musical brain.
7. Envisaging the Future of Music Technology: Analysis of Electroacoustic Music
The emerging field of music neurotechnology draws from research into the neurosciences to develop new technologies for music. For instance, ongoing developments in this field include brain–computer interfacing technology that uses electroencephalogramFootnote 20 information to steer generative rules to compose and perform music (Miranda Reference Miranda2006). This research is opening new possibilities in recreational and therapeutic devices for people with severe physical disability but able brain function.
In this section we speculate on possible developments that may have an impact on the electroacoustic music community: new technology for the analysis of electroacoustic music.
Despite tremendous advances in computing technology, the tools available for electroacoustic music analysis lag far behind the theoretical work of scholars such as Smalley (Reference Smalley1997), Landy (Reference Landy2007) and Emmerson (Reference Emmerson2007), to cite but three examples. The great majority of tools available for the analysis of electroacoustic music are based on the Fourier transform paradigm, which offers a spectrographic representation of a sound. For instance, the Acousmographe, developed by Le Groupe de Recherches Musicales (GRM) in Paris is probably one of the best tools available for the visualisation and annotation of electroacoustic music. Yet the best it can offer in terms of signal analysis is the spectrogram.
In a paper read at the AHRC ICT Methods Network Workshop on New Protocols in Electroacoustic Music Analysis, held at De Montfort University, in June 2007, this author speculated on the unprecedented possibility of building tools for the analysis of electroacoustic music based on neurophysiologic models of our auditory system. These tools would reveal the ‘representation’Footnote 21 that our brain produces at various stages of the auditory pathways. To the best of our knowledge, apart from the cochleogramFootnote 22 there have been no significant developments towards tools for the analysis of electroacoustic music based on neurophysiologic models. However, research into building computational models of auditory brain functioning is an area that is evolving rapidly, and progress at this front is bound to emerge in the near future.
Having described the journey that sounds take through the early auditory pathways in section 2.1 above, below we enumerate the analysis tools that could be developed based on the various stages of this journey and the types of information that they would elicit. We envisage at least four of such analysis tools: the cochlearnucleigram, the olivogram, the thalamogram and the auditory corticogram.
The cochlearnucleigram would give information related to the activity of the cochlear nuclei (2, in figure 1). It would be a powerful source separation tool based on minute onset and spatial information. The cochlearnucleigram would be able to provide precise onset information and trace the behaviour of the sounds in the horizontal and in the vertical planes.
The olivogram would give information related to the activity of the superior olivary complex (3, in figure 1). This analysis would provide further information about sound localisation using two types of mechanisms: one based on timing information and another based on sound intensity. Identification of sound location through the use of timing information would focus on low-frequency sounds whereas the use of intensity information would focus on high-frequency sounds.
The thalamogram would give information related to the activity of the MGN in the thalamus (6, in figure 1). As we have seen earlier, the thalamus controls attention: it enables the brain to suppress information in order to focus on particular aspects of incoming sounds. The thalamogram would reveal salient sound attributes that would be deemed more important than others in function of specific contexts or conditions. One could imagine the possibility of being able to specify such contexts as analyses parameters for simulating the focus of the thalamus under different contexts or conditions. This would reveal the impact of different sensorial modalities on the auditory signal. A number of interesting analysis parameters could be envisaged here: the amount of modulation from other sensorial modalities, the types of modalities to be considered and/or ignored, modalities priorities, and so on.
Finally, the auditory corticogram would give information related to the activity of the primary auditory cortex (7, figure 1). The auditory corticogram would extend the aforementioned thalamogram by building representations and anticipations. One could envisage some sort of ‘thalamocortical control panel’, which would allow for building representations combining different levels of attention to various sound features, influences from other sensorial modalities, and exposure. Ultimately, the thalamocortical control panel would allow one simulate and predict the types of representations that would emerge from different ontologies. The analysis could be furnished with different listening strategies based on exposure to different sound worlds.
8. Concluding Remarks
One of the principal contributions of the burgeoning field of neuroscience of music to our understanding of our musical brain is the elucidation of various neural pathways through which the auditory signal is processed. At the introduction, the challenging task of understanding how the brain processes music was metaphorically compared with the task of assembling a complex jigsaw with missing parts. In this paper we made an attempt at putting together a few pieces of such a jigsaw in order to assemble a picture of how our brain listens to music.
There are various stages in the neural pathways through which music is processed, each of which extracts different types of information from the signal. Basically, once the sound enters our ears, our auditory system decomposes the signal into a number of attributes on its way up to the brain cortex. Then the brain puts all these attributes back together. Interestingly, these attributes are modulated by other sensorial information before they are put back together.
More research is needed, however, to bring to the surface more evidence to back the hypothesis that mental imageries elicited by music are by-products of the inherent abstracting and predicting properties of the brain. However, there is enough work out there suggesting that the brain strives to find or impose structure on sensory information (e.g., Stroh Reference Stroh1983; Bressler Reference Bressler2000; Hampe and Grady Reference Hampe and Grady2005). In order to do this efficiently, it needs to make abstractions to fuel relentless processes of making internal representations and anticipations, aimed at optimising the amount of neural processing the brain needs to make sense of the world as the time goes by. Behind these processes there is an impelling force to organise sensory information, which is driven by the physiological nature of our brain combined with its own internal representations, or models, of the world. In this sense, mental imageries elicited by music can be regarded as a scaffold for music perception.
Finally, research into the neurosciences is driving the development of a number of new technologies for all sorts of applications, including music. However, wider channels of communication need to be established between neuroscientists and musicians in order to ensure that the former address the right musical questions and the latter understand the implications of neuroscientific answers. This would create a fertile ground for musicologists and music technologists to develop better theories and tools for the art of organised sound.
