2. Background: a choreography of sound

Imagine a choreography of sound, an experience in which a sound could communicate what a bodily gesture does on a dance performance stage. A sound that is perceived in space and motion, away, towards, or above the listener. A sound that can communicate not only its directionality and trajectory of motion but also degrees of strength, size, shape, speed; an experience that conveys music, space, and movement semantics. That is and has been indeed the dream of a composer.

Sound has an innate ability to suggest motion through rhythm, speed, phrasing, articulation, and dynamics, but special attention has emerged on sound’s existence in space emanating from its source, on how sources could move, and on how that is different from the innate motion perceived through musical semantics.

Space has been used as a strategy tool in composition: Gabrieli’s choirs in S. Marco Venice (Bates Reference Bates2009: 115) allowed for spatial positioning of portions of the choirs in the different aisles of the church to play call and responses in the music across the whole church; Xenakis’ glissandos movements across orchestra sections (Xenakis Reference Xenakis1959) featured spatial organisation within the orchestra, expanding on a strategy already experimented by even Mozart and Haydn; Stockhausen’s three orchestras work (Stockhausen Reference Stockhausen1957) located the audience in between three orchestras, (ibid: 116) while Stuart Dempster used the space of a 144 foot deep abandoned water cistern as a key element for his trombones and other instruments composition (Dempster Reference Dempster1994).

The major developments in recording technologies and the use of loudspeakers facilitated the birth of acousmatic music – a practice originating in Schaeffer’s explorations in the late 1940s (Schaeffer Reference Schaeffer2012; Kane Reference Kane2014) – and electroacoustic music, a term that became established by the 1960s (Roads Reference Roads1996). In these practices, sounds could be panned or freely placed within a space delimited by loudspeakers, a spatial exploration continued by composers such as La Monte Young (Reference La Monte1964), Alvin Lucier (Reference Lucier1969), and Ryoji Ikeda (Reference Ikeda1999). Following the earlier examples of the Osaka and Phillips pavilions (Bates Reference Bates2009), the typical 6/8 speakers in pairs setups have dominated computer music conferences for a while, while complex diffusion systems such as the B.E.A.S.T. system (2025) or the Acousmonium (Bayle Reference Bayle2018) also became very popular in the 1990s and 2000s. The increased interest in licensing Spatial Audio by numerous companies nowadays and the emerging production and mixing practices that are quickly populating the relevant aesthetic and technological scholarly scenes (Bates Reference Bates2016; Lee Reference Lee2021; Jerez et al. Reference Jerez, Moore, Dewey, Lee and Scheps2024; Dewey, Moore and Lee Reference Dewey, Moore and Lee2024) is also a further confirmation: Music can now be heard through space and at different and dynamic locations within the loudspeakers perimeter and that has become a creative territory to explore.

Arts based on movement, such as dance and dance choreography, provide an interesting framework for a comparison. A great theorist of dance, whose findings are still in use nowadays, Rudolf Laban, spoke about Choreosophia, as the science of the circles (Laban Reference Laban1966), from which emerged his landmark movement notation system. Two of Laban’s key concepts are those of Effort and Action Mood, in which a gesture made by a body could take directions, intensity, size, and speed (the Effort), be combined in groups in the construction of a creative piece (Choreography), and be linked to a particular ‘archetypal’ expression (the Mood) (Laban Reference Laban2011).

A great fascination with spatial sound is derived exactly from that, in how a movement as generated through sound compares with that originated through a body or an ensemble of bodily gestures. Could sound use the same ‘effort’ and ‘action mood’ as in dance? How could movement generated through sound achieve comparable expressions to those of the gestures made by a body in motion, and how could that ever be technically achieved? Can the semantics of music and sound connect with the semantics of space and movement? And when they combine, could a novel meaning emerge?

A choreography of sound is about designing and combining sonic movements with those aims in mind, navigating the territory of sound in space, movement, multimodality, intermediality, and musical meaning.

3. Theoretical framework

Spatial audio composition takes its premises from previous compositional explorations, incorporating the experiences of the Avant-Garde, Musique concrète, electroacoustic and acousmatic music, sonic arts, and interactive and immersive audio. Its palette is thus composite, with many contributors coming from many fields of practice, including artists and thinkers such as Schaeffer, Stockhausen, Berio, Maderna (Emmerson Reference Emmerson1986; Wishart Reference Wishart1996), the contemporary Avant-guard (Cage, La Monte Young, Lucier, Boulez) but also the Soundscape Movement (Schafer, Truax, Westerkamp) as well as the Computer Music and Electronic Music emerging fields. It inherits the traditions and settings of those practices, where loudspeakers sound diffusion is central.

The first important context to consider is precisely the state of the current reflection on spatial audio composition, which will be useful for measuring the contribution of sonic movement practice to the field. In the literature of the most significant reflections on the processes of spatial composition, there are more discussions about spatiality and conceptions of sound in space than about movement, movement per se, or the perception of movement through sound. There is emphasis on the type of spatial image one would want to create using sound and the desired accuracy of such image. Some employ (Smalley Reference Smalley1997; Barrett Reference Barrett2002; Kendall and Ardila Reference Kendall, Ardila, Kronland-Martinet, Ystad and Jensen2008) suggestive concepts such as illusion, allusion, sound identity, and spatial schemata; however, they don’t directly point to aspects of movement per se neither to multimodality nor intermediality, if not just briefly.

Most sources seem to agree on the fact that ‘changes in spatial perspectives are a means of delineating musical structure’ (Smalley Reference Smalley1997: 122) and that ‘merging electroacoustic composition with spatial form expresses an area of underdevelopment in compositional theory and technique’ (Gottfried Reference Gottfried2012). Baalman (Reference Baalman2010), though, points to technological limitations and the opportunity to create hybrid technologies to fulfil the composer’s needs and Gottfried calls out the primary importance of empirical studies of spatial cognition (Reference Gottfried2012). This suggests the need for a deeper understanding of the listener and the spaces where the listening occurs. Writings on spatial audio composition do pay attention to how to conceive, control, and understand spatial sonic perspectives, but do not directly discuss the listener’s experience and make assumptions as if that was a normalised experience across all the cohorts of listeners and listening arrangements. This gap in conceptualisation and practice is a critical area for the current discussion on sonic movement.

Spatial hearing in Psychoacoustics, the study of the spatial perception of sound, is a key framework for the perception of movement through sound. Sound is conceived as having a leading or complementary role to the final percept (Blauert Reference Blauert1997: 288–367), based on the dynamic hierarchy of the senses, that is, when multiple senses operate and contribute to the same percept in a specific order of importance. Head and body movements are also a fundamental and not accessory aspect of perception. A slight tilting of the head helps our auditory sense to provide our brain with the best information about the location of sound (ibid: 178), sufficient, for example, to pass from an undefined stimulus to a precisely located one, resolving front and back ambiguity, causing shifting from the median to the lateral plane.

Still within the psychoacoustic realm, the word ‘familiarity’ (Blauert Reference Blauert1997: 116–142) groups a number of auditory and spatial hearing aspects that vary amongst the listeners: when the listener knows or have experienced the sound already before, when the sound source is familiar to them, or when they have expectations about it or there is a contextual suggestion that aids imagining the sound better (e.g., a narration or even the programme notes of acousmatic concerts?), all these allow for a better appreciation of spatial audio qualities (Rumsey Reference Rumsey2002). This is important when we think about multimedia and intermediality: the presence of multiple media alongside sound is something that can trigger familiarity. Having an expectation, a suggestion, coming from the environment or other factors or media, it could result indeed in a much more defined impression of the perceptual stimulus.

One example of that is provided by the audiovisual. As Chion explains it (Chion Reference Chion1994), the simultaneity and synchronicity of the stimuli to auditory and visual perception produces an added value that is more than the sum of the two components. Processes like ‘synchresis’ or ‘magnetisation’ occur when we watch and hear, and they show how, in a multimodal context, our precepts enhance synergistically. Furthermore, the articulation of complex stimuli also is dependent on how our perceptual system understands and organises auditory streams (Bregman Reference Bregman1990) and is directly related to the musical content as well as the spatial, including even paradoxes, illusions, and perceptual mistakes (Deutsch Reference Deutsch2026).

On the other hand, the general concept of immersion (Lee Reference Lee2025) cannot fully explain the perceptual context of sonic movement because it is lacking direct reference to the spatial, musical, and movement domains, but it provides a clear background of what the components that affect perception are and the engagement that results from them, from the listener’s point of view. As such, it works as an important context to account for.

Lastly, the synesthetic design, as found in Haverkamp (Reference Haverkamp2013), is important for its focus on cross-sensory correlations and the possible strategies for connection, such as conscious construction, semantic coupling, iconic coupling, and cross-sensory analogies. These offer an interesting connection with the concepts of presence and involvement (Lee Reference Lee2025) and with the issues raised from the psychoacoustic, psychological, and audiovisual models.

All these frameworks and contexts are relevant to how we make sense of the spatial location and movement of a sound source. They show that perceptual factors, psychological, and sensory stimulation are thoroughly at play in our understanding of the spatial location of virtual sound sources and their movement. In many listening situations, we can appreciate that some (or many!) of these perceptual conditions are somehow ignored, remain undeveloped or unstimulated, or worse, they are suppressed. This will be an important fact to examine with regard to the context of the design and reception of a Spatial Audio Composition.

4. Experimental practice in Sonic Movement

The following are three practice-based projects organised to engage the audiovisual, multimodal, and interactive domains and developed through cross- and interdisciplinary collaboration. They are analysed here to demonstrate the creation of a spatial audio composition alongside the concepts of the listener’s agency, listening arrangement, multimodality, and intermediality.

A documentation is provided in-text with direct time references to moments in the performances, so that the reader could appreciate as closely as possible what visually and aurally has happened in the real space. Please wear your headphones for a better effect.

4.3. The Song of the Bees

Later, after a break, in 2024, a further experiment was done for designing another type of sonic movement experience. This is the Song of the Bees (SoB) (Perego Reference Perego2024) project in which the listener is no more passive, but neither just a participant: the listener is now the player of an open world game in first person. That means that the player-listener makes decisions on where to look, what to listen to and what to do, to achieve goals set independently by any guidance. In the game, the player takes the role of a bee in a virtual open world, using its body to fly around, collecting pollen and interacting with objects, places, and other creatures. Spatial sound is used at locations surrounding the player and at interaction events.

For what concerns virtual sound sources played at a location, the bee normally orients in space through smell and magnetic sensitivity. Their vision is limited and only activates in very special circumstances. In the project, we swapped the olfactory and sensing abilities of the bee for the human’s auditory system. When the players are in the game, they don’t see but only hear and orient themselves through the sounds positioned in the virtual space. So, the location of sound and sound in space gained a primary role, affecting the actions and decisions of the player alongside the ludic aims. For what concerns movement perception, sounds interact when approached by the player at their exact location and create new actions and scenes. For example, a player approaches a sound, and, when approached, the sound triggers many others with spatial and gestural qualities, similarly to when, in real life, a bee approaches a flower, collides with it, and causes pollen to burst in many places. In this way, the player makes sense and engages with the gameplay led by the sonic designs, one event after the other.

The significance of this development is that the agency is brought to its fullest, moving away from a performance of a piece of music or choreography into a much bigger, open-ended world in which not only narration and audiovisual aspects flow, but also the gamification and ludic mechanics. This creates an impasse for the spatial audio composer that wants to centre the composition on the movement sensation generated through sound, as there are so many more elements that can lead the engagement of the listener (who is now not just a listener but a player) away from sonic movement.

This work, still in prototype and developmental stage, offers thus much to reflect upon, presenting another challenge to sonic movement creation. The way forward for this type of project would be to integrate generative and interactive compositional strategies, as fixed composition would not be fit for purpose any longer because of the open world design of the game. It is also clear that the sensation of movement per se should be embedded in the ludic component and strategy design, meaning should be included in the gamification aims.

Table 1 below should recap visually the key differences between these works, looking at six areas of development common between them: the listening arrangement, the listener’s agency, the technology used, what constitutes the intermediality, how the multimodality is achieved, and the type of sonic and intermedial design.

Table 1.

A table comparing six areas of development across three different works: Stranded, IHUCM, and Song of the Bees

Table 1. Long description

A table with six columns and three rows. The columns are labeled Listening arrangement, Listener agency, Technology, Intermediality, Multimodality, and Design. The rows are labeled Stranded, IHUCM, and Song of the bees. Row 1: Stranded, Proscenium, Passive, Multiple Perspectives, Wave Field Synthesis, Body and Sound, Audiovisual, Synesthetic, Joint Choreography of sound and body. Row 2: IHUCM, One person inscribed in a cube with projection on the floor, Active, First Person Perspective, Ambisonics First Order, Image, Sound, Participation, Narration, Audiovisual, Synesthetic, Participatory, Proprioceptive, Audiovisual Narration, Proprioception engagement, Instruction-based exploration. Row 3: Song of the bees, Virtual Open World, Active, Interactive, First Person Perspective, 3D Audio in A VR Game Engine with Haptic Feedback and Head Tracking, Image, Sound, Interaction, Ludic, Audiovisual, Synesthetic, Proprioceptive, Interactive, Gameplay, Audiovisual Narration and Gamification, Proprioception engagement.

5. Comparative discussion: multimodality, intermediality, and experience design in a spatial composition

Following the above analysis and descriptions, it is possible to draw a diagram (Figure 1) to map out the areas of development for the creation of works on sonic movement, based on:

• • the composer’s decision process (which includes the technology choice, intermediality, and sonic design)

• • the experience (which happens through different engagements of multimodality)

• • the listening context (which includes the listener agency and listening arrangements).

Figure 1.

Mapping spatial audio composition.

If we look at the composer’s decision process (left), three components are available: the choice of technology, the intermedial and sonic designs. For intermedial design, we refer to the use of multimedia interleaved with the presentation of sonic gestures and musical passages. For the sonic design, both the sound composition and spatial design are meant. Every composer who approaches a spatial audio composition should reasonably use those components to create their musical and spatial structures. However, in many practices and performances, when in the sonic design both the musical and spatial semantic are at play, the intermedial design is absent or just jotted.

That could be partly caused by the type of music performance (e.g., acousmatic music) or by the settings and the listening arrangement (e.g., sharing a listening space with a number of people). Interestingly, Christopher Small talks about how a member of the public makes a contractual agreement when entering a venue for a music concert to maintain silence and ensure lack of motion, to avoid disturbance to the other spectators (Small Reference Small1998: 24–27). In many music listening situations, it seems that we are making such an agreement to essentially lock or minimise our proprioception for the common enjoyment, compromising our success for a full appreciation of spatial sound, as we know from chapter 3.

In the rightmost section, instead, it is illustrated the listening arrangement – a direct result of the technology choice made – and the listener’s agency. These two relate to where and how the listener experiences the spatial audio work and what does the listener do when they listen to it. In the centre of the diagram, we have the experience in itself, that consists of stimuli and sensory engagement that affect the perceptual modality or multimodality. That is a factor depending on the sonic and intermedial design choices made, as it appears from the left section.

The areas shown in colour in the diagram are critical areas that can illustrate the gap in current literature, as gathered from the lessons learned during sonic movement research practice. In the composer’s process (left), once the choice for technology is made (including hybrid technology choices) it is in the design of both sonic and intermedial aspects that the sonic movement takes place.

From the doctorate’s practical work (Perego Reference Perego2017) and the latest Song of the Bees prototype, the following were the key relevant findings in working with sonic movement:

• • That a perception of movement through sound must be organised with different media for the best effect

• • That a perception of movement through sound must be organised with different modalities of perception for the best effect (like audiovisual stimuli as in Stranded, proprioception in the IHUCM instance), providing a strong element of immersion bringing the physicality of sound and its spatiality into the light.

• • That the perception of movement through sound is dependent on the use of technology combined with the design of the experience, because technology and design would allow or not, or would be more suitable or less for different listening approaches (passive, active-participant, involved and leading) and perceptual clarity (sweet spot sound diffusion, physical waveform reconstruction, virtual environment and immersion)

It can be argued that intermediality, listening arrangement-technology and the listener’s agency are thus important factors for the inclusion of spatial audio in a composition. Although sonic movement and working towards a choreography of sounds are indeed a special focus within spatial audio composition practice, they have demonstrated that a sonic spatial gesture is complex and to achieve its clarity and intelligibility, a spatial audio composer should carefully consider the psychoacoustic, synesthetic, and proprioceptive frameworks.

Intermediality seems key to produce multimodal sensory engagement in relation to the arrangements chosen. Jensen broadly highlights that intermediality relates to ‘the interconnectedness of modern media of communication’, adding that ‘as means of expression and exchange, different media refer to and depend on one another, both explicitly and implicitly’ (Reference Jensen, Jensen, Rothenbuhler, Pooley and Craig2016: 1). And all of this should also include the staging, the technologies, the scripts, the virtual environments, all that is required by the composer to architect the experience (Clüver Reference Clüver, Arvidson, Alexander and Fuhrer2007: 46). Lee’s reference to immersion as a multidimensional construct (Reference Lee2025: 2) and Chiaramonte’s reference to the ‘synergic relationship’ of the audiovisual (Reference Chiaramonte2024) further support these views. It could be argued that sonic movement is the result of a syncretic discourse as described by Hoek (Clüver Reference Clüver, Arvidson, Alexander and Fuhrer2007: 26), for which a fusion or union is produced between the different media.

The result of combining technology, sonic and intermedial design could be comparable to what is defined in synesthetic design, as the ‘stimulation of sensations in a certain sensory channel’ which ‘results in sensations further felt in one or more other sensory channels’ (Haverkamp Reference Haverkamp2013: 14). In this manner, for example, the perception of visual shape and colour can additionally stimulate a sensation of sound.

With regard to current literature (theoretical and practical) on spatial audio composition, there seems to be a constant assumption about human hearing, for which even very complex auditory scenarios could be designed and discussed without much consideration of the listener’s perceptual modality and agency within the specific listening arrangement. Smalley refers that spatial distribution can enhance the notions of growth and motion in sonic textures (Reference Smalley1997: 116) and that ‘spectromorphology’ becomes the medium through which space can be explored and experienced. These are semantic interpretations of the spatial quality of sound but are done separately, aside from considerations on agency, modality, and listening arrangements technologies. It feels there is a belief that hearing would work with the spatial aspects of sound as it would with any other sonic situation but that contradicts with what established by researching the listener’s multimodal engagement, the level of immersion, and the different shapes and expressions as allowed by the multimedia staging.

At some point in other discussions, the listeners are encouraged to take care of their ‘personal projection’ (which is seen as a hurdle) (Barrett Reference Barrett2010: 2), to facilitate their understanding, somehow shifting the responsibility from the composer to the listener. We think, instead, from the research undertaken, that how the multimodal stimuli are presented is essential for a successful perception of movement through sound and that necessarily involves the intermedial design of sound, the multimedia stimuli, the staging of the overall experience (Clüver Reference Clüver, Arvidson, Alexander and Fuhrer2007: 46), including the listener’s agency.

If the above then constitute the balance that produces the sonic movement, questions remain open about how movement perceived through sound compares to that expressed by the body. Were there comparable elements between what can be achieved through dance and through sound? Was the semantics of music, its pitch, dynamics, timbre, and phrasing, in any relation with the development of the semantics of space and movement when constructed as such?

In a way, sound cannot escape being sound, because when a sound source can be perceived moving in a direction reasonably easily, to fully apply Laban’s descriptors, we encounter the limitations of our perceptual conditions (the ear does not perform with movement as the eye does). So, a full comparison with the body’s ability to create movement effects was not possible and perhaps is not even fruitful. One would immediately argue that the movement of sound, because of the psychoacoustic limitations, can be quite limited compared to a body’s large palette of gestures. However, the fact that sound can more easily cross the perceptual and engage with the semantic, the composers find themselves with still a great palette to work on.

The intelligibility of a specific choreography of sound would thus benefit from a partnership in the creation of a percept, and if anything, this excursion on the practical experience of sonic movement has highlighted new challenges for the composers and lovers of spatial audio composition.