Press ‘A’ to Start

It all begins with a gesture — the push of a single button. Super Mario bounds across the screen, thrust forward by my agency. His run is goofy and overzealous. There’s something slightly incongruous about it, in the cartoonish way that a character’s legs might spin to imply rapid movement. The world he is in, the world we both share, is bright and brilliant: stark blue skies contrast with vibrant greens and reds, and Mario himself is immediately distinguishable, dressed in his stylized plumber’s overalls. The sound is what hits me — small, almost delicate affectations like raindrop-heavy pizzicato plucks operate independently over a musical soundscape that is inviting, joyous, playful. The music signifies something to me, something sensational and magical. It wants me to engage, to play within the parameters of its ludic confines. It wants me to move and be moved.

Super Mario World (1990) was the first game that really affected me, forming my initial love for the medium of video games.Footnote ² In Stephen Sexton’s lucid poetry quoted above, the glow of the television screen in the corner of the room is an affective device, drawing the young poet into its light. Sexton’s formative video-game experience involving Super Mario World mirrors my own. But for me, the experience of immersion in this simple sense was largely down to the ludomusical pull I felt to the audiovisual text — it was the playful sound world that drew me in. Memory can be an inexact phenomenon, yet I clearly remember the bright colours and sounds paired with that visual identification; it was Koji Kondo’s music, above all, that hooked me.

My own energetic eagerness to play is embodied by the musical soundscape: the sound world is light and my virtual movement as Mario is freeing. I am unable to turn away, anxious to continue playing, staring agog at the CRT television screen in front of me. I am transfixed, immersed, cocooned. Mario is at the mercy of my agency, yet I am playing by the game’s ludic rules. However, there is more to it than that: when one is engaged with a video game, the connective tissue that binds the player to the avatar they control can almost appear as a fused state. Michelle Grosser hints at such when she notes that in games, ‘the player identifies with a character and constructs [a] virtual identity as an avatar’.Footnote ³ In that fused player/avatar state, I am immersed in this zany, kaleidoscopically coloured world, too engaged to be pulled away by the concrete and real space around me. And it all makes sense somehow — the bright colours, the cute, rather than terrifying, dinosaur hatching from a speckled egg, the spaceman-like, gravity-defying jumps I (as Mario) can perform, and the monsters I instantly know to be such. How? And why? How am I so convinced of this world, so engaged in it, so immersed in its pixelated ludic environment and soundscape? And why do I stay there, sat looking at the screen in the corner of the room, unable to pull myself away?

Another gameplay memory comes to the fore. Just a few years after my encounter with Super Mario World, I was invited over to a friend’s house. A Super Nintendo Entertainment System, similar to my own, was pulled out from the TV shelving unit and plugged in, the dust blown away and our new favourite Game Boy Advance systems put aside. This time a different game was booted up, a new world coaxing me in from behind the boxed pane of glass of the television screen. Super Metroid, as the game turned out to be, has continued to frustrate, fascinate, and enthral me, and that initial memory of the game world, so dark and foreboding in contrast to Mario’s virtual land- and soundscape, continues to resonate.

The world of Super Mario World is joyous and playful, a palette of childhood carefree abandon, in which you can enjoy the challenge of progressing from left to right at speed, bounding across artificial mounds or even on top of the heads of enemies so that they fall from the screen, as if being dropped out of a claw machine at the arcade. You collect shining gold coins, swim without having to worry about how long you can hold your breath, all while being able to don a cape that can fulfil a ‘Superman’ fantasy of flight, albeit in pixelated 2D. By stark contrast, the opening moments of Super Metroid (1994) are full of panic-inducing affectations: a siren blaring as you try to make your way to the exit as an environmental self-destruct procedure begins, the timer ticking down on screen forcing you to comply rapidly with the ludic rulesets.Footnote ⁴ The colour palette of the game is characterized by dark and morose significations, and underpinning it all, a rumbling soundscape drives the player forward, the bass chorus of synths murmuring sinisterly below, generating a sense of urgency. Once again, the game encourages you to play, and once again, the music wants me to move and to be moved, albeit with a more pressurized, urgent sense of drive, in contrast to that of Super Mario World’s playfulness. It is the musical gestures — the sense of movement — that are a fundamental part of how these two games provide different aesthetic experiences, for all their superficial similarity as 2D side-scrolling platform games.

Such early gameplay memories indicate the pull of musical immersion and the ways in which music can encourage, enact, and soundtrack player movement and gesture. However, few scholars have traced the ways games can immerse players through specific musical gestures.Footnote ⁵ More pertinently, where musical movement has been analysed, the lack of a ludomusical framework specifically targeting game music and sound has resulted in analytical techniques borrowed from other disciplines that ill-fit the subject matter. This article posits a new theory of musical gesture for video-game music and its potential visualization, presenting a new adaptable and bespoke analytical framework. This theory can help us to understand player experience of involvement in the audiovisual environments they inhabit through a process of ludomusical embodiment.

Music’s potential to reach out to the player, pulling them into the virtual land- and soundscape, is thus the main focus of the framework proposed by this article. A major concern here is the development of a theory that can point to precise musical features, styles of listening, and ludic engagement that may lead to a player feeling immersed in the landscape and soundscape they (as the fused player/avatar) might find themselves in.Footnote ⁶ Ludomusicology and wider scholarship has certainly considered ludic engagement and immersion, with Karen Collins notably highlighting the concept of ‘kinesonic congruence’ that emerges when game sound successfully maps onto player action or gesture.Footnote ⁷ Importantly, Collins’s analysis shows how the musical material of games has the potential to be interpreted by the player through a gesture-based lens. This gives the player an audiovisual impetus, driving the engagement with the game and our play as Mario (in Super Mario World) or Samus (in Super Metroid). Specifically, the player harbours a space between musical gestures and the character on screen: they are both interpreter and performer.

This new theory will approach musicological theoretical models and ludomusicological analysis using a newly reworked and remapped gestural analytical framework, borrowing from previous gestural analytical methodologies and expanding them to incorporate new concepts and interdisciplinary insights as well as the audiovisual ludic mode. This project addresses a fundamental question I and other scholars have always held in regard to gaming: what does it mean to be engaged and immersed in the virtual soundscape, landscape, and thus world of a video game?Footnote ⁸ It is the human ability to find embodiment in an audiovisual text and various stimuli, primary among them musical shapings and gestures, that is paramount to the theory I will present and thus to the possibility of engagement and immersion a player might feel, ultimately leading to what I will term ‘ludomusical cocooning’.

Finally, in considering the ‘human’ interaction here, the bodily movement as analogous to musical gesturing, the theory additionally draws from new interdisciplinary links between musicology, choreology and dance pedagogy, behavioural sciences, and ludomusicological research. As such, this analytical framework presents a model for the visualization of musical gesture, one that is tailored for and by ludomusical texts. By visualizing the musical gestures we play with and embody, gesture can be understood as a key primer for game involvement and traditional ideas of ‘immersion’. This gestural theory considers the concepts of soundscapes and virtual landscapes in a new, fused light and highlights a new approach to engagement, listening, and player interpretation. Fundamentally, this theory of gestural potential, as I will reveal, reworks and reexamines the conceptualization of gesture in music.

Why Gesture?

Before I investigate previous theories of musical gesture and propose a new theoretical framework, an initial question may concern the term itself and why the manifestation of and interaction between music and perceived gesture is most fruitful for the subject matter of this article. Why might gesture be useful for concerns about immersion in and engagement with interactive audiovisual media? Fundamentally, music is as much a visual-kinetic art form, a vessel for the translation of spatial and virtual shape into sound, as it is a purely aural one. Musical shaping, be that through traditional concepts of phrase, the rise and fall of a melody, the loudening and quietening of musical intensities, and the ebb and flow of harmonic patternings, is analogous to the gestures of humans and to those we might find in nature. Robert Hatten’s work sheds light on the fact that music has long been recognized as having gestural qualities. For Hatten, the gestural shaping in music characterizes it, breathing metaphoric life into the musical material. For example, grief as an emotion might be expressed with ‘downward and heavy gestures, and elation by means of upward and light gestures’.Footnote ⁹ As such, musical gestures that map analogously onto physical human movement mean that ‘energetic musical gestures can begin to feel like gestures of the body’.Footnote ¹⁰ In games, this translates from the metaphorical to the actual. Virtual, performed movements show the thematization of gestures, from the fused player/avatar actually doing gestures and musical shapings that highlight their agency, accompany their movements, or soundtrack their play. Gestures, in my own definition and in the simplest terms, can be viewed and understood as bodily and virtual shapings within the fixed (but equally ludically malleable) parameters, both temporal and spatial, of an environment. It is the push and pull, rise and fall, and unique agency of these musical shapings within a three-dimensional space, charged with virtual or real tangible kinetic energy, that harbour the potential for the plurality of musical meaning.

My concern with the connection between musical meaning and gesture can be seen in Richard Leppert’s foreword to William Cheng’s Sound Play, where he asks, ‘What does music do to us? And why does it do to us what it does?’Footnote ¹¹ Leppert’s questioning stems from Cheng’s desire to ‘understand music’ and to trace why we do what we do in games through an examination of the aural component of the art form. However, we can also focus on a further dimension: how does music do what it does to us? The purpose of this article is thus to approach music, musical meaning, and musicological analytical methods through a newly adapted lens, one that considers broad inter-disciplinary concepts while also showing a precise yet holistic form of gestural analysis. To return to the goal of this research, i.e. its desire to confront the question of immersion and engagement in games by analysing various ways in which players might feel connected to them through the music that affects their gameplay experience, gesture’s role in such affectation is crucial. Gesture and its ramifications, its metaphorical playfulness, and its ability to pull us into actions and dialogues in everyday life are worthy of such analysis because of its very ability to masquerade within music as a primary force for emotional captivation. Musical gesture coaxes us into the sound world, holds a light to potential meaning, and opens our eyes to possible imaginative worlds. To consider gesture and to provide a musical theory and mode of analysis that can highlight its role in player immersion and engagement is to solve fundamental research questions that consider why players play the games they do. This article’s analysis of musical gesture and foundational theory helps to reveal how music can grab our attention and cocoon us, enticing us to play and continue playing. Such a theoretical framework would need to present a remapping of musical gesture theories within what we might consider a rapidly altering modern world, in which primarily audiovisual technologies of virtual entertainment and escape are competing for our attention.

Remapping Theories of Musical Gesture

One of the foundational texts for gestural analysis of music is Hatten’s Interpreting Musical Gestures, Topics, and Tropes: Mozart, Beethoven, Schubert. Hatten’s understanding of musical gesture is concerned with two fundamental pillars: fixed temporal dimensions and momentum-based contours. For him, gesture concerns our ability as humans to ‘recognize the significance of energetic shaping through time’.Footnote ¹² His theory thus encapsulates gestural movement and momentum within the space and time in which they operate. Musical gestures are ‘founded in human affect and its communication — they are not merely the physical actions involved in producing a sound or series of sounds from a notated score’.Footnote ¹³ Instead, gestures can be interpreted as shapings that characterize or distinguish musical sounds from one another, and it is these shapings that imbue or facilitate sounds to harbour or produce expressive meanings. Crucially, gestures may ‘be comprised of any of the elements of music’, including ‘specific timbres, articulations, dynamics, tempi, pacing’, along with how these elements synthesize with concepts such as rhythm, melody, and melodic structures more broadly.Footnote ¹⁴ Hatten’s concept of gesture and the idea of musical contours manifesting within space informs my own remapped theory.

Notably, Hatten interprets ‘musical gesture’ as ‘a fundamental and inescapable mode of understanding that links us directly to music’s potential expressive meaning’.Footnote ¹⁵ He thus sees human gesture as being critical to the interpretation of musical gesture. Arnie Cox gives further credence to this view, noting how a listener’s ‘implicit knowledge of the physicality of the sound source regularly shapes musical meaning’.Footnote ¹⁶ The body, or human, is necessary in Cox’s and Hatten’s models for musical meaning, imparted by musical gesture, to take shape. This is further evidenced by Hatten’s insistence on the presence of a ‘humanely characterized virtual agent’ in order for the manifestation of musical gesture to both carry meaning and enact potential embodiment.Footnote ¹⁷ This latter observation is supported by Candace Brower’s remark that music ‘is capable of evoking such powerful and varied emotions […] that it reflects upon embodied experience at a very deep level’.Footnote ¹⁸ Brower’s ‘cross-domain mappings’ highlight how human bodily experience can map onto ‘musical-metaphorical concepts of musical space, musical time, musical force, and musical motion’, again exemplifying the role of the corporeal body in the equation of music and gesture.Footnote ¹⁹ Hatten’s theory that virtual embodiment is shown by ‘our capacity to identify with actual human agents who are singing and perhaps dramatically gesturing’ leads him to consider embodiment as involving two human bodies — that of the virtual musical agent (as a body) and that of the composer/performer, who in turn informs the listener.Footnote ²⁰ The listener, in Hatten’s model, thus becomes the recipient. Although Hatten does acknowledge that there has to be a level of comprehension for the listener to hear gesturally what a composer may want them to, his insistence on the importance of the composer/performer in this model has its problems.

In the first instance, gestures do not need to be actioned by a human agent, virtual or otherwise. Though gesture as a non-musical concept is tied to human movement (the idea of inferring meaning through motion), musical gesture can be detached from solely human corporeal movement. Indeed, Steve Larson’s analysis of ‘musical forces’ describes ‘gravity’, ‘magnetism’, and ‘inertia’ all within the context of human embodiment rather than presupposing a human actant.Footnote ²¹ The body, and embodiment of sound, is necessary for connoting meaning and for the receiving of sound, rather than inherently tied to its actioning. What Hatten’s theoretical arguments (especially those related to gestural theories presented by Raymond Monelle and Leonard B. Meyer) ironically seem to miss, given his emphasis on metaphor as vital for the inference of musical meaning, is that non-human entities need not mirror human behaviour or gesture at all to be understood.Footnote ²² The sound of a waterfall, for example, can be gesturally represented in music without the need for the listener to question human influence.Footnote ²³ Kate Galloway explicitly notes this, as the player/avatar in A Short Hike explores their natural surroundings, ‘[filling] in the details with their imagination and mak[ing] sensory connections’ through their understanding of both the game environment and the ecological land- and soundscape it depicts.Footnote ²⁴

Non-human entities, not just music, can act as human ones. Hatten’s theory can thus be remapped to remove the insistence on a human performer. In video-game environments, music might allude to human emotion, as will be explored extensively in this article. Equally, it might simply signal towards the geographical space and ludic environment the player/avatar finds themselves performing in. In the virtual world of video games, the wonder and magic of the human imagination is the only interpreter necessary to translate musical gestures into potential meanings. Therefore, if we are to interpret musics that are ‘non-human’ in their performance (the musics of nature, soundscapes, and musical expressions of the natural ambience, e.g. game music that accompanies vast, open vistas and is seemingly not performed by an individual or collective but rather exists within the virtual temporal frame of the character),Footnote ²⁵ the question that arises is a simple one: how can we understand gesture as an innate musical feature that is detachable from the human? Fundamentally, a part of this article’s premise is to argue that gesture is imbued in music, harboured within its textural, temporal, and dynamic capacities, and that this gesture is neither human nor non-human — it is only necessary that the interpreter is human and able to comprehend possible gestural shapings in order to infer meaning through musical embodiment.

As previously alluded to, Hatten’s theory is seated in the traditional and skewed hierarchy of composer, performer, and listener. This is no triumvirate; the listener is necessarily relegated in Hatten’s theory to the role of singular interpreter, understanding the gestures as composed and performed by the former two roles. Remapping this structural viewpoint, fundamental to Hatten’s theory, is necessary, for his position does not allow for a plurality of listeners and potential interpretations, particularly in the context of video-game play. His listener is an ‘ideal’ one, understanding of learned styles, able to comprehend the meanings embedded within musical signifiers and traditions. To further highlight this: both Ratner and Monelle note how musical forms and semiotic signals can connote musical meaning. For Ratner, styles such as ‘military and hunt music’ had particular connotations linked to nobility, communal events, and the countryside.Footnote ²⁶ His writings also acknowledge musical gesture as functioning in ways that might affect musical meaning: he speaks of gestures having ‘boldness’ and notes their ability to ‘draw the listener’s attention’.Footnote ²⁷ For Monelle, hunting music (in the western art style) similarly alludes to nobility and power, yet ‘the hunt style […] can suggest the morning’ too.Footnote ²⁸ Though Monelle concurs with Ratner’s associations of nobility and the pastoral, a listener’s level of comprehension and their knowledge of and exposure to the music they are interpreting can lead to different levels of inference. Unique listeners, with all their individualities, biographical disruptions, personal traits, and even lack of comprehension of styles that can define the musical meanings, can draw different semiotic implications from the same musical gestures.

Hatten does note that musical gestures are ‘grounded in human affect and its communication’ and that physical gestures, such as gesticulating or common human interactions such as waving hello and goodbye, can be replaced musically by the ‘characteristic shaping’ that gives music its expressive meaning.Footnote ²⁹ Music’s ability to connote rather than explicitly denote is thus of significance here. Importantly, as noted by Monelle, the idea of connoted meaning through sound is a cultural one; cultures will interpret sounds differently according to their own understandings of musical shapes.Footnote ³⁰ For both Monelle and Umberto Eco, objects and entities characterizable through music can only be considered cultural units in semiotic frameworks — objects existing within cultural norms and conventions.Footnote ³¹ It is thus vital that we do not assume cultural competence when we analyse musical gestures and their implications and potential meanings; ‘cultural gestures’ or ‘learned styles’ must also be considered when analysing musical gesture, as form can heavily contribute to musical meaning.Footnote ³² For Hatten, therefore, musical gesture is the gateway to the interpretation of musical meaning; should the narrative and/or dramaturgical trajectory be discernible through music, it is gesture that allows for its virtualized reality to be witnessed and heard.

Within my own theoretical framework, the listener is the ultimate recipient. Each listener carries with them instances of behavioural comprehension, societal norms, literacies, biographical disruptions, and their own personal make-up of likes, dislikes, interests, and learned styles. As Stephanie Lind observes, in video-game media, ‘fictional universes [created in games] are self-established and self-referential, linked to player imagination’.Footnote ³³ The listener is thus multifaceted and certainly not homogenous: individuality is key to understanding that musical gesture, along with the inferences, signals, and semiotic meanings it may carry, is there to be interpreted by rather than thrust upon the listener.

Remapping Hatten’s vision of the composer/audience/performer relationship to the context of video games blurs his tripartite categorization of those roles. This merging of composer, audience, and performer is an updated aesthetic foundation, one that posits the listener’s inference and potential imaginative capacities as paramount. This thus aligns with Grosser’s observations around player/avatar fusion in games. When we experience music or sounds in a game, the in-game avatar ‘is the only thing that is actually influenced by the sound effects and the musical constraints’, while the ‘Player virtually coexperiences these restraints with the Avatar, reinforcing the Avatar/Player’s subjective experience’.Footnote ³⁴ The player, through embodied experience, aligns themselves with their avatar, creating a feedback loop for musical gesture and empathetic interpretation. The game music connotes, the avatar experiences, and the player, embodying sound in the player/avatar fusion, understands both the music and the ludic implications of their actions. Game music can suggest ideas but also ludic possibilities. The game composer Winifred Phillips talks about music as having the capacity to

The role of the listener has now changed when considered within video-game listening and playing; they are now viewer and listener combined. As Kasey Collins writes:

Thus if the player of the video game is both player and listener, we see a fusion of roles emerging — that of the player and performer, or player/avatar as a fused state. The lack of the presence of a performer as separate from the player is crucial. The question in video games is not ‘who’ is doing the performing, but rather the meaning of the music in relation to the player as performer. The player is engaged in an attentive way, both ludically and in terms of aurality. The crucial tenets presented here thus concern the role of the listener in their new fused state, the new mode of musical conveyance in the form of the audiovisual video-game medium, and, finally, the plurality of possible meanings due to the non-homogenous nature of individual listeners as a collective.

As is becoming clear, any theory must account for the possibility that there is no one single reading of a video-game music score. Different listeners will infer various meanings from the audiovisual medium, and the uniqueness of gestural interpretations to the listener is important to note. Thus, although we must not assign one particular analytical reading to a video-game audiovisual text, while also not enforcing one gestural interpretation, by examining how musical gestural potential could be realized, we can begin to understand how listeners/players may become cocooned in the virtual sound- and landscape and engaged with and immersed in the medium.

In conceptualizing how a listener might become immersed in the game through the ‘gestural potential’ of music, this analytical framework will also consider the question of exactly how musical features might carry such meanings for the listener. What in the music encourages a certain tactility, a willingness for a player to want to engage physically with the game? What makes them want to pick up a controller and play? Additionally, what exists within the make-up of music that suggests meaning, that infers signs and semiotic signals, that promotes the possibility of embodiment so that the player may feel a part of the ludomusicological text and want to engage with the game?

Musical gesture in this context will be examined as an identifiable, traceable, and determinable musical element that can generate feelings of intense immersion and engagement, and can harbour the ability to reveal possible musical meanings. Crucially, specific terminology is necessary to identify potential inferences and the semiotic signals musical gestures can connote. It will be seen that music possesses what I term gestural potential, and that various musical features and the ways in which these work together can generate gestural energy (or charge), leading to an overwhelming sense of virtual-physical ‘pull’, causing players and listeners, whether wholly engaged with the aurality of their experience or not, to feel immersed, engaged, and ultimately, ludomusically cocooned within the virtual imagined environment they find themselves in. By reconsidering how we listen to game audio and how we play with both ludic parameters and musical shapes, we can ultimately trace gesture as a force (in Larson’s model of musical forces) that encourages us to both move virtually in a game and be moved in turn through emotive, affective states.Footnote ³⁷

Towards a New Theory of Musical Gesture

Remapping and reconsidering theories of musical gesture in an engaged audiovisual context

It is worth summarizing the main facets of previously outlined theories that need to be reworked and remapped in order to best fit a new framework for the context of video games. To break down and observe musical gestures at play in the ludomusical texts, we need to:

1. 1. Remap the role of the corporeal vessel as the responder to audiovisual stimuli.Footnote ³⁸

2. 2. Recalibrate the listener as both interpreter and inferrer of musical meaning as well as agent, player, and, to an extent, author of the audiovisual video-game medium.Footnote ³⁹

3. 3. Reinterpret musical understanding and meaning away from historical western art concepts, derived from the above two reinventions in the audiovisual frame, in order to remap how meaning is derived and from what source.Footnote ⁴⁰

4. 4. Reconsider the concept of musical embodiment to account for the virtual realized self as manifesting in the form of the player/avatar fusion in the game’s sound- and landscape framework.Footnote ⁴¹

5. 5. Examine interdisciplinary fields of scholarship that might yield further possible lines of investigation in regard to musical gesture, meaning, and immersion. These include dance and dance pedagogy, art styles, behavioural science, and psychological fields examining musical embodiment and engagement.

6. 6. Devise a new set of terminological denotations, including a newly coined concept and mode of analysis, to better understand the musical gestures being identified.

To undertake this work, firstly, the human body as a corporeal vessel for gestural manifestation must be considered in more detail. As highlighted, the body’s response to stimuli results in tangible gestures, be these physical manifestations or emotional reactions. In order to identify how players become immersed in games, we must first identify how and why musical stimuli initiate a corporeal response, urging the player to engage in play, pushing buttons to play in the tactile sense, and finding a meaning or reason to fight off enemies, explore worlds, and achieve gamified goals.

Secondly, by reconsidering the role of the listener as the interpreter of musical signs, semiotic signals, and connotations, we can begin to remap a theory of musical gesture with the listener at its centre. The listener’s role must be critically examined, along with the messiness of biographical individualities and learned interpretations that they bring to the act of listening and play. Linked to this, the player’s ludic response will be seen to be governed in part by the unique identification of musical meaning as deciphered and derived by the listener/player. This links to the third reinterpretation of musical gesture theory and will necessitate a closer examination of audiovisual stimuli as well as learned styles that players might bring to the act of play.

The above concepts cannot immerse and engage the player without them feeling embodied by the music and present as an imagined virtual participant in the sound- and landscape of the game. A meticulous focus must be put on how players can feel embodied by the music, even when only passively engaging with the music that soundtracks their play. Once such an embodiment has been established, we can begin to identify the individualities of musical features that, through musical gesture, can lead to player engagement and immersion. Such analysis will in part be brought about through a consideration of the fifth aspect addressed above, namely a utilization of interdisciplinary scholarly fields that already concern themselves with musical embodiment. Though dance has been briefly discussed, further examination is necessary, along with a look towards styles and concepts in art as well as behavioural science and psychological analyses of how music can embody the listener, and in this case the player, engaging them in a sensory experience and allowing them to find meaning in the sounds they hear.

What musical gesture theory has not had, until now, is a tool that can be used to identify and explicitly highlight these gestures as operating within music. This article will introduce new terminological denotations, including the concepts I term the gestural potential of music and ludomusical cocooning, and a new analytical model able to graphically visualize musical gestures made manifest. Graphical models will mark and reveal musical gestures, allowing us to visualize sensations previously identified as motion- and momentum-based. Building on the third observation above, a reinterpretation of western art music concepts, including traditional modes of analysis, will allow us to see the wider machinations of musical gestures, the layers of which culminate and result in a complex web of musical connotations, signs, and semiotic signals, leading to possible corporeal and ludic responses on the part of the players.

Inspired by Richard Middleton’s model of musical ‘groove’, this graphical work is central to my theory.Footnote ⁴² Middleton’s graphical model helpfully identifies the overall musical feel and how we might interpret vocal lines and rhythmic impetuses as functioning within popular music tracks. However, the nature of the video-game medium is more complex, with extra agential and audiovisual layers combining to create the interactive game environment. Middleton’s analysis only reveals rhythm and melodic contour as key components, but this is not all the player might experience in a game. Though his graphical depictions are a solution, they can be identified as only the initial blueprint, which needs extensive reworking to adjust for a wider array of variables in the video-game format. To begin with an outline of the eventual graphical analysis this article presents, one might follow Middleton’s formatting to present the musical playfulness of the ‘Yoshi’s Island 1’ theme from Super Mario World in its reduced ‘groove’ form (see Example 1).

Example 1.

A graph depicting overall gestural shaping in Middleton’s ‘groove’ identification style. Blue lines outline the overall phrase shaping in a simple antecedent–consequent phrase structure.

However, the musical ‘groove’ alone is not what we must analyse here: how does the music move in a gestural sense, and how does this motion then in turn move and affect player interactivity? A graph depicting only the musical ‘feel’ and motion does little to highlight melodic contour beyond overall shape. Musical texture is barely discernible here, and there is little to identify harmonic movement or timbral shaping. Despite being a helpful guideline for overall musical groove, methodologies such as Middleton’s must be remapped and recalibrated for the video-game format, and more must be done to help analyse and trace the musical gestures at play.

Gestural potential and ludomusical cocooning: A theory of music, gesture, and meaning

A theory of musical gesture must identify and accept the messiness of its topic area. Players, as a collective, interact individually with games and the audio dimensions of virtual land- and soundscapes. The diagram in Example 2 demonstrates the structure of my theory of music, gesture, and meaning as a linear, though often overlapping, model, opening with the initial stimuli of music’s gestural potential and concluding with the final state of the player as immersed in their game environment via audiovisual synthesis, in a state of ludomusical cocooning.

Example 2.

A model of musical gesture in an engaged audiovisual context.

This structure denotes the necessary, identifiable stages relating to how a player might become immersed and engaged in the video-game land- and soundscape. The kernel of my theory of musical gesture can thus be found in what I term gestural potential — that is, put simply, the ability of music and sound to harbour possible meaning, which can be identified in and traced through movement-based shapings. In this article, musical gesture aligns closely with a remapped version of Hatten’s conceptualization of ‘energetic shaping through time’, where it is linked more specifically to music’s ability for movement-based suggestion, connotation, and impetus. As Iain Hart notes, when we listen to the music in games, the sonic element of the audiovisual text is ‘filled with the potential […] for communication’, a reminder of music’s adaptability and potential for semiotic implications as well as something more.Footnote ⁴³ In harbouring connotative allusions, music has potential for communication beyond a reader/listener/player’s initial observations. Summers’s observation of ‘playful listening’ is helpful here: ‘the music we hear has the potential to be otherwise’, highlighting music’s adaptability and possibility for connotative impetuses.Footnote ⁴⁴ We can identify music as having the capacity to create a space for gestural possibility, allowing players/listeners/avatars as a fused state to individually respond.

This therefore leads to the concept of ‘potential’, the idea that music does possess gestural qualities but the unlocking of these facets for the listener, and the player’s ability to understand and comprehend them, is untapped until awoken. This concept is given further credence by Monelle’s aforementioned identification of ‘cultural units’:

Cultural units combine to form a culture, as words combine to form a language. Culture defines society, and society operates within history. In order to describe musical topics, there must be a full account of cultural mythology, of literary genre and symbolism, and of social history.Footnote ⁴⁵

Monelle views music’s ability to connote as a necessary point of musical understanding: if a listener attaches to the symbolism, culture, and social history behind a specific cultural unit, they will comprehend it. Player/avatar embodiment in games highlights how the avatar can respond to the aural stimuli through the understanding of the player. The player responds cerebrally to a ‘cultural unit’, to borrow from Monelle, causing the avatar to react on screen when the embodiment of player/avatar fusion is active. Yet the reaction to the audiovisual stimuli does not have to be solely cerebrally driven. In her discussion of the meaning of shapes and gestures in dance and how dancers can react to musical stimuli, Ann Cooper Albright notes the work of Maurice Merleau-Ponty, another observer of such cultural units.Footnote ⁴⁶ She argues that we should

try to understand the reality of our existence by recognizing our perception is interactive with the world. Thus, the table around which my family gathers is not just an inert object made of wood — it means something and calls those meanings forth when I sit down at it […]. Our bodily experiences […] provide the ground for our thoughts and not vice versa.Footnote ⁴⁷

For both musicologist and dance pedagogue, art is embedded with signs that connote meaning, and it is in musical gesture that this meaning is harboured, ready to be perceived and comprehended by the listener. Equally, through ethnographic research, DeNora discerned that ‘music’s “effects” come from the ways in which individuals orient to it, how they interpret it and how they place it within their personal musical maps, within the semiotic web of music and extra-musical associations’.Footnote ⁴⁸ Culture, and an individual’s relation to it, vastly affect their personal comprehension of music and potential meanings.

However, even though cultural associations and learned styles, to recall Ratner’s propositions, can invite further musical-emotional inference, the corporeal response to music highlights the role of musical gesture without cerebral interference or layered meaning. As Marc Leman astutely observes, ‘even if the cultural meanings of the gestures are unknown, it is still possible to experience and feel the corporeal meaning because the music has its foundation in a physical constitution that all humans share’.Footnote ⁴⁹ Returning, therefore, to musical gesture as interpreted through dance, Thelen and Smith’s argument that ‘movement must itself be considered a perceptual system’ is crucial.Footnote ⁵⁰ The ear and mind listen, the body moves; we perceive, we link our biographical individualities, and we respond. As in dance, when we play games, the complex mechanisms that form our response systems, driven by the human ability to connote action and meaning, urge us to respond to the audiovisual stimuli. Musical gesture is a potent messenger, delivering stimuli the listener and player feels compelled to respond to through their complex interpretive web. The result, in games, is what I term the imagined virtual. Footnote ⁵¹ In other words, the placing of oneself within the game world, sharing the experience with the avatar, leads to a projection of virtuality. The player feels a sense of embodiment and, through the musical gestures they identify and connect to, engages and immerses themselves in the game world.

Gestural potential is thus a crucial part of the individuality of musical listening and ludomusical play: players experience game music and register gestural connotations in ways intrinsically tied to their contextual individualities, biographical peculiarities, and their own recognition of cultural units, as alluded to at the outset of this article. Collective culture and individual experience shape player understanding, and the gestural potential of music can tap into this sense of comprehension, allowing players to connect uniquely with the musical material used in the game.

Gestural Potential and Corporeal Embodiment: Playing Mario and Samus

Once a player engages with the music of the game, they can connect with the virtual sound- and landscape to a far greater extent. Their gateway to this sense of connection and potential immersion is enabled by music’s gestural potential but also by their identification with the avatar. As identified previously, players play as a fused state, at once avatar and agential controller, two identities combined into one. This fused identity is significant here. Helen Kennedy notably discusses how ‘distinctions between the player and the game character are blurred’ in this sense.Footnote ⁵² Indeed, game music can be seen to combine Anahid Kassabian’s ‘assimilating’ and ‘affiliating’ forms of identification in film music.Footnote ⁵³ Players can be seen to fuse with their in-game avatar, aligning their identity, personality, will, and behaviour with them in order to ludically succeed.Footnote ⁵⁴ Musical embodiment is a key component of such identification, which can aid us in our identification of potential embodiment, from the player on one side of the glass pane of the screen to the avatar on the other.

When the player plays, they push analogue buttons, swipe on interactive screens, swing infrared-tracked Wii remotes, or step on colourful interactive arcade-machine dance pads, to name a few modes of virtual tactile engagement. The result is a tangibility, a tactile dimension to the virtuality of the sensory experience, again highlighting the embodiment that must take place for the player to leave the concrete affective world for the virtual. To best explain this sense of corporeal embodiment and the effect of music’s gestural potential on the player, it is fruitful to return to the opening case studies, Super Mario World and Super Metroid — two games with superficially near-identical gameplay conceits and control schemes but highly contrasting affective experiences and potential meanings.

Super Mario World opens with ‘Yoshi’s Island 1’ in a burst of colour and joyful musical gestures. Kondo’s music, tripping over itself with eager anticipation, gleefully soundtracks my ludomusical journey. More than this, its gestures drive my action. At the beginning of the level, a Hollywood-style sizzle following the words ‘Mario Start’ fires the initial gestural starting pistol. A villainous-looking orange-and-blue blob cascades towards me, and my first action is to do something, anything, to avoid it. Golden coins, sparkling and alluring, shimmer as I collect them, while colourful, oddly shaped hills scroll across my screen. Later in the same gameplay I come across a shell. It is in stasis, its gestural energy untapped. So I hit it, and it spins into life. And I get points for this, thus feeling rewarded. Throughout my experience, monsters, as I identify them, file towards me, attempting to usurp my gestural impetus. I am going one way, they are pushing back. Their direction, their gestural momentum, is opposed to mine, and my identification of these creatures is therefore a negative one. They thus form a ‘cultural unit’ in my mind and have a certain disagreeable semiotic implication.

It is the music and the audiovisual combination of gameplay and sound that spurs me into motion. Kondo’s sound world is light and frenetic. Basslines meander playfully while a driving staccato accompaniment mischievously threatens to usurp the melodic contour’s foregrounding. There is an airiness to Kondo’s music here, a lack of grounding. It is clever music, intricate and involving, designed to move players and be moved to. My will to play, my eagerness to engage whimsically in the game world is embodied by the musical gestures. The music and sound world is freeing and light. Mario’s own gestures, his agency, has been transposed to me. I have to play within the game’s ludic boundaries and rulesets, but the connection I feel to the character of Mario is one of player/avatar fusion and embodiment. Through musical and environmental connotation and gestural shapings, I identify tropes that, on the one hand, could connote an evil presence, and on the other reveal this as a fun space for play due to the sensations tied to the ‘cultural units’ at work. The cute, rather than terrifying, dragon enemies are as playful as Mario’s knowingly comedic gesture of clinging on to his hat as he falls from a height. When I jump on them, squashing them down and earning points for my achievement, I’m greeted with buoyant sound effects, always moving melodically upwards, unburdened by cadential closures or weighty timbres. My understanding of Mario’s playful nature, combined with my comprehension of the game’s spirited sound world, is vital. My avatar, moving left to right towards his goal, is spurred on by rising melodic shapings accompanied by a bouncing bassline, exuberant and jocund. Mario’s gestures and the musical gestures function symbiotically, and through this interaction I am able to form my own ideas of musical meaning and, importantly, am able to reach a stage of embodiment. Player and avatar fused, I now play the game as Mario. Certain musics have semiotic implications for certain players and audiences, and these implications are contained within the gestural potential of the music — they have to be comprehended, tapped into, in order to be fulfilled and carry meaning for the player.

Contrast this example with that of Super Metroid. As the game opens, the player is required to flee the facility they find themselves in so as to escape the destruction of the virtual environment. The game’s heroine, the suited Samus, must run backwards, retracing the path the player initially forged. As alluded to previously, the game environment is full of panic and unease. The speed at which the player must flee the facility has gestural implications, stoked by the ominous timer in the top-right corner of the player’s screen along with the cultural unit of the siren sound, blaring atop a musical texture that ominously traces the sense of fear, even dread, the player feels as they move swiftly towards the exit. The intensity of the burnt-out colour palette meshing with jagged metal corridors is far removed from Mario’s rolling plains, and the lifeless bodies appearing throughout further highlight the seriousness of the narrative situation. With the game environment collapsing, the facility begins to tilt, disorienting the player somewhat. When the player finally reaches the last environment to ascend, the door explodes behind them, reinforcing the idea that there is now no turning back. The screen begins to slope and the chaotic final climb is marked by its sense of urgency and potential annihilation. Gesturally, the feeling matches that of the music, one of panic and anxiety as the environment collapses. Moreover, the game environment aims to disrupt the sense of gestural flow. On countless occasions, Samus can be seen stumbling backwards as fragments of the game’s environment fall upon her or steam prevents her from ascending towards an escape.

The mechanics of the game thus feed into the gestural tactility. The control input must additionally be considered in a gestural sense. The player performs the reverse functions in relation to their entry into the facility. Compared to Super Mario World, the ‘jump’ mechanic is weighed down, the gravity too strong or perhaps the momentum too weak to jump as high as in the Mario platformer. The feeling in Super Metroid is one of impending doom, again showing how gestures, physical and musical, can highlight stark differences in game feel and the ways in which we can interpret the gestural potential of music as affecting player experience. Kenji Yamamoto’s music is stricken with oppressive musical gestures, weighed down, as I will analyse, by heavy synths and debilitating rhythmic shocks. An unstable time signature, rhythmic slippages, awkward crescendo effects in both the musical soundtrack and accompanying sound effects, and cumbersome basslines threaten my avatar, Samus, as she attempts to escape. There is a claustrophobic quality to Yamamoto’s composition, the music feeling as though Samus has to wade through it to reach her goal. Its oppressiveness, its tension, and its failure to allow for a moment of respite are what stick with me most. The gestural potential of the music combines with the interactivity of the game and the player/avatar fusion to give different affective experiences.

Embodiment across the concrete/virtual boundary is thus key for players. It is the bridge between the acknowledgement of the gestural potential of music and possible immersion and engagement for the player. A key component of embodiment is empathetic connection, something that is difficult to trace in the previous example. I, a player who has never been to another planet, am expected to empathize with Samus, trapped on an alien facility in outer space. How and why should I empathize with her? Is it possible for me to even do so? Games frequently find players of all ages, genders, abilities, and cultural proclivities engaging in their art forms, and it is thus necessary to discern how a player removed from an avatar’s sense of identity might find themselves playing as them in the fused player/avatar state while also caring enough to move and be moved, connecting with the musical gestures in order to become immersed in the game.Footnote ⁵⁵

The idea of cultural units has proved fruitful in this article for extrapolating such issues. Combined with this, visual art studies and dance studies highlight a further connection between the art, meaning, the perceived character, and the individual. For the philosopher Robin Collingwood,

The game and its audio relay the gestural potential of the music. These imaginary gestures are received and interpreted by the player, who engages in its ‘tactile’ nature.

Further extrapolating this idea, cultural units and their meanings can give further weight to ideas of embodiment. As Cynthia J. Novack pronounces,

Movement, and the gestural meanings it harbours, when performed by a dancer who is performing their own self-expression along with prescribed movement rules, thus introduces a complex web of potential meaning. To return to the concept of play and games, Albright argues that dance and play have many similarities. In noting the playfulness of her art form, she acknowledges that ‘play is a mess. But there is a mesh in that mess, which holds, but does not bind, that mess together.’Footnote ⁵⁸ Taking the complex web of potential meaning alongside this idea of a binding mesh, we can begin to see how games and play exemplify a heightened virtual version of this extended idea of embodiment and virtual self-expression. The player plays with their own individuality and personal histories, combined with those embedded into the identity of the avatar. Simultaneously, cultural units and musical gestures have the potential to be read and understood by the player performing in the game world as a fused player/avatar. The result is a sense of embodiment that is quasi-real, the virtual becoming engaging as the music embodies the player’s sense of musical understanding and connoted meanings. It is through this lens, this interconnected web of embodiment, that I understand how and why I play as Mario and how and why my play as Samus feels so different.

Embodiment, and the line between actual experience and the virtual self-expression a dancer relays when performing, are vital here. In discussing the ‘unravelling [of] meanings […], movement, bodies, and identities’, Albright further notes that ‘cultural values resonate throughout the bodies that constitute them, and often these structures are physically internalized and thus rendered as “essential” elements of human nature. Yet the body is at once social and personal, internal and external.’Footnote ⁵⁹ For Albright, therefore, the body is the vessel through which identity is both constructed and revealed. Virtual self-expression through dance allows the exponent of the art form to at once identify with source material that is not immediately their own, while also enabling them to portray their own emotions through musical and gestural embodiment. Such findings thus give credence to two elements in gameplay embodiment: firstly, that players, when performing in games, can be seen to embody the avatars on-screen, leading to the fused player/avatar state even when the avatar is not immediately identifiable with the player’s own sense of self; and secondly, that the gestural potential of music can have an instigating effect on such embodiment and identification, allowing for cultural meanings and personal values to play out through both the avatar and the player, leading to a state of embodiment that allows for virtual self-expression.

Furthermore, discussing how dancers might put their emotions into prescribed movements and gestures, Susanne Langer notes:

This is analogous to the game player — the player plays with actual movement, with real emotive gesture. As Katherine Isbister notes in How Games Move Us, ‘the player moves through the game world taking actions as this person [avatar], adopting his or her concerns and struggling towards his or her goals’.Footnote ⁶¹ The way the player feels, the emotion behind such movement, is a form of virtual self-expression. The player therefore does not need to be Samus, they only need to perform as her. The sense of embodiment a player might feel could simply be linked to the need to surpass something, to overcome a hurdle. Samus’s escape from the facility could be analogously linked to a player’s concrete urge to complete a challenge. Embodiment therefore comes through identification with musical gestures, empathetic impulses, and the perception of cultural units.

We have therefore observed how players can identify musical gestures as harbouring the potential to connote signs and semiotic implications; this leads to possible engagement with the avatar on screen and thus allows for a fused player/avatar state to occur, revealing itself to be an act of virtual embodiment. The following section, outlining the final stages of this theory of musical potential and immersion in games, points to how immersion can be achieved for the player in a virtual environment, with musical gesture a key component and instigator of this engaged state.

Musical Meaning: From Gesture to Immersion

Immersion is a slippery term that, unhelpfully, avoids precise definition, particularly in the act of and discussion around video-game play. It has become a buzzword, a catch-all to define the sense of engagement a player feels when playing the game. A theory that concerns the gestural potential of music and how this might engage the player, immersing them in the game world, needs to identify the term and give weight to its meaning, particularly given the ‘messiness’ of each individual player and the ways in which they engage with games in a unique way, as I have already identified.

In their paper entitled ‘Player–Video Game Interaction: A Systematic Review of Current Concepts’, Loïc Caroux et al. sought to provide a summary of player interaction with games, based on a collection of empirical papers concerned with player engagement. Crucially, the review found that ‘sound and music may influence players’ performance, immersion, flow, and emotional state [in games]. Regarding input information, the way of game control, such as motion, tactile, tangible […] may influence players’ performance, engagement, presence, perceived realism, and enjoyment.’Footnote ⁶² These findings therefore highlight much of what has been investigated in this article, namely the combination of music and sound stimuli with tactile, tangible engagement. With players comprehending and unlocking the gestural potential of music, game environments feel ‘enveloping’, and the interactive environment, with its ‘stream of stimuli and experiences’ — notably, both audio (musical gestures) and visual (as in the quirky, colourful environments of Super Mario World) — combine symbiotically to inform the player of potential meanings. For my own definition of immersion, one that engages more with the phenomenological aspects of play, I turn additionally to Gordon Calleja’s concept of immersion as ‘incorporation’. For Calleja, immersion as identified by the studies above points to one key element: a ‘state of deep involvement’ in the game that ‘results in the shortening or disappearance of distance between player and game environment’.Footnote ⁶³ Rather than adopting ‘immersion’, an already loaded and prickly word, as a key term in regard to player involvement in the games they play, along with their identification with the game environment as comprehensible, we can instead look to Calleja’s model of ‘incorporation’ to identify what is at stake here. For Calleja, the aforementioned ‘shortening’ of the distance between the player and the virtual sound- and landscape of the game can lead to players ‘interpret[ing] the actions of their avatars as being their own actions in the game world. The sense of being in the environment is what has been referred to as immersion.’Footnote ⁶⁴ Calleja’s concept is linked specifically to game engagement and highlights the role of musical stimuli within the framework:

The result is a gameplay experience that sees the player embodied through audiovisual stimuli, a performer within the game’s land- and soundscape able to virtually self-express within the game world.

Calleja’s model, when adapted specifically to the ludomusical frame and music’s function within game spaces, thus leads us towards what I propose as ‘ludomusical cocooning’. To feel ‘enveloped’ in an immersive sense requires both audio and visual engagement. For Rod Munday, ‘sound surrounds the listener, blending and combining in ways that visual information cannot emulate’.Footnote ⁶⁶ Audio envelopment, a concept that has been identified and empirically investigated by the psychologists Sean Zehnder and Scott Lipscomb, is critical when the player is immersed in the audiovisual text.Footnote ⁶⁷ Through a combination of this envelopment and the aforementioned idea of embodiment, we can begin to understand the player as part of the virtual world through immersion. Indeed, Isabella van Elferen notes that ‘immersion in sonic virtual worlds is not merely emotional […] but physical and geographical as well’.Footnote ⁶⁸ This concept of immersion, its sense of deep connection that is both emotional and physical/geographical when informed, heightened, and in some cases instigated by music and sound, can be remapped and redefined as ludomusical cocooning.

When a player is tuned into the musical gestures and gestural potential of the music while also feeling embodied through the tactility of the game controls in combination with the virtual self-expression they experience, they move towards new levels of immersion. We could argue that they are cocooned, much in the same way that audio cocooning occurs in a car when the radio is playing, for instance.Footnote ⁶⁹ This concept has been further investigated by Michael Bull, who notes that through habit or effective impulse, drivers tend to switch on their personal radios, thus appearing ‘to prefer inhabiting an accompanied soundscape’.Footnote ⁷⁰ A similar phenomenon occurs when I walk down the street oblivious to external sounds because of the AirPods I am wearing. I am aware, to an extent, of the outside world but am not engaged in the aurality of that space. For Marjorie Kibby, such an audio-enveloping experience can be equated to a ‘cocooning’ effect for the listener.Footnote ⁷¹ Both car and personal audio listening highlight the aspect of engagement and immersion this article has identified, that of virtual self-expression within a newly created space for the listener. For video-game players, this can be reidentified as a form of listening and engagement that combines audio and visual stimuli, where the player is at once embodied by the medium and enveloped in its land- and soundscape.

Playing games like Super Mario World and Super Metroid, I am cocooned in my own sphere of aural immersion in much the same way as when, staring agog at the CRT television screen in front of me as a young player, I was unable to peel myself away from the act of play and the audiovisual experience I was performing within. Ludomusical cocooning thus constructs a middle-space fusion between two realities, enveloping and embodying the player in the game world through the initial kernel of meaning they discern through the gestural potential of music. The phenomenon of ludomusical cocooning has been seen to stem from player identification with and comprehension of the gestural potential music may harbour to connote possible meaning and semiotic implications. With this remapped theory of musical gesture established, we can begin to trace a line of inference, of connotation and potential meaning, from the smallest of musical stimuli, imbued with potential meaning, to an overall affective experience for the player. Analysing musical gestures in graphical form, we will begin to see both broad and exact musical features that highlight gestural potential and the possibility of ludomusical cocooning.

Marking Musical Gesture: Graphical Depictions of Potential Musical Meaning

The best way to understand musical gesture, or perhaps to show how it actually functions in a musical sense, is to visualize it — thus to graphically depict the gestures at play. This allows the analyst to view the playful shapings of music across time, in a quasi-analogous way to player experience of such audiovisual movements, stimuli, and momentum-based gestures. The graph in Example 3 highlights a section of a gestural analysis of the Super Mario World theme.Footnote ⁷² Inspired by Middleton’s graphical model, its shapes and annotations have, however, been extensively remodelled to better suit my concept of the gestural potential of music in addition to the audiovisual idiosyncrasies of game music as a style and genre. In addition, the graphical depictions have been inspired by artists like Paul Klee, who notably engaged with gesture in his work, again demonstrating the interdisciplinary nature of this article’s field of study. For Sibyl Moholy Nagy, the ‘beacon, guiding Klee through his [artistic] adventures […] is the line in action’.Footnote ⁷³ Gesture and momentum thus play a crucial role, and Klee’s use of the arrow, in particular what his ‘Black Arrow’, is significant. For Klee, ‘the father of the arrow is the thought: how do I expand my reach?’ The Black Arrow is the epitome of this, where colour intensifies to a point of immense energy and momentum; Klee noted that this ‘extraordinary increase in energy […] is decisive for the direction of movement’.Footnote ⁷⁴ Indeed, this links back to dance studies and their intersection with gesture and movement. Rudolf von Laban said that ‘the expression of movement depends […] on several factors — space, location, including shape, and dynamic content, including effort’.Footnote ⁷⁵ Direction and shape are key to both the artist’s and the dance pedagogue’s principles and thus crucial to the identification and enactment of gesture. In my own graphical work, the arrow and other shapes analogous with it denote the direction of movement and connote the possibilities of gestural potential. The graph-form analysis is intentionally idiosyncratic in nature, an expression of individual interpretations of gesture, reflecting the methodological observation of individuality present in gestural experience and comprehension.Footnote ⁷⁶

Example 3.

Musical gesture graph highlighting the gestural potential of musical shapings and possible meanings for the player. Bars 6–9 of the ‘Yoshi’s Island 1’ theme from Super Mario World.

Example 3. Long description

The graph is organized into three vertical sections.

Top Section: Melodic Line.

- Features a standard musical notation snippet at the top left.

- Below it, a series of abstract black symbols including circles with upward arrows and rectangular boxes.

- Annotations state: Black lines denote shaping of gesture; Upward arrow shapes indicating gestural uplift and playful buoyancy; Boxes indicating important tones, again with gestural playfulness.

- Blue curved lines arc over the symbols, labeled: Blue lines denote momentum of gestural potential.

- At the far right, a zigzagging line is labeled: Sliding line with an emphasis on forward momentum.

Middle Section: Middle Texture Rhythmic Line.

- Displays eight repetitive, thick-bordered rectangular blocks containing two horizontal arrows each.

- A small musical notation of triplet chords is positioned at the start.

- A central annotation describes these as: Fast, chordal, repetitive shapes in a driving dotted rhythm, emphasizing player anticipation. Perhaps mirrors the unknown but exciting possibilities ahead.

Bottom Section: Bassline.

- Starts with a bass clef notation.

- Features a series of black dots connected by angled lines, resembling a jagged waveform.

- Annotations include: Bass is rhythmically simple but with a buoyancy that suggests anticipation; Playful chromatic inflections; and Surging gestural energy in bass, which points to large upward-curving blue and black arrows at the end of the sequence.

- A final note on the right states: Fast tempo is playful and buoyant. Lightness of gestures overall should be observed.

The musical gesture graphs are to be read from left to right, with simultaneously sounding gestures aligning, as in chordal representations in traditional notational forms. Text boxes can be added to further indicate gestural patterns and the links between graphs. New gestural shapes, as determined by the analyst, can also be explained using annotations. At the base of the graphs, short lines with markers denote a timeline.

At the top of the graph, the melodic line can be seen, the specifics of its musical contours shown in three distinct shapings. The first, an upward arrow figure topped with a circle, highlights the immediacy of the melodic impetus. Upward arrow figures are more buoyant than the middle texture’s forward arrows, indicating anticipation. The bulbous shape, like the superimposed circle, indicates the size of the gesture. In comparison to simple lines, these bulbous gestural shapes are variably denser in textural terms or carry more directional weight within the context of the gesture. The second shape, a box with an upward arrow, emphasizes a new rhythmic texturing, one that, as indicated by its directional arrow, reveals the anticipatory nature of the gesture — in this case, upward arrows imply that the gesture seeks to move beyond its limited pitch group, to gather more momentum before returning to the initial bulbous shape. Between these gestures, arcing lines more obviously denote rising or falling melodic gestures. The final gestural shape located in the melodic contour is that of the final semiquaver collapse of the melody. Here, a number of musical elements are occurring, gesturally speaking. The through arrow indicates the momentum of the gesture. Musically, the line seeks to continue the playful sense of whimsy the melody has so far expressed. The rest at the end of the four-bar phrase has the potential to disrupt this, but the gestural energy is retained firstly by the initial scalic fall’s playfulness, and secondly by the walking bassline, its rhythmic impetus increased to make up for the deficit in the melody. The arcing line indicates the rise and fall of the melody proper and, surrounding this gesture, starker straight lines capture the contour, highlighting its limitation in regard to pitch, direction, and flow. This is not an upward, buoyant gesture as seen previously, but rather a collapsing one that seeks to anticipate the return of the phrase in the following bar.

At the centre of the texture, a rhythmic line is heard, quietly operating as a rhythmic impetus, its purpose being to enhance the anticipatory feeling of the gestures at play. The vertical lines marked by their dashes symbolize the rhythmic charge of this gesture, its triplet semiquaver rhythms bundling and bouncing their way into the next rhythm set. Boxes denote the chordal nature of this playful charge, with forward, rather than upward, directional arrows further highlighting the anticipatory feel. Below all this, underpinning this momentum is a stable walking bassline. Circles indicate the notes proper, while upward arrows indicate the bounce of the line as it tries to rise through the texture. In the section’s final bar, the arcing line indicates the drive of the bass as it seeks to compensate for the aforementioned loss of gestural charge and momentum in the melody line.

Further gestural shaping is shown by the blue lines, which indicate the momentum and direction of the music’s gestural potential. In the case of Super Mario World, the graph highlights the buoyancy of the gestures: it is playful music that seeks to engage the player in a sense of gleeful delight as they surge through the game environment. Kondo’s sweeping musical gestures, charged with energetic playfulness, not only accompany the on-screen action but additionally exude an abundance of musical colour, complementing the bright and cartoonish tone palette of the virtual world. The tempo is fast and engaging, the gestures are light overall, the chromaticism in the jazzy walking bass is loaded with quirkiness, and the gestural potential of this passage can be seen as spurring the player into action.

Contrast this with a graph from Super Metroid depicting the spinning theme that changes time signature from 6/8 to 2/4 to 5/8 in the final two bars depicted here (see Example 4).Footnote ⁷⁷ Again, the tempo is quick, but it now emphasizes the feeling of doom. The thickness of the line borders reflects the gravitas of the musical gestures. Downward arrows highlight the heaviness of the gestural momentum. The chromaticism is now intense rather than playful, and the bass is the main textural feature. Gone is the playful rhythmic melodic line; what the player hears is the weight and solemnity of the bassline, its chromaticism, and its irregular time signature, which almost seem to weigh the player down while simultaneously alluding to the need to escape, fast. Notably, the spinning melodic line collapses, folding into the thrumming bass. The result is a menacing cocktail, charged with gestural potential and momentum. Meanwhile, a siren, a cultural unit of immense semiotic value and impetus, blares throughout, combining with the on-screen timer and the musical features that complement this sense of dread.

Example 4.

The ‘spinning’ motif. Musical gesture graph from Super Metroid, highlighting the gestural potential of musical shapings.

Example 4. Long description

The graph is organized into three horizontal layers along the vertical axis.

* Top Layer (Siren): Labeled as a cultural unit above texture. It is represented by a long horizontal arrow with a crossed circle at the start and diagonal slashes along its length. Blue curved lines expand outward from this path. A text box on the right describes the siren as an essential driver of gestural potential and doom-laden momentum.

* Middle Layer (Melodic Line): Labeled as reduced in the latter part of the phrase. It begins with a musical staff in 6/8 time followed by a series of triangular shapes with diagonal slashes. A text box describes these as spikier melodic shapes marked by heaviness and swirling triplet figures. A large black arrow curves downward from this layer toward the bassline. A central text box explains that the melodic line collapses into the bass, causing a surge of downward energy and a collapse of the upper texture.

* Bottom Layer (Bassline): Labeled as the dominant line. It starts with a bass clef staff and a series of oval shapes that rise diagonally. A text box notes threatening rising bass in diminished and chromatic shapes indicating a sense of dread. This transitions into a series of rectangular blocks with downward arrows, labeled as weighted bass with chromatic inflections.

* Overall Synthesis: A large white arrow points toward the upper right, cutting through the layers. A text box at the bottom center summarizes the fast tempo, feeling of dread, and semiotic implications of the gestural potential. Blue lines converge toward the bottom right, where a text box notes all gestural energy surging with forward and downward momentum.

The shapings of these contours and their stark differences from the Super Mario World example are important to note. Firstly, the cultural unit of the siren is marked by its position at the top of the graph. The sound cuts across the audiovisual land- and soundscape and is the principal driver of momentum and charged gesture in this section of the game. The player hears the signal, derives meaning from it, and notes its charge. The result is a player experience very different from Super Mario World — there is a driver to play that is charged with negative semiotic implications. Triangular shapes also dominate the main melodic line, which is no longer depicted with arrows denoting buoyant gestures but rather downward-facing triangular contours, highlighting the melody’s spiky, urgent nature. The rhythmic symbol of the line marked with dashes is retained from the earlier graph to show rapid rhythmic momentum, in this case denoting the triplet spiralling figure. Notably, at the centre of the graph, a pervasive downward arrow cuts through the depiction, from melody to bass. Here the melodic line’s collapse can be clearly seen through musical gesture, and the result is twofold. Firstly, the melodic gesture is lost after this, meaning that the impatient 5/8 bassline percussively permeates the texture. Secondly, the gestural momentum shifts: we move away from a spiralling melody, perhaps denoting the collapse of the facility, to the chromatic bassline, its sinister shaping a sign of impending disaster.

A final gestural depiction worth expanding upon is the bassline itself. Marked again by box textures to denote its tightly contained pitch set, locked into the chromaticism it performs, the gesture is marked by two types of arrows, one filled, the other marked by an absence of colour. The white arrow has greater weight, while the smaller black arrows have spikier, though lighter, gravitas and gestural pull. The arcing line through the white-arrow gesture highlights this, and the result is a bassline that seems violent and anticipatory, serving to further player fear and anxiety in this level of the game.

A graphical analysis model depicting the musical gestures at play, building on initial concepts founded in musical gesture theory and musical analysis, can demonstrate how we might remap the listener and player as the corporeal vessel responding to audiovisual stimuli. Graphs such as these reconsider musical embodiment to account for the player/avatar virtual realized self and to highlight how they might interpret video-game sound. No two analyses should be the same, as players in this model can be seen as individual interpreters of musical meaning as well as co-authors of the audiovisual game text. By considering the interactivity of the video-game medium and highlighting the playfulness of its music, the graphical analysis depicted above allows us to understand the musical gestures that make up the whole, not only serving to show how music moves but how players, in turn, are moved by the audiovisual artwork they themselves co-author.

Conclusions: The Gestural Potential of Music and Ludomusical Cocooning

Charting a path through both modes of musical analysis and phenomenological concepts relating to ludomusicology, dance study, and everyday human gestures more widely has highlighted this article’s main area of research, as well as its unique fusion of interdisciplinary studies. Dance studies, examples from visual art, and their tangible, tactile links to human gesture are crucial to understanding musical gesture. Using Hatten’s theory of musical gesture in combination with the aforementioned research, we can begin to remap our ideas of music and its gestural potential to better consider music situated within an interactive audiovisual frame. This extrapolation of musical signs and semiotic significations allows us, through the new framework, to understand why we play the games we play and how we play them. What informs our play? What drives us to move in the virtual space? How do we connect to and feel embodied by our virtual avatar representations on-screen? Fundamentally, then, why do we move within games and how can we be moved by them?

Music clearly plays a role in such functions, as has been shown above, not only in the reading of the sound- and landscape of the game environment, but in addition, contributing to players’ feelings of embodiment and their understanding of the game, its characters, and the virtual world. Musical gestures have the potential, through harbouring their semiotic and functional capacity, to quantify, connote, and reveal emotional signposts, narratological devices, tropes, and thematic constructs for the player. Depending on the player’s own sense of comprehension and in some cases prior knowledge, the result is a game experience that is incorporative, in Calleja’s sense of the word. Game music and its gestures thus have the potential to captivate the player, ludomusically cocooning them, resulting in a deeply engaging and rich ludic experience.

To further summarize this article’s premises, it is important to note that this is not ‘Mickey-Mousing’ at play, with musical gestures equating to a synchronization of sound and on-screen action. Rather, gestures can be read, interpreted, and understood uniquely by the player, creating a deeper, more meaningful gameplay experience. Simple musical gestures can be understood to inform players about the game’s make-up as well as the actions they might take when performing in the virtual space. Linked to this, the mode of analysis posited here enables discussion around and consideration of the diversity of individual listeners and their biographical individualities, and their personal interpretations and understandings of musical gestures and sound in games. The theoretical framework and its gestural graphical depictions are specifically designed with adaptability and uniqueness of expression in mind; they accommodate the individual yet are still specific and systematic — as every mode of analysis is, since analysis is always an interpretive act.

Ludomusical cocooning has been revealed to be a potent form of incorporation and envelopment, leading to player involvement and the ability of games to blur the boundary between concrete reality and virtual self-expression. I would posit finally that such gestural interpretations need not be limited to analysis of video-game music. Rather, wherever musical gesture can be seen to operate as enveloping the user in the aesthetic of the sound world such music conjures, musical gesture analysis, presented by this remapped theory, may be best placed to reveal individual interpretations, providing evidence of music’s pull, both metaphysically and metaphorically, on listeners and their emotions. Gestural analysis could be readily applied both in and outside audiovisual contexts, and given the current time, perhaps understanding the affective qualities of music on individual and collective listenership across multimedia boundaries is particularly apt. Furthermore, musical gesture analysis has the possibility for expressing individuality and also, conversely, collective experience, as we engage with game music just as well in a visual-kinetic manner as in an audio one. Thus, sitting in the light of my admittedly more modern television screen, a mere push of a button is all it takes to begin my gesturally charged interactive journey into a virtual video-game world filled with music, play, and adventure.