2. THE ROBOT SOUNDSCAPE

To get a high-level overview of the medium of sound in HRI, one may ask: (1) what type of sound occurs across HRI scenarios, and (2) what the creators of said sounds aim to communicate. A possible map of the robot soundscape, focusing on the different types of robot sound, is shown in Figure 1.

Figure 1.

A top-level view of the robot soundscape.

Sound Uttered by Robots is the dominant form of robot sound. It comprises regular speech and various types of non-verbal utterances. The latter include categories such as Musical Utterances or Gibberish Speech and have been thoroughly surveyed by Yilmazyildiz and colleagues (Yilmazyildiz et al. Reference Yilmazyildiz, Read, Belpeame and Verhelst2016). Sound Associated with Robot Movement includes the often unintentional sonic byproduct of motors and joints and movement sonification, artificial sound intentionally added to accompany robot movement. A comprehensive review of both can be found in prior work by the authors (Robinson, Velonaki and Bown Reference Robinson, Velonaki and Bown2021). Sound Performed by Robots comprises robots using various instruments to create music, on their own or in collaboration with humans. A comprehensive survey of robot musicians was created by Bretan and Weinberg (Reference Bretan and Weinberg2016). As we move beyond these three categories, applications become rarer and less associated with HRI research. Sound as Background to HRI covers applications such as shared listening (Hoffman and Vanunu Reference Hoffman and Vanunu2013), ambient infrasound (Thiessen, Rea, Garcha, Cheng and Young Reference Thiessen, Rea, Garcha, Cheng and Young2019), or subtle haptic vibrations (Komatsu, Kobayashi, Yamada, Funakoshi and Nakano Reference Komatsu, Kobayashi, Yamada, Funakoshi and Nakano2018). Sound Responsive to Human Actions uses sensor data about the robot’s environment to provide immediate auditive feedback to the listener. Schwenk and Arras, for example, continuously sonify the distance between robot and human using sine-like tones (Schwenk and Arras Reference Schwenk and Arras2014). In the robotic artwork Echidna, created by artists Woolf and Bech, visitor gestures and proximity are translated into a range of sounds emitted by the fist-sized robot.

The purpose of these various types of sound are diverse. Among others, they are meant to convey artificial emotions, engage the listener and communicate internal robot states. They also aim to direct the attention of the listener, and guide and manage expectations. It is common to achieve these aims by imitating, referencing or creating associations to other living things, fictional characters or technological artefacts. We describe this strategy as creating a robot fiction, explicitly or implicitly conveying a set of characteristics to be perceived by the human. Well-known social robot Kismet communicates with humans through child-like babbling vocalisations (Breazeal Reference Breazeal2000). Social robot Vector sounds like an animated character taken directly out of a Pixar movie, and faithfully recreates the outgoing personality we are accustomed to from these types of characters on the big screen. Cha and Mataric use simple sine tones with different pitches to communicate urgency on the iRobot Ava, approaching the conventions of user interface sound (Cha and Mataric Reference Cha and Mataric2016). The underlying reference could be described as a large smart device on wheels.

We argue, however, that highly refined versions of all the preceding references already exist, be it film characters, smart devices or humans themselves, and that all of them are better adapted to their specific environment, and better equipped to meet our expectations. Social robots are not fictional characters, smart devices or humans, and this referencing thereby restricts what associations we can create through sound, and how we can manage the expectations that come with them. We therefore set out to draw from a different set of references and take a new perspective on robot sound, breaking with existing listener expectations and searching for ways to communicate that may be uniquely applicable for this very particular application area. The approach we take – viewing robot sound design as electroacoustic composition – attempts to harness the music’s ‘bewildering sonic array ranging from the real to the surreal and beyond’ (Smalley Reference Smalley1997: 107) and use it to create richer and more refined sonic HRIs. These ideas, and a demonstration of them, are presented in the following sections.

3. CRAFTING THE VOICE OF AN INTERACTIVE ROBOTIC ARTWORK

Robotic artwork Diamandini (Figure 2) was created between 2011 and 2013 by the third author. She is approximately 1.5m tall, slender and her exterior resembles aspects of classical sculpture, combined with futuristic undertones (Silvera-Tawil, Velonaki and Rye Reference Silvera-Tawil, Velonaki and Rye2015). Via an omni-directional mobile base, she proceeds through her environment in a floating fashion, gently moving between the people she encounters. Diamandini uses a wirelessly connected overhead cameras to track her location as well as those of the people around her. Through markers on the robot’s head, the remote cameras can track its gaze direction, allowing it to face or turn away from people. Additional proximity sensors built into the robot near floor-level provide a high-level resolution in its immediate environment. With these sensor capabilities in place, the robot does not follow pre-programmed motion paths or sequences, but instead uses a basic intelligence to autonomously respond to changes in its environment, dynamically choosing people to either approach or avoid.

Figure 2.

The robotic artwork Diamandini explores the sharing of physical space between robotic agents and humans.

Diamandini was built to explore how people interact spatially around a robotic agent that moves among them. A gallery setting was used to gather data on these movements outside the traditional laboratory settings in a public social environment. This spatial data could then be used to gather behavioural data such as time spent or proxemics and infer comfort level and interest, among others. These explorations were facilitated by the robot’s ability to communicate through movement. She could approach and move away from people in various ways, leaving the audience to guess her intentions and underlying characteristics. So far, all this happened silently, and we pondered how the addition of sound and music may expand and enrich the robot’s communication; how it could shape how humans perceive the robot’s awareness, presence, fiction, intent and affect, among others. Therefore, Diamandini now serves as a platform for our ongoing exploratory work on creating a sonic language for robotic agents. While the term ‘language’ usually refers to words exchanged between humans, we will use it in this article to describe the pallet of sounds an agent emits with the intention to communicate. The language of Diamandini is treated as a piece of electroacoustic music, played through a non-standard multichannel speaker distribution. The timeline of the piece is non-linear, driven by people and events in the environment. It can be seen as an adaptive score to a narrative emerging during interactions. This score is distributed across eight channels spread across the robot’s exterior and interior.

Video Example 1 illustrates what encounters with the robot sound like. The excerpt shows a stereo reduction of the robot sound output as it is approached by a human: As we enter the space, the robot awakes in the other corner of the room. After watching her idling in the distance for a while, we approach her and are greeted with a sudden absence of sound elements, accentuated by what resembles a vocalisation, signifying awareness of this new interactant. We subsequently circle the robot, while it idles. As we remain close to the robot for a longer period of time, its sound becomes more lively. We then leave and the robot sounds decays, before the interaction is ended with a final utterance. We encourage the reader to view and listen to the media examples in the supplementary materials before continuing, as they will be referenced throughout the following sections.

4. SOURCE MATERIAL AND ROBOT FICTION

If we treat the sound emitted by a robot as an electroacoustic composition, we may start by choosing a source material and then derive a language from it by shaping the material into gestures, phrases and larger temporal structures. This source material might be recordings of a physical object or of a family of related materials.

Consider, for example, the approach found in Jonty Harrison’s composition Internal Combustion (Harrison Reference Harrison2007). Here, various car engines form the core material, and throughout the piece are introduced, developed and disassembled. Later sections introduce additional sounds of the built environment, referencing broader themes of technological process and environmental concern. Another example is David Berezan’s Buoy, which paints a hyperreal picture of a realistic setting: a buoy with a bell floating in the ocean (Berezan Reference Berezan2013). Over the course of the piece, this scene is developed and processed into an increasingly unnatural soundscape, bordering on the surreal, and making distant references to notions of loneliness associated with the original setting.

Applying these approaches to robot sound has a range of implications. As mentioned earlier, it avoids the expectations and constraints that come with referencing established robotic sound sources and, instead, gives us access to a significantly wider pallet of sound materials – Smalley’s ‘bewildering sonic array’. We may also think in terms of Kulezic-Wilson’s description of non-musical sound in film. In her words, soundscapes can be crafted and shaped ‘with so much attention to their rhythmic and musical properties that they themselves become music: non-sentimental, non-domineering, non-illustrative, but strikingly effective and visceral’ (Kulezic-Wilson Reference Kulezic-Wilson2008: 130). Instead of picking familiar sources and adjusting them to an application context, we may choose arbitrary sources, and grow them into a language that is custom-made for what needs to be communicated.

Another implication of this approach is that it allows us to harness what Smalley calls source bonding: the human tendency to associate sounds with imagined causes and sources (Smalley Reference Smalley1997). With the dominance of utterance-based robot sound, the most obvious source association is that of an imagined vocal tract. However, as we emit sound from a physical, moving object – far removed from the presentation via loudspeakers that would encourage reduced listening (Schaeffer Reference Schaeffer1966) – source bonding of any sound and the robot body are possible. This allows us to invent the internal workings of the machine, to invent which materials are involved, and what processes these materials go through, whether in a realistic manner, or, more interestingly, simply in a convincing one. In Emmerson’s words, we are ‘challenging the listener to re-create, if not an apparent cause, then at least an associated image to “accompany” the music’ (Emmerson Reference Emmerson and Emmerson1986: 18). While this targeted pursuit of referential listening may appear un-acousmatic to some, O’Callaghan notes how many composers do not see referential listening and acousmatic music as in conflict with each other and identifies a symbiosis of these two in some of Smalley’s works, comprising both ‘real-world elements [in] abstract sound worlds’ and ‘abstract sounds … shaped according to real-world ecological syntaxes’ (O’Callaghan Reference O’Callaghan2011: 61).

One might ask, what are the benefits of freely drawing from an almost limitless pool of source material, deriving a language from it, and putting it in relation to the body emitting it? Why would we want to create meaningful connections between robot sound and the physical robot structure? One goal is the creation of a believable object, using sound to create a convincing fiction that is self-contained, without referencing other things. Possible – and in our view, misguided – fictions can include a jukebox on wheels, a moving smart speaker, a human made of metal, or an animated character made of plastic.

A second, equally relevant goal is to be deliberate in the choice of associations we aim to evoke. These associations might be states and behaviours, such as reconfigure, depressurise, calculate, strain, learn, shift and charge. Or, they might be characteristics, such as digital/analogue, precise, powerful, old, soft, organic, tense and smooth. It is not difficult to imagine scenarios where communicating these behaviours and characteristics benefits the interaction. They represent relevant aspects of communication about interaction context and the robot itself. At the same time, the language of electroacoustic music has countless ways to convey these notions without relying on established references such as auditory icons (Gaver Reference Gaver1986), or human prosody. An even more direct link can be made between robot gesture sonification (Robinson et al. Reference Robinson, Velonaki and Bown2021) and the motion vocabulary found in Smalley’s spectromorphology, including push/drag, flow, rise, drift and float (Smalley Reference Smalley1997: 117). One may, for example, consider relationships between the perceived energy trajectories of a robotic gesture with Blackburn’s expressive visual interpretations of Smalley’s vocabulary (Blackburn Reference Blackburn2011).

Populating the robot’s language with musical gestures that aim to create these specific associations can also be viewed through the lens of Wishart’s notion of landscape, which he defines as a sound’s or group of sounds’ imagined source (Wishart Reference Wishart and Emmerson1986). He gives the example of a forest soundscape, whose individual sound sources – wind, wildlife and vegetation – are gradually replaced by arbitrary sonic objects. At some point, the ‘disposition of the objects remains realistic (in the sense that we retain the image of the acoustic space of a “forest”) yet the sound sources are not real in any sense of the word. Here we have the first example of an imaginary landscape of the type “unreal objects/real space”’ (Wishart Reference Wishart and Emmerson1986: 48). Taking this perspective on robot sound, one could say we populate the real space of the robot – a language emitted from a physical object – with objects that convey the various states, behaviours and characteristics mentioned previously. These objects may be considered real if they are made of, for example, stone recordings with little processing applied to them. Yet they may also be considered unreal objects if they comprise heavily processed material that barely resembles its underlying sources.

To illustrate, let us consider the core fiction and chosen source material of Diamandini. The key characteristics defining the robot’s fiction are its marble-like exterior, and its fluid traversal facilitated by its mobile base. This combination of rigid materials and flowing motion led us to define Diamandini as a solid structure which resonates in response to external forces. The chosen source material was stone, both in the form of stone instruments excited through wind, friction or impacts and in the form of recordings undergoing more natural physical processes such as falling, chipping or being dragged. A selection of source materials is shown in Figure 3.

Figure 3.

Deriving Diamandini’s language from various stone-based sound sources and instruments.

Many of the musical gestures are associated with Diamandini’s motion trajectories, and therefore play with the acceleration and deceleration of heavy, rigid structures. Phrases are then determined by the robot starting, continuing and ending movement. An example of such a trajectory can be heard in the supplementary materials in Sound Example 1. Musical gestures also occur in other contexts. Simple interactions such as a greeting require deliberate utterances from the robot and are therefore independent of motion. This introduces a notion of agency which is reflected in the sound material as a more obvious use of audio processing, that places less emphasis on representing believable physical processes. Examples of utterances juxtaposing the naturally occurring energy–motion trajectory of falling pebbles with more deliberate cuts and loops can be heard in the supplementary materials in Sound Example 1. Finally, musical gestures take the form of sonifications of events and processes in the environment, which is the focus of the next section.

5. ANIMACY AND CONTEXT-AWARENESS

As the name suggests, HRI is inherently interactive. This interaction, however, is mostly a turn-based back and forth, due to the nature of dialogue as well as technical constraints – robots need time to understand what we tell them and unstructured exchanges between humans and machines remain a challenge. A significantly less common form of interactivity is one focused on immediate experiential qualities, creating a tight feedback loop between the robot, the space it inhabits and the people around it, and conveying this feedback loop through sound. The goal is then to make sound dependent on external factors and have the impact of these factors clearly reflected in the sonic behaviours emitted in a way that is noticeable, but not overly literal. While the collection of sensor data in this context is trivial, the challenge lies in creating meaningful relationships between this data and sound. Did the environment, the robot, or the human cause of this sound? Is the relationship convincing and believable? How well does the sonic feedback integrate into the overall language of the robot? Answers to these questions depend heavily on associations created by the listener, which encourages the use of a sonic vocabulary that can effectively create these associations.

The language of electroacoustic music appears well suited for this. In Smalley’s words, ‘the invisible freedom of spectromorphological content and motion creates a [wide and variable] pool of extrinsic, behavioural references’ (Smalley Reference Smalley1997: 118). These references can include energy–motion trajectories learned from natural physical processes in the environment, or archetypal behaviours found in living things. What makes them valuable in this context is that they are intuitively understood by the listener and that electroacoustic music provides a vocabulary to describe them. Drawing from Smalley’s vocabulary of behavioural relationships, for example, we might consider a sound that is dominated by a robot’s current movement state, while at same time being loosely coordinated with a human’s presence and pressured due to obstacles in the environment. Conveying this using the sonic array provided by human utterances, for example, would be challenging to say the least and interpretations would likely be based on the implicit information contained in human vocalisations. The language of electroacoustic music, on the other hand, provides a much richer vocabulary and pool of references to describe these behaviours, which in turn widens the array of possible creative interpretations. From a technical perspective, it also provides an established toolkit of synthesis and processing methods to realise these behaviours. We therefore argue that harnessing this pool of behavioural relationships to effectively convey the ways robot sound is interlinked with the environment and the listener is a worthwhile endeavour.

The motivation for this is twofold. One goal is to communicate notions of context-awareness and animacy. By having the robot respond to human actions, humans are informed that their presence is perceived and acknowledged, which provides a continuing and subtle response to the question ‘Does this robot know I am here?’ Conveying animacy is a particular concern. We are aiming to create meaningful social interactions between humans and what are essentially large, moving computers. In that context, the illusion of life is a highly relevant perceptual parameter. Envisioning a future for interactions between artificial agents and humans, Rinaldo describes a ‘cybernetic ballet of experience, with the computer/machine and viewer/participant involved in a grand dance of one sensing and responding to the other’ (Rinaldo Reference Rinaldo1998: 375). This ballet can, and should, go far beyond turn-based dialogue.

The second motivation is to maximise variation, to create a language consistent enough to be coherent and recognisable but featuring enough complexity that it never shows its true nature: a synthesis algorithm or a collection of audio assets. A promising way to achieve this is to tap into variety present outside of the machine and use it to inform internal parameters. An illustrative example of this can be found in the robotic artwork Echidna (Woolf and Bech Reference Woolf and Bech2002). The robot is made up of a handful of wires, connected and placed on a podium. While making up the robot’s body, the wires also function as a sensor that picks up changes in the surrounding electromagnetic field. Movement of bystanders (proximity and gesture) is picked up by the antennae and sonified, essentially creating utterances that are closely linked to changes in the robot’s environment. The noisiness of the sensor creates a broad pallet of sound, which, in Woolf and Bech’s words, ‘reflects not the sophistication of the underlying electronics, but the complexity of the environment in which the sculpture is situated’ (Woolf and Bech Reference Woolf and Bech2002: 32). Another example of this approach can be found in the works of sound artist Peter Vogel, which could be described as responsive robotic artworks. In his piece ‘Sound Wall’, a large sculpture of openly visible analogue circuits, light sensors detect movements from bystanders and translate them into rich soundscapes, realised using fully analogue means (Vogel Reference Vogel1979). As a result, the piece comprises both the sculpture itself and its reaction to the presence of the visitor.

In light of these considerations, we approach Diamandini’s interactive sound as follows. Encounters with her involve no functional word-based dialogue. Communication therefore happens through location, and through non-verbal sound, which is determined by several robot-external parameters and their interplay. These parameters include the number of humans present, the frequency of human contact, the presence of humans in the field of view, potential obstructions in planned motion trajectories and the distance to individual humans (see Figure 4).

Figure 4.

Diamandini’s language is informed by various external real-time parameters.

Human Presence, for example, is sonified both continuously and once the listener crosses a specified threshold. From a purely functional perspective, the distant robot emits high-frequency material to convey its presence in a way that is easy to localise, while reducing low-frequency content to minimise the leakage that results from the reduced directivity in that range. When being approached, the robot slowly grabs more and more attention through the addition of new sound material, before swiftly withdrawing to provide a quiet sonic backdrop to the encounter. The actual sound material shifts from high-pitched granulated debris into vibrating stone resonances, before retreating into subdued static. This sequence of transitions – accompanying slowly approaching the robot from a distance of 7m to 1m – is shown in Video Example 1. The Engagement parameter is used on a broader time scale. It lets Diamandini distinguish between being engaged with and being ignored and respond to both states accordingly. This takes the form of ‘idle chatter’, short musical phrases occasionally emitted during sections of low engagement and absent during interactions. Engagement also more continuously affects the phrases by having them sound increasingly rough and machinic as the robot is ignored for extended periods of time, thereby conveying the idea of the human ‘breathing life’ into the robot through interacting with it.

6. USING SPACE ACROSS THE ROBOT AS A CREATIVE DIMENSION

With sound material and interactive behaviour discussed, we now turn our attention to the rendering format. Space has long been a creative dimension in electroacoustic music, be it embedded into a multichannel tape piece, or diffused in real time across an acousmonium. And, since the movement of sound through space has received considerable attention from composers for decades, it appears obvious to try to benefit from this wealth of creative potential. While the dominant listening situation is ‘people in a room surrounded by speakers’, a robotic structure can certainly be seen as a space in itself, which can be filled with sound, and across which sounds can move. In doing so, we may not be able to expect Envelopment. Yet, we can certainly expect detailed spatial trajectories across the robot body, with a similar level of control over spatial resolution that a loudspeaker orchestra provides. Therefore, we may benefit from making the active decision to treat a robotic structure as a space in itself – rather than an object, a point source – and explore how this new perspective shapes the creation process and outcome.

This treatment of robot body as a space has two motivations. First, it expands the range of creative opportunities by adding a spatial dimension to a communication channel that is usually treated as a point source. This spatial dimension can introduce many of the notions described in Smalley’s spatiomorphology (Smalley Reference Smalley1997). Even on this smaller scale – multiple speakers across a sculpture in front of the listener – concepts such as orientation, breadth/depth, intimacy/distancing, focus, and contiguous and non-contiguous space all still apply and promise a wealth of creative options. Putting these in relation to robot motion and gesture yields additional interesting juxtapositions. Second, it provides us with a significant tool to work towards creating a more believable fiction. One could say, it allows us to reproduce a space more convincingly – space as in Wishart’s notion of space within an imaginary landscape. This space might be a faithful reproduction of a physical material. In a literal sense, it could be a resonator getting excited by forces applied internally or externally, such as a metal body ringing from elaborate mechanical processes taking place inside it. Perhaps more interestingly, it may be a fictional resonator with the sound characteristics of any imaginary instrument, or simply a space that confines the movements of more autonomous sound sources within it, like a bird cage.

Diamandini emits sound through a non-standard 8-channel distribution of small drivers positioned across her structure, and larger speakers integrated inside. After exploring a wide range of distributions in a prototyping environment, we applied two different configurations to the robot, as shown in Figure 5. Space is used in two main ways. First, in line with the source material, it conveys the characteristics of a human-sized stone instrument. To the listener, much of the sound Diamandini emits is caused by excitation through air friction at the robot’s mobile base, which then causes the entire structure to vibrate. These associations are a result of the electroacoustic sound material, as the robot’s motors create almost no sound on their own. The second use of space involves explicit human ‘input’, like a touch, sending ripples across the sculpture, moving outwards from the initial touch point. Making this response more or less inharmonic provides a way to encourage or discourage further interactions.

Figure 5.

Left: Using drivers on a magnetic whiteboard to experiment with different distributions (Robinson Reference Robinson2020). Centre and right: Diamandini with different loudspeaker positioning.

Broadening our definition of space, we can also draw a link to Smalley’s concept of source-bonded space – which he describes as not only a spatial zone, but also a mental image built around the notion that ‘space carries with it an image of the activity that produces it’ (Smalley Reference Smalley2007: 56). When considering the use of sound to convey information that relates to the space a robot body inhabits, we should not only examine the physical space we create by placing sounds at different locations, but also the conceptual space inherent to the sound material we use. In Smalley’s words, while a listener will ‘intuitively pick up various cues of position in space … these cues are not the prime space-bearers. It is the behaviour of the [sounds] themselves that transmits the main spatial information’; sound emitters ‘produce space through their action’ (Smalley Reference Smalley2007: 38). Looking at the previously mentioned bird cage as an example, the space – cage, in this case – can be conveyed by the rattling of metal bars, but it is also implied by the behaviour of the birds inside it.

Smalley provides an extensive taxonomy of spaces, and, in the context of this article, an obvious link can be drawn to mechanised space, source-bonded space which is created by machines that emit sound. He mentions that a key characteristic of these machines is that, although they may often be controlled by humans, they can to some degree emit sound independently of us and thereby produce their own space. His examples include sound sources across transport, work and manufacturing, and robotic agents fit well within this categorisation. There is, however, another space applicable in this context: utterance space. According to Smalley, this space produced by vocal sound can be intimate, personal or social. He clearly differentiates sound created by machines and sound created by humans. However, a key characteristic of social robots is that humans are drawn to interpret the sound they emit – their utterances – as social cues. Read and Belpaeme, for example, show that people tend to interpret robot sound across a small number of categorical prototypical emotions (Read and Belpaeme Reference Read and Belpaeme2016). In other words, we hear a sound emitted by a robot and immediately wonder what emotion the robot is aiming to convey. This creates an interesting intersection of Smalley’s utterance space and mechanised space, indicating that the latter may very well be intimate, personal or social, even if it is not created by humans. Two relevant questions in the context of robot sound and source-bonded space are therefore: What conceptual spaces do we want the robot to create and what sources do we want to be perceived as creating this space?

7. OBSERVATIONS ON THE CREATIVE PROCESS

We conclude this article with some observations on the creation process of Diamandini’s language. Overall, sound editing and processing was done in a digital audio workstation (Pro Tools), which was either plugged into studio monitors or directly into the robot’s audio system. Sound was then exported in layers and segments and implemented into a game audio middleware (Wwise) to be mapped to the various interaction scenarios. Finally, sound and mappings were exported to run on a small audio server (Latte Panda). This server is integrated into Diamandini and connected to the robot’s control system and the various loudspeakers. This workflow is generally similar to the creation and implementation of video game sound. Hence, we were able to benefit from tools created for that context. Where it differs, however, is the site-specific nature of sound reproduction, the physical robot.

An overview of the creation process is shown in Figure 6. Authoring environments are at the top, sound reproduction systems are at the bottom. The two elements on the right, the creation of mappings and playback through the physical robot, are where this process differs from traditional electroacoustic music practice. Throughout the creation process, jumping between these four contexts – Pro Tools/Studio, Pro Tools/Robot, Wwise/Studio, Wwise/Robot – allowed us to mitigate the blind spots of each. After a while, it also became easier to anticipate the effects creative choices in one context would have in another. We would like to highlight the interplay between the two authoring environments, Pro Tools and Wwise, and the two sound reproduction systems, the studio and the physical robot.

Figure 6.

An overview of the creation process of Diamandini’s language. Creative Decisions are made in the authoring environments above, and then (re-)evaluated on the sound reproduction systems below as back and forth.

7.1. Interplay between authoring environments

The first observation concerns the interactive context’s effect on the composition process. Fundamentally, material is created in layers and segments, which are then integrated into the robot’s audio engine and recombined, faded in and out, layered and processed in real time (see Figure 7, right). A very similar process can be found in the creation of sound effects and music for interactive systems such as video games or media installations. Here, practitioners first create material (e.g., audio files containing footsteps, birds, vegetation, wind, rain and music) and subsequently integrate it into the system, where elements need to be tuned to work in harmony. However, the difference for this robot context is that there is no ‘ground truth’ that needs to be accounted for. The need for a faithful reproduction of real and familiar sound is replaced by a pursuit of aesthetic considerations and a convincing presence. So, if certain elements work against each other, one can just be left out. More interestingly, if the recombination or randomisation of arbitrary elements or mappings results in more interesting sonic results, such creative choices are as valid as any other. In the context of a video game, this could mean removing footstep sounds because they clash with the vegetation, and instead, having each footstep trigger the next 500ms of the musical score. This might not have been planned during asset creation, but it may be interesting.

Figure 7.

Right: After being created in a traditional studio environment, sound is adjusted for Diamandini’s speaker distribution. Left: Sound is broken down into layers and segments to be mapped to robot behaviour and environment in game audio engine Wwise.

In the case of Diamandini, that meant, for example, taking all robot sound output, re-pitching it, applying randomised amplitude modulation and sending the output through a resonator whose size is determined by robot movement speed. The result is a soundscape comprising muffled, hollow, choppy, temporally smeared remnants of the familiar robot output. This might not have been a planned part of the composition, in fact, it might have been difficult to imagine or time-consuming to recreate. Instead, it was discovered during the implementation phase during interaction with the robot. In this way, the rather pragmatic process of asset implementation turns out to be just as creative an endeavour as the composing itself, enabling the discovery of sonic possibilities that where hidden during work in the digital audio workstation. Moving back and forth between these two contexts – creation and implementation – would thereby enable a continuous development and reinterpretation of existing material. Using the language of Verstijnen and colleagues’ research into creative discovery, the implementation could also be seen as a sketching process, which supports the restructuring of previously created sound material (Verstijnen, van Leeuwen, Goldschmidt, Hamel and Hennessey Reference Verstijnen, van Leeuwen, Goldschmidt, Hamel and Hennessey1998). To illustrate this implementation process, Video Example 2 shows the sound of Diamandini simulated in the game audio engine Wwise. Input events are shown on the left, while the playback and processing of different audio elements are shown in a schematic on the right. The simulated scenario is as follows. The robot first awakes, before starting a motion trajectory. After the motion is completed, the robot notices a human in the environment and starts moving once more. During this second movement section, the robot starts to run out of power, which affects the sound output in the way described previously.

It is worth noting that this recombination or randomisation of arbitrary sound elements or mappings could also take place in real-time during interactions with the robot – something which we did not implement in Diamandini. This inclusion of procedural elements could create variety to break up potentially repetitive sound elements and thereby facilitate more engaging long-term interactions. While we consider this an intriguing prospect to keep a robot’s sonic profile interesting, it may make it difficult to maintain a clear and coherent sonic identity for the robot. Creating sound that continuously reinvents itself, while at the same time conveying a core robot fiction is an interesting challenge and may be facilitated by recent forays into procedural robot sound using machine-learning techniques such as recent work by Savery and colleagues (Savery, Zahray and Weinberg Reference Savery, Zahray and Weinberg2020).

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit https://doi.org/10.1017/S1355771822000358