2.1. Digital multi-user instruments and network performance

The interactive dance system in the telematic framework of instance (detailed in section 3.2) provides an interface for the performers to play a multi-user digital musical instrument (DMI) in the telematic performance space, drawing from the tradition of DMI design in networked performance. The histories of DMIs and networked performance are inextricably linked (Jordà Reference Jordà2005). John Cage’s Imaginary Landscapes No. 4 (1951) for 24 performers on 12 radios (Cage Reference Cage1960) was one of the first networked music performances (Muller, Burger, Giovanni and Ziegler Reference Muller, Burger, Giovanni and Ziegler2019) and an example of an electronic multi-user instrument (Jordà Reference Jordà2005). Bringing Cage’s concepts into the digital realm, the League of Automatic Composers performed with the idea that their network was both one multi-user instrument and an ensemble of individual instruments (Bischoff, Gold and Horton Reference Bischoff, Gold and Horton1978). Similarly, instance incorporates a network interface and the sonification of multiple performer inputs, existing as both a networked performance and a multi-user instrument.

2.2. Multi-user instruments and multimodal interaction paradigms

Existing in the telematic performance space, the DMI in instance embraces perceptual shift and is conceptually and experientially similar to acoustic paradigms in the African performing arts. For example, the interaction between musicians playing from opposite sides on the same xylophone, or the interaction between a dancer and drummer in moment-to-movement improvisation. These interactions illustrate the concepts of flow and agency in relation to the felt-experience of the artists.

Interlock is a central concept in many musical practices in Africa (Agawu Reference Agawu2014), such as the mbira practices of Mozambique and Zimbabwe. Mbira features prominently in instance, as detailed in sections 4 and 5. In traditional mbira practice, the player composes a quick, cyclical pattern in constant pulsation. An underlying melody weaves through the pattern, interlocking between the notes played with the left and right hands. Skilled mbira players improvise or compose variations on the original pattern, to bring out different elements of the melody (Berliner Reference Berliner2020). A composition can be played for hours at a time. Because of the cyclical nature, those who are listening or playing along could experience the melody in any number of ways, depending on their felt perception of its starting point.

Perceptual shift, also referred to as simultaneous multidimensionality (Locke Reference Locke2009), is evident in many African music practices. For example, perceptual shift resulting from interlock in a multi-user instrument context is evident in the Kiganda amadinda xylophone practice. One musician starts to play a melody in regular pulsation. A second player sits on the opposite side of the instrument and joins with another melody, in constant pulse, falling halfway between the first player’s pulses in time. The combination of the two interlocking parts forms a melody. A third player joins in the upper register of the same instrument, doubling this resulting melody (Kubik Reference Kubik1969). The first two musicians experience the beat in exactly opposite temporal locations rather than as syncopation, like one might in Western-based music practices. This embodied experience determines their perception of each individual component of the music and the performers’ awareness of the combined sonic output. Playing a single instrument, the felt and sonic experience of performers are contributions to a shared aesthetic output, and there is no single absolute way to experience the music in time.

In a similar manner, the interplay of drummer and dancer in the Baganda Baakisimba practice expresses a dynamic give and take where both performers lead and follow, cueing each other in both deliberate and unintended synchronicities and juxtapositions. As much as the musician, the dancer defines and shapes the aesthetic moment through an improvisatory performance practice (Nannyonga-Tamusuza Reference Nannyonga-Tamusuza2015).

The elements of perceptual shift emerge from the organic principle of interlock, the dancer’s significant agency over a shared sonic output, and improvisation towards unintended synchronicities and juxtapositions. These elements comprise an accurate model for the multi-user DMDI and music-dance interactions in instance. We incorporated these embodied concepts into our composition, rehearsals and performance, using soma-based design strategies to explore the temporal separation (Chafe, Cáceres and Gurevich Reference Chafe, Cáceres and Gurevich2010) caused by network latency.

2.3. Digital music-dance instruments

The Digital Dance and Music Instrument (DDMI) is a conceptual tool for the design framework of a digital music instrument (DMI) played through dance (Tragtenberg, Calegario, Cabral and Ramalho Reference Tragtenberg, Calegario, Cabral and Ramalho2019). We use the term ‘digital music-dance instrument’ instead because our approach has roots in music–dance practice. As a combination of a DMI and an interactive music system (Tatar and Pasquier Reference Tatar and Pasquier2019), the concept of the DMDI enables the designer to integrate both an instrumental relationship with music and a bodily expressiveness with equal importance into the design. Though not explicitly designed as a DMDI, another example of a multi-user music–dance instrument is the ‘body-machine instrument’ in Vrengt (Erdem, Schiand Jensenius 2019), where the musician alters the sonification of the dancer’s movement. Hereby, the performers are focused on both movement and sound, playing the system together as a single instrument. In instance, the multimodality of music–dance performance fosters bi-directional sonic agency between musician and dancer through a multi-user telematic instrument. Similarly to the sonic agency of the dancer in Baakisimba improvisation practice (detailed in section 2.2), the dancer in instance plays the DMDI, moving with the intention of altering the shared sonic output.

3.1. Design motivations and specifications

Our design motivations are influenced by the limitations of social-distancing during the COVID-19 pandemic. The dancer had to rehearse and perform from home in complete isolation under the South African national lockdown restrictions, with no in-person technical assistance. Hence, we aimed for a technical framework with a simple technology set-up on the dancer’s side, running all complex technologies on the violist’s side.

Beyond the framework of an audiovideo stream of a networked performance, we were curious to explore multimodal interaction possibilities of music–dance performance in a telematic framework and to express the philosophical qualities of telematic performance through sound and movement through the design of DMDIs.

The performers’ experience of flow was an important consideration in our design process. We centred the DMDI designs around resemblance to familiar performance instruments to minimise the impact of the technical constellation on flow, deciding that neither performer should physically touch the interface beyond their instrument of expression to play the DMDIs. Rather, synthesis parameters are controlled through mapping of body-tracking data and viola audio features, towards a representation of familiar instruments in the multi-user DMDI controls.

3.2. Telematic framework

We first needed a technical framework that satisfied the design specification of simple technology on the dancer’s side, to facilitate our iterative design processes of DMDIs. Conceptually, the telematic framework is designed in such a way that neither performer touches the interface beyond their own body or acoustic musical instrument, bringing to light the intangible and ephemeral qualities of music within the space and time of the telematic performance environment.

Next, we explored a signal processing framework to be implemented on the musician’s laptop. We used Max MSP for body-tracking and audio signal data processing, and utilised TouchDesigner for routing video data over NDI (Figure 1). We used a screen-capture of the Zoom call to input the dancer’s video stream to TouchDesigner (pipeline 1 in Figure 1), then routed this video stream to Max MSP using NDI (pipeline 2). A Max patch hosted an implementation of PoseNet – the n4m-posenet (Yogo Reference Yogo2021) Max external that connects to the PoseNet browser-based demo using Node for Max. Using the dancer’s video stream, PoseNet tracked the dancer’s body (Figure 2), which is one of two inputs of the multi-user DMDI.

Figure 1.

Telematic framework of instance.

Figure 2.

Body tracking using PoseNet.

Viola audio signal is the second input. The signal processing of viola audio is a real-time audio feature extraction using the Max MuBu (Schnell, Röbel, Schwarz, Peeters and Borghesi Reference Schnell, Röbel, Schwarz, Peeters and Borghesi2009) and PiPo externals (Schnell, Schwarz, Larralde and Borghesi Reference Schnell, Schwarz, Larralde and Borghesi2017). The implementation is available as open-source (Tatar Reference Tatar2021). Our feature set includes bow velocity and acceleration, pitch detection and loudness. We used the fzero∼ Max object to detect pitch and the loudness calculation is in digital dB scale. Although we tried the YIN algorithm (de Cheveigné and Kawahara Reference de Cheveigné and Kawahara2002) using PiPo to detect the pitch, we observed that fzero∼ was more consistent for the low pitch register of the viola. The bow velocity calculation is a simplification of the technique presented by Pardue, Buys, Edinger, Overholt and McPherson (Reference Pardue, Buys, Edinger, Overholt and McPherson2019). We first take the absolute of the viola signal, then apply a sample by sample integration with a window size of two samples. Finally, we add a lowpass filter with 79.2Hz cutoff frequency. From this result, we calculate the acceleration from the first order differential of the velocity signal. Mapping these features to the synthesis parameters creates a pipeline for sonification of viola gestures within the multi-user DMDI.

We routed a mix of viola sound and synthesised audio from Max into the Zoom call with Soundflower (pipeline 3 in Figure 1) and selected the ‘use original audio’ feature on Zoom. We streamed the performance to YouTube with a Zoom Pro account (pipeline 4).

4.1. Space and time in compositions for telematic performance

Müller et al. (2019) discuss fragmentation of the time–space continuum in telematic performance, through digitalisation, reflecting on the continuous and the discrete, and the reordering of events. Telematic performance is deconstructed from the dimensions of reality in the continuous medium of spacetime where the performers exist, then reconstructed in another physical location from discrete data. A symbolic juxtaposition can be drawn between the acoustic instruments that provide the sonic materials in instance. The viola can play continuously variable pitches and is evocative of the analogue. The mbira creates only discreet pitches and evokes the digital.

In his philosophical writings on time and reality, Rovelli (Reference Rovelli2019) proposes that we assume time flows continuously because we are unable to distinguish the smallest temporal intervals as a result of our inability to escape subjective reality – the limitations of the human interface with space–time. Inspired by Rovelli’s proposition, we centred our artistic exploration of the temporal limitations of the telematic performance space within the human body itself as our most direct interface with space–time. During this exploration (detailed in section 4.2), we further explored how this proposition relates to the telematic interface: since temporal continuity has only been assumed, it is possible that a digital telematic interface captures and re-orders events between discrete dimensions, rather than fragmenting a continuous reality through digitalisation. Almost 100 years ago, Schoenberg first justified atonality with the ‘emancipation of the dissonance’ (Schoenberg Reference Schoenberg1952). Telematic performance enables the emancipation of performance from the order of time, offering creators of telematic performances an opportunity to perform with subjective temporality.

Performers’ embodied experience of time is important in the study of performance excellence and musical expressivity, even outside the telematic medium. Mental imagery can allow performers to ‘feel the musical process’ (Stachó Reference Stachó2018), towards an embodied performance practice, while a distortion in the perceived experience of the passing of time is an indicator of flow (Araújo and Hein Reference Araújo and Hein2019). Similar to the perceptual shift in African music practices, a performer’s ability to shift their attention forward and backward in time on any time scale through a mental navigation process is associated with performance excellence (Stachó Reference Stachó2018). Deliberate focus on mental navigation during performance is often associated with visio-spatio and gestural imageries (ibid.). For this reason, we started the iterative design process for instance from the body, using mental imagery to explore movement qualities the telematic performance space might evoke.

In network and telematic performance research, the internet is an acoustic and musical medium (Chafe Reference Chafe2009) and a performance space (Lemmon Reference Lemmon2019). One way that performers can adjust their qualities of expression to better suit the telematic performance space is to anticipate larger rather than smaller intervals of time. In a network music study focusing on latency, researchers observed that musicians in different locations could play together with greater ease when they focused on the shape of phrases, compared to when they focused on smaller divisions of temporal alignment (Chafe Reference Chafe2009). Although playing with longer musical phrases is also a strategy for musicality in Western classical music performance, telematic performance can bring new aesthetic possibilities of a shift of performance focus from smaller to larger temporal intervals.

The speed of light is faster than the speed of sound and visual information precedes auditory information in the natural world. Yet in network performance, this order is reversed (Chafe Reference Chafe2009). We hear the effect of a performer’s action before we see the action. The relevance of cause and effect is explored further in digital artworks such as the Weblogmusic platform (Morris n.d.), a web-based venue for time-shifted improvising ensembles. In instance, we further explore the agency of performers in the telematic medium and express the disconnect between intention and effect with an African improvisation model towards aesthetic synchronicities and juxtapositions.

4.2. Iterative design and composition process

The telematic framework facilitates a data pipeline, inline with the specification of low-technical requirements on the dancer’s side. Building from the foundation of the telematic framework, the iterative design process that we clarify in this section focuses on the sonic exploration of the music–dance interaction in the telematic DMDIs in instance. Our design interest lies in the embodied experience of both performers. Accordingly, we use soma-design strategies (Höök et al. Reference Höök, Caramiaux, Erkut, Forlizzi, Hajinejad and Haller2018) within the RtD model (Zimmerman, Forlizzi and Evenson Reference Zimmerman, Forlizzi and Evenson2007).

4.2.3. Iterations towards DMDIs

In the iterative design process for the instance DMDIs, we first used a combination of bodystorming and physical movement sketching activities under the embodied sketching umbrella (Höök et al. Reference Höök, Caramiaux, Erkut, Forlizzi, Hajinejad and Haller2018) to elicit a collection of movement qualities that the telematic performance space could evoke, with reference to the conceptualisations in section 4.1. Overall, this iteration produced three distinct movement qualities. Granularity evoked swift, punctuated movements in quick succession, in contrast to the light and breath-like characteristics of the other two movement qualities, suggestive of the ephemerality of the telematic performance space: the directionality of time brought continuous, floating qualities to mind; and the metamorphic process of fragmentation and transformation through digitalisation produced oscillatory, breathing qualities of dilation and contraction.

In the previous section, we elaborate on the association between gestural and visio-spatial mental imagery and the embodied experience of temporal navigation for musicians during performance. In the iterative process, the use of gestural mental imagery bridged an organic connection between conceptualisations on time in the telematic medium and sonic explorations. Inspired by our collection of movement qualities, we first constructed acoustic sound vocabularies of viola and mbira (Figure 3) to lay a foundation for the subsequent electroacoustics. We refined the design of the DMDIs in viola improvisation sessions, then in music–dance improvisation sessions.

Figure 3.

The mbira sampled in instance was made by master mbira player and maker Edgar Bera (Giminez 2018).

To establish an acoustic sound vocabulary for the viola, the violist translated gestural imagery to musical affects by recording several short sonic sketches in a fluid, improvisatory process grounded in the embodied movement qualities described previously. The gestural imagery relating to granularity inspired pizzicato motifs and short, non-pitched sounds, such as tapping and bowing on the tailpiece and body of the instrument. When applied to arco techniques, this imagery evoked abrupt changes in bow acceleration. The continuous, floating gestural imageries elicited sustained lines of harmonics played with consistent bow speeds and slow, cyclical patterns. Inspired by the imagery of dilation and contraction, the violist explored the extremities of viola pitch range using overpressure to create subharmonics and playing on and behind the bridge to produce squeaks and scratches of varying lengths in the extreme high register. She smoothed the transitions between pitch ranges by varying bow speed, pressure and sound-point with each bow direction change to create oscillatory effects. By improvising, the violist discovered natural groupings of techniques towards a strongly contrasting selection of sketches, which later guided the design of electroacoustic elements in instance.

The dancer – a mbira player himself – composed and recorded a short piece on mbira with a continuous, steady pulse, which comprises variations on an interlocking pattern, each variation highlighting different elements of an underlying melody that weaves between the interlocking parts. These characteristics are intrinsic to many styles of African music and are touched on in section 2.2. The cyclical structure of this composition also alludes to the oscillatory and cyclical breathing qualities, while notes plucked on mbira keys are naturally reminiscent of swift, punctuated movements in quick succession. In addition to the composition inspired by gestural imagery, we sampled each mbira key and captured the resonance of the instrument.

Before exploring the electroacoustic possibilities of the collection of affects, we tested the interactive qualities in improvised acoustic telematic music–dance sessions. The dancer’s contribution as a somatic connoisseur (Höök et al. Reference Höök, Caramiaux, Erkut, Forlizzi, Hajinejad and Haller2018) helped to refine the expression of musical affects through movement, defining and shaping the sonic aesthetics of instance. The dancer’s agency over sonic material is an African concept, detailed in section 2.2. Since the sessions centred on dynamic qualities in relation to expressed meaning, this iteration is a choreography of interaction activity (Klooster and Overbeeke Reference Klooster and Overbeeke2005) under the Embodied Sketching umbrella in soma-based design (Höök et al. Reference Höök, Caramiaux, Erkut, Forlizzi, Hajinejad and Haller2018).

Having refined an acoustic vocabulary, we explored audio processing and synthesis techniques to create a prototype collection of three DMDIs, as described in the following. In our initial investigation into the electroacoustic possibilities of the mbira material, we created a digital version of the mbira using the sampled keys. We analysed one of the samples, and used the frequencies and amplitudes of the ten loudest partials from the sample’s spectrogram to re-synthesise the sound through additive synthesis. Drawing from the continuous, floating imagery, we used the synthesised sound as a drone in the composition, transcribing the metallic timbre of mbira into the digital realm, outside its framework of discreet pitch constraints in the natural world. This material is implemented in DMDI 1 (Figure 4).

Figure 4.

Additive synthesiser – Scenario 1.

The first DMDI comprises a synthesised drone sound constructed from the timbre of mbira and a ring modulator in serial connection. The loudness of this additive synthesiser is mapped to the loudness of the viola input. The dancer adds timbral manipulations to the mbira drone sound by controlling two ring modulators for left and right channels where the diagonal expansion of his body controls the modulator frequencies. The sum of left wrist to body distance and right knee to body distance controls the modulator frequency of the ring modulator on the left channel. The mirror image mapping controls the modulator frequency of the ring modulator on the right channel. We mapped modulator strengths of both channels to the dancer’s z-axis. This mapping scheme allows the violist and the dancer to control different parameters of one audio synthesis pipeline, while influencing the drone sound together.

Inspired by the dancer’s swift, punctuated movements in quick succession, the second DMDI takes the form of a granuliser that processes live viola audio with chaotic elements in delay time (Figure 5). When paired with pizzicato motifs and short, non-pitched sounds, the granuliser produced discontinuous, erratic clusters of movement, which is in contrast to thecontinuous texture that was conjured with expansive and cumulative phrases when the granuliser was paired with sustained harmonics (Sound Examples 1 and 2). This DMDI adds polyphonic elements to the composition, in contrast with the drone sound generated by the additive synthesiser in DMDI 1. The dancer controls reverb feedback amounts of both channels independently, using the distances of his left and right wrists to his body, respectively. This allows a different interaction strategy where the dancer controls one dimension of viola audio processing; thus, both performers create a single sound output together.

Figure 5.

Granular synthesiser – Scenario 1.

The expanding and contracting gestural imagery inspired spectral transformations with ring modulation and set in motion an exploration of time-based manipulations on the dancer’s mbira composition recording, which is incorporated in DMDI 3 (Figure 6). This DMDI employs an audio player that is further processed by a ring modulator in serial connection, using the mbira audio material composed by the dancer. Additionally, the bow acceleration feature of viola controls the modulator frequency of the ring modulator. The audio output of this DMDI is demonstrated at 6 minutes 20 seconds into the recording (Video Example 1).

Figure 6.

Sample playback and processing – Scenario 2.

The dancer manipulates playback speed of the sample by expanding and contracting his body. The speed is mapped to positive and negative values to enable playback forward through the sample and in reverse. When the dancer moved in such a way that the sample was extremely slow, he could control each note as an event in time. This mapping strategy enabled an expression of ambiguity in the directional flow of time. The dancer explores the ambiguity of the directional flow of time at 7:07 in Video Example 1.

6.1. Rehearsals

In the preparation, the performers used a combination of soma-based principles, informal discussion sessions and concepts in the African performing arts to play with the idiosyncrasies of the telematic medium.

Rehearsal sessions for instance began when the DMDIs were still being conceptualised, and were intrinsic to both prepared and improvised compositional processes. Early rehearsals took the form of instant-composition play sessions on Zoom, where the performers used soma-based processes to guide the performance interaction, both with and without the DMDIs, identifying peculiarities and challenges of performance practice in the telematic medium. One of the play sessions without the DMDI is documented in Video Example 2.

At first, the performers struggled to find a sense of flow, producing unintentional pauses from the delay of action and reaction due to latency, which was between 100 and 150ms due to Zoom video connection, well above the ‘edge of playability’ for rhythmically synchronised musical duos of 55 to 66 ms described by Chafe, Cáceres and Gurevich (Reference Chafe, Cáceres and Gurevich2010). This was particularly evident when playing with shorter gestures and in moments when they were each unsure if the effect of their actions had been successfully portrayed to the other performer. In addition, the motivations behind their gestures were often constrained by the technology rather than by artistic decisions. As a result, they used less pause than when practising music–dance outside the telematic medium, reflecting that they often produced sound and movement simply to test whether sound and movement could still propagate through the network. The performers experienced both of these peculiarities as indications of a lack of trust, between each other, and in technology too.

To build a foundation for an embodied empathetic interaction towards flow, the performers held informal discussion sessions before and after each rehearsal. Conversation flowed freely between topics related to telematic performance, and topics related to day-to-day life. Without the luxury of playing together in the same space, verbal communication helped the performers to construct contexts around each other. Alongside music–dance rehearsals, they perceived these discussions as a necessary contribution to building rapport, towards the perception of a shared creative experience of flow when playing together.

As the development of gestural and musical affects progressed during the iterative design process (section 4.2.3), the dancer noticed changes in the violist’s telematic improvisation, in comparison to when they had played together in the same physical space. Musical transitions were smoother and more gradual, indicating a shift of focus from smaller temporal divisions to larger ones and from a focus on individual gestures to an overall sonic output. The violist was improvising musical compensations for network latency by altering the way she communicated expression of emotional affect through time, which is in line with the observations of the network music trial between Norway and the United States, mentioned in section 1 (Chafe Reference Chafe2009). The violist’s temporal adjustment also reflects a shift of focus from ‘what is’ to the embodied human experience; as discussed in section 2.2, Rovelli (Reference Rovelli2019) writes that we might experience space–time to flow continuously only because the limitations of the human interface with reality do not allow us to distinguish the smallest temporal intervals.

Though the performers could lengthen their phrasing to adjust to network latency, they still experienced ambiguity of cause and effect in the telematic medium (section 4.1). To explore solutions, the performers improvised with a variety musical qualities (section 3.3.3). Playing in slow tempi with longer notes and smoother gestures, the performers felt they could keep track of action and reaction more easily than when using more pause and playing shorter notes and gestures. However, maintaining a sense of flow became easier when they each focused on their individual contributions to their exchange of musical and gestural quality and character, rather than on any definitive composition of sound and gesture in their shared sonic output. Here, the performers played with African performance principles in mind, embracing perceptual shift and unintended synchronicities (section 2.2) to explore the effect of fragmentation on the order of time in the telematic medium (section 4.1).

Although performers’ focus on a shared sonic output is a defining feature of a multi-user instrument (Jordà Reference Jordà2005), it is equally important that performers embody concepts of perceptual shift within their individual experience, when the multi-user DMDI is used in the telematic medium. During rehearsals, the strategies used to transfer music–dance practice to the telematic medium were simultaneously used to learn to play the DMDIs, as a result of the resemblance of familiar instruments (outlined in section 3.3.2) in the DMDI design.

6.2. Performance

The performance discussion explores the impact of the addition of an audience on the performers’ experienced interaction with the DMDIs, the performance being the resulting artefact from the iterative design process inline with RtD approach (section 3.3).

Towards the design specification of minimal technical set-up for the dancer, the musician’s laptop handles all computational processing. The dancer’s video stream is used for the DMDI controls, for visual feedback to the violist and for the YouTube stream to the audience (Figure 1). This asymmetrical design leads to contrasts in the performance experiences of the two performers, described in the following.

Although the violist played the DMDIs through playing her viola, she was in charge of running the telematic framework and preparing and triggering the compositional scenarios (section 4.2.3) on her laptop. She experienced these brief interactions with her laptop as abrupt breaks in her experience of flow. One such moment occurs at 4 minutes in the video of the public performance (Video Example 1). To play around these distractions during the performance, she used a strategy similar to the change in interest soma-design strategy described by Höök et al. (Reference Höök, Caramiaux, Erkut, Forlizzi, Hajinejad and Haller2018) by turning away from the camera and the video-feed of the dancer to focus her attention inward: first to her own body, taking note of her breathing, then gradually moving outward to her muscles, then to the points of contact between her body and the viola. By drawing her attention inward to her own body, she re-centred herself. From focusing on her embodied experience of the familiar interaction with the viola, she shifted her somatic awareness outward, towards an empathetic connection with the dancer.

When performing Scenario 1, the performers became frustrated that their input to play the multi-user instrument was imbalanced. The mapping design dictated that the violist had to trigger all the electroacoustic material. When she stopped playing, the sonic expression of the dancer’s gestural input disappeared. Although this imbalance could potentially facilitate an artistic exploration of balance in agency, a solution for future performances would be to map one of the dancer’s controls to the loudness of the drone sound, so that his movements can produce sound when the violist is not playing. Although the dancer moved closer and further from the camera in rehearsals, he positioned himself at his maximum possible distance from the camera throughout Scenario 2 in the performance. His decision meant that his whole body was visible to the camera, allowing for precision in his DMDI controls through PoseNet gesture detection (section 3.2), yet it also resulted in a lower ring modulation strength (section 5). An audience member with no prior knowledge of the mapping design would be completely unaware of the DMDI’s full potential for timbral manipulation through ring modulation, including the musician’s ability to affect ring modulator frequency with her bow acceleration and the dancer’s ability to control the DMDI with his distance from the camera. The violist also found herself resisting the urge to demonstrate all the possibilities of each DMDI in a short space of time, using the comparison that when she plays acoustic improvised or Western classical music, she does not play every possible pitch and tone quality on her viola, so perhaps it is not necessary to expose every sonic possibility of a DMDI during performance.

The performers reflected on the multimodality of their gestures. During the performance, gestures served three functions simultaneously: to control the DMDIs, to contribute to the aesthetic performance output for the audience and to communicate with each other. The multiplicity of gestural effect was experientially different for the musician and dancer because of the nature of their different modes of expression. While the dancer’s movements were naturally intended as both performative and communicative through dance, the violist conceptualised her gestures first as necessary for the production of sound and then as performative.

The telematic interface in instance dictated that communicative and performative gestures were by default perceived by the audience as performative gestures, where any small gestures intended as visual cues were equally visible to the audience and had an effect on the shared sonic output. As a result of this design framework, the violist had no more visual or sonic feedback from the dancer than the audience, and experienced a negative impact on her rapport with the dancer as a result of the addition of the audience in the performance setting, compared to her experience in rehearsals.

Audio transferred through a network can naturally exist throughout another space, while movement transferred through video material is bound to the constraints of a screen. In addition to the audio feedback from the dancer’s input to the DMDIs, the musician relied heavily on visual feedback from the dancer’s video material. The dancer relied more on sonic feedback than visual feedback from the musician and could move freely about the space, without the need for a conscious interaction with technology to feel connected with the musician. With a simple technical set-up, the dancer felt that he could focus on his somatic experience in a similar way to the rehearsal context.