2.1. The joint multi-instrument: facilitating technology for new musical expression

The basic performance system consists of a computer, an audio interface and different interfaces for controlling real-time processing of acoustic drums and electric guitar. In its basic form we can compare the system to a digital live mixer with built in effects, where one musician live processes input sources in real time using various controllers. The instrument does not produce sound or playback on its own and is therefore dependent on input signal from the performers. The choice of processing techniques spans from extensive use of different convolution techniques, granular synthesis, pitch manipulations, delays, reverbs and overdrives to name but a few.

The system can in many ways be compared to a tradition within electroacoustic music, especially related to a mixed music scenario, where acoustic instruments are captured by microphones and manipulated by different processing techniques in real time. Similarities in this comparison are that the processing system itself is pre-composed and the system can be activated or not to change the timbre or to accompany the acoustic instruments during performance. One of the main differences is that both performers in this situation are detached from the direct sound of their instrument during performance. This has consequences on many levels and will be further discussed later in this article.

Figure 1 shows the system design in its most basic form when both musicians are at the same physical location. The musical interaction is based on improvisation, and there is a series of effects that can be activated and manipulated both on single instruments and on the total during performance. This means that the instrument is not put in a static setting but affords several combinations and manipulations of effects that can be activated or not during an improvisation. This undetermined structure affords a large degree of flexibility and variation, and in addition, the instrument is always in development to meet new musical needs. Examples of live performances showing the interplay with the joint multi-instrument can be seen at a presentation Magnify the Sound gave at a conference arranged by Kulturrådet (Norwegian Cultural Council)Footnote ⁵ and a concert at the International Conference on Live Interfaces (ICLI).Footnote ⁶ To look for further artistic achievements and new possibilities of artistic expression, we have extended our instrument by integrating the network as a crucial part. Thereby, the network has become a critical material of our artistic work.

Figure 1.

Joint multi-instrument. Continuous lines being audio signals and dotted lines control messages.

2.2. The CN-instrument

Our CN-instrument came to life by merging the two described practices that were existing within NTNU: the NMP competence and framework and the joint multi-instrument of Magnify the Sound. There were two distinct reasons for creating this new instrument. The fundamental artistic motivation was to make the network an integrated part of our joint multi-instrument to explore new forms of music-making and to seek new artistic practice and expressivity. Second, the musicians were faced with an uninvited practical need of a networked instrument because of the physical separation between people caused by the COVID-19 pandemic, which struck Norway in March 2020.

Since the CN-instrument grew out from our already established artistic practices, it became natural that these perspectives gave the premises for the instrument’s system design. First, because the musical interplay is not based on pre-composed material or fixed synchronisation points, the need for a tight response time between the performers was of high importance. Second, the response time between the acoustic input and the joint multi-instrument needed to be kept as low as possible to support the gestural interaction and perceptual connection between the musical intentions and the sounding results. These artistic conditions led to the technical premises for constructing the CN-instrument: it should have low latency, a large flexibility concerning signal routing, and listening conditions that detached the performers from the direct sound of their acoustic instruments. The development of the CN-instrument was done in two steps.

The first step was to recreate a traditional studio session for recording an album and at the same time expand the instrument to include a concert performance in an NMP context.

As shown in Figure 2, the system for recording and live performance involved a third performer, an audio engineer, performing live production behind a home router by remotely controlling signal routing and mixing through the NMP studio at NTNU. The drummer (musician 1) was physically placed in the NMP studio listening through headphones. Musician 2 (guitar, live processing) was running the joint multi-instrument behind a home router. This set-up was used for two recording sessions and a concert at the NowNet Arts Online Performance Series 2020 on 2 May.Footnote ⁷

Figure 2.

Enabling for recording and live performance. Continuous lines being audio signals and dotted lines control messages.

The second step of developing the instrument involved the inclusion of live visuals by inviting a video artist as a fourth performer.

As shown in Figure 3, musician 1, musician 2 and the audio engineer have the same functionality and level of control through the CN-instrument as in Figure 2. In addition, the video artist remotely controls the live visuals software running inside the NMP studio. The remotely controlled live visuals software receives live video from musician 1 and musician 2, and these live video feeds are mixed with pre-recorded videos and then synchronised with the sound. This system was used when performing two concerts late in 2020: DokkhusetFootnote ⁸ and NowNet Arts Conference.Footnote ⁹

Figure 3.

Enabling for live audio/video performance. Continuous lines being audio signals, medium dotted lines being video signals and small dotted lines control messages.

4.1. A networked interactive music system

Our CN-instrument is an interactive music system (IMS) and is as such part of a tradition within electroacoustic music that has lasted several decades. A very interesting presentation and outline of the development of the IMS is given by Drummond (Reference Drummond2009). He points out that the term ‘interactive’ is used with quite different meanings within the field of music and new media arts. According to Rowe’s definition in his book Interactive Music Systems (Rowe Reference Rowe1993), a characteristic feature of an IMS is a computer music system ‘listening to’, and responding to, a performer, whereas Chadabe, referring to his own interactive work, proposes the term ‘interactive composing’ and underlines that the musical product from these interactive composing instruments is a result of a shared control of both the performer and the interactive system (Chadabe Reference Chadabe1997). As Drummond (Reference Drummond2009) comments, interactive systems are often classified related to different degrees of participation/inclusion of performer, system and audience, and there has also been a focus on describing interactive systems in terms of digital musical instruments, where mappings between inputs and outputs (e.g., gestural input and sonification) are typical constituents (cf. Miranda and Wanderley Reference Miranda and Wanderley2006). Summing up, Drummond concludes:

A fundamental consequence of our interaction with the CN-instrument is, by definition and construction of the instrument itself, that this interaction takes place over the internet. What then characterises our networked performances? Generally speaking, an NMP is commonly said to establish an interaction of three spaces:

1. 1. A local stage (where each individual participant is physically placed).

2. 2. A remote location (referring to the position of the others).

3. 3. A mediating, in-between space (where the networked activity is happening).

Figure 5 illustrates a typical NMP set-up for Magnify the Sound. The two musicians are separated by physical/social distance from each other, as well as from the audio engineer, the video artist, the network engineer and the audience. They are all connected through the mediating space.

Figure 5.

Illustration of an NMP set-up for Magnify the Sound.

Interaction with the CN-instrument happens at neither of the performers’/participants’ local stages but takes place in the mediating space. When the participants in the artistic interplay illustrated in Figure 5 are playing on their own personal/individual instrument, or interacting with controllers situated at their local stages, they are all at the same time performing on, and interacting with, a common networked interactive system. This has dramatic consequences for the performance of the different participants and affects the way the performers experience their artistic contribution.

4.2. Fragmenting and reshaping the acoustic and electroacoustic premises

In Figure 5 we show the connection between the participants placed at different local stages. All local stages have different acoustic environments in terms of room size, acoustic treatments and so on. They also have different technical frameworks ranging from number of microphones used and microphone placements, to listening conditions and network quality. The drummer performs from a medium-large studio acoustically treated for sound recording. The drums are captured through multiple microphones, mixed, summed and sent back to the performer through headphones. The choice of listening through headphones in this situation is important for two different reasons: first, for separating the microphones and drums from other sources in the room to avoid bleeding between microphones; and second, to detach the performer from the direct sound of his acoustic drums. From the CN-instrument perspective this affords the possibility of having separate control over the acoustic drum production and at the same time controlling which parts of the drums are sent as input for live processing. Further, the lack of bleeding from other sounds in the room prevents live processing of unwanted sources on the same channels as the drums during performance. From this performer’s perspective, the detachment from the acoustic instrument brings the auditive and performative focus closer to the CN-instrument. The drummer’s headphones mix is identical to an audience perspective and corresponds with what is sent out from the CN-instruments main output.

The electric guitar and live processing are performed from a home studio connected to the CN-instrument via the public internet. Neither the guitar nor the joint multi-instrument uses microphones placed at this location and therefore avoids challenges related to separation and bleed through microphones. Since the acoustic sound of the electric guitar is inconsiderable in this context, it is also easier to detach the performer from his instrument in order to get closer to the CN-instrument. The performer listens to two separate speaker systems in order to create the illusion of playing with the drummer in the same physical room. One system represents the acoustic sounds of the drums and the other represents the combination of processed drums and guitar. The joint multi-instrument receives input from the acoustic drums and the guitar and then transmits the processed results on separate channels back to both one of the speaker systems and the CN-instrument.

The audio engineer monitors and controls all in and outputs to and from the local stages, and mixes all channels before sending to the main output of the CN-instrument. The video artist listens to the CN-instruments main output while monitoring and controlling all video feeds from the local stages before mixing them and feeding them to the CN-instruments main visual output. In addition, there is a dedicated communication system with sound and video for keeping fluent communication between musical activities. This system is running outside the CN-instrument and the content is in this case represented identical at all local stages (cf. Figure 6).

Figure 6.

Illustration of the listening and communication system.

As seen in Figure 6 the homogenous acoustic and technical environments normally experienced at a concert stage or in a recording studio are changed. The traditional practice of balancing the performance inside the same physical room is shifted towards balancing several physical rooms into one virtual room which in this case represent the totality of the performance. When performing in a traditional context of playing amplified music together at the same physical stage, the possibility for separation between sound sources, bleeding between microphones and controlled listening environment for individual performers are difficult to obtain, and in most cases impossible. At the same time these same issues are further reinforced through the processing system since bleeding between separate incoming live inputs are further reinforced when adding different processing techniques. During the described activities, all participants communicated through two separate systems: one system dedicated for musical performance (sound and video) where the main musical and visual output also reach the audience, and one dedicated for communication between the performance activities (sound and video).

In any musical interplay there is a potential danger when the focus of the performer is pointed more towards the instrument itself than on the musical output. Balancing this dual focus has been a challenge throughout the whole work. Fortunately, the time span between building, learning and interacting with the CN-instrument has gradually decreased during the process, and as a result led us to focus more on instrumentality and new directions within our artistic practice. In other words, putting time into rehearsing and performing through this system has moved us closer to our common instrument, resulting in a more intuitive and direct interaction in performance situations. In addition, expanding the ensemble with an audio engineer and a video artist has extended the functionality of the CN-instrument and at the same time changed the interaction between the musicians and between the musicians and the instrument. The musicians no longer only react to the sound but also to the output from the video artist and vice versa. This interdependence between musicians, audio engineer, video artist and instrument leads to a mutual responsibility for the interaction in the performance situation. By taking these elements into account, we could argue that the network itself has become an inherent part of enabling interaction within the ensemble. Instead of preparing a conventional concert in a room with physical placement on the stage, we are separated and can arrange the local stages into a new flexible virtual stage. Furthermore, the feeling of a closer connection to the sound of the CN-instrument has helped to shift our focus more towards instrumentality than on the instrument itself. Because of this, and even though we have been physically separated, the sense of being present at the same ‘virtual’ stage has gradually increased during the process. So, how do these changes affect our experience of performance and the perception of our roles in the musical interplay?

4.3. New experience of performance: a phenomenological and (or) somatic transformation

When the drummer in our NMP setting is hitting a drum, the audio output is not necessarily an ordinary ‘boom’ drum sound, but rather a sound created because of the guitar player processing the sounds of the drum set. This is something the drummer must consider when playing. The drummer’s instrument is in this situation not just the drums but is also the integration of the drums and our common interaction with the CN-instrument. This establishes a totally new musical situation as compared with performing on the acoustic drum set.

In the ordinary, usual way of performing music, each musician interacts with her or his instrument in a communicative interplay with the other musicians, and this has been made possible due to years of practice through which the instrument has become a prolongation of the musician’s body. This is also pointed out by Alperson, when he argues: ‘Musical instruments are not objects divorced from performers’ bodies, so much so that, in some cases, it is difficult to know where the body ends and where the instrument begins’ (Alperson Reference Alperson2008: 46). Furthermore, he states: ‘The performer’s musical instrument is better understood as an amalgam of material object, the performer’s body, and bodily disposition as habituated by the developments of various musically related skills’ (ibid.). However, in musical interaction with our presented joint, CN-instrument, this interlocking relationship between musician and personal, individual instrument is radically changed. The musician’s interaction with own instrument has consequences for, or is manipulated by, the sound or action generated by another musician, in unilateral or bilateral ways. Thus, the musician loses control of the sonic output of her or his instrument, and the unified body of musician and instrument is chopped up. This may, indeed, be very frustrating: your usual, habitual and embodied interaction with your instrument may now not give the musical results that you want and expect. Such a feeling of losing control in music performance may often happen when performing with an IMS. As touched upon earlier, Drummond (Reference Drummond2009: 124) points out that a different notion of instrument control is presented by interactive systems from that usually associated with acoustic instrument performance and refers to the concept of shared control as more appropriate for many of these settings.

In Magnify the Sound we not only have a common responsibility for the musical development but also have a shared responsibility for the sound itself. In particular this is the case for the output of the processed drum set sounds. Based on this basic premise for our musical interaction, we have experienced that in our performances within various contexts with our CN-instrument there is a new and very exiting potential of musical expression related to a change of our perception of our performance and the way we interact with our instruments. Instead of I am performing on my own instrument, we shift to we are performing on our common networked instrument. In this way the somatic experience is not my instrument being a prolongation of my body, but rather an extended instrument, consisting of the totality of our musical/technological set-up, being a prolongation of an enlarged body of all participating musicians. Seen as such, the network establishes a new kind of embodiment in our musical interaction, and we perceive a phenomenological shift from individual instrumentality to shared instrumentality, which is ‘involving an assemblage of multiple instruments and combining the intentionalities of more than one performer’ (cf. Peters Reference Peters2017: 75). This phenomenological and (or) somatic transformation also challenges our experience of proximity: to our own instrument, to our fellow musicians and to our audience. To exemplify: When the drummer is interacting with the CN-instrument, he is, so to speak, moving away from his drum chair and into the mediating space, thereby losing proximity to his local stage, and gaining larger proximity with the remote locations. A similar phenomenological transformation is experienced by the other participants. Thus, the fundamental musical and interactive constituents of our performance and performative actions take place in the mediating, in-between space, with ‘equal’ proximity to each participant’s local stage and remote locations. It is also in this mediating space we meet our audience.

Although we argue that artistic proximity and our meeting with the audience happens in the mediating, in-between space, we are physically and socially separated from each other and from the audience in our NMP. This does make a difference in our perception of the communication, with our fellow musicians/artists as well as with the audience, as compared to a situation where we are all together in the same physical room. This has phenomenological as well as somatic aspects. Interesting studies of effects of physical separation on the subjective experience of musicians have been undertaken by Iorwerth and Knox (Reference Iorwerth and Knox2019). They conducted empirical investigations of three pairs of classical musicians that were recorded in acoustically isolated spaces with audio and video links. The themes found to be most challenging for these separated musicians were musical issues (e.g., tuning and blending, and reduced risk-taking), communication and social interactions (ibid.: 297). In the various performances with our CN-instrument we can, to different extents, relate to the findings of Iorwerth and Knox. One important difference between the musicians in the Iorwerth and Knox investigation and Magnify the Sound is that the former are classical musicians whereas we, the participants of Magnify the Sound, have background from various genres within rock and jazz. We have years of experience with studio recording, and the musical situation of physical separation is not unfamiliar to us, in contrast to what is the case for the classical musicians. We did not feel tuning and blending as problematic themes, and contrary to the classical musicians we often experienced an increased risk-taking in our performance due to a feeling of being safe and not ‘disturbed’ by sensing the physical presence of other musicians –or the audience.

One very challenging issue within NMP is the less degree of visual communication. In a networked performance with Magnify the Sound we do see each other, but the transmission of video has a latency that excludes the possibility of performing synchronised physical gestures among the musicians based on visual cues. Moreover, we lose more intimate nuances of facial expression and body language in the communication of musical intention and interaction. We, thereby, have to rely (even more!) on listening and auditive information in our musical interplay. These aspects of NMP are also discussed and studied by Mills and Beilharz in a situation of a networked improvisation where the participants do not see each other, and they argue that the sound artists and musicians ‘must rely on listening and the semiotics of sound to mediate their interaction and the resulting collaboration’ (Mills and Beilharz Reference Mills and Beilharz2012: 16). It is interesting to note that Iorwerth and Knox (Reference Iorwerth and Knox2019: 289) also report that the musicians in their study rarely used the video link while playing. Moreover, the minimal use of the video link when playing is stated as a common reaction to NMP by Cáceres and Hamilton (Reference Cáceres and Hamilton2008). However, whereas we used visual communication to a very little extent during performance, the visual communication was important when we were rehearsing and discussing musical choices and strategies between recording-takes (see Video Examples 1–4 in section 3.1). A similar observation is also commented by Iorwerth and Knox (Reference Iorwerth and Knox2019: 294) where they remark that even though the video link was rarely used when playing, it was more often used between takes.

A very exciting somatic and phenomenological experience is also related to a combination of our shared control of the CN-instrument and the reduced degree of visual communication: sometimes when playing we have a feeling of losing ourselves, we are not able to identify our own individual contribution to the music, and we hear musical results that we could not predict. Moreover, we can, at times, be very surprised when we in retrospect listen to a musical session we have performed and experience that we hear or perceive the presence of a ‘third musician’, a contribution to our musical expression that neither of us, individually, are able to identify with. This ‘guest’ into our musical interplay lives in the mediating space and emerges out of our shared, common interaction with the CN-instrument. This experience is intimately related with the fact that we in our performances are real-time composing/improvising with our interactive CN-instrument. Already in 1997 Chadabe writes of early examples of interactive instruments: ‘and when the music is interactively composed while it is being performed, distinctions fade between instrument and music, composer and performer. The instrument is the music. The composer is the performer’ (Chadabe Reference Chadabe1997: 291). Following up, Drummond points out that this statement focuses on the shared creative aspect of IMS, and he states: ‘The musical output is created as a direct result of this shared interaction, the results of which are often surprising and not predicted’ (Drummond Reference Drummond2009: 125).

4.4. The network as an artistic material for music-making

In its nature, NMP establishes a physical distance and separation in the relationship between the participants in a performance situation. However, if we look at the traditional acoustic environments and technical frameworks found in most concert venues and performance stages compared to the potentials of constructing a virtual stage combined of different local stages, the proximity to the sounding and visual result can in many ways be perceived closer in the virtual concert venue than in the traditional concert hall. Also, this separation is often a goal in many studio and live recording situations. As we have discussed, physical separation between performers and between performers and the audience affects both musical and interpersonal communications in many ways, but there is also a large degree of artistic potential both concerning sound and visual appearance within this gap. In that respect, trying to recreate already existing practices will not be as fruitful as establishing new practices since it is impossible to overcome the aspects of latency and physical separation within NMP contexts. In our experience there is no doubt that NMP opens for several creative possibilities concerning acoustic environments and controllable sound separation:

• Instead of imagining one physical room there is a potential to blend different rooms into one. ‘Rooms’ in this context are not limited to the acoustic environments on each local stage, but any additional room where the sound, or parts of the sound, can be projected into a room and retransmitted through microphones.

• The listening conditions can, as demonstrated, be individualised, and not exclusively reproduce the same result to the different participants.

• As also touched upon, the separation of real-time video feed from the performers opens a variety of different possibilities of visual representation that would be challenging, and in most cases, impossible in a performance situation where the participants are sharing the same physical stage.

Up to this point we have focused mostly on new aspects related to physical separation. We will now discuss how material properties of the network influenced our performative choices of musical parameters within a context of tele-improvisation.

4.5. Tele-improvisation, internet jam and a perception of sound as affordances

As mentioned earlier (section 3.2), after a concert with Magnify the Sound at the NowNet Arts Online Performance Series 2020, we performed two internet jams with members of the NowNet Arts Lab Ensemble. The participants in these jams were musicians and a visual artist from Canada, Belgium and New York and us (Magnify the Sound), who were as usual situated at different locations in Trondheim, performing on guitar and drums feeding into the CN-instrument. We had never met or played with these artists before. It is important to note that these ‘external’ participants in the internet jam had heard our concert preceding the jam, so they were familiar with our way of performing and our musical aesthetics. Moreover, since they were all members of the NowNet Arts Lab Ensemble, it is fair to say that they to a large extent shared our musical and cultural references. This is a crucial point to be made because Roger Mills has conducted interesting studies of how cross-cultural musicians experience tele-collaborative engagement in an online jam session, and he concludes that culture plays a vital role in structuring experiences of distributed interaction (Mills Reference Mills2019: 193–4).

The two jams were based on free improvisation. Whereas auditory delay has not been problematic for performances with Magnify the Sound when all participants were at different locations in Trondheim, latency was, indeed, a challenging issue in these two tele-improvisations. Because we had not played together before and latency was problematic for coordinated actions, both jams started with a large degree of horizontal contribution to the musical interplay; in other words, we played ‘after one another’ in a kind of call–response manner, inviting each other into the musical conversation. This is a different approach from the suggestions outlined by Wilson (Reference Wilson2020) in which she references composed music when presenting strategies to enable vertical harmony despite latency in NMP. The approach in our internet jams is also very different from how we (Magnify the Sound) perform networked music together from separate locations in Trondheim. This is due to the fact that we, as a band, have rehearsed together and known each other through years of musical cooperation, and because latency is not an issue in our more local networked performances. Other characteristic features that emerged in our tele-distributed interaction were our focus on sound quality/ texture/ timbre (e.g., bell-like, metallic, industrial, river-like), sound duration, density of events, motivic work and energy/dynamics. When rhythmical elements were introduced – for example, when the drummer was performing a groove/quasi-groove – the other participants often responded by playing something on the top of (or under) the drummer’s contribution that complemented what the drummer was doing or performed a similar rhythmic expression that was not musically dependent on synchronisation; for instance, when the visual artist made rapid-changing facial expressions in a rhythmical interplay with the drummer’s quasi-groove.

It is very interesting to compare our subjective experiences and observations of these two internet jam performances with Mills’s empirical case studies of distributed interaction and tele-improvisation. One of his key findings is that networked performers perceived qualities of sound as affordances in their interaction:

Moreover, Mills writes: ‘articulatory parameters in sound acted as affordances for directing performers through different stages … of an online jam session’ (ibid.), and he presents a table demonstrating different articulatory sound parameters and stages of tele-improvisation (ibid.: 110–11). –In this table we read for instance that Mills discovered that sequentiality (call and response), durational tones, legato articulation, soft to moderate volume, and imitation are articulatory sound parameters that act as affordances in the initiation stage of the tele-improvisation, whereas sequentiality moving to simultaneity, increase in volume, instruments taking figure-like aural perspective act as affordances in the development stage, and homophonic musical texture and repetition of rhythmic patterns are articulatory sound parameters that establish additive musical texture, and build density and tension in the progression stage (ibid.: 110). The application of these sound parameters in tele-improvisation and distributed interaction resonates well with the subjective experiences and observations of our internet jams. Seen as such, our perception of sound as affordances supports Mills’s empirical findings.

It should at this point be noted that only the musicians in Magnify the Sound interacted with the CN-instrument in the discussed performances previously. Our reflections in this section therefore highlights interesting aesthetics related to the material properties of a network in tele-improvisation and network jamming, not the interaction with the CN-instrument in itself.