1. Introduction
Since the beginning of the 2000s the proliferation of network-based tools and mobile computing devices has allowed musicians and researchers to explore remote communal musical exchanges (Barbosa Reference Barbosa2011). Musicians now regularly perform together using audiovisual connectivity across high-speed networks. Social networking approaches allow remote musicians to interact using various messaging and symbolic data sets. NOMADS advances the field of social music-making by enabling an emergent socio-synthesis between musicians and the audience or between members of any large group of participants. Linking multiple stages through NOMADS allows dislocated groups to become part of the same multimedia synthesiser. The approach works in parallel with telematic audiovideo connectivity. This connectivity amplifies the synergistic possibilities of the communal exchange, enabling a new form of network emergent sound art. Because thousands of participants can join NOMADS simultaneously in real time, the energy of the resulting sound can be rich and complex in new ways. Depending on the application, the system may be used by large groups of trained performers to create network-based performances, or by untrained participants such as members of an audience. The aesthetics of the NOMADS approach to socio-synthesis embraces emergent musical behaviour.
2. Background of NOMADS
NOMADS has its origins in the MICE (Mobile Interactive Computer Ensemble), a human–computer ensemble formed at the University of Virginia in 2001. MICE was designed to embrace paradigms of human–computer interaction and multi-performer systems (Burtner Reference Burtner2006), particularly those pioneered by Stockhausen in the 1960s (Stockhausen Reference Stockhausen1965), the HUB in the 1980s (Bischoff, Gold and Horton Reference Bischoff, Gold and Horton1978) and Sensorband in the 1990s (Bongers Reference Bongers1998). MICE was part of an early twenty-first-century trend exploring the varied possibilities of an ensemble in which humans and computers collaborate towards musical ends. Other formative approaches, contemporary with MICE, included Gil Weinberg's Interconnected Musical Networks (IMN), experiments at the MIT Media Lab (Weinberg Reference Weinberg2003), the PLOrk Princeton Laptop Orchestra by Dan Trueman and Perry Cook at Princeton University beginning in 2005 (Trueman Reference Trueman2007), and Nathan Wolek and Matt Roberts’ Mobile Performance Group (MPG), started in 2004 at Stetson University (http://www.mobileperformancegroup.com).
MICE began as a small ensemble (5–20 members) that incorporated a variety of performance tools including acoustic, electric and computer-controller instruments, laptops, and mobile phones (MICE 2009). In 2008 the group swelled into a 200-person human–computer orchestra requiring the development of more sophisticated server technology and new performance interfaces.
MICEtro, the first system developed for the MICE Orchestra, allowed the collection of data from hundreds of participant-performers in order to create an emergent synthesis instrument. MICEtro connects a video-game interface and a robotic ‘conductor’ via a Java server. Performers in the MICE Orchestra interact through the game interface, and MICEtro keeps track of the player's physical locations and behaviour. All the participant data is merged into a single complex synthesiser by the server. The multi-performer synthesiser is played through speakers on the stage while musical sounds from the game are produced by the individual laptops.
MICEtro monitors the network traffic and looks for particular data patterns. Identifying such a pattern, the server sends the physical location of that performer to the robot, which then pivots and faces that location. The robotic face, with a 64-led-illuminated, motor-driven scissor skin (see figure 1), exhibits different aspects of its character by patterned flashing and by expanding or contracting the LED face.Footnote 1
The MICEtro robotic network controller.
The MICEtro robot is positioned at the front of the stage, in the place of a traditional orchestra conductor, but it is not a conventional robotic conductor such as Sony's QRIO or Honda's ASIMO. Rather, it joins in a network feedback relationship with the orchestra. The MICEtro robot is controlled by the performers and it in turn controls their interfaces. Its directional movement is coupled to a network ‘spotlight’ that projects a virtual parabolic beam onto the physical concert space. The beam acts as a filter for network traffic, highlighting activity by users located within the beam and attenuating that of those outside it. With its movement MICEtro can isolate segments of the concert space. Computers within the virtual spotlight react by lighting up and making sounds. Computers outside the network spotlight remain dark and quiet (although their data will still be used in the socio-synthesisers). A narrowly focused beam can affect a single laptop, and a broad beam can elicit light and sound from a large segment of laptops.Footnote 2
MICEtro was premiered at the 2008 Digitalis under the Stars concert at the University of Virginia's outdoor amphitheatre. From MICEtro we learned that a simple rule-based approach works well with a large group of participants. Especially in an emergent musical texture where individual effects on the system are minimised or lost, an interesting interface helps to engage the performers. A gaming approach works well because it entertains and encourages participation.
3. NOMADS
3.1. Networked social music
NOMADS explores the effect of network topologies on the social aspect of music-making, previously considered by Gil Weinberg and Atau Tanaka. Weinberg's Beatbugs allows groups of participants to collaboratively compose, share and modify rhythmic sequences (Weinberg Reference Weinberg2005). Tanaka's Malleable Mobile Music project also allows for collaborative composition, enabling geographically separated participants to create a real-time ‘social remix’ (Tanaka Reference Tanaka, Nao and Ali2005).
David Kim-Boyle links these approaches to the collaborative, improvisatory, open works of practitioners such as Stockhausen, Wolff and Brown in their effort to ‘facilitate an awareness of process and collective becoming’ (Kim-Boyle Reference Kim-Boyle2009: 372). The aim of NOMADS is to empower large and dispersed groups of people, energising the active data-space around the performance by engaging the audience in a socio-synthesis system in collaboration with other musicians. To do this, NOMADS incorporates some familiar social networking tools as well as new social music tools.
3.2. Nomadic computing
Ubiquitous computing, or nomadic computing, refers to the use of personal computer devices over widely available networks (Weiser Reference Weiser1993 and Yang, Lim, King and Helal Reference Yang, Lim, King and Helal2006). In NOMADS, participants join the performance using their own mobile computing devices (phones, laptops, etc.). This interaction is parsed and analysed, and the results are projected in the concert space as sound, projected images, text, animation and lighting.
3.3. Emergent musical systems
NOMADS provides limited control to a large number of participants rather than giving a small number of performers a great deal of sonic control. The resulting sounds are distributed spatially through individual laptop speakers, and/or mixed through the house sound system. An emergent sonic texture results from large numbers of agents following simple rules. In experiments with the MICE Orchestra, we found that, by increasing the ensemble size, this emergent behaviour takes on a different kind of sonic identity. Work with multi-performer computer interaction, for example PLOrk and SLOrk, has generally been confined to smaller groups of performers (Trueman Reference Trueman2007; Wang Reference Wang2009). Laptop ensembles such as the Worldscape Laptop Orchestra (Harker, Atmadjaja, Bagust and Field Reference Harker, Atmadjaja, Bagust and Field2008) double or triple the numbers of performers in the PLOrk model. The MICE Orchestra typically involves around 250 performers and has included as many as 500 performers. At this scale, individual agency cedes to emergent group dynamic.
3.4. NOMADS system architecture
NOMADS exploits Java's robust network API, multi-platform support and web-browser-enabled virtual machine, allowing an almost unlimited number of agents to connect together in real time. We use standard, built-in Java APIs for user interface creation and graphics rendering. For audio synthesis we use Phil Burk's JSyn package (Burk Reference Burk1998). Much like Java itself, JSyn offers a rich and flexible framework for building audio structures.
The NOMADS server consists of a Java program running on a dedicated computer connected to a high-speed Internet connection. All client applications use the same basic template to communicate with the server. Java graphical user interface (GUI) and audio elements are written into each applet. The server records data sent and the index of clients. Control applets determine how the server filters and routes that data (figure 2).
NOMADS system architecture.
NOMADS’ clients consist of two main components, a main program and program threads. This paradigm satisfies several OS and Java security standards, it helps provide convenient bookkeeping for routing data, and it allows applet programs to run asynchronously from network data transmission. Socket connections are monitored closely as spawned threads can sometimes terminate independently unknown to their parents. Dead threads are restarted, if possible, or purged via garbage collection (figure 3).
NOMADS data routing.
NOMADS relies on the Java network socket stream objects. For ease of use, NOMADS currently employs the UTF data streaming functions. These fold seamlessly into Java's thread mechanism, which allows the system to support hundreds of simultaneous connections between clients. It also provides the ability to send and parse data as ASCII text. When a client launches an applet it establishes a socket connection to the server. As soon as the server receives a connection, a corresponding thread is launched specific to that client. This allows the server to easily reference client-specific data (figure 4).
NOMADS threads.
All application classes of the API have identification tags (IDs). As soon as a client establishes a connection to the server it sends this ID. The server in turn immediately caches this information, associating it with the corresponding spawned thread. Based on specific implementation needs, data is then routed to and from clients. A client sends data to the server, which then sends data out to corresponding connected applets based on the ID of the incoming applet and the server's current data routing configuration. Dynamic arrays are maintained by the server process, allowing it to quickly reference senders and recipients, as well as their data.
3.5. NOMADS interaction hierarchies
In a typical NOMADS performance three types of applications work together: one for the performer/participants, one for the conductor, and one for the merged audiovisual projection (figure 5). This design is similar to that used by Jason Freeman's LOLC ensemble (Freeman Reference Freeman and Akito2011). A participant engages with an applet locally on their laptop or mobile device. The conductor applet provides control over participant applets and sets the data routing configuration for the server. Display applets receive merged data and offer a real-time visual and audio representation of group input.
Participant/conductor/display paradigm.
In the Sound Mosaic instrument, for example, each participant interacts with a grid-like interface to sequence rhythms. Participants may control their own synthesisers, or in another mode may alter a merged-group step sequencer. In yet another mode, the conductor can control all of the participant computers. Figure 6 describes the participant/conductor/display architecture.
Participant/conductor/display paradigm in NOMADS Sound Mosaic.
Applets are organised into a customisable ‘Bindle’ (a nomad's bag of tools), presented as a small toolbar when the participants access NOMADS. The Bindle is customisable for each context and new tools can be developed for specific purposes. Clicking on each button will open the corresponding program in another window. The conductor enables and disables applications using a control panel. The Bindle offers a solution to the problem of distributing software to participants. While not completely dynamic, like Steven David Beck's GRENDL (Beck Reference Beck2010), the conductor's ability to enable and disable tools through the Bindle allows any module to be readily available in performance. The conductor can enable a particular application and then it can be launched by participants. Figure 7 shows a Bindle with the first four buttons enabled.
The NOMADS Bindle.
3.6. Performance issues and considerations
NOMADS sends symbolic control data that is not as time-critical as audio data. Audio in NOMADS is synthesised or played back locally by each client applet. Unlike live ‘network jams’ (Cáceres, Chafe and Gurevich Reference Cáceres, Chafe and Gurevich2010) NOMADS performances can tolerate occasional latencies that would otherwise degrade a live musical performance.
Applets run in real time, without a scheduler. In order to compensate for network latency, we first calculate the average return time of message transmission from the server to the clients and back, similar to Roger Dannenberg's CMLO (Dannenberg et al. Reference Dannenberg2007). The main control applet can, when manually triggered by the performance conductor, reinitialise values for latency drift during the live performance using this same method. The server then broadcasts a starting command to client threads using this average as an offset. For applications requiring rhythmic synchronisation such as the Sound Mosaic, client applications rely on the mobile device clock used in conjunction with this latency calculation. More exhaustive testing of latency timing and analysis remains an area of future work.
4. NOMADS In Performance
4.1. Auksalaq telematic opera: audience participation
Auksalaq (2007–11) is a telematic opera composed by Matthew Burtner and produced by Scott Deal.Footnote 3 ‘Auksalaq’, the Inupiat word for ‘melting snow/ice’, explores global climate change from a northern geographic and cultural perspective. Using networked technology, video, music, voice, visual arts and interviews, Auksalaq integrates artistic expression, scientific information and socio-political commentary to create an interactive, multi-dimensional experience. The narrative incorporates fragmented and conflicting perspectives about the state of climate change as experienced in Alaska and the Arctic. The music expresses interlocking environmental forces as eco-musical forms, expressing the profound changes in the ice-flows of Arctic waters.
The opera is performed on five remote stages, interconnected with live audio and video streams such that the performers play together in real time and the audience perceives a local live stage in counterpoint with the other mediated stages via sound and video projection. The performance uses high-bandwidth audio streaming in JackTrip (Cáceres and Chafe Reference Cáceres and Chafe2009), and video through ConferenceXP. The scored music compositions were created specifically for remote performers, taking advantage of the innate characteristics of distance and network mediation.
The concept of the work inspired the use of remote performers and audiences. Auksalaq attempts to convey the close connection we feel to our home places, and simultaneously the sense of distance we feel as a result of geophysical and temporal separation from these places. By conjoining a notion of ‘home’ with human-caused climate change, Auksalaq explores our individual human relationship with a changing planet. The consequences of our actions are not only felt locally; they affect the entire planet. Multiplied by billions of humans even small patterns of behaviour have global consequences.
Auksalaq involves the audience in all locations as active participants in the multimedia performance. The audience contributes to the music, libretto, lighting and social discourse through NOMADS. As the audience members enter the concert hall they are pointed to the client interface URL (figure 8). The interface includes sub-windows for several applications, allowing a participant to move between functions without changing screens. On the server side, these applications can be turned on and off, and the brightness of each can be controlled on the main projection screen allowing overlaid elements to fade in and out.
Auksalaq client interface.
Upon joining NOMADS, each person is assigned a single sampled droplet of a melting Alaskan glacier. The sound emits from the personal device speaker, not from the house sound system. By moving the cursor around the screen, participants can control the individual rate of the droplet and the frequency at which it repeats. This rate is relatively slow (ca. 2–5 seconds between drops), but as the number of participants increases, a distributed sonic collage of melting ice permeates the concert hall.
Each individual appears as a cursor connected by lines to other participants, on the NOMADS display projection. As each person's cursor moves around the screen, the summed image presents a single, evolving graphic structure. Figure 9, taken from the performance, shows the audience interacting through NOMADS on the right-hand screen. As participants move the individual cursors, they simultaneously control a noise synthesis engine working in counterpoint with the scored instrumental Windprints music. The screen on the right of figure 9 shows the resulting graphic generated by the clients’ input into the noise sythesiser
NOMADS in a performance of Auksalaq.
The Auksalaq participant interface also includes a discussion application, Group Discuss, and the Thought Cloud application. During the performance, the singers prompt the audience to enter responses in the Thought Cloud. This application formats the text as a real-time ‘tag cloud’ such that more popular words grow larger and less popular words shrink. Words gradually fade from view if they are not repeatedly entered. Two sections of the opera use the Thought Cloud. In the first such section, a singer asks the audience members to enter the name of their home place. Figure 10 shows a still image from a performance.
Libretto text generation from Thought Cloud in Auksalaq.
The singer constructs the aria for ‘Unganaqtuq Nuna’ using the place-names entered by the audience. In a later section, ‘Auksalaq Aria’, the singer asks the audience to respond to interviews with scientists and village elders. She asks them to describe how it makes them feel. She then draws text directly from the real time Thought Cloud display and sets them to the composed melody (figure 11).
Real-time libretto generation in performance of Auksalaq.
Throughout the performance, the audience can communicate with one another and with the musicians using the Group Discuss application. The discussion text is projected onto the display screen across all of the venues. In figure 11, the small text along the lower left of the image is the beginning of an audience discussion.
4.2. Digitalis MICE Orchestra: rehearsed ensemble
The 2010 Digitalis concert at the University of Virginia showcased NOMADS through the performance of a 260-member MICE Orchestra. The concert featured several performances for NOMADS, including Unity Groove, Swarm Synth and Sound Mosaic with the electronic music duo Matmos. The composition of these pieces reflects David Kim-Boyle's notion that, in environments where networks provide participants with creative control, the role of the composer becomes that of the designer, and the nature of the performance becomes one of play (Kim-Boyle Reference Kim-Boyle2009). These three pieces are based on the interaction between large numbers of performers controlling narrowly defined musical parameters. As in other emergent compositions, these pieces require many individual agents in order to produce the desired musical effect.
4.2.1. Unity Groove
Participants in Unity Groove are given a simple instruction: ‘Match the pitch and rhythm of your neighbour.’ The interface consists of a pulsating pitched noise played out of individual performers’ laptop speakers. The pitch and rhythm of the sound is visually matched with a flashing, colour-changing circle on each performer's screen. Pitch and rhythm are coupled, so that as pitch increases, the tempo increases. Synchronised flashing circles pulsate and change colour in rhythm with the sound. The performance begins with a random starting pitch/rhythm. As performers attempt to match their neighbours, an immersive pulsating sonic texture emerges (figure 12).
NOMADS Unity Groove.
4.2.2. Swarm Synth
While swarms have been used in algorithmic contexts, for example T.M. Blackwell's SWARMUSIC (Blackwell Reference Blackwell and Bentley2002), the ‘swarm’ in the Swarm Synth is made up of individual participants jointly controlling a single FM synthesiser. Participants control the position of their pointer on an X/Y grid that is mirrored on their screen and on the NOMADS display. Position on the X-axis controls the value of the carrier frequency, and position on the Y-axis controls the value of the modulating frequency. The NOMADS display shows the position of all performers on the grid as coloured dots, as well as the conductor, indicated by a larger dot. Reacting to the group's instruction to ‘follow the leader’, a swarm of sound is produced by the shifting sidebands created by FM synthesis. Performers may also engage in ‘free play’ where they learn to work together to create interesting sonic and visual patterns (figure 13).
NOMADS Swarm Synth.
4.2.3. Sound Mosaic
Sound Mosaic is a twenty-four-step sequencer with a bank of up to twenty-five sounds. Individuals may load sounds, turn on and off individual beats, and change the tempo while the rhythmic pattern plays out of their laptop speakers. Users also have the ability to select random patterns, random banks of sounds, and rhythmic divisions for each sound based on 1, 2, 3, 4 … n beats.
Sound Mosaic uses network capabilities in two different modes: ‘Kahn’ control and ‘Tribe’ control. In Kahn control, similar to Gil Weinberg's concept of ‘monarch’ control (Weinberg Reference Weinberg2005), the conductor assumes top-down control of performers’ sequencer instruments by starting and stopping them, syncing the current pattern and sound bank selection, and changing the tempo. Thus, the leader can control the sonic output of a large number of distributed laptop speakers. In Tribe control, a ‘democratic’ system (Weinberg Reference Weinberg2005), the individual participants are able to control the pattern and sound banks of the conductor's sequencer, which is connected to the main house playback system. At Digitalis the electronic music duo Matmos performed the NOMADS sequencer, controlling the MICE Orchestra as a 260-voice spatialised sequencer (figure 14).
NOMADS Sound Mosaic.
5. Conclusion
NOMADS enables large groups to participate in an emergent display of socio-synthesis. The examples discussed here, Auksalaq and the MICE Orchestra, reveal two distinct paradigms for using the technology. In Auksalaq, audiences separated in different concert halls interact through NOMADS in parallel with audiovisual connectivity. In the MICE Orchestra, NOMADS enables a large group of trained performers to collaborate on complex synthesised structures.
In both examples NOMADS breaks down barriers between the performer and the audience. In NOMADS performances, the participants are not necessarily on the stage. In the case of Auksalaq, the participants are self-selecting audience members, and in the MICE Orchestra they are a rehearsed ensemble but they sit around the concert hall, not on the stage itself.
In any given concert, some people sitting in the audience seats will participate and others will not. Active performers sit adjacent to passive listeners. The reformatted arrangement and the interactive nature of the experience has a profound effect on the social aspect of the concert. The atmosphere of the concert hall becomes less formal. Participants and non-participating audience members alike feel inclined to interact with their neighbours. Because sounds come from personal mobile devices, the audience members assume some ownership of the sound, and they feel more confident to express themselves in the musical process. The constantly evolving spatial diffusion encourages movement and exploration. In outdoor or gallery performances, people often walk around to experience a modulation of the changing texture. The music is accessible to audiences who enjoy playful interaction with it. And, for experienced listeners, the emergent texture rewards sustained and repeated listening.
In his introduction to the Network Music session at the 2008 International Computer Music Conference in Belfast, Ireland, Andrew Gerzso challenged telematic art by saying ‘The audience must not simply be a witness to the event’ (Gerzso Reference Gerzso2008). Like others, he points out that audience experience in teleperformance is weakened by communicative limitations of audiovisual transmission across networks (Camurri, De Poli, Leman and Volpe Reference Camurri, De Poli, Leman and Volpe2005). NOMADS repurposes the audience, offering them agency in the music structure through socio-synthesis, enabling a critical mass of participants to energise the performance.
