1. Introduction
Composing's one thing, performing's another, listening's a third. What can they have to do with one another? (Cage Reference Cage1961: 15)
In a cultural landscape increasingly dominated by user-generated content and social media, we believe that the traditional feed-forward model of music production and consumption – in which a composer notates a fixed score, one or more musicians interpret it in performance, and each listener comes to a personal understanding of the resulting sound in concert or on recording – risks becoming increasingly anachronistic. (Fixed-media electroacoustic composition typically bypasses the role of the performer in this process, but the essential feed-forward structure remains.)
With the help of new musical interfaces, sophisticated algorithms and ubiquitous digital devices, anyone can now create, remix and share music, with no formal training, using only the commodity hardware they already own (e.g. Wang Reference Wang2009). At the same time that technology has democratised access to music-making, it has also problematised the act of concert performance: when music becomes so dependent on studio techniques, how can a work be replicated in live performance, and, more importantly, how can performance continue to be an authentic, risky, unique and virtuosic activity (e.g. Auslander Reference Auslander1999)? The frequency with which both popular (e.g. de Moraes Reference de Moraes2004) and classical musicians (e.g. Gresham Reference Gresham2012) have been ‘caught’ faking live performances attests to the magnitude of this challenge.
Our interest lies in examining the relationships connecting composers, performers and listeners in live concert performance and investigating the ways in which new technologies and creative practices can transform these connections. We seek to create performance environments in which composers, performers and audience members are linked together through novel participatory interfaces, integrated visual feedback (dynamically rendered music notation and data visualisation), and expressive sound (both electroacoustic and acoustic) to create complex feedback networks in which each constituency continuously influences the others, and each other, in real time during each performance. The concert then becomes a unique product of the shared experience that transpires in the moment of performance. There is no longer any reference score or media on which the performance is based – only a set of processes and materials that define a space of possibilities.
Over a decade of artistic practice and research in this space, we have identified and focused on core challenges in the design of works in which large concert audiences engage with one or more onstage musicians to collaboratively shape a musical performance in real time. Large audiences are made up of unique individuals whose musical knowledge, expectations, tastes and egos vary widely. So how does one design an interaction that is engaging, expressive and musically satisfying to the group as a whole? Fundamental challenges include participatory interaction design and low-floor, high-ceiling interfaces; reflecting a multiplicity of inputs in a single output stream by maintaining the importance of both individual contributions and a coherent musical result (Freeman Reference Freeman2010); and motivating participants without prioritising competition over musical listening and reflection. These issues tend to become increasingly pronounced as the number of audience participants grows.
Our explorations of these questions have been tempered by practical constraints in deploying systems to support large-audience participation. We typically must develop a new software and hardware solution for each project, accounting for the interface requirements and limitations on budget, venue, and setup time. The amount of time we invest in creating new systems for each new project – and often in adapting those systems to each new performance venue – works against the experimental approach of rapid iteration and assessment that we would prefer to employ. Thus, we have always been in search of a practical system that could be quickly adapted to each new iteration and each new project we undertake.
As the capabilities of smartphones grow and they become increasingly ubiquitous, they provide a compelling opportunity to facilitate these interactive, expressive and unique collaborative experiences in live musical performances. In this paper we discuss massMobile, our flexible, scalable framework designed to facilitate creative large-scale audience participation in live musical performances, using smartphones and existing wireless network infrastructure as the communication link between audience members and performers. massMobile is not in itself a solution to the daunting artistic challenges inherent in such work, but rather a framework designed to enable rapid, iterative design, development, deployment and evaluation of systems that explore solutions to those challenges. We also hope that massMobile will help us (and others) to analyse the group behaviour of large audiences, deriving more generalised knowledge about collaborative creativity similar to what Mathew (Mathew, Rogers and Lloyd Reference Mathew, Rogersand and Lloyd2011) and Jaimovich (Jaimovich, Ortiz, Coghlan and Knapp Reference Jaimovich, Ortiz, Coghlan and Knapp2012) have done in different participatory contexts.
massMobile facilitates the connection of audience members to performers and to each other through a smartphone-optimised web application client, a server and an onstage computer. Audience members access the massMobile client using a web browser on any recent computing device (most commonly a smartphone or other mobile device). By interacting with the web application, they send data in real time to the onstage computer, which collects and analyses this data to drive tasks such as information display to performers, electroacoustic sound generation, video projection or lighting. The onstage computer can also send data back to the audience web application, such as visualisations of collective audience activity and dynamic changes to the web app's user interface. Both the audience and performer interfaces are highly customisable for specific implementations of works that employ this framework.
In this paper, we first address previous work in audience participation in live performance, followed by a discussion of massMobile's design goals and implementation. Finally, we describe and evaluate its use in a series of projects and performances.
2. Background
We define audience participation as engagement in music from those not directly involved in its conception and planning. Such participation typically implies a redistribution of responsibilities within the traditional roles of composer, performer and listener, a modification of the links that connect these entities together, and, potentially, a blurring of the distinctions between those roles. The more actively audience members involve themselves in the performance, the less clear these distinctions tend to become. In this section, we briefly outline modes of audience participation, mechanisms to support them, and the contexts in which these practices have developed.
2.1. Simple modes of participation
The simplest form of participation involves listening to fixed media. While this is often a private activity, there is sometimes a social element as well, whether choosing songs to play at a party, creating a mix tape, or sharing listening activities through online music services such as Spotify and Pandora (Spotify 2012, Pandora 2012). Listening to music in a concert setting can likewise remain a largely private activity – despite the shared time and place of the event – in performance settings where audiences are expected to be quiet throughout the performance. But in other settings, audience members may engage their presence more publicly.
The most common way in which audiences publicly engage is through applause, where they have the opportunity to acknowledge the performer, composer or both. With applause, group collaboration is not only evident from adding to the mass of sound, but audiences’ clapping often moves in and out of periodic synchronisation (Neda, Ravasz, Brechet, Vicsek and Barabasi Reference Neda, Ravasz, Brechet, Vicsek and Barabasi2000), demonstrating a subtle awareness of and connection with each other. The cover of the 1974 album Before the Flood shows a stage view of audience members waving lit cigarette lighters above their heads (Dylan Reference Dylan1974). As with applause, this tradition communicates approval and shows the desire to communicate with the performers, but using a quiet and ubiquitous device. Dancing moves beyond applause and lighter-waving in the opportunity it provides audience members to express themselves more individually and creatively in response to the music.
Singing enables audiences to contribute directly to the musical performance itself. ‘Sing alongs’ are seen in several contexts from religious and athletic gatherings to performances of art music to popular music performances. At sporting events, large groups often sing or chant in unison without any sort of conductor (Heaton Reference Heaton1992). Such activities have been appropriated by contemporary music composers as well: in Hasse's, Reference Hasse1986 piece Moths (1986), audience members create the music by whistling along to a graphical score and following the instructions of an onstage conductor.
2.2. Networked audience participation
Barbosa (Reference Barbosa2003) describes a classification space for networked music along two axes: interaction (synchronous or asynchronous) and location (co-located or remote). These same axes apply equally well to networked audience participation.
Asynchronous participation generally occurs before a musical event, and is usually remote, with participants engaging from a variety of locations. In Virtual Choir, participants from around the world submit webcam recordings of themselves singing one part of a composition; the recordings are synchronised and played back as one choir (Whitacre Reference Whitacre2012). UrbanRemix invites participants to submit geo-tagged field recordings to a website where they can traverse this aural landscape via an interactive map (Freeman, Disalvo, Nitsche and Garrett Reference Freeman, Disalvo, Nitsche and Garrett2011). Participants in Converge gather recordings, pictures and text from their daily lives, and this material is visualised and sonified during the performance (Oh and Wang Reference Oh and Wang2011). Other types of asynchronous participation works allow audience members to influence the musical result in performance, rather than directly contribute musical material. Graph Theory and Piano Etudes, for example, allow users to rearrange precomposed musical fragments into a new version of the score (Freeman Reference Freeman2010).
Synchronous participation occurs during a musical event, and can be either co-located or remote. Max Neuhaus, in his works Public Supply (1966) and Radio Net (1977), utilised telephone networks and radio stations to gather and broadcast input from distributed participants (Neuhaus Reference Neuhaus1990, Reference Neuhaus1994). In a similar fashion, Negativland's Over the Edge Radio asks listeners to contribute to what it calls ‘receptacle programming’, phoning in audio content to a live radio programme (Negativland 2012). Utilising the web, Telemusic #1 allows participants to remotely trigger sounds in a physical performance space via their web browsers (Young Reference Young2001). The World Opera uses dedicated high-speed communication networks to connect distributed performers for real-time performance (Olmos, Brul, Bouillot, Benovoy, Blum, Sun, Lund and Cooperstock Reference Olmos, Brul, Bouillot, Benovoy, Blum, Sun, Lund and Cooperstock2009). In No Clergy, co-located audience members in a concert hall use web browsers to influence the notation read by performers (Baird Reference Baird2005).
massMobile's main focus is on synchronous, co-located use during a performance, establishing a real-time network within the concert space, since that is where its plug-and-play deployment approach is most beneficial. But its architecture does allow for synchronous and asynchronous, and for co-located and remote, participation.
2.3. Contexts for participatory practices
Our interest in networked audience participation is grounded in the context of open-form composition. The open-form scores of composers such as Karlheinz Stockhausen (Reference Stockhausen1957) and Earle Brown (Reference Brown1999) negotiated a redistribution of the roles of composer and performer by challenging musicians to make expressive interpretive decisions about the order in which score fragments were to be played in concert. Composers such as John Zorn (Reference Zorn2002) and George Lewis (Reference Lewis2000) went a step further by creating structured game-like environments for improvisation in which a formal architecture influenced how musicians improvised and how they communicated with each other in a network of mutual influence. These practices emerged, at least in part, out of new questions about the role of the performer spurred by the growth of electroacoustic fixed-media composition practices (Stockhausen Reference Stockhausen2004).
As we consider these practices in a contemporary culture focused on collaborative content creation, our focus shifts to the relationship of such open scores to listeners. We are concerned that these classic renegotiations of the composer–performer divide leave the audience aside: in fact, many listeners remain perplexed about (or simply unaware of) the novel roles both composer and performer play. Thus, the need arises to reconsider these practices not only as a means to manipulate the composer–performer divide but also to rethink all the boundaries between audiences, composers and performers. Fortunately, novel musical interfaces combined with real-time computing power now make it practical to extend the open-form model to audience participation technologically.
Networked game research also serves as a fertile foundation through which to consider networked audience participation. Similar approaches to interface design and interaction strategies can be found across both practices, and they share many of the same challenges of design and technological implementation. For example, the use of heuristics to evaluate the playability of games described by Desurvire can equally be applied to audience participation designs (Desurvire, Caplan and Toth Reference Desurvire, Caplan and Toth2004). Issues of latency and synchronisation across networks as described by Claypool are equally applicable to networked audience participation (Claypool and Claypool Reference Claypool and Claypool2006). Despite technical and analytical similarities, one of the biggest differences is typically motivation for the interaction. Where games are frequently motivated by competition and accumulating points, audience participation is more oriented towards collaboration. Here, the product is not composed of winners and losers but a single sonic experience shared by the participants.
2.4. Participation devices
There are several approaches currently used to facilitate audience participation, such as sampling the intensity of audiences’ cheering and clapping, tracking the audiences’ movement, and distributing devices to the audience. While some types of audience input can be gauged merely by human perception, technologically grounded tools for detection and analysis open up new participatory design possibilities in areas such as interface design, bi-directional communication, data analysis, data mapping and visualisation.
Computer vision systems have long offered a relatively scalable participation platform at the expense of easy setup and rich interfaces. Maynes-Aminzade describes such a system and its use in large-audience video game interfaces (Maynes-Aminzade, Pausch and Seitz Reference Maynes-Aminzade, Pausch and Seitz2002). Other projects have distributed devices such as light sticks in Glimmer, lighted hats in Flock or coloured paddles in Cinematrix for tracking using computer vision (Carpenter and Carpenter Reference Carpenter and Carpenter1999; Freeman Reference Freeman2008). Reference Feldmeier and ParadisoFeldmeier and Paradiso use disposable wireless sensors instead of computer vision (2007) to more accurately capture motion and to support performance contexts, such as clubs, in which computer vision might be problematic.
Distributing devices such as paddles, light-sticks and custom sensors, though, requires significant investments in equipment and infrastructure as well as significant setup and calibration time at each performance venue. This ultimately limits the feasibility of potential performances. Further, the interface design of each system is typically highly constrained by the hardware, limiting both the depth of audience interaction and the adaptability of the systems to diverse interaction designs and to iterative design revisions.
Instead of relying on a custom hardware setup with distributed devices and/or computer vision, massMobile uses the mobile devices audience members themselves bring to performances, along with existing wireless network infrastructures, to enable more flexible participatory design systems to be deployed with minimal logistical or technical hurdles. Smartphones have compelling features appealing to composers of participatory works. All data is in digital form, making computational analysis relatively easy, and each user can be tracked individually throughout the performance, giving a greater level of personalisation of the data beyond a simple aggregate.
massMobile's use of commodity mobile devices follows a long tradition of their use in participatory performance contexts. Golan Levin's DialTones utilises the audience's mobile phones as a distributed array of loudspeakers (Levin Reference Levin2005). Although the audience's agency is limited, their role in the piece is clearly defined, and each audience member clearly understands how they contributed to the performance. More recent projects use smartphones as platforms for more active participation. In a piece for audience and solo performer, Echobo transforms the audience's smartphones into instruments used to accompany the performer (Lee Reference Lee2012). In Moori, participants respond to prompts from the performer by typing short messages into a locally networked application (Kim Reference Kim2011). TweetDreams acknowledges the difficulty in developing and maintaining mobile applications for a variety of different operating systems (Dahl, Herrera and Wilkerson Reference Dahl, Herrera and Wilkerson2011); it uses a pre-existing framework (Twitter) to gather audience input to influence sound and visuals in performance.
massMobile not only uses commodity mobile devices; it also uses web standards to make it easier for artists and researchers to design and deploy participation systems. As Oh and Wang (Reference Oh and Wang2011) explain: ‘the mobile web provides us with an ideal setting to enable audience participation in music performances in a manner that is software-free (beyond an internet browser) and less dependent on the hardware (that is, in comparison to building device-specific applications).’
2.5. Network infrastructure
In performance settings, mobile devices can be networked over local WiFi networks temporarily set up for a performance or over existing Internet-based wide-area networks.
Local networks typically provide a faster transmission of data and are well suited for applications that demand minimal latency. TouchOSC, Mrmr and Control are examples of frameworks that utilise local networks, with the original design intent of replacing dedicated hardware fader boxes for controlling music software with highly configurable wireless smartphone interfaces (TouchOSC 2012, Mrmr 2012, Control 2009). These frameworks are typically intended for single-user interaction with a single computer but can support multiple users. Jesse Allison describes a system that works in a similar manner but is deployed as a browser-based interface in order to leverage web standards and facilitate multi-platform use (Allison Reference Allison2011).
In the context of massMobile, these systems identify fundamental design questions surrounding wireless networks and web standards. While these applications must minimise latency (at the expense of easy network configuration), massMobile must maximise ease of setup (since audience members must quickly install and configure the software themselves). The use cases for massMobile, in turn, must assume a high latency and incorporate this limitation into their designs. massMobile is also directly influenced by the ways in which these systems use standard tools and reusable, configurable components to facilitate rapid development of interfaces for new contexts.
3. Massmobile Goals
As a generalised, flexible and open software framework, massMobile seeks to further the practice of audience participation in live musical performance and to explore new design possibilities. It uses commodity mobile devices and web standards to create rich participatory musical experiences that create novel feedback connections among large concert audiences, onstage musicians and electroacoustic sound sources. In developing massMobile, we focused on three overarching goals: enabling rapid development of new participatory interfaces; enabling plug-and-play deployment at performance venues; and facilitating simple use by audience members on a variety of mobile device platforms.
Exploring interactive design challenges requires iterative development, evaluation and revision. Rapid development enables iterative artistic experimentation through the design–test cycle. Iterations of the interaction design can be developed and deployed entirely in software code and/or configuration changes, so they take days (or even seconds) instead of months as with hardware-based systems.
Plug-and-play setup is critical for deployment in a variety of venues where there may not be technical infrastructure (such as WiFi networks) or setup time to install and calibrate hardware such as computer vision systems. Having a simple setup can create more opportunities to test a participatory design and present it in performance.
Audience members need the ability to instantly launch an application and participate, ideally in a single step. It is not practical to dedicate a significant amount of performance time to setup and configuration, nor to send assistants through a large audience to help them connect. Our assumption is that most participants are familiar with operating their mobile web browser, as well as connecting to an available network. Rather than depending on the availability of local or ad hoc wireless networks, we support participation over any available Internet connection.
4. Design and Implementation
massMobile consists of a web-client for use by the audience on any recent smartphone or other device with a web browser connected to any Internet connection (WiFi, 3G, 4G), a Max/MSP API for use in software for the performance, and an offsite server that passes information between the smartphone and Max/MSP clients (Figure 1).
Block diagram of the massMobile framework.
4.1. Web application
4.1.1. Web application architecture
We initially developed native applications for iOS and Android for audience participation, but eventually abandoned these in favour of a web application. The additional overhead of developing and maintaining two separate applications was contradictory to our design goal of rapid development in extending the framework, and it lacked support for platforms such as Windows Phone.
The current smartphone web client uses common web standards, including the Google Web Toolkit, HTML5 and REST, and runs without modification on all major smartphone web browsers. (It even runs on desktop web browsers.) While this development approach lacks access to some native phone APIs, it met the needs of massMobile and performs at near-native speeds. Revisions do not require deployments to native app stores, and audiences need not download a native application. (The web site link is distributed to the audience as both a QR code and a short URL.)
Because massMobile connects to a cloud-based server, it requires no special network configuration. Once audience members load the URL, they are ready to participate.
4.1.2. Web client user interface
The user interface has a customisable GUI that can be configured for different uses via a server-side database and the Max/MSP API (see below). When a user loads massMobile, the URL's query string automatically loads the configuration for a particular project and event. In massMobile, projects represent user interfaces that may be used across multiple performances, while events represent a single performance of a project and all of its associated audience participation data.
The main page for an event has tabs for introductory text and for each active user interface element (called a ‘model’ in massMobile) (Figure 2). Models support interaction modes such as sliders, text entry, voting and drawing. Parameters of their display are preconfigured through a server database and can also be updated in real time during an event.
User interface showing the voting model.
massMobile has two ways to provide feedback to audience participants: via a video projection in the performance venue that is generated by the Max/MSP software, or via visual elements that are displayed directly within the web application interface. The latter is particularly useful in venues that lack video projection. Feedback typically shows a visualisation of audience response (e.g. the tallies of votes for different choices) so that participants can consider the collective activity as they use the interface, and so that they can better follow the influence of audience participation on the live performance.
4.2. Max/MSP API
Because massMobile can be used in a variety of performance contexts, we provide a generalised API for Max/MSP rather than a single, monolithic application to drive performances. Our goal is to enable any application developed with the Max/MSP API to be plug-and-play, so that even performers with minimal technical expertise can launch an application on their laptop and perform with massMobile without assistance or further technical infrastructure (though some massMobile projects inevitably require additional resources for things such as electroacoustic sound diffusion).
The Max/MSP API connects to our server to get and process both raw and pre-processed data, update GUI configurations on connected phones, and simulate audience members’ interactions with the system for rehearsal and testing purposes. The API is implemented in Java as an MXJ external. The API also has methods for searching data based on query strings so that sets of data with specific attributes can be returned.
4.3. Controlling the Smartphone Client from the Max/MSP API
In many massMobile projects, it is necessary to change the smartphone client's GUI configuration in real time during an event. Changes might enable or disable all interaction, step through different interaction interface models, or change text instructions or configuration of a single interface model.
These changes are typically triggered from the Max/MSP API and can be done in two ways. The first is through a pre-programmed queue of changes that can be stepped through by pressing the spacebar, using an external controller (such as a MIDI foot pedal), or configuring an automatic timer. The framework also allows for spontaneous manual changes of the smartphone interface through a ‘configuration editor’ (Figure 3), permitting performers to respond to an event's unique dynamic in real time and to tweak the smartphone interface accordingly.
Left: User smartphone interface showing the slider model. Right: Configuration editor in Max/MSP API.
4.4. massMobile server
In accordance with our design goals of scalability and flexibility, massMobile utilises Parse – a schemaless cloud-based data store – for handling requests from the clients and storing all audience and configuration data (Parse 2012). By hosting the server offsite from performance venues, we enable smartphone access over any Internet connection at the expense of latency. By using web standards and a database, we are able to host a robust, scalable server and also to archive participation data for later analysis.
Because latency over cellular networks is both potentially large and extremely unpredictable (and in our experiments can be as long as a second), massMobile projects must be designed to accommodate such latencies. Data sent to the server is time-stamped when inserted into the database allowing for reconstruction of the event sequence as received. We generally expect that the web application will supply input that is not highly time-sensitive, such as votes, text messages, drawings or notes within a looping sequence, while the Max/MSP client handles absolute timing. Such strategies have long been common to address latency in networked music collaborations such as WebDrum (Burk Reference Burk2000).
5. Massmobile Projects
5.1. Early experiments
We first used massMobile in collaboration with choreographer Jonah Bokaer, where we carried out initial testing of the framework in spring 2011. massMobile was used in his work FILTER in performances in Avignon, France, and at the Ferst Center for the Arts at Georgia Tech in Atlanta, USA. In addition to these public performances, we conducted several informal trials with small audiences and a solo dancer onstage as we developed the software.
In the initial tests, audience members voted on lighting preferences. The votes were displayed on a video projection and also used to change the lighting configurations, which in turn cued the dancer to change his choreography. The small groups enjoyed responding to each other's votes and seeing the resulting change in dance and lighting, and a group dynamic quickly developed in which ‘leaders’ would change their votes and then ‘followers’ would gradually shift over to one leader, collectively effecting a change.
During the autumn of 2011, we presented massMobile in a series of demos at Georgia Tech. During these demonstrations participants created music collaboratively through a tonematrix looping grid sequencing interface, selecting points on a grid to activate specific pitches at specific times. massMobile turned this classic sequencer paradigm into a collaborative improvisation as participants composed a constantly changing loop together. There were no performers aside from the co-located participants using their mobile devices to interact with the shared tonematrix. Everyone could see the looping sequence projected on a screen and hear the result through a speaker system connected to the Max/MSP computer.
5.2. Saxophone Etudes
The most substantial performance work realised to date with massMobile is Saxophone Etudes, a 10-minute composition for solo saxophone with live audience participation, composed by one of the authors (Freeman). In this work, we tried to reconcile the asynchronous participation systems used in Freeman's past works such as Piano Etudes and Graph Theory – in which online users navigated open-form scores and by doing so influenced future live concert performances of the work (Freeman Reference Freeman2010) – with the synchronous participation systems used in works such as Glimmer and Flock – in which live audiences used simple analogue interfaces to influence the music performed on stage (Freeman Reference Freeman2008).
5.2.1. Goals
Saxophone Etudes aims to provide audiences the more sophisticated score-based interfaces that the asynchronous systems had afforded, but in the context of an immediate collaboration among audience members, and between the audience and the performer in a shared time and space. For us, the realisation of such a participatory structure only became possible with the development of massMobile, as it enabled us to create a richer real-time participation interface in the concert hall that was easily scalable to large audiences with low fixed costs and no in-venue setup.
Within this design, we deliberately created a simple, almost skeletal interaction, in recognition of the inherent constraint of no prior experience or rehearsal and a short performance. We also structured the performance into two movements, each with a different interface, in part to comparatively assess their efficacy. The performance was preceded by a short verbal introduction and demonstration lasting about five minutes.
5.2.2. Interaction design
The first movement, ‘Harmony’, consisted of twelve screens of music, each containing between two and four noteheads on a single music staff (Figure 4). The set of notes on each successive page always differed by a single note addition, removal or modification. Audience participants voted on which note they wished to hear. They could change their votes as many times as they wanted, and they could see the current vote tallies on their device's screen, both as percentages above each notehead and as semi-transparent circles around each notehead. (The radius of each circle grew with the note's popularity, and the most popular note was highlighted with a different colour.)
Audience interface for Saxophone Etudes, movement I.
The saxophonist read her music from a Max/MSP application running on a laptop. It displayed the same screens of music as the audience devices, with a similar visualisation of each note's popularity via semi-transparent circles (Figure 5). The saxophonist could simply play the most popular note or could play the notes in rough proportion to their relative popularities, always bringing her own ideas about phrasing and melodic structure into her decisions alongside the audience input. The saxophonist chose when to advance from one screen to the next by pressing a foot pedal; the audience screens then advanced automatically.
Performer's interface for Saxophone Etudes, movement I.
Both the saxophonist and audience's screens also displayed a bar titled ‘audience activity rate’; as audience members voted more frequently, the bar's value increased, and the saxophonist was instructed to play at a louder dynamic and with generally faster tuplet subdivisions.
The second movement, ‘Melody’, consisted of twelve screens of music, each with a short melodic gesture between five and ten notes long. Audience members dragged vertical bars on their screen to highlight a section of the melody they wished to hear (Figure 6). As with the first movement, they could change their votes at any time, and they could visualise the collective voting results via a gradient displayed underneath the music. (Darker sections of the gradient indicated more popular sections of the melody.)
Audience interface for Saxophone Etudes, movement II.
The saxophonist, again, saw the same screens, with the same gradient visualisations, on her laptop screen (Figure 7). She was instructed to play each melody in a loop, developing phrasing, articulations, rubato and other expressive elements based on the audience voting, and, if she wished, omitting less popular sections from the melody altogether. The same ‘audience activity rate’ was used as in the first movement, but this time to indicate dynamics and overall tempo rather than a specific tuplet subdivision.
Performer's interface for Saxophone Etudes, movement II.
In designing the audience interface, we deliberated over the inclusion of Western musical notation. We ultimately decided to include it, since each musical gesture was quite short and its most critical characteristics were in the height of each pitch (first movement) and contour of the melody (second movement). We did, however, include a toggle button on the interface so that audience members could switch to a graphical view of the notation if they wished. This follows from a similar practice we employed in Piano Etudes (Freeman Reference Freeman2010).
5.2.3. Evaluation and next steps
Saxophonist Jan Berry Baker performed Saxophone Etudes on two concerts in Atlanta in April and May 2012. At both performance venues, massMobile's technical architecture was instrumental in enabling the performances to proceed: we had no advance access to the venues, just a few minutes for setup and sound check, no Internet access, and no video projection. These constraints presented no problems for massMobile; both audience participants and the saxophonist used 3G/4G connections and visualised voting data on their own screens.
We assessed these performances based on quantitative data collected at them: the server log data on how participants interacted along with a short survey in which participants ranked statements on a 5-point scale (5 = strongly agree; 1 = strongly disagree) and optionally added free-response comments. We also evaluated the performance more informally based on conversations with participants and with the saxophonist and our own observations. Across the two performances, a total of 45 people participated with their mobile devices, but only 7 completed the post-performance survey.
Audience participants used massMobile to participate aggressively in the performances, logging 6,954 votes across the two performances, which averaged to 155 votes per participant. Despite this active participation, many audience members struggled to learn the interface itself or to trace their activities through to the saxophonist, given the short timespan of the performance. When asked if they felt their participation influenced the actions of other audience members, most participants (62.5 per cent) answered ‘disagree’, and when asked if they felt their participation influenced the actions of the saxophonist, the most common response (50 per cent) was ‘neutral’.
One survey participant noted that ‘The second part interface was less intuitive than the first half’. This sentiment was echoed repeatedly in informal conversations with participants and was reinforced by participation data: 80 per cent of votes were logged during the first movement and 20 per cent during the second.
Besides the challenges participants faced in learning the interface itself, some also found it difficult to ‘see the correlation between screen notation and the notes played on sax’. While this is an inherent challenge in a short-form participatory performance, in further discussions with participants we discovered that score fragments – which serve as a powerful paradigm in asynchronous participation contexts – introduce new challenges in synchronous participation, where participants cannot dedicate time to familiarising themselves with the sonic content of each fragment and cannot even aurally preview their contents.
The saxophonist enjoyed the experience but expressed continual frustration that the audience activity meter did not change more dramatically over the course of the performance.
All this feedback, in turn, suggests to us a potential improvement to the work's interaction design: to place dynamics, tempo, tuplet divisions and other expressive parameters more directly under the audience's influence through a two-dimensional crosshairs-style interface, while leaving decisions about the content to feature within each screen up to the saxophonist. This would move the audience interface away from its current focus on notation, circumventing the challenges it poses in synchronous participatory performances, and move towards a paradigm in which the interface remains rich but is easier to learn, audience influence is more readily reflected in the saxophonist's performance, and audience voting is more clearly visualised to more effectively facilitate the emergence of group behaviour. We have employed such an interface in experiments with the Georgia Tech Marching Band (Figure 8) and it has proven to be quite effective in these regards, and we are currently revising Saxophone Etudes accordingly in preparation for follow-up performances and evaluations in spring 2013.
Crosshairs-style interface used with the marching band.
We also recognise, in retrospect, that our evaluation of these performances excluded audience members who did not participate; for future evaluations, we will create a separate survey for them to complete so that we can better understand their experience and the unfortunate but unavoidable divide that smartphone-based participatory works create between those with compatible devices and those without.
As we undertake these revisions, we are again reminded of the benefits of the massMobile system over systems we have developed for prior participatory works. Though our revisions involve a radical change to the audience interface, the type of data collected, and the ways in which it is visualised to both the audience and the saxophonist, we are able to make these changes in a matter of days by reconfiguring and recombining existing components within massMobile. This enables us to iteratively improve, deploy, and evaluate the work rapidly in a way that would have been impossible previously, when each iteration required massive new technical development.
6. Discussion
To date massMobile has been used in four live performances of two different works incorporating audience participation, several demonstrations at Georgia Tech and one in Bournemouth, UK, and through these events we are achieving the fundamental goals of the framework: rapid development, plug and play deployment and simple audience access.
Rapid development facilitates iterative testing of the framework. During one experiment with the Georgia Tech marching band, we were able to deploy and evaluate three different types of interaction paradigms in under an hour. In this case, massMobile allowed us to do quick A/B comparisons of different designs that would have taken substantially longer with other platforms.
The plug and play setup of massMobile has enabled us to use the framework in a variety of difficult circumstances, including performance and demo venues lacking advance setup time or any technical infrastructure (WiFi, projection). Setup time for massMobile is comparable to that required to unpack an acoustic instrument.
Preparing the audience is always a challenging part of participatory performance. We have tried to make accessing the web application as easy as possible through a QR code and a short link printed in programme booklets. Despite such instructions, occasional confusion persists. We have observed audience members searching for massMobile in their device's app store, despite instructions to the contrary. And at one performance the short URL and QR code for the post-performance survey were printed in much larger text than the information for the web application itself, so some audience members launched the survey instead of the web application by mistake. We continue to identify solutions to these challenges, including closer partnership with concert venues in preparation of programme booklets, advance press to encourage audiences to bookmark the web application before the performance, and native applications that wrap the web application.
The benefits afforded by massMobile do come at a price: they are only available to audience members who own (and bring) a compatible device to the performance. This necessarily excludes some members of the audience from participating, and this division can work against the sense of engagement and collaboration that participatory works usually seek to instill. By supporting as broad a range of devices as possible, we have sought to minimise the number of people excluded from participation. And recent data suggests that smartphone ownership is becoming increasingly ubiquitous: in the United States, for example, 46 per cent of adults own a smartphone as of March 2012 – an increase of 11 per cent in just one year (Smith Reference Smith2012).
We are continually improving massMobile to better meet the primary goals of the framework. Currently, the Max/MSP API allows users to select which interface models will be active on the web application, along with the configuration parameters of each model (e.g. explanatory text or labels). In addition to the Max/MSP API, we are also adding a more general Java API to allow for usage in other environments. We are also interested in creating a GUI-based model editor to enable anyone to create and deploy user interface models for the web application without any coding.
We are currently collaborating with large ensembles and individual artists to bring massMobile to new performance contexts, and we also plan to conduct user studies to help us improve both the framework and the projects we realise with it.
We are developing a new project with massMobile focused on collaborative drawing in performance. Participants draw on a shared canvas in response to the music during the performance. Afterwards, animation of the canvas is synced to an audio recording of the performance both to create a ‘music video’ of the audience's collaboration and also to serve as a new graphical score to guide improvisation of a new performance. An initial performance and study of the framework is scheduled for summer 2012 with the Georgia Tech Jazz Ensemble.
We are also planning to use massMobile with Georgia Tech's marching band at a football game halftime show in autumn 2012, with thousands of fans using their smartphones to influence both the music and formations of 350 musicians on the field. One of the authors (Freeman) is also writing a new work for solo piano and electroacoustic sound, with massMobile, for pianist Melvin Chen.
We plan to continue conducting user studies via web-based surveys available from within the massMobile application, as we did with Saxophone Etudes. This feedback, coupled with quantitative analysis of the database event log data, provides needed material for formal evaluations of both specific implementations of the system and the overall framework. Additionally, it gives insight into how audience members perceived their role in the performance, the transparency of the interaction, and the musical result.
We also plan to create a series of new user interface models to take advantage of additional modes of interaction for the web application. Most of our current models focus on touch-based interaction (e.g. text, drawing, sliders, voting), but we want to support other modes such as audio and video analysis, accelerometers and geo-location. The challenge, as always, will be to integrate these into the framework in ways that minimise transmission bandwidth and recognise inevitable latency.
Rather than being a specific implementation of an audience participation piece, massMobile is a configurable, flexible and scalable framework for collecting and processing audience input. Mappings, processing algorithms, audience feedback and musical goals are not hard-coded into the framework but are intended to be quickly and easily configured for a specific implementation. With this framework, novel mapping strategies can be quickly tested, algorithms can be modified in real time to adjust for audience size or amount of activity, and user interfaces can be rapidly developed in order to produce a more collaborative, expressive and creative musical experience.
Acknowledgements
More information about massMobile is available at http://www.gtcmt.gatech.edu/research-projects/massmobile. We expect to make the software available as a free download in early 2013. In addition to the authors, Sriram Viswanathan contributed towards early development of massMobile, and Weibin Shen and Shaoduo Xie helped develop the cloud data store support. massMobile is supported in part by a grant from the National Science Foundation as part of a larger research project on musical improvisation in performance and education (NSF CreativeIT 0855758).
