1. Introduction
The differences between improvisation and composition are complex and controversial (Nettl Reference Nettl1974), but for the purposes of this paper we will use the two concepts to refer to different ends of the spectrum between musical decisions made as they are performed and decisions made prior to performance. The key element of the distinction is along the dimension of time, but, for live musical performance, the two extreme ends of the spectrum are merely idealisations (all musical decisions made at the instant they are rendered in sound, or every aspect of sound determined prior to performance). Live music is generally the result of some balance between elements composed prior to performance and elements determined by musicians as they play. The focus of this paper will be on music-making within an explicit window of time existing between the two extreme idealisations. We have developed the Anticipatory Score system, wherein notation is entered by musicians during the period of time after a performance has begun and prior to when it is rendered in sound.
In a survey of experienced musicians, Biasutti and Frezza (Reference Biasutti and Frezza2009) found the ability to anticipate upcoming musical events to be one of the most important skills required for successful improvised performances. While performers of composed music rely on notation to see and prepare for what is coming and for coordinating activity across potentially many performers, improvisers rely more on a variety of other techniques for coordinating musical activity during performance. For example, they frequently develop various performance plans during rehearsals prior to performance, and then during performance rely on a variety of cues such as eye contact and other kinds of ‘body language’ in addition to sonic cues to indicate intent and coordinate their activity.
Some kinds of pre-performance notation (e.g. chord charts for jazz, or graphical notation for electroacoustic music) are used in improvisatory contexts that function to organise and coordinate musical activity while leaving much responsibility to the musicians. Given the variety of meanings with which graphical symbols used in musical notation can be imbued, and their potential for conveying musical intent and coordinating temporal behaviour, it seems arbitrarily restrictive to limit their scoring to the time prior to performance when they by definition cannot be responsive to musical activity and sound that happens only during performance. Our goal is to design a mechanism by which musical notation can be responsive to ongoing sonic musical activity and, at the same time, to contribute new possibilities for anticipation, coordination, and collaboration in an improvisatory context. If improvisation can be considered as ‘real-time composition’, then why not integrate notation, the primary mechanism for representing and communicating musical ideas in composition, within the practice of improvisation? This is not to make a value judgement about the notationless modes of communication typically employed by improvisers currently, nor is it conceived of as a ‘solution to a problem’ that improvisers may or may not face. Rather, the goal is to create an environment in which the time-tested musical communicative capabilities of notation can be exploited in a new context and subsequently to explore its creative and musical potential.
Toward this end, we have developed a media platform, the Anticipatory Score, that supports the creation of graphical notation by participants during performance. Notation is written to the Anticipatory Score ‘just in time’ prior to its rendering in sound, and is visible to all participants as soon as it is written by any one of them. In this way, notation can be responsive to the unfolding sonic activity of the performance as well as to notation that other participants have written but which has not yet been sonified. The notational tools for scoring are thus in the hands of the performers themselves, allowing each to influence the performance by annotating the real-time scrolling score. This provides performers with an awareness of each other’s musical intentions and provides opportunities to prepare and respond to them both sonically and through notation. Anticipatory Score notation thus takes on a different role from traditional notation because it is not necessarily prescriptive. Instead, notation is used as a channel of communication between performers and interpreted as indicative of impending musical behaviour which the notator intends to have rendered, and therefore provides information that fellow participants can respond to in a variety of ways.
1.1. Background
The nature of the musical score has been on the move since at least the middle of the twentieth century. Cage’s (Reference Cage1969) book Notations is a fascinating snapshot of scoring practices that were revolutionising the way music was written and performed during that time. Even though the book provides little in the way of detailed descriptions of how each score was used, it is clear that graphical notation per se was not the only innovative aspects of the scores, but that composers were experimenting with new ways for performers to engage with sound and to interact with each other and their audiences.
A recent ‘sequel’ to Cage’s book is Sauer’s (Reference Sauer2009) Notations 21, which reflects the continued expansion of graphical techniques and interpretation strategies in current musical practices. However, a book platform does not do justice to a new class of musical notation that has been recently emerging. Computer technologies in particular have given rise to dynamic scores – scores that are not static, but change in some way over time. For example, Luke Harris’s Animated Score For Quartet projects a film that is read as a score by instrumentalists. It combines traditional notational elements, but, in addition, takes advantage of the spatial movement film affords using graphical elements as a new notational device for performers to interpret. In this kind of ‘filmic’ representation of a score, everything on the screen at a given moment represents the current performance time. Performers do not see an explicit representation of notation in the future (though there might be some continuity of motion that could be used to anticipate the future to some extent). While the Harris score is dynamic, the notation was generated in advance of rather than in the real time of the performance.
Real-time scores, where notation is generated during performance, have been growing in prominence along with enabling technologies such as fast network infrastructure and protocols, touch screens and graphical software. Gerhard Winkler (Reference Winkler2004) documents an early use of computer screens as interactive systems and scores. He developed two different kinds of notation (the ‘Control-Score’ and the ‘Playing-Score’) that each supported different relationships between performers (e.g. a ‘leader’ with the Control-Score). Winkler discusses several aspects of real-time scores that are still open research topics today such as balancing complexity and readability, and the optimal window of time for representing on screens for players. McAllister, Alcorn and Strain (Reference McAllister, Alcorn and Strain2004) invited the audience to generate score material in real time through wireless connections from their PDAs to computer displays in front of instrumentalists on stage. More recently, Freeman (Reference Freeman2008) collected real-time audience input for score generation in Flock (and other pieces) using video cameras. Burtner’s NOMADS system (Burtner, Kamper and Topper Reference Burtner, Kemper and Topper2012) allows audience input with mobile devices to create visual ‘thought clouds’ used in a score-like way by performers, but also to allow audience members to communicate with each other and to generate sound themselves. Because playing an acoustic instrument is generally a ‘hands-and-eyes-busy’ activity, a common architecture for real-time scores employs a single notator who generates the score for the performers, as in Justin Yang’s piece Webworks and as in pieces made with the eScore system (McClelland and Alcorn Reference McClelland and Alcorn2008). An algorithm might also serve as the real-time notator for acoustic instrument performance (Didkovsky Reference Didkovsky2010; Eigenfeldt and Pasquier Reference Eigenfeldt and Pasquier2012). Algorithmic generation of notation may be influenced by performer activity such as in Nick Didovsky’s (2002) Zero Waste (Hajdu and Didkovsky Reference Hajdu and Didkovsky2009), using various types of sensor data (e.g. Winkler Reference Winkler2010).
Scoring systems that can be viewed as ‘anticipatory’ include MIMI (François, Chew and Thurmond Reference François, Chew and Thurmond2007) with a scrolling score displaying performance-time-generated notation from an algorithm using input material from a human performer. Barri (Reference Barri2009) developed Versum, which is a 3D navigational score system where a notated space represents possible futures. Versum also allows the solo performer to be playing sounds (by navigating over graphical icons) while at the same time entering notational data that might be rendered into sound later.
We are particularly interested in the real-time score as a multi-way communication channel between improvising performers. When multiple performers engage in real-time notation, the score establishes relationships between participants (Small Reference Small1998; Harris Reference Harris2013) different from the relationships established through other types of real-time scores (single central notator, instrumental performer-influenced algorithmic notator). The Anticipatory Score situates performers in a specific kind of relationship with each other as they collaborate to plan and guide the unfolding of the piece based on a combination of sonic and visual information from their fellow performers. The relationship between performers through notation impacts real-time collaborative group and performance dynamics such as mutual engagement, awareness, cooperation and consensus building. We want to understand the elements that influence this real-time notational mode of communication so that we can improve mediated support for compelling musical experiences.
2. The Anticipatory Score System
The Anticipatory Score system is based on a ‘scrolling’ score paradigm where performers can see notation corresponding to a temporal window that stretches from a point in the recent past to a point in the near-term future. The exact duration of the visible time window in either direction is one aspect of the system under investigation, but it typically extends in the neighbourhood of 5–10 seconds into the past, and 10–20 seconds into the future. As notation scrolls from future to past (right to left), it traverses the ‘now’ indicated by a fixed visible line in the score (see Figure 1). As notation passes the ‘now’ line it is rendered into audio via electronic synthesis and/or interpreted by a performer or both (depending upon experimental or creative goals).
Elements of the Anticipatory Score showing notation from two different notators, one having drawn several line contours and a circle of events labelled with ‘.0’, and the other having drawn a complementary contour (stippled). Notation is flowing right to left, and small ticks along the top and bottom of the score mark seconds. The drawing mode is selected from the radio buttons on the upper right, and drawing can be parameterised (in this case, with line thickness) using the slider to the left of the score.
The Anticipatory Score system is built on a platform of networked machines that function as both display and as input devices, typically tablet computers with touch-screen capabilities. Notation is entered directly on a display by any and/or all of the participants, and then all displays are updated so that the score portion of the display shows the same information to each of the participants. Sound can be generated by automatically mapping the notation to arbitrary controls for sound synthesis. The notation can also be interpreted by live performers.
The ‘anticipatory’ window of time where notation is entered blurs the distinction between composition and improvisation because:
-
a) notation is entered prior to being interpreted and/or rendered in sound (as in composition);
-
b) planning and temporal coordination across performers is done with explicit notational support (as in composition);
-
c) the temporal order and rate in which notation is entered need not correspond to the temporal order and rate in which it is later played – that is, notation is generated ‘outside time’ (as in composition);
-
d) musical decisions are made during performance (as in improvisation); and
-
e) multiple participants engage in real-time mediated interaction (as in improvisation).
3. Awareness and Communication
Improvisation demands significant cognitive effort from the individual musicians, such as requiring them to make sense of the direction of the whole performance, to take into account what has been played already, and to consider the moment-to-moment contingencies to decide what to do next. Group or ensemble improvisations require the individual musicians to coordinate and communicate with each other. This is generally done non-verbally and often indirectly, with communication taking place through the sound of the music itself and with the support of body language. In short, being able to improvise on stage with others requires a great deal of experience and practice, and a wide variety of specialised communication skills.
Different kinds of dynamic scores support mutual awareness in different ways (Sudarshan and Wyse Reference Sudarshan and Wyse2011). To explore and push the boundaries of some of the challenges surrounding music improvisation, we posed the following questions to guide the development of our prototype system:
-
∙ How can we provide musicians with information to anticipate upcoming musical events during a musical performance in new ways?
-
∙ How can we capture and display creative decisions made in real time during a performance?
-
∙ How can performers coordinate ensemble activities that involve multiple forms of simultaneous communication including real-time score events?
Bryan-Kinns (Reference Bryan-Kinns2012) identifies several design features for supporting mutual engagement in music systems: mutual awareness of action, annotation, shared and consistent representations, mutual modifiability and spatial organisation. These features were identified by experiments with social musical systems approached from the perspective of computer-supported collaborative work (CSCW). The difference between what is now sometimes called computer-supported collaborative play (CSCP) (Wadley, Gibbs, Hew and Graham Reference Wadley, Gibbs, Hew and Graham2003) and CSCW, is the former’s emphasis on experience as opposed to outcomes. Common to both areas is a focus on ‘awareness’ of participant activities and intentions in a collaborative activity.
To position the Anticipatory Score with respect to the design features for social music-making identified by Bryan-Kinns, mutual awareness and the identifiability of authorship are addressed by distinct colour-coding of the notations from different notators. This issue is particularly important with electroacoustic music where performers draw on an open-ended palette of sounds rather than being identified with a single instrument timbre. The question of ‘who makes what sound?’ and its role in awareness in electroacoustic improvisation was addressed by Merritt, Kow, Ng, McGee and Wyse (Reference Merritt, Kow, Ng, McGee and Wyse2010), though the emphasis was on visualising the immediate past rather than the impending future as we are focusing on here.
Another factor contributing to mutual awareness in the Anticipatory Score is that participants see each other’s notations as they are entered. Even contours that may take several seconds to enter are delivered to coparticipants as they unfold rather than only after they are complete. This new graphical information appearing on the screen provides shared awareness and feedback that is easy to visually attend to over the other unchanging elements (Dourish and Bellotti Reference Dourish and Bellotti1992).
Bryan-Kinns’ ‘shared and consistent representation’ is implemented by drawing on a small set of icons for representing events and gestures and by displaying the exact same score data for each participant. Bryan-Kinns’ ‘annotation’ feature refers to an extra channel of communication outside the ‘product’ (in this case the scored notation). The Anticipatory Score system does support arbitrary textual notations (e.g. ‘begin crescendo’) that can be placed anywhere in the score. They were designed as notational elements themselves, but could be used as an ‘extra channel’ of non-musical communication between performers as well. Spatial organisation refers in part to providing both some public (shared) and private (not shared) space on the display where participants work (Fencott and Bryan-Kinns Reference Fencott and Bryan-Kinns2010), which is incorporated in the design of the Anticipatory Score by making only the score itself public, not the sections of the display where sounds and drawing modes are selected and parameterised. The concept of ‘mutual modifiability’, where participants can alter each other’s contributions, is supported through a ‘copy and paste’ operation that works on gestures contributed by any coperformer and is a kind of direct collaborative engagement through the notation system that exploits and communicates mutual awareness.
3.1. Notation and timeframes
When notation is composed prior to performance, composers work ‘out of time’ in the sense that they are free to notate performance gestures at a different rate and in a different order from which they are sonified at performance time. For example, notators can spend minutes to notate 1 second of sound, or 1 second to notate minutes of sound. In the design and development of the Anticipatory Score system, we have found it useful to distinguish between three distinct timeframes:
-
a) Composition time – the ‘real time’ lived by the composer as notation is entered,
-
b) Score time – the point in the score where the notation is entered (represented in the horizontal dimension),
-
c) Execution time – the time in which notation is sonified (by performers or synthesisers)
We have explored two modes of notation in the Anticipatory Score system and a few additional mechanisms to manage the relationship between composition time and execution time. One mode is for events (where the ‘release’ phase of the sound is triggered immediately following the ‘start’ phase, and represented as circles in Figure 2). The other mode is for gestures (with an extended duration between the ‘start’ and ‘release’ phase of the sounds, and represented as a curve in Figure 2). When the notator draws on the score in the event mode, events are generated at 50 ms intervals as long as the drawing action continues (as long as the notator’s finger is in contact with the device). Of course, since the drawing gesture can move across the scrolling score in any direction (or not move at all) the 50 ms intervals in composition time are entered at whatever score times the drawing gesture happens to be traversing. The notator could, for example, draw a circle of events on the score (as shown in Figure 1). As the circle crosses the ‘now’ line at execution time, events along the top half and the bottom half of the circle would overlap temporally playing in a different order from which they were entered with a circular drawing trajectory at composition time.
Buttons (lock time, +time, and 0 time) determine the compose-time input mode. Notated events (circles) and contours (curves) are scrolling right to left toward the vertical ‘now’ line seen on the left of the figure. Everything to the right of the ‘now’ line is the ‘anticipatory window’ where notation can be entered. Shown here is the ‘lock time’ mode where the notator has positioned a curser at a fixed interval from the now line where the input gesture appears as it is drawn. By restricting the input to a single horizontal screen location over the scrolling score, a gesture is created ‘in time’ (albeit in advance of when it will be rendered), so that it is sonified with the same temporal structure with which it is drawn.
The second mode is for contours. The contour notation appears as an extended curve on the score, and is used to represent a parametric change in a sound over its duration. For gestures, the liberty to move forward and backward in score time during composition is restricted because the resulting gesture could have multiple values for a particular point in score time and thus make interpretation of the gesture as parameter movement difficult. In order to free the composer from having to carefully avoid multiply traversing a single point in score time while drawing a contour, the gesture is automatically restricted to unfolding in increasing score time. Individual gestures are thus always a singled-valued function of score time. Different gestures can overlap in time to create a polyphony of different sounds.
A common complaint from early users of the system was that it was difficult to notate a pattern that would unfold in execution time with exactly the same temporal structure used to notate it in composition time. That is, notators did not always appreciate the ability to move forward and backward in score time. Another way to express this is that notators wanted ‘composition time’ to correspond (albeit with a delay) to ‘execution time’. For a gesture to be notated on a scrolling score and be rendered with the same temporal structure with which it was notated, the drawing gesture needs to move over the score at the same rate the score crosses the ‘now’ line. In terms of the scrolling score of the system, this means that the gesture must emerge from a single horizontal position on the screen. This was simple to implement with a ‘lock time’ mechanism that the composer can select for either gesture or event entry (Figure 2).
There are two other time-constraint mechanisms (shown as buttons in Figure 2). One is ‘+time’ which allows drawing only in the positive direction in score time (always enforced for gesture mode as described above, but optional for event mode). The other mechanism is ‘0 time’ which is exactly the same as a ‘lock time’ with the cursor positioned at the now line. It is used for ‘playing live’, rendering sound as the input gesture is made.
3.2. Implementation
The reference version of the Anticipatory Score has been developed entirely with HTML5 technologies and derives much of its networking flexibility from being so (Canning Reference Canning2012; Wyse and Subramanian Reference Wyse and Subramanian2013). Code is written in Javascript using the node.js library on the server, and the Web Audio API on the clients. The system runs in a browser so that anybody with an Internet connection can join a session simply by navigating to a URL. The browser-based clients send notated score data to the server using JSON. The server is essentially a simple ‘chatroom’ that receives notation data from individual clients and distributes the information back to all other participating clients.
Although there is a delay associated with sending and receiving notation data through a remote server, the anticipatory window structure makes the delay irrelevant. Notation may show up on one participant’s screen 250 ms after it was entered by another, but this is a short time relative to the length of the anticipatory window, so participants still have time to respond with locally generated notation that can be precisely synchronised with network-generated notation before sound is rendered locally. Thus the system is both a ‘latency accepting approach’ (Renaud, Carôt and Rebelo Reference Renaud, Carôt and Rebelo2007) to network music, but at the same time allows for the accurate synchronisation of sound between participants who might be geographically distributed. Furthermore, when the Anticipatory Score system is run with collocated participants, the server can also be on a local area network so that delays in displaying notation sent between ensemble members are negligible.
Only ‘control data’ (notation) is exchanged between clients. All audio is synthesised locally based on the notation. We have experimented with using instrumental performers to sonify notation (as described below), but the most natural way to render sound is automatically through synthesis. In this case, the device (e.g. tablet computer) simultaneously functions as a display, a notation medium and as an instrumental interface. There is no conflict then between hands-and-eyes-busy instrumental control and notation because there is no separate instrumental interface. The interface for entering notation is identified with instrumental control.
The system is running at animatedsoundworks.com:8020, and has been tested with up to 15 simultaneous participants. All client, server and synthesis code is available as open source (Wyse Reference Wyse2014).
4. Anticipatory Scoring with Acoustic Instruments
One realisation of the Anticipatory Score we explored was specifically developed for an acoustic guitar ensemble. The network and collaborative scoring strategy were similar to that described above, but instead of synthesising sound when notation crossed the ‘now’ line, instrumentalists responded with performative gesture (most of which made sound). Although the sounds were produced by traditional note-oriented instruments, and the notation referred to traditional musical qualities, the specifications allowed for a wide berth of interpretation, and ensemble members flowed in and out of different listening and synchronisation strategies so that the result had a textural rather than melodic or harmonic structure.
What characterised this exploration of the Anticipatory Score was the inclusion of instruments that were ‘hands busy’. Instrumentalists could not play and notate at the same time. To maintain the goal of exploring notation-based collaboration and novel listening strategies for the piece, performers alternated between playing and performing (which became a theatrical element as well), and at least two performers notated at any given time (Figure 3).
A close-up of two out of six ensemble members working with the shared graphical score – one playing while the other is notating.
For this version of the system, the score was broken into six ‘tracks’, each corresponding to pseudo-traditional musical parameters such as pitch centre and distribution, tempo and dynamics, and improvisatory solo/accompaniment role indications (Figure 4).
A version of the Anticipatory Score designed for use with acoustic instrument performers. Different performance parameters are notated on separate tracks.
What we found from these explorations and follow-up conversations with the musicians was that:
-
a) time sharing (between playing and notating) might work for an individual piece, but is not a generally satisfying musical paradigm;
-
b) notators (who were untrained) had a tendency to compose without necessarily listening to what was happening sonically;
-
c) notators were able learn to coordinate their notation with other notators; and
-
d) complexity was the biggest challenge and manifest itself in several ways:
-
i. notating multiple parameters simultaneously is difficult and stressful; and
-
ii. integrating multiple ‘tracks’ of parametric information is a challenge for players.
-
The Anticipatory Score system can be used with ‘hands-busy’ acoustic instruments in this way, but only by separating notation from instrumental performance. For this reason, we consider this system to be ‘electroacoustically native’, most naturally used when sound is synthesised under parametric control by signals generated from the notation as the score scrolls through time. In this case, the interaction device functions both as a display for the collaborative score and as an instrumental interface translating a performer’s physical gestures into sound.
The Anticipatory Score system can also be used in ways that have been developed in other dynamic score paradigms. For example, the ‘just in time’ scrolling display window could be notated algorithmically, or used as a space for audience members to collaborate on creating scores for performers. However, the system seems to distinguish itself most clearly from other systems when its users are responsible for generating both the notation as well as the sound through synthesis because that is when it most clearly demands a two-way sonic and visual engagement between participants.
6. Conclusion
We have been exploring the creative possibilities of the Anticipatory Score system, which brings together elements of composition (notation and pre-performance musical decision-making) with elements of improvisation (collaboration in performance-time musical decision-making). Notation becomes a multi-way mechanism of communication between performers that can play a role in planning and coordinating musical activity while being responsive to the real-time sonic unfolding of a performance.
Some earlier work in dynamic scores have explored the musical possibilities of animated score elements, and others have developed techniques for incorporating audiences, performers and/or algorithms into the score-generation process. The distinctive characteristics of the Anticipatory Score system, with its emphasis on performer generation of ‘just-in-time’ notation during performance, focuses attention on musical group dynamics such as mutual awareness, indirect communication, and planning in an improvisatory context. Perhaps notation, after being conceptualised as the soundless abstraction of composition for so many years, can be redeemed to take a place alongside sound as a vital real-time communicative component of musical experience.
