2.1. Early scoring attempts

My early attempts at graphic scoring for telematic works (Whalley Reference Whalley2010, Reference Whalley2011), beyond deciding on participants and defining frequency, spectrum, timbres/‘instruments’ – and archetypes like rhythms, spectral manipulation, pitched information and so on – involved making hand-drawn graphic scores that could be reduced to one A3 page for participants to follow. At at a macro level these scores illustrated the sections of the work, timing only in minutes, a key centre for each section to allow any spectral or pitched information to be in agreement, volume and tempo. The scores included parts written on one line for each participant, indicating who was to play and when, to show the ongoing dialogue. Each part was also prescribed motives to play and subsequent development (1, 1a, for example), as well as the gesture that a motive might take. Players could then see all of the parts. Participants might subsequently add details to their own copy of scores, such as patch changes or details on motive transformation, to aid with their performance.

Logistically, the composer’s score would then be distributed to all participants, with perhaps more details of what was intended for each performer for their part on a separate sheet. Through rehearsals, players then improvised their parts within the structure and prescribed motives and gestures, adjusting their parts in relationship to others’ as performances were refined. This method allowed for composition at a macro level, and comprovisation (Dudas Reference Dudas2010) at a micro level. There would be variation in the detail between performances, and rendering was dependent on the ability of participants to improvise and adjust to others in real time in an ongoing dialogue. Different combinations of participants may then also result in varying renditions of the same score.

While exact note-based rhythmic timing in this system was non-specific, one lead performer could count in macro sections as each section approached its minute timing mark through participants’ network-connected video monitors. Timing within each score section would be approximated by participants using any form of digital stopwatch placed next to the score.

2.2. Scoring problems/research questions

While using these hand-drawn scores proved useful in earlier telematic works (Whalley Reference Whalley2010, Reference Whalley2011) they had limitations within the aesthetic approach adopted.

If people had suggestions to improve their part of the score and wanted to change it so others could see these, they needed to send alterations back to the composer to alter then redistribute the master score to everybody. Similarly, if alterations were to be made by the composer, the master score needed to be changed and sent again to all. A paper-based score may then go through several iterations before a final performance. A real-time solution to update scores based on multiple computers spread over different countries was required to allow multi-directional editing as part of an ongoing dialogue. While some shared online drawing programmes were experimented with to address this, the tools were not specific enough for the purpose intended, the method of entry in using them often cumbersome, and they were musically closed systems.

A second problem was to keep time, even if not entirely accurately, since the use of hand-drawn scores required participants to glance between a digital stopwatch and the score. In addition, timing was imprecise within sections as there was no means to follow changing tempo indications on hand-drawn scores. A metronome system was then needed for tempo that could also be altered, and a scrolling system needed to keep all participants located. Furthermore, the system needed to include a ‘conductor’ to count in both performances and rehearsal cues.

The hand-drawn score method used also assumed a directed dramaturgy approach, where one composer would act as an author, defining the macro structure of a piece and allowing alteration of its details by participants. A more flexible distributed system might allow for multiple authors, and in line with Weinberg’s (Reference Weinberg2005: 31–2) notions, accommodate a musical structure/composer-controlled approach (machine agents and/or humans filling prescribed musical or performance outcomes), and process-based approach exploring possibilities or fulfilling goals through collaboration or competition where experiences may differ between renditions.

Most importantly, while dialogue in the hand-drawn score system showing which participants should be conversing could be indicated, there were limited possibilities for the representation of affective communication within each part without using complex words or a simple two-dimensional matrix model. These would clutter the score and make details difficult to change. An efficient way to represent this affective dimension, extending prior work (Whalley Reference Whalley2009, Reference Whalley2011), then needed to be implemented.

A further problem was that hand-drawn scores were closed in that information within them could not be linked to other systems that might use the information in musical ways. An open system and common communication method was then needed to explore this possibility. Yet a representation system needed to be comparatively simple despite the complexity of information it contained.

Finally, the current hand-drawn scoring system, like an orchestral conductor’s score, was largely a way to coordinate participants. A means of rendering scores for both participants and audiences was then sought. Beyond being an artisitic choice that the technology facilitates, this may further involve audiences in the performer-to-audience experience. And it may also show something of the dramatic structure of works in a way that could be quickly grasped, allowing further audience engagement the narrative experience.

2.3. Electroacoustic music representation

Problems of electroacoustic sound representation are not unique to networked sound art. This is partly because electroacoustic music gestures can be independent of pitch yet also expand on traditional pitch-based gestures – and there are no common symbols for types of synthesis, space and effects. While methods of electroacoustic music representation have been proposed (Couprie Reference Couprie2004; Patton Reference Patton2007; Blackburn Reference Blackburn2011) based on spectomorphological understanding (Smalley Reference Smalley1997), there is currently no standardised method for this in the way that there is for pitch/duration music. Accordingly, net-based scoring representations, like graphic scores for acoustic music, are often idiosyncratic to their composers.

A second problem is that while electroacoustic music has common sound archetypes, it has few common ‘instruments’, where sound is associated with identified sources/makers. Indeed, participants might also compose their instrument as well as improvise their part (Dudas Reference Dudas2010). And ‘instrument’ or ‘sound object’ building constititutes a good deal of current networked music practice. Moreover, timbre in electroacoustic music is often spoken of in affective terms from a semiotic perspective, but this is rarely represented in graphical scoring systems.

What is often represented at a macro level in many net-based electroacoustic scoring systems then, are aspects of the basic the elements of sound making, such as frequency, tempo, archetype (rhythm, pitched material, spectral information, non-pitched sounds, timbral information), gestures and effects.

The challenge was then to represent some of this range of these inputs in a simple, efficient and integrated way – accepting that it was likely to be idiosyncratic to this system.

2.4. Screen scoring systems

Both Sudarshan and Wyse (Reference Sudarshun and Wyse2011) and Vickery (Reference Vickery2012) have recently reviewed computer-based systems, putting forward conceptual overviews. Usefully, Vickery (Reference Vickery2012: 131) notes that a screen score from a compositional perspective may be sequential, permutative, transformative or generative; that a performer’s role may be interpretive or immanent (read in the moment); and that scores may be segmented (form or content), scrolling or real time.

Continuing and extending the hand-drawn graphical scoring system used in prior works, GNMISS (Graphic Networked Music Interactive Scoring System) had to accommodate a primarily sequential paradigm once single or multiple participants had constructed the score from a creator’s perspective, but also allow for permutative and transformative elements in real time. The performers’ role was to remain largely interpretive, but could also be immanent if changes could be applied in real time. Again, once constructed, the GNMISS paradigm would be able to accommodate a scrolling score, but perhaps allow for segmented real-time changes with future developments.

3.1. Timing: scrolling and conducting

The starting point of the new scoring system was to address how it was to look, and how scrolling and tempo information would be addressed. While hand-drawn scores used a rectangle shape on an A3 page to represent all aspects of the score, an influence on the GNMISS rendition, although not its method, was Justin Yang’s use of a circle to represent works (Yang Reference Yang2010) in Webworks1. In GNMISS, the circle was adopted because it proved to be an efficient method of representing the structural relationship between affective parts in a space – illustrated later in this paper. Furthermore, parts requiring less detail could be given less space on the score, and vice versa, making it an efficient way to represent information.

In terms of timing, outside circles with one line per participant turn at a constant speed in clock time, so that noon is always current time and the noon line segments all parts (see Figure 4). Participants can then anticipate events approaching the line while they are playing. In contrast, the metronome in the centre counts in clicks per minute – no beat measures are included – and tempo can be manipulated independently of absolute time while the score is scrolling, similar to most sequencer packages. This allowed all participants to see others’ entries, and keep in tempo where rhythmic information (regular, asymmetrical or floating) needed to be observed.

A traditional metronome arm is used for the centre circle, which displays a flashing lower half-circle when the arm reaches each side, as participants found this the most efficient way to anticipate the tempo while playing. A conductor’s role in controlling timing is incorporated by allowing a ‘count in’ function, a ‘move to section’ function for rehearsals, and tempo changes which can be manipulated by an editor independently of clock time.

To synchronise all computers across the net for the timing and the sharing of all score information, Bencina’s OSC Groups server is used to allow multicast data output to be transmitted to all group members by any client attached to the group. All aspects of score information are then sent and received by clients in OSC format. This accepted and worked within the delays that networks impose with distance – approximately 165ms, for example, between Hamilton in New Zealand and Calgary in Canada.

3.2. Adding emotion

Beyond a formalist view, adding an affective element assumes that sound as language has affective/emotional elements (Juslin and Sloboda Reference Juslin and Sloboda2001), and also that this language is not self-contained but has semiotic qualities by implied associations with other communicative means. Accordingly, the first layer of the score draws on emotion theory research, and extends the argument put forward in theoretical work (Whalley Reference Whalley2009) on linking affective computing and sound art.

In a prior networked composition (Whalley Reference Whalley2011), the affective uses of sounds in a structural sense were mapped by the author at a macro score level rather than for individual parts, using Mehrabian’s (Reference Mehrabian1995) three-dimensional P-A-D model that covers i) pleasure/displeasure (positive and negative emotions), ii) arousal/non-arousal (activation and alertness) and iii) dominance-/submissiveness (control and influence emotion makes). This model is also used in some aspects of music informational retrieval research. However, the descriptors made it difficult to map onto a two-dimensional score and read in real time effectively.

Applying any emotional model to sound is of course fraught with conceptual concerns. Acknowledging these, there is no attempt to represent a universal view or categorisation of emotions in implementing an affective approach in the GNMISS system, because emotion is culturally embedded, filtered and expressed. Furthermore, it is accepted that different languages might have different emotional taxonomy or different words for emotions that are unique to a culture, and might differentiate or not differentiate between emotions that reflect particular cultural perspectives. GNMISS only attempts an English-speaking western and contemporary view of emotional ranges. And even within this selected gamut, it is accepted that models of emotion are contentious, despite many composite approaches now covering mixtures of valence, arousal and intensity.

Notwithstanding this, affective computing is an ongoing part of human–computer interaction research (Cambria, Hussain and Livingstone Reference Cambria, Hussain and Livingstone2012, for example), and there are some common models that are used for this. So to move forward pragmatically, Plutchik’s (Reference Plutchik2001) model of emotion that builds on Russell’s (Reference Russell1980) work was adopted in GNMISS. Plutchik (Reference Plutchik2001: 349) puts forward an evolutionary basis for emotion and a three-dimensional circumplex model that ‘describes the relations among emotion concepts, which are analogous to colours on a colour wheel’.

In the two-dimensional view of Plutchik’s diagram (Figure 1) based on word categorisations, Plutchik arranges emotions based on four pairs of opposites. The cones represent intensity, the eight sections the primary emotions, and the mixture between cones as they extend outward allows for blending the two adjacent emotions, facilitating more complexity.

Figure 1

Plutchik’s circumplex – two-dimensional and black and white version.

While the colours are not illustrated in the black and white version given in this paper (Figure 1), the colours attributed by Plutchik to the circumplex are also adopted in GNMISS. Again, such colour mapping is accepted as contentious and culturally embedded, although not without some tested support in video game theory (Joosten, van Lankveld, Spronck Reference Joosten, van Lankveld and Spronck2010). Similarly, the idea of mapping colour as a form of visual notation in sound (Poast Reference Poast2000, for example) or consonance/dissonance relationships (Alves Reference Alves2012) is one that has been extensively explored.

The pragmatic simplicity of adopting Plutchik’s model for affective communication in GNMISS, despite the acknowledged limitations of doing this, is that it provides a range of emotions to explore, a set of associated words, and a set of associated colours as the basis for an efficient means of communication once the map is read and understood by participants. The associations are then relatively quick to learn.

With the initial deployment of Plutchik’s model in GNMISS the two-dimensional diagram was reoriented in terms of x/y orientation – this change being made because feedback from initial tests with participants suggested it was a more musical way to think in terms of the x/y axis (Figure 2). Also, the initial system only used colours in the primary spokes of the cones, and has yet to venture into colour blends and possible words between the outer spokes of the wheel on the diagram, although players might explore these blends as part of their own performance inputs.

Figure 2

Rotated map from the GNMISS player.

The emotional range for each part/participant is mapped on a circle. This allows everyone to see the affective/dramatic structure of the work in mapped colours without the associated words, and the basis for dialogue between the different parts. In affective terms, this is analogous to revealing the structure of a dramatic play, with players, character parts, entries and exits, but mainly in implicit rather than explicit terms (see score example, Figure 4).

3.3. Key centres

An initial text layer added to scores gives key centres, if any, aligned to the macro structure of works. Rather than being used in a pitch/duration sense as the basis for tonal scales, this allows for any spectral parts/participants to align to the emotions indicated in a harmonious or inharmonious way, depending on the emotion indicated in each part. It also allows for instrumental participants to gravitate to different notes at different sections to work with or against the spectral information. It also allows for works to be tuned to the structure of an instrument if needed, an approach I had previously used in networked compositions (Whalley Reference Whalley2010, Reference Whalley2011). Key centre indicators are placed closely to the metronome so participants can follow both in near proximity (see Figure 4).

3.4. Motives development, frequency

The first text part above each prescribed emotion line allows for a motive to be indicated, along with motive variation or repetition, and frequency range (high, middle, low): for example, 1aH = development of the first motive played in high range of the instrument/sound object used. Text can also be used to indicate patch numbers for synthesis changes. These are given at the beginning of each emotion part displayed, and participants have the discretion of spreading their interpretation of them out or segmenting them as they see fit over the duration of the emotion part indicated. Using a combination of an affective structure approach and motive development, the macrostructure of works can be set by a composer or composers.

All this presupposes that selected comprovisors, once the score is established, will bring a possible set of potential sounds and gestures that might mesh as an ensemble in an integrated fashion to first rehearsals. The careful selection of participants and prescribing/negotiating roles is then vital to the process of rendering works.

3.5. Archetypes and gestures notation

A second text layer above each emotion line allows for composition/performance indicators that represent both musical archetype and gestures. The approach was influenced by Penderecki’s shorthand used in acoustic scores in works such as his Threnody to the Victims of Hiroshima (1960). For example, he used a black triangle to indicate playing the highest note possible on a string instrument, and developed a system to represent microtonal clusters for a string orchestra. Technically the influence for this came from the development of the Sonova sound representation font (Toresen and Hedman 2007; Thoresen Reference Thoresen2010) where this is used for the spectromorphological analysis of sound object based on an adaption of Pierre Schaeffer’s typomorphology. But unlike Thoresen’s work, which is a practical articulation and implementation of Schaeffer’s TARTYP classification of sound categories, the GNMISS font required something simpler and less descriptive to allow quick changes and greater participant flexibility, so it does not follow Schaeffer’s classifications.

Consequently GNMISS symbols cover only basic archetypes: texture, spectra, effect, pitched material (tonal, atonal, microtonal), gesture shape, rhythm shape (symmetrical, asymmetrical) and volume. To represent this using a font maker, a font collection of sixty-two symbols was created with common shapes for indicators within each archetype (see Figure 3). This included circles for effects, squares for textures, multiple lines for spectral information, triangles for volume, lines/dots for pitch and rhythmic gestures, and double lines for non-pitched shapes.

Figure 3

GNMISS symbols font.

In the font used (Figure 3), the top line covers basic effects: reverb, delay, filter, filter delay, flange/phase, distortion, granular and spectral, and includes pan and pause. The next two lines cover rhythmic and melodic gestures for single- or multiple-line participants, and the next line the shape and sense of direction for non-pitched gesture/sound-cluster participants, which can again also be used by single-line participants. The line below this indicates different types of texture and gesture within them, particularly for non-pitch participants. The second line from the bottom is for spectral players, showing how the spectra and spectral rhythm might work: static, rising, falling, static with regular rhythm, static with irregular rhythm, shape within outside outline, undulation internally. The final line is an abbreviation of some basic indications for volume changes.

The simplicity of this approach is twofold. Text characters can be combined to make directive sequences, in the same way that standard notational sentences are constructed. And they can also be used as a mnemonic for material that the performer may have worked through in rehearsals. Practically, separating out the frequency information in the prior line of the part results in the archetype/gesture line taking up much less space. In addition, when the number of symbols are restricted and grouped into categories, participants have a limited number of them to learn.

The integration of all parts of the score enables participants to see the dramatic structure of the work (Figure 4). And as with a standard notation system, they all use the same symbolic scheme as a way of understanding the detail of what each is contributing.

Figure 4

Sensai na Chikai (Whalley Reference Whalley2012a) rendered score example – black and white version.

4.1. Performer interaction

Using the first version of the GNMISS system, performers construct and improvise their parts by interpreting the information provided in the macro score. To do this, they may work in isolation from one another first using a standalone version of GNMISS that gives macro timings, the structure of the score, and details of their part. A comproviser, having navigated the emotional circumplex and associated words, then selects timbres that they think would reflect the affective range of the colour or words prescribed within the frequency range designated on the score and the sound makers they have or might build for a work. They interpret the motives sonically indicated in the text font, also guided by the archetype/gesture font detail. Separating the score in parts/layers allows different participants to concentrate on one or more archetypes: textural information or rhythm, for example (see Figure 4).

As noted previously, this system relies on the ability of participants to improvise, and their ability to make or adapt instruments or to manipulate sound objects to play their part. The process is an interpretative one and subjective. Two people may play the same part differently, and a single participant may play several versions over the course of several performances.

Participants also follow the development of motives prescribed in a compositional sense. If they are unfamiliar with improvising, as has been the experience of some classical players, a composer may further prescribe the details of a singular part with more specific gestures over time, and a participant may adapt these.

Score interpretation by players, as with the hand-drawn approach used previously, however, is not just undertaken in isolation. When all are brought together, some who may or may not have rehearsed their parts to various degrees, interpreting the parts becomes interactive as participants adjust to each others’ contribution. The process is an iterative one, and works evolve through the rehearsal process. So a score may have many collective interpretations, depending on what combination of participants has been assembled for each performance. The outcome also depends on the character each participant brings to their part, which may force several players to rethink their contributions. With the new system, there is less concern with timing, however, and the symbolic system is far more precise.

Also aiding performance, apart from a common score, is the use of visual cues whereby players can see each other through multichannel HD units. Performance feedback then involves sonic and visual gestural information as people interact in real time. Telepresence, relationship building and nuanced communication then become vital to the experience of making works.

4.2. Rendition and audience involvement

For performances, all scores are rendered and distributed through the Internet, a participant needing a network-connected computer and GNMISS client with OSC Groups link to join in. Each person can make adjustments to the size of the score at their node, and this does not influence its size at each of the other nodes. This allows adjustment of settings locally to aid player performance needs (see Figures 5 and 6).

Figure 5

GNMISS player version rehearsal and score adjustment window.

Figure 6

View of Sensai na Chikai score and vCom from the Hamilton node.

As all computers are co-ordinated through the Internet, multiple units can be added at each node to render the score. Scores can then become part of the visual performance when projected from a computer – onto a screen behind performers, the floor or ceiling, for example – apart from what a performer alone might see.

Through public projection of scores, the audience can see the macrostructure of works, where colours particularly lay out a dramatic overview and the dialogue players are following. This facilitates greater public engagement with the nature of the work, as audiences can also anticipate different entry and exit points in the score. They can also see the HD video parts, and local live part where performers interact and interpret the score. This provides a way of engaging wider audiences in electroacoutic music, and this is one means of addressing some of the concerns put forward by Landy (Reference Landy2009) regarding the need to create greater public engagement. Through the multidimensional telematic experience and the combination of experimental and traditional music practices (Whalley Reference Whalley2012c), the telematic art medium can then forge collective understanding that is unique to the idiom.

4.3. Rendering

Programmed largely in MAX/MSP, GNMISS is rendered in two versions: player and editor. The player is a receive-only version of GNMISS (see Figures 2 and 5 for the main windows), which loads score versions saved as text files that any editors can make. This is useful for audience-only scores, and for participants who may not need to make macro score editorial decisions. The version allows participants to see Plutchik’s matrix in the first instance (Figure 2), as can the audience, to understand the relationship between colours and word mapping used for the affective score. Once the score is loaded, the tempo changes over the duration of the work can also be seen (Figure 5). For rendering in real time, the circle score is brought to the front, or can take over the screen (see Figure 4 as an example).

The receive-only version interface, like the editor version, allows for standard rehearsal mechanisms such as ‘count in’ or ‘move to section’, as well as change tempo in real time to test how various sections might work at different speeds within a scrolling time that is constant. Any connected person can change these aspects for all scores, to aid with the rehearsal process (Figure 5).

The editor version allows for any participant to make or alter any aspect of the score as well as macro timing. In two main frames, it has an entry window for all score details (Figure 7) for rapid entry of parts. In addition, all frame, circle lines and circle background colours can be changed to suit the visual rendition of the score, to compensate for lighting arrangements in distributed venues (Figure 8). From tests, a black background seemed to work best for scores, as it highlights the affective parts of the score well.

Figure 7

Editor window of GNMISS for score details (black and white version).

Figure 8

Editor window of GNMISS for timing, tracks, length, display colours (black and white version).

Technically, GNMISS client programmes, either participant or editor versions, are distributed between telematic nodes, or between machines at the same node, so that multiple players can follow the score. Participants with edit versions of GNMISS can then make changes to the score details, and these update all other scores on attached machine, sent as OSC data. Scores can be saved at any point and reloaded, so that different versions can be kept.

4.4. Composing, dialogue, rehearsal

As GNMISS is both narrative- and dialogue-based, representing the affective structure of the work implicitly similar to how a dramatic play may represent things in words written explicity, a single creator might begin composing a work from a narrative idea. From this, the number of tracks/participants/actors can be defined, and the length of the works set (Figure 8), and the macro dialogue set by arranging the emotional narrative as a potential dialogue between participants. The system has been tested using up to eight parts, allowing for a wide-ranging dialogue-based ensembles.

Figure 9

Ian Whalley and Hannah Gilmour playing Sensai na Chikai at the Hamilton node.

Following this, it is possible to set the way parts might be further articulated from a sonic perspective – taking into account frequency ranges of ‘instruments’ to be included, the general sonic archetype that each part is to carry, and the way the gestural parts might be articulated. Timing at a macro level is also set in relationship to the dramatic structure of the work using a timeline that the metronome will follow (Figure 8).

When a score is completed, the conversation with participants can begin. From this conversation, players may make individual suggestions of alterations to their parts. Changes can be made by the composer as main editor of the system, or by performers directly in conversation with all players. This reinforces participatory reward in terms of identifying the contribution one makes – a vital part of the telematic experience (Bryan-Kinns and Fencott Reference Bryan-Kinns and Fencott2010).

However, the GNMISS system also allows for the group composition of scores, either at the same site, or at a number connected telematic sites, using the shared editor. Composers /players/instrument-makers might then build a work in a non-linear way as part of a creative dialogue, as well as make changes to scores as individual parts evolve through rehearsal processes. This reflects Rebelo’s (Reference Rebelo2009) notion of projected dramaturgy, allowing group or individual authorship and direction that is collectively shared. And it caters for Weinberg’s (Reference Weinberg2005: 32) notions of networked based process-centred approaches in the making of scores, and moving to structure-centred approaches in arriving at the musical outcome.

The system also allows the possibility of changes to be made to the score in real time, so that intervention in what has been previously decided on can altered as performances are in progress, such as deleting or adding information a part by performers or outside editors. But the extent of this has yet to be fully tested at this point, and would move the systems use in to aspects of distributed dramaturgy proposed by Rebelo (Reference Rebelo2009).

5.1. Sensai na Chikai (Whalley Reference Whalley2012a)

Two versions of this work were carried out – one with people I had worked with before to see how it improved on the hand-drawn approach (Figure 9), and one with a new participant to check how the new concepts might be adopted and realised quickly.

The Japanese titled Sensai na Chikai (Fragile Vows) was a sole authored work of twelve minutes for MUSICACOUSTICA 2012. Performers in Hamilton, New Zealand, included Ian Whalley (wind controller, foot controller, synthesiser programing) and Hannah Gilmour (keyboard controller, pad controller) triggering synthesis parts and traditional Maori instrument rhythm samples from Richard Nunns. In Beijing, China, Bruce Gremo played Shakulute and Shakuhachi (Figure 10).

Figure 10

Bruce Gremo at the Beijing node for Sensai na Chikai.

The process began by constructing a sketch of the score on a rectangular A3 sheet, but this was soon abandoned, as it was easier to rapidly enter the score in to the GNMISS editor. Players could then load the score and check their parts. In the initial dialogue in Hamilton, this proved a quick way to alter parts of the score as discussion and rehearsal progressed.

As in prior works, this one explored a conversational model in telematic music, realised within the broad shape, structure and motives prescribed by the composer. Different lines of the score emphasised different musical archetypes – rhythms, pitch material, sound objects, textures – but there was some overlap between these, allowing a layered approach to composition. Sonically, Tui (a New Zealand native bird) and Maori rhythm instruments from New Zealand were contrasted with the shakulute and shakuhachi played in Beijing, interceded with spectral and textural lines. From a dramatic perspective, the idea was to explore the changing nature and fragility of relationships as circumstances change – the musical dialogue between the three participants traversing an emotional and sonic gambit of shifting interests laid out in the affective scheme of the score (Figure 4).

Participant feedback suggested that it was relatively quick to learn the language used for the score and the system had technically solved prior timing problems, but participants wanted to add more detail for their parts on the initial score as mnemonics. We were able to achieve this through the ongoing dialogue leading up the performance, and direct entry of changes by individuals.

Of interest was the array of ways participants interpreted the emotions, timbrally and in sonic gestures, for each of their parts, and it was a view that could also change in real-time rehearsals as they adjusted their interpretations to others as part of the performance dialogue. Also found was that once in real-time performance mode, players often zoomed in on the score to show only one quarter of the circle, so distance from the score could be accommodated to clearly read the font symbols displayed.

Finally, the flexibility of results could vary in early rehearsals, as players came to terms with different symbols and attempted to follow the tempo indicator at the same time. Participants who could only produce one sound and gesture at once were often the best at staying close to the score, and others adjusted accordingly. With more rehearsals, renditions began to stabilise. Metronome timing seems more difficult to follow for textural and spectral players, and less effort was spent on timing issues so more time could be devoted to interpretation.

A second rendition involved playing Sensai Na Chikai (Walker and Whalley Reference Walker and Whalley2012) with only two players, and me working with a performer I had not worked with telematically before. In this rendition, I played all the electroacoustic lines live from Hamilton, New Zealand, and flautist Sally Walker played live to a local audience in Newcastle, Australia. The work was also broadcast/streamed to audiences in Singapore, Linz, Beijing and Hamilton in real time.

For this rendition, I provided the flautist with additional macro gestural details for her part, to help with improvisation. Again, this relied on having a very competent player who could also improvise.

This test demonstrated two things. First, the system was rapid to implement and use for rendering works. Within a five-day period, it required only three conjoint sessions. This involved a short technical discussion and rehearsal to implement/synchronise the score at both playing sites, together with a short performer discussion on the nature of the system and requirements for playing. A second session was a rehearsal of the work between players. The final meeting was the performance. Again, player preparation time was then essential and performers could rehearse to a stand-alone version of the GNMISS system. Second, the deployment also allowed both the beaming of the score to the Internet audience in addition to the projected score being made available at each physical location as part of the performance backdrop for audiences, enhancing audience engagement (see Figure 11).

Figure 11

Sally Walker and Ian Whalley performing Sensai na Chikai, vCom streaming view.

5.2. SymbolAct

Using GNMISS again directly for score input, I wrote SymbolAct (Whalley Reference Whalley2012b) – see Figure 12. This was performed between Hamilton (New Zealand) and Sydney (Australia). For this, I played spectral, rhythmic, and textural and pitch material, again using hardware controllers including a wind synthesiser, iPads and foot controller – all triggering software-based instruments from New Zealand. In Sydney, Ivan Zavada played the violin part in Sydney. The system again proved a remarkably robust way to realise works in a short time frame, bearing in mind that both participants were capable composers and players (Sound example 1).

Figure 12

SymbolAct score, black and white version.

Of interest in this test was that the work was rendered as part of the CyberArt festival of networked-based new media art, whose audience comprises non-specialists in electroacoustic music. The rendered visual performance and audio was captured and broadcast/streamed in real time to 14 physical audience nodes spread across multiple countries, as well as any audience who could log on to the Internet stream from a fast Internet connection, creating a worldwide audience.

Written notes for the work were provided on the Internet streaming page, these notes indicating how the approach taken was based on conversational roles to negotiate relationships in a sonic narrative, and how this was based on the composer’s platform GNMISS. The notes also indicated how participants were ideally meant to prepare and respond to the graphic flow of symbols for emotion, sound and gesture prescribed by the composer, but also that players altered their musical responses based on others’ inputs to the piece. The intention of new methods producing net-based interactive electroacoustic music and broadening the gestural/semiotic base of the electroacoustic medium was also indicated.

In this rendition, the score was beamed onto a wall at the Hamilton and Sydney nodes, and captured on streaming cameras (see Figure 13, top screen), so the score as well as performers became an integral part of the experience streamed live for the Internet audience. On-screen chat windows on the streaming page also allowed audience feedback to further enhance audience involvement, and performers could also see the comments with an eight-second delay on the Internet streamed version, allowing feedback as works progressed.

Figure 13

Ian Whalley and Ivan Zavada rehearsing SymbolAct, view from Hamilton node.

Based on the informal feedback of audience members of using the score as part of the experience, it was apparent that the colour system used quickly communicated a sense of the macro structure of the work and its individual parts. Secondly, the colour system helped the audience see where and how different parts interacted and the timing of these with the use of circle rotation and the central metronome. Finally, the individual symbols used by performers on their lines were generally intriguing to audience members, but interpretation of them sonically tended to be enhanced if one had some experience of playing a musical instrument.

6.1. Scope

The GNMISS system initially set out to address practical timing problems in coordinating network-distributed participants following a score, and implementing an Internet2 composition method established in prior works (Whalley Reference Whalley2009, Reference Whalley2012c). Addressing these issues resulted in controlling scrolling in absolute time, metronome independence, and OSC group multicast synchronisation. Following on from this, it implemented an affective way to approach timbre and gesture that could also be used as a structural dramatic device rendered in a conversational manner between participants, with further detail provided in a text-based shorthand to represent frequency and motive development, sound archetypes and gestures.

One objective in the development of the system has been to reduce complex ideas to a simple representation in a practical and robust way, almost to act as a mnemonic for participants, yet retain a good deal of flexibility of deployment for people and machine players to yield very complex outcomes. It is accepted that the danger of simplifying the interface both results in leeway for interpretation, and is based on broad assumptions particularly in representing emotion and sonic symbols. Like any language system, however, communicative value is the primary concern here. GNMISS provides one approach to a common language to undertake Internet2 sound art.

Enhancing telematic sound art, participants/composers can build works in a ‘coordinate and cooperate paradigm’ that is suited to Internet2. Reflecting Follmer’s (Reference Follmer2005) ideas for NMP again, the works here were partly realised through hybrid instruments and collective decisions as to aspects of form and/or content through dialogue between participants. Developing the works through rehearsals was also a significant part of the process.

Apart from aspects useful to players, the usefulness of GNMISS is in providing a means to enhance the communication of electroacoustic telematic works to audiences through the public projection of scores during performance, improving multi-sensory involvement. The GNMISS system, then, not only mediates in terms of sonic notation shared between composers(s) and performers (s), but also serves as mediation between them and the audience seeing/listening to and perhaps participating in a performance. Used in combination with a layered approach to sonic language in composition and production (see Whalley Reference Whalley2012c) this may point to Internet2 production and broadcast/streaming providing a means to bridge electroacoustic music practices with wider audiences.

6.2. Limitations, on-going work, speculations

The current tests were largely sequential from a composer’s perspective, interpretive from a performer’s perspective, and scrolling from a score perspective (Vickery Reference Vickery2012: 131). While the score can be constructed and synchronised by groups or individuals, artistic tests on adding or deleting aspects of the score as part of real-time performance need to be conducted, allowing for an immanent response from performers. Furthermore, changes to the macro structure of the score as part of real-time performance need testing artistically, thus expanding the scope of the GNMISS application as a screen-scoring device, which may require further system development.

The vocabulary of font symbols for micro gestures needs extending within each archetype to accommodate greater detail. The initial set used only a limited part of the alphanumeric keyboard, and more symbols may provide the ability to make more complex gestural sentences.

While intelligent multi-agent machine input has been used extensively in my prior work (Whalley Reference Whalley2009, Reference Whalley2012c), the technology needs to be integrated to link with to the dramatic/affective structure of the system here. The open score approach used, where information is communicated through OSC data, allows for intelligent-agent systems to join in with and adapt to other participants. And machine agents may develop their own interpretation of the affective scheme in GNMISS to enhance the conversational approach adopted between human/machine participants as they follow a part on the score.

From a reception perspective, further tests are also needed on audience responses to the score and sonic interpretation, both from electroacoustic initiates’ perspectives and general audience perspective. This may provide a means to examine common agreements between the affective scheme put forward, and the common ways that they may be interpreted electroacoustically. Internet streaming of works, incorporating an audience feedback system, facilitates this.

Finally, it would be useful to do more testing from a production perspective on a broader range of digital instruments that might be better suited to networked music (Braasch Reference Braasch2009) but that can also interpret the affective scheme, and to develop computer-based behavioural objects (Bown, Eldridge and McCormack Reference Bown, Eldridge and McCormack2009) as sound makers that can construct their own sonic interpretations of the affective map used.