3.1. Complex systems

While distinctions between performer, instrument and environment are blurred within interactive systems, the notion of feedback remains integral. The computer – whether it is considered an instrument or performer – is expected to make decisions regarding the input it receives, decisions that result in a variable sonic output. Such decisions are typical of improvisational participants; as Casal suggests, ‘improvisation happens in an environment full of snap judgments, where previous experience, cultural heritage, and current information acquired through listening all help enable the improvisor to make decisions quickly’ (Casal and Morelli Reference Casal and Morelli2007: 3). If the performance ecosystem expands beyond a single performer and instrument, to modelling the complex interactions of a collective – Lewis’ Voyager, for example – the nature of the model will be crucial to its success. Earlier interactive systems, such as those by Lewis and Chadabe, relied upon constrained random procedures (Winkler Reference Winkler1995), a limitation that I discuss more fully elsewhere (Eigenfeldt Reference Eigenfeldt2007).

Newer systems have explored complex systems within performance. Casal has chosen to use elements of Artificial Life (A-Life) and Music Information Retrieval (MIR) within his system Frank. Impett (Reference Impett2001) has implemented a swarm system, which he influences in performance with his extended trumpet. Swarm systems also have been musically realised by Blackwell and Young (Reference Blackwell and Young2004). Young (Reference Young2008) and Bown and Lexer (Reference Bown and Lexer2006) have implemented neural networks that learn during performance; A-Life strategies have been employed by composers, including Biles (Reference Biles1994), Beyls (Reference Beyls2009) and Weinberg, Godfrey, Rae and Rhoads (Reference Weinberg, Godfrey, Rae and Rhoads2008). Designing complex systems and influencing those systems in performance – thereby designing the interaction as well – moves beyond the improvisational to the compositional; however, the nature of the complexity is a defining aspect. Within the above-mentioned examples, the complexity of these systems is organised (Weaver Reference Weaver1948) – as it is achieved through the interaction of its parts (and has the potential for emergent properties) – whereas the complexity of earlier constrained random systems’ is disorganised, as it is generated (and can be understood) by statistical models (i.e. random probabilities). As Impett and others have demonstrated, controlling an organised complex system in performance is feasible.

While the individual elements within such systems may behave in complex and unpredictable ways, their macro-level interactions are governed by a defining principle. In designing such interactions, we are thus embracing Di Scipio's reformulation of interactivity as ‘a by-product of carefully planned-out interdependencies among system components’ (Di Scipio Reference Di Scipio2003: 27). Such complexities of interaction emerge from free improvising human ensembles – however, the organisation is collective; if organisational methods are defined by an individual, it is an act of composition. Therefore, if singular (as opposed to collective) and significant (as opposed to stochastic) control over complexity can exist within live performance, this new performance ecosystem is compositional, and can be considered real-time composition.

4.1. Multi-agents

Rather than model human musicians, some systems have modelled the interactions between musicians: multi-agent systems have provided some exciting developments in this regard. Intelligent agents are elements of code (programs or patches) that operate without direct user interaction (they are autonomous), interact with one another (they are social), interact with their environment (they are reactive), and make decisions as to when they should operate, and what they should do (they are proactive) (Wooldridge Reference Wooldridge2009). Since these are also attributes required of musicians in improvisational settings, the use of agents to emulate human–performer interaction has proven to be a fertile field of research. Whalley (Reference Whalley2009) gives an overview of the current state of software agents in music and sound art.

4.2. Artificial Life

Musical agents can exist within a variety of contexts: one that relates to performance ecosystems and offers potentially exciting new models is Artificial Life. A-Life is a discipline that studies natural living systems by simulating some of their biological aspects on computers (Langton Reference Langton1997), and has subsequently fostered the development of a pool of research tools for studying complexity, including modelling tools for interacting musical agents (Miranda and Todd Reference Miranda and Todd2003). See Whitelaw (Reference Whitelaw2004) for an excellent discussion on art and artificial life.

4.3. Sonic ecosystems

Bown (Reference Bown2009a) discusses failures of the traditional A-Life paradigm of the interactive genetic algorithm for artistic creation, and points to new approaches based within social learning, cultural dynamics and niche construction that offer potential solutions (McCormack and Bown Reference McCormack and Bown2009). Rather than exerting direct control over evolving systems through aesthetic selection, Bown suggests that an ecosystem approach ‘might generate sonic works that continue to develop and transform indefinitely, but with consistent structural and aesthetic properties’. Instead of designing methods of selection, an ecosystem approach allows artists to ‘design complex generative dynamics in an exploratory manner’ – in effect, designing the environment and the way in which sound-making individuals interact, so as to produce interesting artistic results. The significant contribution of this approach is the interactive nature of the design – which allows artists to ‘tinker’ with the system until it produces desirable outcomes – as well as the fact that the entire system is conceptualised within its medium: a sonic ecosystem operates in the medium of sound rather than being a sonification of a process. RiverWave (McCormack and Bown Reference McCormack and Bown2009) is an example installation that demonstrates their concept of a sonic ecosystem.

Bown describes a system (2009b) which ‘animates the sounds of a man-made environment, establishing a digital wilderness into which these sounds are “released” and allowed to evolve interdependent relationships’. The analysed acoustic environment of the installation space, which includes the projected sounds of the agents, is treated as the virtual environment in which the agents exist. A continuous sonogram is made of the acoustic space, and agents attempt to inhibit low-energy regions, thereby creating an interesting dilemma: as the agents make sound at their occupied frequency region, they deplete the available resources at that region. It is the interaction of agents competing for fitness and available resources that creates the potential for emergence, and thus provides the musical evolution.

The sonic ecosystem (SE) is an exciting RTC model, as it allows for the design of a complex system that serves a musical purpose. Although interaction is not inherent to SEs, it is not excluded.

5.1. Gestural interactions and plans

One important aspect of composition is the detailed specification of the interaction between musical gestures. When serendipitous relationships occur between performers during improvisation, they can appear magical; within fixed compositions, such correlations are predetermined (although potentially still appearing as magical to the listener). These interactions can be incorporated into the design of RTC systems through the use of plans, an interesting capability of agents using the Belief-Desire-Intention model (Rao and Georgeff Reference Rao and Georgeff1995). This model expands on the social attribute of agents, in which they form beliefs about their environments, individual desires and goals, and plans to be executed. Each of these attributes are continually adjusted in performance (thereby offering further parallels between the virtual and the physical performer); however, the ability to share (and adjust) these plans moves beyond human capabilities.

Preliminary research has been undertaken to codify the gestural interactions within acousmatic music (Basanta and Eigenfeldt Reference Basanta and Eigenfeldt2010) and mixed-media works (O'Callaghan and Eigenfeldt Reference O'Callaghan and Eigenfeldt2010) in order to incorporate such potential relationships within future RTC systems.

5.2. A preliminary system: kinetic engine

The creation of a true RTC system has been an ongoing investigation by the author, and grew out of work in improvisational interactive performance systems. The first research project created Kinetic Engine (KE): its development, both technically and artistically, is discussed elsewhere (Eigenfeldt Reference Eigenfeldt2009c). However, some of the salient points can be reframed here within a performance ecosystem.

The first version (KE1) was an attempt to manage high-level decisions within software, generating rhythmic compositions in real-time. While successful in achieving these goals, it also removed the composer from the performance ecosystem, something with which I felt (at the time) somewhat uncomfortable. It was presented as an installation, producing continual short (three- to fifteen-minute) compositions; a great deal of effort was placed in logging the generated parameter settings (i.e. tempo, time signature, timbre, section durations, degrees of variation) of each composition, and fuzzy logic was used to avoid successive iterations from being too similar. Lastly, in an effort to maintain some interactivity, certain performance parameters could be influenced by the audience via sensors. As the first audience comprised computer music creators and developers,Footnote 2 I had assumed audience members would attempt to delicately influence the system, listening to the subtle shifts in relationships between agents; I was (perhaps naively) surprised that those audience members bold enough to interact with the system immediately tested the limits of interaction and control, varying parameter settings to their extreme, essentially making the system choke.

Having removed myself from the performance ecosystem, my role as passive observer allowed me to take copious notes on the successes and failures of each short composition, and audience reactions to them. Prior to the next day's presentation, I adjusted many of the parameter ranges so as to maximise success and minimise failures (which can be understood as uninteresting music). This was a curious delayed (or perhaps anticipatory) feedback, which, in the end, resulted in an artistically unsatisfactory performance ecosystem.

The next system, KE2, returned to an interactive model that placed the composer within the performance, using immediate feedback to guide the system at a high level by influencing agent interactions (see Eigenfeldt Reference Eigenfeldt2008b for a detailed description). However, KE2 was not a true RTC, but rather an intermediary step between interactive multi-voiced instrument and intelligent composition; the agents had no plans or desires, other than to interact in rhythmically interesting ways.

Both versions of the system raised a design issue concerning the balance between elements of surprise versus guaranteed success. As it was operating under a ruleset, the more defined this set became, the more likely it was to generate music I wanted to hear. At the same time, limiting this rule-set also limited the potential for genuine surprise.Footnote 3 During performance, KE2 was a consistent performer that could be relied upon to generate complex and musically interesting rhythms; however, it was not very versatile, and, while its details continued to surprise me, it never went anywhere unexpected.

My solution was to create a new system that allowed not only for performance interaction, but also an adaptive and dynamic ruleset through corpus-based recombinance. Kinetic Engine 3 (KE3; Eigenfeldt Reference Eigenfeldt2009b) used a genetic algorithm to continually evolve derived rhythmic phrases; another (Eigenfeldt and Pasquier Reference Eigenfeldt and Pasquier2010) used a steerable Markov model to generate harmonic progressions. Like all interactive systems, the design of the interaction – between agents, in this case – was a compositional decision; the selection of the corpus to be used was a further compositional act. Performance actions were limited to very high-level choices, such as when to change sections, and influencing agent interactions in global ways. Unfortunately, the amount of calculations required to continually generate an evolving population of individuals in real-time proved to be too demanding for performance, and KE3 was limited to a non-performance role, producing fixed works.

6.1. Coming Together: Beauty & Truth

The first Coming Together system is conceived as an autonomous negotiation between melodic agents, which, given no predefined material, converge towards a musically meaningful cumulative output, whose content is generated and determined by the actions and interactions of the agents themselves. The result is a continuous series of compositions (usually between 8 and 15 minutes each) which either end in success – marked by the instantiation of additional agents – or failure, in which the agents collectively acknowledge failure and start over.

An individual agent's desire is to generate a repeating melodic phrase that has a meaningful relationship to all other agent phrases. By musically meaningful, I mean harmonic, melodic and rhythmic associations that could have been produced by a human composer: Ritchie's value property (Ritchie Reference Ritchie2007). Agents move towards this collective goal by listening to one another's musical output – which can be MIDI data or audio – as well as messaging one another, altering their own musical output based upon formed beliefs as to other agent's states. The end result is a negotiated solution that demonstrates a unique melodic, harmonic and rhythmic environment that has been arrived at collectively.

Constraints are placed upon agent choices to engender reasonable outcomes. Although complete unpredictability may be desirable from a standpoint of A-Life research, the choice was made to use a sonic ecosystem approach in which the environment was designed to produce artistically satisfying results. While the design bears a resemblance to the first KE system, there is a great deal more intelligence in the interaction between the musical agents, resulting in a higher degree of novelty within the musical details; however, the composition's overall structure, a consequence of the converging agent desires, produces the clear notion of a single composition continually re-interpreted.

The performance ecosystem in Beauty & Truth is certainly challenged, as several components seem to be missing. Feedback, integral during the development and fine-tuning stage, is neglected during performance. The composition is set in motion, and its outcome cannot be influenced until the system succeeds or fails. While each composition will be different, the system does not take into account any environmental aspects, and will happily perform for an empty room, thereby ignoring Impett's model for interactive music as ‘music instantiated in real-time on the basis of local performance and environmental information’ (Impett Reference Impett2001: 108). By this account, one can question whether performance is actually involved, and whether this work is instead simply the sonification of a complex process.

An important design decision in Beauty & Truth was the deliberate avoidance of performance. Western notions of concert performance imply structures with a beginning, middle, and end, and a predictable duration. Installations, on the other hand, can be viewed as ongoing: generative installations – in which Beauty & Truth situates itself – allow for the continued revisitation by an audience, stepping in and out of the ecosystems, so as to listen to its diversification over time. I have found the installation model to often be more suitable to RTC, as it can then operate outside the bounded time constraints of the concert presentation; interestingly, other creators of A-Life systems have similarly chosen to present their work in such a fashion (McCormack Reference McCormack2001; McCormack and Bown Reference McCormack and Bown2009; Bown Reference Bown2009b).

6.2. Coming Together: Freesound

Rather than manipulate events, as Beauty and Truth does, Freesound uses soundscape recordings as its source material. Agents communicate their selections from the pre-analysed library of recordings taken from freesound.org by sending metatag data; agents also have the ability to analyse the spectrum of other agents’ audio. During performance, agents listen to the sonic environment, and select material from their unique database based upon their perception of the current conditions (which is formed from both spectral analysis and interpretation of the collective metatag data). At different points of the composition, selection methods vary.

An important contribution to RTC design in Freesound is the use of a defining structure that is generated at the beginning of the performance. Freesound is framed as a performance (rather than installation), and seeks to balance agent negotiation with predefined compositional structure without the necessity of user interaction to guide it. Certain aspects of the environment's evolution in time are predefined:

• • Prior to performance, the user indicates an overall duration from which four sections are generated algorithmically. Agents are aware of their current position within the structure, and behave differently depending upon the point in time.

• • The global evolution of certain parameters use preset tendency masks.

• • The overall increase in resonant filtering is a defining audio feature of the composition.

• • The initial selection of recordings is from freesound.org.

Freesound demonstrates characteristics of a sonic ecosystem in that the environment is carefully calculated, yet the interaction between its components remains complex and unpredictable. Musically, Freesound generates surprising and varied compositions. Ironically, the composer's role is limited to adjusting amplitude levels of the four agent's gestures, similar to the performative diffusion of fixed acousmatic works; however, unlike such works, the ecosystem is dynamic and mercurial, the content being negotiated by the agents. And while the performative actions cannot influence the direction of the work – the feedback is one-way – they certainly contribute to its success (or failure, if one neglects to listen and react appropriately).

6.3. Coming Together: Shoals

This RTC is composed of a population of autonomous agents that exist within a sonic ecosystem derived from real-time analysis of live audio. Agents search for food consisting of CataRT unit analyses (Schwarz Reference Schwarz2007), within a two-dimensional projection in a descriptor space of spectral flatness and spectral centroid. When an agent finds food, it broadcasts this information to the community; all agents within that agent's neighbourhood – a globally defined radius – can move towards the location in which the food was discovered. Agents become active when they find food, which is then consumed through granulation.

Individual agents are initialised with random synthesis parameters, but communicate these parameters to agents within their social network. Networks are formed initially through proximity (i.e. an agent's neighbourhood), but later through more advanced selection methods, which include food supply and total numbers of agents within the network. Agents within a social network converge their parameters, thereby creating unified grain streams. Separate gestures thus emerge through the self-organisation of the population.

The musical objective of Shoals is the continual generation of distinct gestures that are arrived at through self-organisation, which itself is generated through convergence of agent parameters using a social network. Agent interaction is dependent upon the performance environment and the existence of new audio input. Shoals has the ability to be configured over a network, with audio and agent parameter data gathered from geographically displaced locations.

Like the previous described RTCs, Shoals is a sonic ecosystem whose components have been carefully adjusted so as to produce musically interesting outcomes, while at the same time providing for novelty and surprise, even given the same environmental conditions. However, the importance of the environment to the sonic outcome has been greatly increased, in that live performance can directly influence the agent actions. Similarly, feedback is two-way, with performers able to react to the evolving timbres. The composer's role in the performance ecosystem is, like Beauty & Truth, non-interactive, but clearly evident within the design of the interactions themselves.

A question can be raised as to whether any composed improvisation could be considered real-time composition, if a main criterion is defining the complexity of interactions. I believe the answer to be yes, to the extent that the interactions are clearly demarcated. However, within an RTC, the limitations of agent behaviour can be clearly defined, as well as altered, during performance in ways that are not possible with human performers. The desire within RTC is to balance a fixity in compositional choices (inherent within the design of the system itself) with a fluidity in performance possibilities, and it does so through a new performance ecosystem in which composition is the main constituent.