2. ARCHITECTURAL CONSIDERATIONS

Our experience can be illustrated by a simple demonstrative objective: to take a standard flocking model and visualise and sonify it across a large array of lights and speakers. The original ‘boids’ model of flock and swarm behaviour showed how three rules of local interaction – separation, alignment and cohesion – produce complex emergent patterns exhibited naturally in some bird and fish species (Reynolds Reference Reynolds1987) (Figure 2). There are different ways you might render a flock of boids on a static system of lights or speakers, the most obvious being to illuminate any light or play sound through a speaker that is located near a boid (assuming that the boids model is superimposed onto the spatial configuration of the lights and speakers). The sound and light might further display specific aspects of the boid’s behaviour such as its speed (e.g., faster boids are redder in hue or brighter in tone). In a typical multichannel approach, focusing only on audio for the moment, one might use a real-time audio development environment to run the boids model and calculate the audio for each speaker.

Figure 2.

A two-dimensional flocking simulation (Reynolds Reference Reynolds1987), where each nearby ‘boid’ is an agent that moves according to Newtonian forces with three basic flocking principles: align velocity with nearby, keep distance from nearby boids and move towards centre of nearby boids. This model is an example of complex systems dynamics, with rich global patterns emerging from these simple low-level rules of interaction.

By contrast, imagine that each of your speakers does not just receive audio from a central computer but also incorporates a MIDI (or open sound control (OSC)) controlled synthesizer or sampler, and you can only communicate to these devices via MIDI (or OSC). This is essentially the technical constraint of a distributed IoT system – we cannot stream separate real-time audio signals to each speaker. We can, however, pre-program each synthesiser to respond to incoming messages to effect real-time control of multiple audio devices.

The boids example is interesting from a distributed computing point of view in highlighting some technical constraints. First, as a deterministic complex system its state may be determined by any number of steps into the future, but it cannot be computed without running through all of the steps in the model. This means that if each device were to run the whole model to save bandwidth, then every device must remain synchronised, and if any device needs to catch up, it must either run the required updates in the time given or receive an update of the entire system state over the network in sufficient time. It is also a model that requires a lot of data to represent its state compared with other models that produce rich patterns (such as a composite of sinewaves), and sharing that data across a network at a high framerate imposes a potential bandwidth constraint (though much lower than the bandwidth of streaming each speaker’s audio). Lastly, although the model is a multiagent model, we don’t benefit from having each boid’s state computed on a separate processor, because the boids generally must know the state of all other boids at each time step to compute their next step.

These factors show how a creative goal is tightly interdependent with a network architecture solution. An artist declares ‘I would like to render a simulated flock of birds above people’s heads’, and the constraints and affordances of the system dictate how they might go about achieving that. They may even drop the idea in place of something more simple, given the cost and associated risk of having to build framework code that works for this task.

Working through examples such as this, we are building a set of tools to reduce these risks, while retaining as many creative possibilities as we can. This is a practical task to do with creating tools for a new creative domain, but in such an open-ended space as this the solutions to such problems can contribute to the definition of the medium itself. In McLuhan’s terms, cited in this special issue’s editorial, media organise in layers, ‘the content of a medium is always another medium’ (McCluhan Reference McLuhan1964), and within the rich space of multiplicitous media systems there are multiple points where we (any artist setting out the architecture with which they work) can creatively redefine a medium.

We can consider the case of the flocking model further. Given that our hardware configuration includes a large number of devices capable of computation, we could seek to parallelise the model, but will always be constrained by the bandwidth and speed of delivery of data needed to compute the next step of the model. Failing that, we could consider running the entire flocking model on each device. This makes massively redundant use of CPU resources, with many CPUs calculating the same thing, but potentially minimises the reliance on sending data around over the network. If the CPUs are not busy doing something else, then there is only an extra energy cost in doing this. The problem with this approach is what happens if a device is interrupted and falls behind. Since we cannot jump ahead in the model – we have to calculate each consecutive step – the device would either need to catch up, or other devices would have to share the current state of the model, and do so quickly enough that the lagging device would then be back in step with the others. Evidently these computation-on-device approaches, if they are to be robust against interruptions, still intermittently pose similar bandwidth requirements as if we were sending the state every timestep.

We are left with a solution where a central computer computes the model and sends the state, at every timestep, to every other device, which interprets it to create light and sound. In our experience, this also happens to be conceptually and practically a manageable approach. The trade-off in this approach is a constraint on the number of boids in the model, given the bandwidth and latency of the network sent from the central computer. But importantly, what works best for a boids model is not necessarily a good general solution for networked devices. Other models might be parallelisable by allowing the division of computation between devices, while others might be computable ahead of time making it easier to keep multiple devices in sync, with each model requiring a specific approach to distribute the computation. (Another caveat here is that, naturally, we can also embrace a glitch approach, where what has been described here as failure – a message not getting through in time, a device falling behind – is recontextualised as something that might be creatively fruitful.)

Through this example we have hopefully illustrated how the medium of multiple networked devices presents a rich space of design decisions that interrelate with our aesthetic decisions. The core issue in our research is that the task of making these ideas is influenced by more detailed design decisions, not just the hardware but also the platform that is at hand to work with: all of these ideas require time and involve risk, as software development is notoriously unpredictable. Even if clearly tractable, they pull the creator away from creatively fluent work such as more open-ended exploration. How does one free the creator from the technical work of designing software and network architectures so as to offer a fluent and richly creative workspace?

Are there simplifying architectures or working paradigms that simplify the technical work of realising any particular creative goal, while managing to remain open to the diversity of possible things one might wish to do? A holistic view of creative practice might recognise that such systems need not necessarily cater for every possible creative idea, and may in fact be most usable if they actually offer constraints (Simonton Reference Simonton2003; Stokes and Fisher Reference Stokes and Fisher2005; Dalsgaard and Halskov Reference Dalsgaard and Halskov2010; Stokes and Gibbert Reference Stokes and Gibbert2020). From our experience we suggest as a starting point that, considering the following three different attitudes with which a creative practitioner may engage in a technology-based project, the third is a more realistic account of how we see creative practitioners adopting and responding to the technologies they have to hand. We entertain the way in which this attitude influences how we approach a general system design for networked devices:

1. 1. ‘I have a creative goal, find me a system that can realise it.’

2. 2. ‘Show me the conceivable limits of what this system can do and I will find my creative space.’

3. 3. ‘Show me some things that this system can do, and I will create something within that space.’

The first attitude represents a kind of artistic ideal, which assumes that the given goal is realisable. It is overtly goal-based. The second is a kind of platform ideal that may imply that the most creatively powerful platform is desirable. The third, we claim, may be considered closer to what artists actually do, familiarising themselves with certain narrow spaces of possibilities and adopting those that inspire them to develop ideas. Entertaining the third attitude points not to seeking the most versatile system designs but to seeking the designs that are most fruitful in opening up some space of creative possibilities. You can do anything in MaxMSP (or PD, or Supercollider), but what also makes these tools powerful in the hands of their most practised users is the specific possibilities they place close to hand.

3. DESIGN REQUIREMENTS

With this in mind, we draw on literature from creativity support tool and creativity studies, as well as our own experience, to list broad requirements for a successful creative process that we believe a well-designed system should exhibit, despite its constraints.

• Integration of modular compositional elements. The creator should be able to work in a modular way and be able to create their own modules, with modules that are appropriate to the domain (Resnick et al. Reference Resnick, Myers, Nakakoji, Shneiderman, Pausch, Selker and Eisenberg2005). In a DAW, tracks are common modules. A creator can make as many independent tracks as they like via direct manipulation of the DAW interface. They can copy and paste tracks from one project to another. The act of mixing tracks is effortless. In some DAWs, segments of MIDI or audio elements and signal chains of effects are also common modular elements that can be effortlessly arranged. However, integrating two entire project documents may be less trivial. For example, a live performer working with Ableton Live has no way of seamlessly mixing two Live Sets. In programming environments such as Processing or MaxMSP, sketches (Processing) or patches (MaxMSP) can be considered modules (as are classes and objects in text-based programming languages), but the integration of these is less trivial. A coder who has made three different animated visual sketches in Processing has no easy way of layering or mixing between them. In general, more open-ended and unconstrained tools can make it challenging to manipulate modular materials effortlessly.

• Reducing the cost to prototype. Rapidly prototyping a new technical idea (usually involving programming in some capacity) should be as painless as possible (e.g., see cognitive considerations in Blackwell and Green Reference Blackwell, Green and Carroll2003). If no out-of-the-box device exists for your needs, then the engineering work of building systems that do what you want to do is unavoidable, but can be made less burdensome with great programming languages, libraries and developer tools, as well as clear design patterns. Creative development environments employ many techniques to improve the relationship between engineering and doing creative work and sometimes blur that distinction entirely. For example, MaxMSP is designed in such a way as to allow you to real-time audition algorithms as you build them. This has the interesting effect that even in the midst of solving some technical design problem with a specific goal in mind, you may be more receptive and alert to new unforeseen creative possibilities as you go, including discovery by accident.

• Extensibility, evolvability and hackability. Once a system is made it should be easy to extend it to be used effectively in multiple works. This is a conspicuous challenge for many creative technology platforms, where it is common for an artist to create a system for a specific work or composition and later hack that system to create different variations of the work. Professional development environments provide powerful ‘refactoring’ tools that allow you to rearrange program designs without impacting functionality, but these are still hard to use and do not necessarily give insights into how a design should be refactored to improve it. Often it makes sense to extrapolate reusable libraries from one’s initial sketches, but this introduces the challenge of distinguishing and maintaining a working reusable API as well as the immediate project. This may not be feasible, even though it would in theory support greater extensibility.

• Direct manipulation. Interaction with the devices in the network via direct manipulation should be readily accessible in many different ways and at as many different stages as possible. Testing, probing and exploring creative spaces is arguably more easily done via immediate ‘direct manipulation’ interfaces and sliders, dials and other non-programming widgets (Shneiderman Reference Shneiderman1983, Reference Shneiderman2010). However, making direct manipulation interfaces interoperate with programming activity is not straightforward.

• Recall/all-in-one-place-ness. It should be possible to save all the work in a creative project so that it can easily be stored and recalled from a single archive. Assuming a fully virtual studio, a DAW typically enables you to save a composition in a single project folder, which can be perfectly recalled. Typically, all the audio files and presets are collected in the project folder. System configurations and third-party plug-ins may not be, but system alerts will generally help resolve anything that is not set up correctly. In an analogue or MIDI hardware studio, this is typically not something that can be perfectly automated in a system design, though typically USB connected devices, and MIDI devices more generally, can be centrally managed.

4. A CANDIDATE ARCHITECTURE

Building on our approach to implementing a flocking model, and considering how generally applicable this approach is to other circumstances, we have narrowed our focus on a high temporal granularity architecture that we feel facilitates a suitably broad range of designs while remaining conceptually simple and practically manageable. Essentially, this design is loosely similar to a MIDI studio, with a number of remote devices being orchestrated from a central scheduler device, typically a DAW. Our architecture requires programming the devices by defining a renderer, which can be thought of as akin to a programmable synthesiser, and then additionally creating a device sketch that loads the renderers and manages the job of interpreting incoming control data.

Renderers, like MIDI instruments, are reusable and versatile low-level modules that can be played directly as if they were an instrument. They are equivalent in scope to modular synthesiser oscillators: monophonic, state-based machines with a rudimentary interface that directly controls the oscillator parameters. In our work, we have built a generic renderer for both light and sound in which the user can choose between sample playback, granular sample playback and FM synthesis, and similarly different modes for pulsing or shimmering LED colours. They can set many parameters including the ability to loop sounds in time with a globally synchronised clock. More complex envelopes and sequences, if needed, require additional programming (as with a modular oscillator). This renderer is reusable in a vast number of contexts, with the aim of minimising the need for the creator to think about low-level control detail, while remaining hackable for users to program variations or start their own renderer from scratch.

In the device sketch, the user declares which renderer is being used, and the device then loads an instance of renderer for each output (light or speaker) that it has control over. The user then adds additional code to sequence the renderers. Typically, this would involve call-back functions that respond to incoming OSC broadcast messages. The function can interpret the OSC message to control each output, and when doing so can query the output’s physical position (stored in a spreadsheet), if any spatial calculation is necessary. For example, the sketch function may receive periodic updates of the state of an entire flocking model, then calculate the light and sound intensity for each of its outputs based on the relative positions of outputs and boids.

Unlike the renderer, the device sketch code is likely to be composition-specific, and we have developed an API with many functions that help keep the code relatively short. It is expected that while the renderer would be commonly reused, the sketch code is short enough and specific enough in purpose that it can be throwaway, and simple enough that specific functions can easily be copy-pasted to other compositions.

The final component is a central sequencer. We use Ableton Live, but anything that can send OSC can be used. Ableton’s MaxForLive extension enables sending custom OSC messages via small Max patches embedded in tracks within Ableton. This in turn taps into Ableton’s strong framework for managing automation lanes, manually recording in automation data and switching between automation control sources, in particular overriding and reinstating arrangement control rapidly. Having written a device sketch that responds to specific OSC parameters and setting up those parameters in Live, the creative process can fall back into a more familiar DAW workflow, arranging or live performing a reduced number of control parameters. Again, this does not preclude making interesting non-linear, interactive or emergent processes to control the system, alongside more traditional timelines.

5. CREATIVE CASE STUDY: THE MIND AT WORK

We now consider some more types of content we might create beyond flocking models. In previous papers we have attempted to define basic classes of relationships between output elements in a large network based on the kind of content we might wish to put on them (by output elements we mean specific lights or speakers – we do not say ‘device’ as in our architecture a device may control multiple of these output elements, and it is the output elements that we want to specifically orchestrate). In Bown, Fraietta, Ferguson and Tillman (Reference Bown, Fraietta, Ferguson and Tillman2020) we proposed that four useful concepts to describe types of content were forests, fields, compositions and substrates. In Bown and Ferguson (Reference Bown and Ferguson2018) we also discussed self-organised systems, which we present here as a fifth concept.

In forests (Figure 3a), outputs are all executing the same behaviour and there is no systematic variation between them. For example, the behaviour may take the form of randomly occurring clicks and timed pulses, and the overall effect would be a natural cacophony. Even without any coordination between devices, this blending of the same behaviours may take on new properties at scale: the random clicks may form a granular wash of sound. Technically forests are the simplest type of content to put onto a multiplicitous media system. They are also the most robust in the face of network failures since they do not rely on network interaction. Our system already provides programs with a global clock to synchronise to, and we can send well-synchronised broadcast messages to all devices, so it is possible to incorporate some coordination between devices (clock synch) without having to explicitly code this in our programs. Such designs would no longer be strict forests. (We acknowledge that this name may misrepresent real forests, which actually do involve much interaction!)

Figure 3.

(a) Forests run the same program without systematic relations between the elements. Any variation must therefore be random. (b) Fields run the same program with the additional possibility of introducing variation through spatial relations. (c) In compositions, the elements may be heterogeneous, running different programs which are related. (d) Substrates run programs that primarily serve to render images. (e) We can also consider self-organised behaviours where the elements interact with each other over the network.

In fields (Figure 3b), outputs are all executing the same behaviour, but they vary systematically according to their spatial relations; for example, through gradients and waves. Consider a colour gradient, fading from blue to red across one spatial dimension of a physical installation. To achieve this we write a single program, as with forests, but that program utilises the spatial position data of outputs (which is accessible in our system via a database of output locations for any given installation configuration, pre-loaded onto the devices).

In compositions (Figure 3c), outputs are assigned different roles that are coordinated in a top-down manner. In this case we may write different programs for different classes of output, orchestrating different behaviours that complement each other. In terms of implementation details, though, we may not need to write different programs, instead writing a single program that directs different behaviours in response to factors such as position or at random. Compositions are just forests or fields with more discrete heterogeneity of behaviour.

In substrates (Figure 3d), outputs work together to render an image. This involves a conceptual shift from forests and fields towards a virtual canvas where things can be drawn. Each output is not an independent entity but serves to render the virtual image. Outputs collectively form a substrate through which images are rendered. This is what the pixels that make up a screen do, and the speakers in a surround audio system. A single boid flies through the space of the installation, rendered as a blob of light and a localised sound. This mapping requires a creator to write a program for each device that knows how to render the given content, as well as a separate (centralised or distributed) sketch that provides the content for the device.

In self-organising systems (Figure 3e), a single program or multiple different programs are run on devices, including some inter-device communication that significantly influences the unfolding of events. For example, we may build a cellular automata model with each light or speaker updating its state periodically in response to the state of its nearest neighbours.

5.1. Content examples

We now present four types of content that were used in the installation artwork The Mind at Work (Figure 4), which launched at the Casula Powerhouse Arts Centre in Western Sydney in November 2020. These were chosen to constitute four movements in a continuously looping 14-minute composition. They were not chosen to neatly represent the different preceding content types, but derived from creative composerly thinking about an exploration of the aesthetic possibilities with the given materials and their spatial contiguity as networked devices.

Figure 4.

The Mind at Work installation at the Casula Powerhouse Art Centre, Western Sydney. The installation has 200 LED globes and 100 speakers. Available at: https://tinyurl.com/y599zaf2. Image courtesy Casula Powerhouse. Photographer: Chantel Bann.

The piece commences with scattered bursts of light and sound in the form of localised bubbles, small gestural flickers with no rhythmic pulse, typical of the brooding beginnings of a live improvisation (Movement 1). Gradually these bursts crescendo until a single shimmering form expands to fill the entire space, sounded as a flood of noise. From the structural flatness of the noise, small regular pulses start to emerge. One by one each light begins to enter into a regular flashing sequence, with each speaker doing the same with gentle plucks or pops, until there is a cacophony of flashes (Movement 2). Initially these seem chaotic, but they are actually only out of phase by a few milliseconds. Gradually the phase difference reduces and a spatially systematic rhythmic phase pattern comes into view. Pulses of light and sound start to sweep across the space. Eventually, the lights and sounds become tightly synchronised in a single periodic burst, before drifting off out of phase again, as layers of tiny bell chimes mix into the rhythmscape. Once again, the cacophony of rhythms smears into another blanket noisescape out of which gradually emerges smooth floating ripples of light and sound, relatively flat in structure except for its calm drifting around the space (Movement 3). The sonics become centred around gently shifting drones. The movement again gently evolves into a bright lava-lamp-like gradient of hues that slowly rotate through the structure, accompanied by droning sonics (Movement 4). The piece stops unexpectedly and suddenly and immediately loops to the beginning, as if the improvised bursts of Movement 1 had interrupted the calm.

One of the great challenges of creating this work lay in achieving seamless transitions between the phases described previously. Each of the phases themselves requires programming different system behaviour. With resource-limited devices, in so far as audio processes go, simply crossfading between concurrent model processes was not feasible. In much the same way multiple projects in a DAW cannot be seamlessly mixed, a method to switch between movements without a visible transition point was needed. To do this, an architecture was required that allowed specific models to be made that can be easily switched during the performance.

Movement 1: this was implemented by sending specific event data similar to a combination of MIDI note and continuous controller data. Each event would specify the coordinates of the centre of a shape plus its diameter, followed by a series of other parameters including what sound file to trigger and what colour. These events would be updated continuously so that gestural control of shape colour, size, position and sound could be achieved. Because we were able to send controller data over WiFi at 50fps, a smooth gestural control was achieved. It was also possible to directly record or draw in gestures into the controller computer running Ableton Live. Gestures were defined in automation lanes, and the value of each automation parameter was sent as part of an OSC message. Some global parameters were then configured to enable the transitions. A low frequency oscillator (LFO) modulation depth parameter was used to control how much a noise oscillator was fed into a ring modulation effect on the audio output. This LFO depth parameter was also mapped to a ‘sparkle’ value that dictated how much random shimmer the lights produced. At maximum noise level, the original samples were inaudible, creating a cloak for switching to a different movement playing different sounds. A final shape grew to envelop the entire installation, at which point this noise transition took place, filling the entire system with sparkling light and white noise.

Movement 2: upon receipt of a global message to start Movement 2, each device set a random timer up to 10s, at which point it would switch to playing a loop with a phase offset proportional to its x-axis location, multiplied by a global phase-spread parameter. After this 10s duration, all devices had entered their ‘Movement 2’ state, playing a phase loop, and the global LFO modulation was gradually brought down to reveal the new soundscape of acoustic guitar plucks and handclaps, playing phase-offset rhythms. The global phase parameter was then slowly brought down over a few minutes, during which time the rhythmic output of the entire system constantly morphed. The same LFO mod variable was later increased again to use the same effect to morph out of this movement. The global phase and LFO depth parameters could be directly edited (or controlled directly from a slider or input device) to fine tune the composition.

Movement 3: a flocking model was run in MaxMSP from within the Ableton Live project. The state of the model was sent at the 50fps control rate over the network. Each device responded to the incoming flocking model state by illuminating lights and playing sound based on the positions of 10 boids moving in a flock. A separate global parameter dictated the dynamic variation in the light and sound levels: at zero, all lights were at a constant brightness and all sounds were at a constant volume, uninfluenced by the flock. At its maximum level, the brightness and volume of each device was entirely dictated by the flock. Starting and ending this movement with the global dynamic value set to zero enabled a smooth transition into and out of this movement. At the end of the movement, all lights and sounds were again constant.

Movement 4: one global parameter controlled the base hue value. Another global parameter controlled the extent of the variation in hue across the x-axis of the device’s renderers. To start, the base value was set to the same value as the hue of the preceding movement and the variation parameter was set to zero. Again this meant that we could switch to this movement without anything visibly or audibly changing, and gradually introduce a varying spectrum to the hue across the installation.

Following our preceding categorisation, Movement 1 can be thought of as a substrate-based content design: each gesture sent over the network must be rendered as a spatial graphic and sonic object. Movements 2 and 4 are field-based: the content takes the form of a set of identical rendering processes, controlled globally and individually varied according to the device position. Movement 3 is again substrate-based: an ‘image’ (the content of the flocking model) is broadcast to devices and rendered by the devices interpreting the model state. No movement satisfied the ‘forest’ category, which is natural since we were always interested in working with coordinated relations between the devices: forests are technically the easiest systems to program but with our architecture set up to enable the real-time sharing of global parameters we always opted to build in mappings between global controls and spatial positions. Aspects of the composition category were present in Movement 2: the devices were randomly allocated different sounds to play to make up a rich instrumentation.

In reality, all these movements are implemented as specific actions that are applied to a set of simple state-based audio-visual rendering machines, by writing a composition sketch that interprets incoming OSC commands from the master controller program (Ableton Live broadcasting OSC messages). A central movement parameter tells the system when a new movement has commenced, which sets about further state changes (such as turning on and off looping, changing which samples are being played, and other playback settings). In Movement 1, we also have an event-based code that recognises when a new object has been created and attaches specific sound files to that event. In addition to these discrete events, the entire system is continuously controlled and once the sketch code has been written and installed, a large amount of real-time exploration of the composition space can be undertaken to finesse the system from the relative ease of the Ableton Live arrangement and device parameters. In particular, Movement 1 could be composed in fine detail through the creation of live gestures that are recorded as Ableton control parameters to give an organic feel and precise control of events.