1. Introduction(s)
This paper offers an overview of intermediality and the authors’ interest in electroacoustic music and cinematic traditions, then discusses audiovisual practices, the role of iteration/modularity and musicians as makers. A diagrammatic representation of how we formulate audiovisual relationships in our work beyond the duopoly of sound and vision is provided. An overview of our ever-evolving modular system for audiovisual interplay is outlined, and two works created with this system are discussed in the light of Chion, Boucher and Piché. We advocate for future work that looks beyond media relations and suggest that both electroacoustic and audiovisual discourse could better acknowledge physically present human performers.
The authors (who collaborate as Nhoj Elocin) focus on improvising with lively and semi-autonomous processes using custom software and self-made instruments. We perceive hands-on and tactile interfaces as vital, but hardware and algorithmic complexity provide stimulus. We regularly perform live electroacoustic audiovisual sets that range from textual to beat-based. The overall agenda is to traverse noise aesthetics, contemporary computer music and groove-based forms. Please note: video documentation of Hex on Data (Nhoj Elocin 2024c) and Modular Audiovisual Performance Systems (Nhoj Elocin 2024b) is available on the author’s website, and also included as video examples 1 and 2 in supplementary material. Our overall goal with this paper is to situate live performers and their technologies within an intermedial audiovisual framework.
2. Intermediality
Grishakova notes that intermediality ‘retains a connection with the avant-garde understanding of a work of art as an experiment, happening or performance and with the “intermedia” as a fusion of two or more art forms in a new, innovative work’ and argues that intermediality ‘has always been and still is an incentive for experimentation with new materials and cultural forms’ (Grishakova Reference Grishakova, Bruhn, Azcárate and Vieira2024: 14). Further discussion notes Glaser’s (Reference Glaser and Glaser2009: 12) use of the term intermediality to ‘describe creative works that resist classification into the “pure” or conventional categories of literature, music, or the visual arts’ and outlines Baetens and Sánchez-Mesa Martínez’s (Reference Baetens and Sánchez-Mesa Martínez2015: 292) suggestion that intermediality is ‘not only the general term that defines the relationships between autonomous media, it is also the term that identifies the internal plurality of each medium’ (Ibid.: 14–15). Méchoulan deploys Martin (Reference Martin2011: 37) to discuss cinema as one example of a historical medium that has outgrown its attachment to a single pure medium: ‘Consider the example that cinema provides: “its medium-specific possibility seems to have been well and truly overrun by its tendency to intermediality, its fundamental impurity. That is where its true materiality-effect, today, is situated ….”’ (Méchoulan Reference Méchoulan2015: 3). This leads to the suggestion that the purpose of an art or a specific medium may change due to intermediality. Further discussion warns against conflating intermediality simply with multiple forms of media, going on to suggest, ‘Every art or media thus seems to find its means in other arts and media, unsettling the expected borders between them’ (Ibid.: 3). Méchoulan also examines the etymological route of the ‘inter’ prefix and discusses commonalities across intertextuality, interdiscourse and intermediality; the reminder that ‘Intertextuality seeks to retrieve the text from its presumed autonomy’ (Ibid.: 3) is particularly relevant to our final reflections regarding whether intermediality is about the author’s intent or audience perception.
The authors acknowledge that the word intermediality was not at the foreground of their thinking when they created the works discussed in this paper. However, they have come to find value in this concept, especially in regard to how their work may be interpreted and/or understood by a perceiver. While we cannot claim that intermediality was a conscious consideration at a design or compositional level, from our perspective, intermediality is clearly inherent within our collaborative work. This paper is an important step in our reflexive research process, and our future work will configure intermedial relationships as a direct catalyst for new artistic explorations.
3. Electroacoustic
The call for this thematic edition leads with the word electroacoustic but places emphasis on musique concrète thinking that is ‘abstracting materials from reality while retaining traces of their origins’. However, the word electroacoustic has long been associated with practices that look beyond the acousmatic, as Waters (Reference Waters and Emmerson2000) and Adkins et al. (Reference Adkins, Scott and Tremblay2016) have argued. Nevertheless, to situate our relationship with the term electroacoustic and set up later discussion, an overview is necessary: Pierre Schaeffer explored how radio and recording made a new experience of sound possible, diminishing the causal link, allowing sounds themselves distinct existence from their sources. In an acousmatic performance, the loudspeaker replaces live performers and associated performance rituals, encouraging the listener to concentrate solely on sound. Listening itself becomes the point of origin for meaning; musical meaning is accessed through perception and individual reading of the abstracted sound itself, not its external reference or causal identity. The question is, what exactly are you hearing? While the term electroacoustic is sometimes conflated with acousmatic conventions, Simon Emmerson, in his introduction to Living Electronic Music, extends previous definitions of ‘electroacoustic music – a music heard through loudspeakers or sound made with the help of electronic means’ to include: ‘amplified acoustic music where the amplification changes, in essence, the experience of the sound and is integral to the performance’ (Emmerson Reference Emmerson2007: xii). That Emmerson’s definition so clearly includes amplified acoustic music is important to the authors because when an amplified acoustic instrument is legible as a source, the live human aspect is often foregrounded. While we deploy electronic and digital materials, our focus is on creating and performing using bespoke tangible objects, and our overall agenda highlights the role of the human performer in live electroacoustic audiovisual composition.
4. Interpreting cinematic traditions
Our work is situated at the intersection of electroacoustic and sound/vision techniques and theories, and Chion is a useful point of orientation. While Chion’s theories of practice and analysis are more easily applied to cinematic realism, the authors have applied them to abstract non-narrative visual forms and real-time interactive compositions. Our work embraces acousmatic concepts articulated by Chion, treating the revelation of sound sources (primarily tangible instruments) through the visual medium as a compositional technique. We consider the interplay between physical and virtual instruments within the overall compositional structure of tangible objects, sound and vision. Because acousmatic is closely associated with electroacoustic composition, it is pertinent to discuss relevant differences between Schaeffer’s and Chion’s concepts of acousmatic and their relationship and relevance to the authors’ work. Schaeffer’s acousmatic concepts constitute a methodological framework for reduced listening and analysis that relies primarily on hearing a sound without seeing or knowing its source and for sonic morphology rather than causality. Chion adapts Schaeffer’s concept for the visual medium to situate the relationship between sound and vision towards perceptual and narrative devices that address tension, spatialisation, symbolism and meaning. Whereas Schaeffer applied the acousmatic condition to negate the source as a narrative device, Chion applies the acousmatic process as a compositional device where the interplay of source revelation may be used expressively to drive audiovisual dramaturgy. This interplay in the authors’ work is used to structure performances and visual material where the tangible instruments are often the direct visual source material, such as in Hex on Data (HoD) that integrates live camera feed and processed images of the instruments or through sound-reactive visualisers in Modular Audiovisual Performance Systems (MAPS).
Chion and Gorbman (Reference Chion and Gorbman2019) plot the interaction of narrative sound devices and classify them as acousmatic or visualised to demonstrate spatial relationships among sound types such as speech, environmental/ambient sound and sounds of transmission. The soundworld for HoD and MAPS is primarily electronic; we do not employ what in cinema would be considered standard narrative sound types; however, from our perspective, the process of composing the visuals considered sound as either sitting in the acousmatic or visualised space, and we therefore find Chion a useful lens through which to develop and articulate our work.
Additionally, the authors utilise Chion’s theories of extension and synchresis in their compositional approach. In defining extension, Chion states, ‘Extension of the sound environment is my term for the degree of openness and breadth of the physical space suggested by sounds, beyond the borders of the visual field and also within the visual field around the characters’ (Chion and Gorbman Reference Chion and Gorbman2019: 85). Extension is a spectrum and can be used to infer environments beyond the visual frame (vast extension) or to create an intimate space that exists only within the visual frame and perhaps only the soundworld or perspective of a single character (null extension). In Chion’s framework, sound is a tool to either augment or restrict environments, with the potential to create layers of environments or stark intimacy. Chion’s framework largely applies to how sound affects the perception of the visual space.
However, the authors consider that the projected visuals can augment the environment in which the tangible instruments reside. Chion’s concept of vast extension may be applied to augment the physical instruments’ existence through scale, presence and temporal and spatial effects. The sound from the instruments is made larger and immersive by projecting the sound (dry and processed) through loudspeakers. In a similar way, projecting a version of the physical instruments can create a larger-than-life virtual double. This double can then take on its own life through visual processing. The virtual double exists as both a real-time camera feed, which details interaction between the performers and the instruments, and a processed, echoic depiction that alludes to the instruments existing outside the real-time interactions.
A third concept of Chion’s that the authors explore is synchresis, which Chion defines as ‘the spontaneous and irresistible weld produced between a particular sound event and a particular visual event when they occur at the same time’ (Chion and Gorbman Reference Chion and Gorbman2019: 64). In narrative cinema, synchresis is mainly a practical sound application that offers perceptual context explaining how Foley, environmental sounds and other post-production techniques are perceived as authentic sounds created by the diegetic objects or events in the visual frame or sources implied by the narrative. While the effect is a product of timing (sound and vision synchronising), the perceived outcome also relies on the established narrative, diegetic elements and symbolism that provide the necessary context for the audience. Chion states that synchresis can be demonstrated through experimental video, in which sound and vision lack tight representational coherence. The authors have used this as a starting point to consider synchresis in their works between tangible objects, sound and projected visuals. Various types of synchresis that illustrate relationships between timing and framing are used to investigate the function of visualised tangible objects in HoD, which are created through real-time camera feed and the abstracted images of the instruments. Synchresis is also observed in MAPS through scenes and motifs that visualise sound via frequency and amplitude tracking. Myriam Boucher and Jean Piché have established a precedent for applying Chion’s synchresis concepts to experimental works, stating, ‘Our comprehension of synchresis extends to encompass all situations where a viewer-listener can accept the suitability (as opposed to the believability) of a time-based association of sound and image’ (Boucher and Piché Reference Boucher, Piché and Knight-Hill2020: 13). These categories are phenomenological, describing time or physical relationships that the audience perceives. Similar to Schaeffer’s sound objects, relationships may be vertical, horizontal or overlapping and may range from instantaneous to extended in duration. They have proposed ‘ten types of synchresis: direct, on cut, anchoring, expected, delayed, gestural, ghost, behavioural, congruent movement and metric’ (Ibid.: 19), as well as ‘seven types of diegesis: direct, synthetic, musical, parametric, incongruent, sublimated and monomedia,’ (Ibid.: 23). The authors will describe the types relevant to their work in Section 8.
5. Audiovisual terminology and alternatives
Harris has examined audiovisual experience with emphasis on media hierarchy, suggesting that: ‘composing audiovisually describes the act of creating with sound and vision materials in such a way as to consider them as being of equal importance in the conception, development, realisation and outcome of the work’ (2021: 196). Finding parity in the relationship between sound and vision while avoiding media hierarchy is a clear agenda, and the focus is on composing and curating media relationships with little reference to contexts where live musicians or visual artists are evident. However, Harris’s article Audiovisual Coherence and Physical Presence emphasises the relationship between audiovisual media and physical (human) presence. She notes that when attempting ‘to translate a fixed audiovisual practice into a live performance context and subsequently, somewhat perversely, back into a fixed audiovisual document’, some concerns emerge around the relationship between fixed and live contexts. Harris goes on to question, ‘[w]hat and where was the work?’ and then notes: ‘When editing the document together, I cut between three different angles of myself with the screen showing the computer-generated imagery, and a render of the visual component that was taken during the performance itself. Consequently, there were times when the performance environment, and myself as the performer, were not visible in the document, the performance at that point being effectively the same as one of my fixed audiovisual works’ (Reference Harris2016: para 23). These concerns resonate with the authors, who have adopted the term audiovisual and who also question what a definitive mediatisation of their work might entail (hence the contrasting approach to documentation in this article). However, when conceptualising overarching audiovisual relationships, rather than a duality of sound and vision, we configure this (somewhat reductively) as a triangle. From our perspective, and as per Figure 1, the sonic output (a) and on-screen activity (b) are often equal at two corners of a triangle, but the third corner (c) is the theatre and interaction of the human performers and their instruments/tangible objects. While we may sometimes conceive, develop, realise and perceive the outcome of a work as an equilateral triangle, extreme weightings are possible, and there is significant moment-to-moment variation; the angles of influence range from acute to obtuse, often resulting in non-congruent triangles. This seems quite compatible with Harris’s suggestion that ‘experiencing audiovisual works necessitates transperceptual attention’, which she defines as ‘sonic and visual components of the work both in isolation and as they relate to, and affect, one another….’ and to ‘attend to sight and sound and their interaction, and encounter other perceptual and contextual modes in light of that interaction’ (Ibid.: 197). We suggest that new perceptions of media relationships emerge through what might be conceptualised as intermedial perception and that the relationships among media, human performers and their instruments point toward the productive role of intermediality in creating and performing live electroacoustic audiovisual compositions.
Diagrammatic representation of how the authors formulate audiovisual relationships in their work: sonic output (a); on-screen activity (b); theatre and interaction of the human performers and their instruments/tangible objects (c). January 2026.
While in-depth discussion is beyond the scope of this paper, it’s worth noting terms beyond audiovisual: in The Practice of Live Cinema, Mia Makela quotes from the Transmediale Festival 2005 press release to suggest, ‘Live Cinema today stands for the simultaneous creation of sound and image in real-time by sonic and visual artists who collaborate on equal terms…’ (Reference Makela2008: 1). Ana Carvalho and Corelia Lund unravel this and other related terms in their 2015 book The Audiovisual Breakthrough. The term live cinema sits across various practices and is also associated with the work of Katie Mitchell in the context of hybrid productions incorporating real-time cinema and live theatre (Ledger Reference Ledger and Reilly2018: 69). Multimedia performance is another term that resonates with the authors, but we concur with Olivier in an article titled Intermedial Cognition that discusses Nicholas Cook’s (Reference Cook1998) suggestion that ‘a multimedia work is only truly understood as a mix of various types of media when their meanings are not the same. Cook argues that if the images are so closely linked with the sound that they point to the same topical meaning, the medium is more closely akin to a synesthetic experience’ (Olivier Reference Olivier2016: para 39). The net result for Olivier is that intermedia is a more appropriate term than multimedia. Dixon’s (Reference Dixon2007) book Digital Performance is another useful text and Gibson et al.’s (Reference Gibson, Arisona, Leishman and Tanaka2023) Live Visuals: History, Theory, Practice offers a rich, overarching history of real-time visuals, and many of the artists discussed therein could be argued to be related to the author’s work.
6. Musicians as makers/intermediality in modular workflows
The author’s audiovisual practice starts with the tangible instruments; it is from the design of the instruments – as controllers and sound-producers – that the larger work emerges. Our intermedial systems are created through iterative development; the tangible instruments shape not only the soundworld, but (to a slightly lesser extent) the visual outcomes as well. While DIY and maker practices foreground experimentation and embodied interaction (Anderson Reference Anderson2012; Chippewa Reference Chippewa2016; Breaux Reference Breaux2017), this approach is also fundamental to research on tangible interfaces, where physical objects function as mediators between embodied action and computational processes in performance contexts (Ullmer and Ishii Reference Ullmer and Ishii2000). Intermediality and negotiated control are realised through both the inherent functions of our tangible objects and the performers’ interaction with the instruments and with each other. The prototyping stages that address form and function are central to developing playability, informing how each modular component functions both individually and within the larger system.
The audio software central to both HoD and MAPS consists of Max/MSP and Ableton Live. These software packages include both MIDI (musical instrument digital interface) mapping and audio effect processors. While software often remains more fluid and open to ongoing reconfigurations than hardware, we also adopt a modular approach to software. Carroll has developed a number of modular effect processors implemented as Max for Live devices within Ableton. This workflow allows customisations and the use of similar processing devices across different works utilising the same tangible objects. This enables design efficiency across hardware and software and provides continuity of soundworlds across works. It also allows the performers to become intimately familiar with the instruments by expecting repeatable interactions and outputs, while maintaining a platform for continual development and experimentation. Many of our boxes run code Ferguson wrote for Arduino-compatible microcontrollers based on the Control Surface and Fast LED libraries, which support highly layered controllers with computer-controlled lights, accelerometers and standard pots and buttons. Ferguson also handles TouchDesigner programming, where a modular framework runs individual elements in isolated modules, enabling selective activation, adaptable performance integration and computational efficiency. In MAPS, sound reactivity is foregrounded, while HoD uses TDAbleton to facilitate communication over a network via Open Sound Control.
When the artist-maker creates hardware and software instruments intended for their own use, that freedom translates into unique one-off hardware and constantly evolving software. Bespoke and idiosyncratic systems thus lead to unique electroacoustic audiovisual works with personal trademarks and signature aesthetics.
7. Tessellating systems: What?
Tessellating Systems (TS) is an ever-evolving modular system for audiovisual interplay. A growing range of boxes tessellate across a surface, as illustrated in Figure 2; some produce electronic sound directly via the infamous 40106 CMOS chip, the ‘hex Schmitt trigger’ (see Tate et al. Reference Tate, Brown, Ferguson and Della-Bosca2024), while others function as custom controllers – with knobs, buttons and accelerometers – connected to Ableton, Max and TouchDesigner software to create a unique audiovisual system. Each sound-producing box has its own circuitry, sensor configuration and control interface to maximise the potential of the 40106 CMOS chip. While several of the boxes produce the raw timbres associated with the harsh square wave output of the 40106, the sonic output is ultimately shaped by real-time processing in Max and drum programming in Ableton. The accompanying visuals created in TouchDesigner augment the performance system, integrating a live camera feed and amplitude- and frequency-reactive visual motifs based on the tessellating patterns of the box structures. Computer-controlled LEDs (light-emitting diodes) on the tangible objects and the projection visuals together produce rhythmic ambient light that feeds back into the light-controlled boxes, thus completing the audiovisual feedback loop; see Nhoj Elocin (2024a) 4’20”–5’59” for a clear example.
Tessellating Systems (2023) a modular instrument collaboratively created by the authors (Photo credit: authors).
7.1 Tessellating systems: Why?
The authors’ approach to system design also considers materials in terms of both consumerism and personal connection to the work and calls into focus issues surrounding sustainable approaches to music technology. Creating bespoke systems is a response to the growing market of off-the-shelf music controllers. While these mass-market devices may suit the needs of many, they rarely possess the physical properties that inspire unique performances. In designing novel devices, the goal is not simply unique gestural control but to build an instrument and system that the artist-builder-performer knows and understands. Considering authorship and connection to the instrument, this understanding informs the composition and performance processes as well as the design process. As each box is designed, tested and integrated, new control functions and performance affordances emerge. The fluid and iterative nature of the design process is mirrored in the performance practice: the ‘work’ is not simply the performance, the physical instrument or the code, but the audiovisual outputs as they evolve and the open-ended, ever-transient system. We situate the work in the moment of audiovisual experience – in the audience’s encounter with the system, whether live, fixed media or edited documentation. This flexibility is an initial design consideration: a modular system that can be utilised across various creative outputs and formats. This approach follows in the system-as-composition practice of Nicolas Collins, David Tudor and Composers Inside Electronics, where the system is an indeterminate flexible score, a map of endless possibilities. This is similar for Gordon Mumma and Fillion, who notes that his instruments are not general-purpose but are specific to particular pieces and designed for a niche function: ‘[m]y electronic music equipment is designed to be part of my process of composing music. I am like any composer who builds his own instruments, though most of my “instruments” are inseparable from the compositions themselves. My “end-product” is more than a package of electronic hardware: it is a musical performance for a live audience’ (Reference Mumma and Fillion2015: 184). Reflecting on the idea of building resistance into instruments and navigating the locus of tension, Mumma’s performance-centric approach, as opposed to a performer-centric one, is an appropriate frame for our work.
8. Analysis introduction
Some similarities/differences between Hex on Data (HoD) and MAPS: HoD is documented using five camera angles that capture projections, performers and the tabletop. While the authors made decisions in post-production about which angles to show, there are no edits in time to the work itself, and the documentation is not a composite that is three versions of the work that were recorded and the ‘best’ was retained. MAPS is documented with a single static camera at the back of a concert hall, so all elements are presented in a single frame, warts and all. These differing approaches impact the character of each work; the documentation of HoD may appear idealised while MAPS may seem more live/authentic. However, both works were explicitly developed for live performance, and both are performed with a stopwatch and a text score with specific timestamps.
Although HoD and MAPS originate in improvisation, the refinement process is systematic: we consider the effect of the score, code and interactions on the audiovisual relationships. Analytical frameworks for this work remain underdeveloped relative to established compositional practice. Boucher and Piché’s elaboration of Chion’s synchresis provides a precise vocabulary rooted in Chion’s theories. The Sapir–Whorf hypothesis suggests that language influences thought and perception; we argue that tailored terminology for audiovisual practice shapes how relationships are conceived, discussed and realised. Naming these relationships informs intentional compositional decisions. The analysis reveals both deliberate and emergent moments of synchresis, demonstrating that Chion’s framework is a dynamic mode of engagement rather than a retrospective justification.
8.1 Hex on Data overview
Hex on Data (15’18”) (Figure 3) is an interactive audiovisual composition that explores networked systems, performed live-in-concert by the authors at The International Symposium of Electronic Art (ISEA) in Brisbane in 2024. From the authors’ perspective, parallels with HoD are observable in both commercial music-making and academic research. Key influences include artists who utilise TouchDesigner and network communication: Amon Tobin’s stage setup and approach to performing ISAM (2020), Joel Zimmerman’s (Deadmau5) cubev3 and Nine Inch Nails’ Light in the Sky Tour in collaboration with Moment Factory. Carroll’s computer runs Ableton and Max for Live custom devices; Ferguson’s computer runs bespoke software written in TouchDesigner – we use TDAbleton for integration, giving each of us access to every parameter on the other’s computer.
Nhoj Elocin (2024c) performing Hex on Data, Queensland Conservatorium Griffith University, Brisbane, Australia (Photo credit: Kai Mohrholz).
8.2 Hex on Data analysis
Hex on Data layers direct simultaneity, morphological resemblance and data-driven correspondence to produce an audiovisual work where multiple modalities of synchresis emerge, often concurrently. From the outset (0’00”–2’23”), the work establishes a strong relationship between performance gestures, abstracted visuals and circuit behaviour. Sonically, the work commences with hands-on manipulation of a custom synthesiser based on the 40106 CMOS chip that is the power switch of blue knobs, which is repeatedly activated and deactivated, and the resulting sound of voltage draining from the circuit via a large capacitor is foregrounded. This is accompanied by textures from Ableton and a slow bass drum pulse, alongside auto panning and red knobs, and a 40106-based circuit that utilises amplitude modulation. From a visual perspective, the work begins with a top-down live camera feed processed with edge detection and coloured purple to highlight the hexagonal form and tessellating arrangement of our tangible objects. At 0’36” a yellow hexagonal grid is flown into view from behind the camera; from 0’46” this is deformed by manipulating mass and force parameters applied using a spring algorithm to simulate physics. This allows a performer to set parameters, then trigger a scenario and watch the result unfold; that is, the hexagonal grid continues to deform and move because of the mass and force parameters until the oscillation dies out. The algorithm is reset and retriggered occasionally, so the performance of this visual element appears related to the activation/reactivation of the electronic circuit that causes the sonic behaviour.
Considering the above in terms of Boucher and Piché, behavioural synchresis is perceived, where ‘[c]onvergence does not obligatorily happen at the same time, but the sound and image have strong morphological correspondences’ (Reference Boucher, Piché and Knight-Hill2020: 21). Simultaneously and on a more extended timescale, sublimated diegesis is observable – this ‘appears when the sound/image relation elevates an audiovisual unit to a role similar to that of congruent music in narrative cinema. It unequivocally abandons any physical relationship between the sound and image but forges a strong primary emotional link’ (Ibid.: 25). However, since Boucher and Piché’s concepts are defined as audience-perceived relationships and this is a live performance with sonic and visual systems running in parallel, an alternative reading is that the live, real-time nature of the work sits better with ‘direct synchresis’ – ‘[s]ound and image are bound by their simultaneity’ (Ibid.: 19). This section demonstrates the potential limits of discrete audiovisual topologies, as multiple simultaneous interactions and readings are observable. This layering of sonic and visual materials and performance gestures might be framed as a stacked intermedial instance, where the audience perceives coherence across layers, suggesting that intermediality is grounded in the recognition of shared behaviours.
2’45”–4’45. At 2’45” both performers activate what Boucher and Piché call an ‘on cut synchresis’ (Reference Boucher, Piché and Knight-Hill2020: 19); this punctuation marks the transition to a beat-based musical section and the emergence of a hexagonal tunnel. The form of the tunnel and the location of the camera/viewer’s perspective are determined by noise data that is manually performed. An ‘all-in’ groove emerges at 3’15”, and the camera shifts from outside to inside the tunnel. Gestural and frantic elements are performed using knobs and accelerometers, and what Boucher and Piché call ‘synthetic diegesis’ may be appropriate: ‘There is a strong concordance between visual and sound materials. Synthetic diegesis concerns abstract images and sounds (i.e., not representing the real world). Abstract images don’t have a specific sound, but they have kineto-morphological features that indicate their appropriateness to the sounds they are presented with’ (Ibid.: 23). Further to this, while the colour palette is set in advance to bring coherence, audio analysis is deployed so low, mid and high frequency bands are mapped to red, green and blue colour values which are then applied as a multiplication of the preset palette. This can be considered in line with what Boucher and Piché term ‘[p]arametric diegesis’, which ‘harnesses the parametric values of one media (colour, blob tracking for visual/pitch, loudness and timbre for sound) scales them and remaps them directly onto the other media to provide a visualisation of sound or a sonification of images.’ It is further suggested that this approach uses ‘transmutability, which relies on the premise that any kind of input data can be algorithmically visualised or sonified’ (Ibid.: 24). Parametric diegesis describes what happens technically at the system level; synthetic diegesis describes what happens perceptually. This section demonstrates a strong kineto-morphological congruence created through both embodied and data-driven interaction.
The musical theme between 6’15” and 8’16” is a groove where individual drum hits are mapped to a random number generator that selects images from a folder; each image is a close-up photograph of a single element from Tessellating Systems. As discussed previously, this might be considered via Chion’s concept of vast extension that is projecting a version of the physical instruments creates a larger-than-life virtual double, the double takes on its own life through visual processing, in this case extreme scaling and tiling alongside various strengths of edge detection to bring colour homogeneity and amplify textual visual details while visually matching the wildness of the flashlight controlled electronics with fairly jerky visual zooming. At a system level, this might seem compatible with Boucher and Piché’s parametric diegesis as discussed above, but the overall relationship is perhaps better aligned with metric synchresis, where ‘image and sound are linked by their common time base’ (Reference Boucher, Piché and Knight-Hill2020: 21). Whereas the tunnel section was marked by fluid, energetic motion, this section features a rhythmic drive. This groove, in conjunction with the scale of the tiled photographs, demonstrates how the effect of metric synchresis can be amplified by the visuals.
The dominant visual theme at 9’23’–13’11” is created through a three-dimensional particle system with edge detection and feedback applied to blur the line between two and three dimensions; that is, on some occasions, the perception of depth is minimal and a flat two-dimensional pattern is perceptible. At a system level, this might be read as Boucher and Piché’s parametric diegesis – the starting location of the particle system is constantly being reset rhythmically – or with metric synchresis, given that this is another groove-based section with clearly synchronised timing. However, networked communication is foregrounded here: either performer can stop or start the music and adjust the tempo, and a scripted ‘game’ is performed that changes radically from performance to performance. This ‘game’ foregrounds the shared negotiation in intermedial performance and makes the behaviour of the system and the underlying relationships clear as Boucher and Piché’s behavioural synchresis is again observed.
8.3 Hex on Data summary
Hex on Data integrates computer-generated imagery, a live camera feed of our performance ecology and photographs of individual tangible objects into the visual sphere, expanding the tabletop into the projected visual space. Edge finding is used selectively to colourise, manipulate and bring continuity. The underlying data driving the music is also mapped to parameters of a particle system, and networked communication enables dual control of the overall system. Hex on Data is an intermedial study of DIY handmade electronic instrument-making and live electroacoustic audiovisual composition. It explores tangible objects where performance gestures and light control and reception enable cyclical sonic and visual motifs. Through design and performance, Hex on Data questions how each element – hardware and software, tangible and virtual – can both enact and respond to generative processes through gestural control and light manipulation and how each contributes to the temporal sonic and visual space.
8.4 Modular Audiovisual Performance Systems (MAPS)
MAPS (10’04”) (Figure 4) explores electroacoustic composition and performance in audiovisual contexts. MAPS was performed live-in-concert by the authors at the Loreto Theatre at The Sheen Center for Thought & Culture as part of the New York City Electroacoustic Music Festival (NYCEMF) in 2024. Sound-reactive visual scenes and tightly composed performer-activated cues exploit synchresis: the brain perceives a relationship by virtue of simultaneity. A range of sonic processing is deployed, and raw timbres are created and performed via 40106-based circuits, some of which are the same as those deployed in HoD. Leaning towards beat-based electronica, the system collectively is ‘on the edge’ of several genres, including noise and electroacoustic music. The soundworld is an amalgamation of direct hardware output and custom real-time software processing in Max with drum programming in Ableton. Interactive visuals created in TouchDesigner complete the work through generative content and sound-reactive visual motifs. The goal is an intermedial work spanning genres, producing cultural references that oscillate around beat-based materials. From the authors’ perspective, MAPS is influenced by the practices of Carsten Nicolai (Alvo Noto), Noemi Schipfer and Takami Nakamoto (Nonotak) and Richard ‘Richie’ Hawtin – situating the work between avant-garde and popular music contexts.
Still image (2024b) from video documentation of Nhoj Elocin performing Modular Audiovisual Performance Systems (MAPS).
Figure 4. Long description
Panel A: A person is interacting with a set of modular audiovisual instruments. The instruments are tangible objects that produce sound and visual effects. The person is using their hands to manipulate the instruments, which are connected by various wires and cables. Panel B: Close-up view of the modular instruments, showing detailed components and connections. The instruments have a complex structure with multiple modules and interfaces. Panel C: The visual effects generated by the instruments are displayed on a screen. The effects include abstract shapes and patterns that change in response to the performerʼs actions.
8.5 MAPS analysis
MAPS departs from the networked communication that characterises HoD and instead embraces techniques similar to those of disc jockeys/visual jockeys (DJs/VJs), who often manipulate pre-composed and pre-programmed audio and visual material. Beyond strict clip triggering, the intermedial logic lies in the distribution of control over these materials and in the tightly scored and performed transitions. While substantial sonic elements are pre-programmed and triggered, the 40106 synthesisers are often running quite freely and need attention/interaction to avoid stasis. The result is a ‘live production’ that blends improvisation with studio-produced elements – the work’s ‘liveness’ lies in the continual negotiation required to sustain audiovisual coherence. The form and character of the work are repeatable, but idiosyncrasies develop and emerge from one performance to the next. The visual elements are a series of interactive scenes, but perceived interaction between sound and vision is often curated/performed manually by each performer in reference to timed triggers and scored transition arcs. The structure of MAPS is fluid, with the audio and visual layers continually overlapping. Precise timestamps are difficult to assign because the sonic and visual layers often appear to operate independently, with the sonic progressing into new material while the visuals sustain the structure. These static visual sections allow a dense audio layer to build without causing sensory overload. The elasticity creates synchresis through approximation rather than strict causality, a reminder that Boucher and Piché’s framework often resists straightforward classification. MAPS tests a different intermedial problem than HoD: how bespoke hardware can operate within visual and sonic pop-adjacent forms and idioms. The system is designed to accommodate multiple engagement modalities: the balance of pre-programmed and live material requires compositional discipline, creating an opportunity to examine form through synchresis and diegesis.
From 0’00” to 3’16”, a visual particle system with a variable light source moves between shadowy silhouettes and full illumination, while granular textures emerge in the sonical layer. This relationship is legible through Boucher and Piché’s behavioural synchresis grounded in ‘morphological correspondences’ (Reference Boucher, Piché and Knight-Hill2020: 21). Both media share morphologies of drift, accumulation and spatial movement that allow for textural density as a motif for development. This establishes a mode of non-causal coherence where both domains evolve through parallel processes that create the perception of integration and interaction. From 40”, a slow bass beat emerges and develops into a rhythmical motif; this increases in volume and timbral presence as a filter opens to full spectrum; by 2’00’’ a grid mesh emerges and is scaled and rotated. While many of the visuals appear to track the audio, no audiovisualisers are introduced until 3’16”; up until this point, the particle system and grid are controlled by generated noise data and manual manipulation, demonstrating that synchresis can be both performed and inferred. The particles carry this section and bear a visual relationship to the grid mesh – the second dominant motif – creating development across both visual and audiovisual dimensions. The opening section invites a reading of expected synchresis, which Boucher and Piché define as ‘a significant event in the sound while there is no change in the image’ (Ibid.: 20) or vice versa. However, the nuance of their definition may be challenged here. While they describe this synchresis as functioning by ‘breaking expectations of continuity’ (Ibid.: 20), the overlapping structures in this instance produce a more seamless, continuous texture in which the sound and visuals flow independently yet maintain an established relationship. By establishing the relationship early in the work, moments of non-coincidence may be read not as surprise or interruption, but as helping to establish partial independence.
3’16”–4’38” contains several implementations of expected and delayed synchresis, creating layered transitions that blur section definitions. 4’38”–6’58” foregrounds beat-based material using drum kit samples. There are four distinct kits and beat foundations that build in timbral aggression and intensity. These grooves provide the underlying musical drive and serve as the primary sound material used to trigger visual motifs via amplitude tracking. While some element of direct sound visualisation is present, the work relies heavily on synchresis to elicit the perception of tracking and audiovisual mapping – the vertical lines, for instance, are controlled by a noise signal within TouchDesigner, manually timed to roughly match the music. Boucher and Piché’s notion of musical diegesis is useful here, defined as a relationship in which ‘[t]he sound/image relationship is drawn by the music itself (pitch, rhythm, etc.). The composition is guided by a musical structure. In this case, the image seems like a visual representation of the music, or vice versa’ (Reference Boucher, Piché and Knight-Hill2020: 24). The image seems to derive its organisation from rhythmic and textural features in the music, even where this relation is only partially enacted through direct sound processing. This reveals that musical diegesis is an effect composed through the interaction of score, interface and performer-to-performer coordination.
The emphasis on performer coordination is important when considering performer error and contingency. Moments of a mistimed control gesture reveal the extent to which intermediality is maintained through embodied attention. These errors contribute to the ‘liveness’ experience, in which they may illuminate the manual labour of creating a coherent performance. That said, Ferguson is aware that there were times when he twisted the right knob on the wrong box, leading to some unintentional moments of confusion.
The final section reinforces intermediality as distributed and negotiated in real-time. From 6’58” to 10’04”, the waveform and spectral visualisers are directly controlled by audio, while the grid mesh motif (that is often reduced to a single line) and the returning particle system are again based on scored events and the manipulation of noise algorithms, as well as performers reacting in time to the musical material. The work concludes with Carroll performing on vertical copper pipes that control a 40106 circuit. The final visual is simply a frequency spectrum that is deactivated when the performance reaches silence, emphasising the liveness of the system.
8.6 MAPS summary
Our overall goal was to co-create audio and visual material that felt coherent in character and movement. Verbal feedback we received at NYCEMF highlighted the novel and unexpected contribution that MAPS made, particularly that it was not in any way ‘fixed media’ but still foregrounded electroacoustic techniques while showcasing a clear commitment to performance and extending the agenda ‘beyond music into multimedia’ with several audience members later commenting ‘there was just so much going on we didn’t know where to look’.
9. Final reflections towards future work
While we have touched upon how MAPS and HoD were created, our overall focus has been to discuss the video documentation of these electroacoustic audiovisual compositions and to situate that discussion within broader theoretical contexts. While some may call for ‘more technical description’, that’s not the goal of this paper. It is all too easy for us to get lost in software patching and hardware making, and sometimes the creative output of makers who are also musicians comes down to whatever we are able to make in that moment, but that’s not the goal here. With these projects, we conceptualised and developed a tessellating system with intermedial affordances, then set about composing conceptually solid and repeatable works that (for example) embrace Chion’s concept of vast extension, where projecting a version of the physical instruments creates a larger-than-life virtual double. The form of these works revolves around agreed timestamps and text scores. We advocate for the value of improvisation and iterative processes, but the main goal of this paper is to discuss the final artefacts themselves.
Overall, our goal with this paper is to situate live performers and their technologies within an intermedial audiovisual framework. Hex on Data suggests that intermediality in live audiovisual performance need not be reduced either to fixed mapping or to perceptual coincidence alone; rather, it can emerge through a networked ecology in which causality is distributed across performers, interfaces and generative processes. MAPS suggests that intermedial coherence in live audiovisual practice need not depend on fully networked reciprocity or direct audiovisual mapping; it can also emerge through scored approximation, embodied timing and the perceptual binding of partially autonomous media streams. We suggest that appropriate analytical frameworks for our live audiovisual work remain underdeveloped, but Boucher and Piché’s elaboration of Chion’s synchresis provides a precise vocabulary rooted in Chion’s theories. From an intermediality perspective, the ‘work’ is not simply the performance or the tangible objects, but the complete system and the audiovisual outputs as they evolve. And as our work develops the thinking inscribed in this paper will be a direct catalyst for a new phase of practical experiments, which recalls a point made earlier via Grishakova, who notes that intermediality ‘retains a connection with the avant-garde understanding of a work of art as an experiment, happening, or performance and with the “intermedia” as a fusion of two or more art forms in a new, innovative work’ and argues that intermediality ‘has always been and still is an incentive for experimentation with new materials and cultural forms’ (Grishakova Reference Grishakova, Bruhn, Azcárate and Vieira2024: 14).
We cannot offer watertight conclusions but will attempt some final reflections: As per Figure 1, we suggest that the audiovisual experience in our work is best represented as a triangle, not a duopoly. While this seems simplistic, if we were to consider the right-hand triangle in Figure 1 as a representation of current discourse, we would need to vertically extend (c) upwards infinitely to represent the current balance between sonic output (a); on-screen activity (b); and theatre and interaction of the human performers and their instruments/tangible objects (c). While we note that Gibson et al.’s (Reference Gibson, Arisona, Leishman and Tanaka2023) Live Visuals: History, Theory, Practice is a vital contribution, we remain concerned that both electroacoustic and audiovisual discourse sometimes seem to overlook physically present human performers and their instruments. The use of a triangle also seems to fit with intermediality, where relational perceptions change frequently and various readings are possible. Influenced by audience reactions to MAPS at NYCEMF and noting that Harris’s notion of transperceptual attention already hints at intermediality, we currently see intermediality as most useful in regard to the perception of an audience. That said, it is also clear that iterating modular hardware and software while simultaneously exploring sonic and visual mappings is creatively productive, and this territory may fundamentally be conceived as intermedial. Finally, by contextualising our work within traditional cinematic frameworks established by Chion and furthered by Boucher and Piché, we hope to contribute to expanding vocabularies in our field. In addition to broadening the scope of the audiovisual experience, we aim to provide a larger palette for discussing and investigating live electroacoustic audiovisual compositions.
Supplementary material
Video example 1: OSCarroll_1_.m4v Nicole Carroll and John Ferguson (2024) performing Hex on Data, Queensland Conservatorium Griffith University, Brisbane, Australia. (Captured by Kai Mohrholz, edited by Ferguson and Carroll). Video example 2: OSCarroll_2_.m4v Nicole Carroll and John Ferguson performing Modular Audiovisual Performance Systems (MAPS) at New York City Electroacoustic Music Festival (Captured by New York City Electroacoustic Music Festival). The supplementary material for this article can be found at https://doi.org/10.1017/S1355771826101575.
Open access
