Hostname: page-component-74656f9884-rngvg Total loading time: 0 Render date: 2026-11-05T10:28:36.008Z Has data issue: false hasContentIssue false

Evaluating media equality in generative audiovisual composition using modality constraint and replay-based reveal

Published online by Cambridge University Press:  03 August 2026

Tom Didiot-Cook*
Affiliation:
Engineering Mathematics & Technology, University of Bristol, UK
Rights & Permissions [Opens in a new window]

Abstract

Coupled audiovisual instruments, in which sound and image are produced from a shared generative process, promise media equality by design. But does causal coupling actually deliver perceptual parity? This article asks how composers experience media equality and intermedial interference when working with such an instrument. We report a practice-based study of the Cellular Au-Tonnetz, a system in which audio and LED light are parallel projections of the same evolving cellular-automaton state. Nine participants composed short pieces under three modality conditions: visual-only, audio-only and audiovisual. Each was followed by replay with both modalities revealed. By withholding one medium during composition and reinstating it at replay, the protocol makes the construction of audiovisual experience observable. Audiovisual access was preferred overall (eight of nine participants), but the audio-only condition most often produced the largest perceived mismatch (six of nine), and replay frequently turned apparent mismatch into coherent re-reading. Mismatch typically took the form of cue conflict: temporal persistence, density correspondence and perceptual masking. Productive interference emerged at replay through rebinding. We position this as foundational practice-based work, offering a method other practitioners can adopt, and design priorities for instruments where sound and image share a single generative substrate.

Information

Type
Article
Copyright
© The Author(s), 2026. Published by Cambridge University Press
Figure 0

Figure 1. Figure 1 long description.System block diagram, comprising the TZ5 instrument and its constituent parts, as well as the supporting infrastructure and instrumentation required to conduct the study.

Figure 1

Figure 2. Exploded view of TZ5 audiovisual instrument. Light from the 91× red, green and blue (RGB) LEDs is guided by a hexagonal lattice and diffused by a 2-mm sheet of acetate.

Figure 2

Figure 3. Tonnetz pitch-class map used for hue mapping. MIDI numbers are given under each pitch class, and computed from the root of C; 36 (bottom, centre), incrementing by +3 semitones moving right and +4 moving left.

Figure 3

Figure 4. Counts of best overall rank, most intermedial selection and biggest mismatch by block condition (A = visual only, B = audio only, C = audiovisual).

Figure 4

Figure 5. Diverging stacked Likert responses for Part A (pre-reveal) items by block condition. Bars show within-condition percentages centred on neutral (4).

Figure 5

Figure 6. Figure 6 long description.Diverging stacked Likert responses for Part B (post-reveal) items by block condition. Bars show within-condition percentages centred on neutral (4).

Figure 6

Figure 7. Stacked bar chart of parameter influence nominations by condition.

Figure 7

Figure 8. Montage of video stills taken from participant outputs across conditions (A/B/C).