2.1 The evolution and limitations of music emotion recognition

Research into MER has grown steadily since the late 1990s, largely within the music information retrieval community. The premise that music functions as a language of emotions is supported by psychological studies showing that listeners often engage with music to regulate affective states (Juslin and Sloboda Reference Juslin and Sloboda2001, Reference Juslin and Sloboda2010). Researchers sought to map acoustic features (timbre, rhythm, tempo, harmony) onto discrete emotional categories or onto dimensional models of valence and arousal.

Several fundamental problems emerged. The categorical approach reduces musical experience to a handful of classes, ignoring subtle and culturally embedded distinctions (Yang and Chen Reference Yang and Chen2012). Emotion descriptors are ambiguous: different listeners describe the same passage differently, and the same listener may be inconsistent over time (Juslin and Laukka Reference Juslin and Laukka2004). Despite advances in deep learning, these limitations persist. Surveys by Han et al. (Reference Han, Kong, Han and Wang2022) and Schedl et al. (Reference Schedl, Gómez and Urbano2014) confirm that state-of-the-art models struggle with generalisability beyond controlled datasets.

The technical trajectory reflects these tensions. Early work used handcrafted features (spectral centroid, mel-frequency cepstral coefficients, rhythm periodicities) fed into support vector machines and Gaussian mixture models (Kim et al. Reference Kim, Schmidt, Migneco, Morton, Richardson, Scott, Speck and Turnbull2010). More recent approaches employ convolutional neural networks (Choi et al. Reference Choi, Fazekas, Sandler and Cho2017), achieving accuracy gains within benchmarks but remaining vulnerable to dataset bias and cultural specificity. What registers as sad in a Western tonal context may be experienced as triumphant in another tradition (Eerola and Vuoskoski Reference Eerola and Vuoskoski2013; Susino and Schubert Reference Susino and Schubert2019). MER has produced technical advances but faces a conceptual impasse: emotions in music are not universal categories waiting to be detected but situated, relational and context-dependent phenomena.

2.2 Affective computing

Picard (Reference Picard1997) argued that if computers are to interact naturally with humans, they must recognise, model and respond to emotional states. The promise lies in systems that are responsive, adaptive and emotionally informed. Applications span education (D’Mello and Graesser Reference D’Mello and Graesser2012), healthcare (Calvo et al. Reference Calvo, Dinakar, Picard and Maes2016) and creative practice.

The perils are increasingly recognised. Critics argue that affective computing rests on dubious assumptions: that emotions are discrete, measurable and universally legible through proxies such as facial expression, voice or physiology (Barrett et al. Reference Barrett, Adolphs, Marsella, Martinez and Pollak2019). In practice, these assumptions have led to deployment in surveillance, law enforcement and workplace monitoring, where algorithmic claims to emotional knowledge carry significant risks (McStay Reference McStay2018; Crawford Reference Crawford2021). Birhane and Guest (Reference Birhane and Guest2021) highlight how such systems encode cultural bias and perpetuate algorithmic injustice. Affective computing holds creative potential, but its dominant implementations treat emotions as extractable data points, obscuring their situated and relational character.

2.3 Interactive performance systems and embedded agency

Within music and sound art, a counter-tradition embeds algorithms not as detectors but as agents within performance ecologies. Rowe’s (Reference Rowe1993) taxonomy distinguished between the instrument model, where the computer extends the performer’s control, and the player model, where the computer behaves as an autonomous partner. George Lewis’s Voyager (Reference Lewis2000) epitomises the latter: an improvisational collaborator whose embedded algorithms generate responsive, unpredictable contributions.

Research within the New Interfaces for Musical Expression community has expanded this approach. Fiebrink and Caramiaux (Reference Fiebrink, Caramiaux, Dean and McLean2018) survey how machine learning can serve as a creative tool, enabling performers to shape models around personal vocabularies rather than universal categories. Eigenfeldt and Pasquier (Reference Eigenfeldt and Pasquier2013) developed autonomous musical agents that interact in ensembles. Tatar and Pasquier (Reference Tatar and Pasquier2019) provide a comprehensive typology of musical agents and their roles in metacreation, distinguishing between systems that extend, collaborate with and operate independently of human performers.

Live coding offers another precedent. Collins et al. (Reference Collins, McLean, Rohrhuber and Ward2003) showed how exposing and manipulating code in real time creates transparency and contestability. Magnusson (Reference Magnusson2014) theorised live-coded algorithms as a form of musical notation, a code score in which the same text functions simultaneously as instrument, score and performed gesture.

The past decade’s advances in machine learning have produced a wave of co-creative music systems that extend this agent-based tradition (Briot et al. Reference Briot, Hadjeres and Pachet2020). Rather than classifying or recognising, these systems generate or shape musical material in dialogue with a human partner. Louie et al. (Reference Louie, Coenen, Huang, Terry and Cai2020) show that novices retain a stronger sense of ownership over a composition when given tools to steer a deep generative model rather than passively receiving its output. McCormack et al. (Reference McCormack, Gifford, Hutchings, Llano, Yee-King and d’Inverno2019) equip an improvising AI with an extra-musical channel that communicates its internal confidence to its human collaborators, and Thelle and Pasquier (Reference Thelle and Pasquier2021) frame their agent explicitly as a partner for creative brainstorming. These systems share a concern with agency and control that TEAL also pursues, but they operate on musical content directly. TEAL differs in making affective inference, rather than direct note or audio generation, the locus of co-creation; musical material is generated downstream from that affective layer.

These practices establish algorithms embedded in performance as agents of composition rather than instruments of recognition. TEAL draws on this tradition.

2.4 Expressive computer music performance and affective algorithmic composition

A body of work directly addresses computational systems for expressive and emotionally informed music generation. Livingstone et al. (Reference Livingstone, Muhlberger, Brown and Thompson2010) developed a computational rule system for modifying musical scores to express target emotions, demonstrating that systematic manipulations of tempo, articulation and dynamics could shift perceived emotional content. Kirke and Miranda (Reference Kirke and Miranda2009) surveyed the broader landscape of computer systems for expressive music performance, including affective algorithmic composition, in which emotional parameters drive generative musical processes.

These systems share a common architecture: they take emotional descriptors as input and produce musical output designed to express those emotions. Livingstone et al.’s rule system modifies tempo, dynamics and articulation to shift a score from ‘happy’ to ‘sad’. In these systems, the target emotion is specified in advance, and the music is generated to express it. Kirke and Miranda’s survey documents dozens of systems operating on this same causal axis: emotion specification drives the generative process.

TEAL inverts this direction. Musical analysis produces emotional parameters; those parameters generate text; the text generates new musical output. The emotional layer is not a target to be reached but a compositional interface between analysis and generation. Where Livingstone et al.’s system asks ‘how should music sound to express sadness?’, TEAL asks ‘what emotional profile does the current music produce, and what text and sound should that profile generate next?’ The distinction is not merely technical. It determines whether the system serves recognition or composition.

2.5 Feedback loops and cybernetic traditions

The recursive feedback loop at the heart of TEAL situates it within a lineage of cybernetic and feedback-based art practices. Wiener (Reference Wiener1948) established feedback as a principle for regulating complex systems. Pask extended these concepts into interactive art, including his Colloquy of Mobiles (Pask Reference Pask and Reichardt1971), where machines and participants engaged in adaptive dialogues. Reichardt’s Cybernetic Serendipity exhibition (Reichardt Reference Reichardt1968) demonstrated the aesthetic potential of feedback-driven systems across media.

In music, feedback principles informed electronic composition and performance art throughout the 1960s and 1970s. Collins (Reference Collins2004) documents how David Tudor’s live electronics demonstrated self-regulating, environment-sensitive systems. Biofeedback experiments sought to make physiological states audible, establishing an aesthetic of recursive coupling between human and machine.

TEAL operates within this lineage. Unlike early cybernetic art, it makes its interpretive limitations explicit. The system does not present algorithmic inference as a universal regulator but as a situated, partial and compositionally determined reading.

3.1 Design principles

TEAL was conceived not as a universal recognition system but as a piece-specific compositional agent, calibrated to the specific harmonic, gestural and timbral language of each work. Four principles guided the design:

• • Embedded operation. Real-time integration within the performance signal chain, with analysis-to-actuation latency consistently under 50 ms; the language-model text layer runs outside this tight loop, at higher latency.

• • Transparency. Selected metrics, text outputs and confidence values are made visible to performer and audience.

• • Personalisation. Analysis windows and mappings are tuned to individual compositional idioms.

• • Bidirectional generation. The system does not merely analyse music and produce outputs; it closes a loop in which generated text is reversed back into musical material.

The central design decision is that emotional inference functions as an extended compositional technique. The composer assigns arbitrary emotional statistics to musical patterns: in other words, it is the composer’s subjective understanding of specific musical gestures and structures that informs the rule set of the emotional inference engine. Examples of typical choices include mapping lingering minor thirds to ‘sad’, recurrence of the Dorian major sixth to ‘light’ or phrases articulated via the Phrygian minor ninth to ‘dark’. These assignments are compositional choices, not scientific claims, combining subjective taste and music theory, primarily based on the perception of harmonic and modal idiosyncrasies. The system tracks these patterns, fuses them with gestural data and routes the composite signal onward (Figure 1). In this sense, the emotion inference layer is part of the score.

Figure 1.

TEAL system architecture. Signal flow from instrumental performance through nested temporal analysis windows (Layer 1), higher-order harmonic analysis (Layer 2), composer-defined emotional recipe, language-model narrative generation (Synthetic Dream) and bidirectional output to Disklavier, visuals and physical actuation. The feedback loop from Disklavier output back to the performance table is shown at the top.

Figure 1. Long description

A diagram of the TEAL system architecture. The diagram represents the signal flow from instrumental performance through nested temporal analysis windows (Layer 1), higher-order harmonic analysis (Layer 2), composer-defined emotional recipe, language-model narrative generation (Synthetic Dream), and bidirectional output to Disklavier, visuals, and physical actuation. The feedback loop from Disklavier output back to the performance table is shown at the top. Layer 1 includes core feature extraction with harmonic, modal, melodic, rhythmic, and playing style analysis. Layer 2 involves higher-order analysis such as modulation detection, cadence recognition, secondary dominant detection, and tonal tension. The emotional recipe layer maps musical features and gestural energy to named parameters. The Synthetic Dream layer is the LLM Emotional Recipe Interpreter (Qwen2:7B-Instruct with a five-turn memory buffer), which interprets the emotional recipe to generate the Synthetic Dream. The actuation layer includes visual prompting, sonic prompting and physical prompting; the sonic pathway drives the MIDI generator (teal_generator.js) and the Disklavier.

3.2 Musical analysis: Nested temporal windows

TEAL’s musical analysis is performed by a 3,462-line JavaScript module running inside Max/MSP (harmony_analyzer.js, v2.5). The module employs a nested temporal architecture (the Matryoshka approach), operating across the micro-to-meso time scales described by Roads (Reference Roads2001), which maintains four concurrent analysis windows, each feeding harmonic, melodic and rhythmic feature extraction simultaneously.

Temporal architecture. A central performance table maintains pitch, velocity and timestamp for each note event, with perceptual loudness weighted using Stevens’ power law (exponent 0.6) and temporal decay functions applied over a 180-second rolling history. This table is the shared data structure from which all feature extraction proceeds.

Four nested windows read from this table in parallel, spanning timescales from onset detection to phrase-level progression. The window durations are controlled by a single timeSeed parameter that scales all four proportionally, allowing the performer to tune temporal resolution to the piece. At the finest granularity, additional micro-windows (50–200 ms) capture onset clustering and attack transients. At the coarsest, the extended window tracks harmonic progressions, modulation events and cadential functions across multiple phrases.

Harmonic analysis. Three key-detection algorithms run in parallel within the sustained window: the Krumhansl–Schmuckler probe-tone algorithm (Krumhansl Reference Krumhansl1990), Gómez’s squared Euclidean distance method using harmonic pitch-class profiles (HPCP) (Gómez Reference Gómez2006) and Temperley’s key profiles derived from the Kostka–Payne corpus (Temperley Reference Temperley1999). A weighted voting mechanism combines their outputs and reports confidence scores. The brief window performs chord recognition via template matching against 37 chord types, including extended jazz harmonies (major/minor 7ths, 9ths, 11ths, 13ths, suspended and altered chords). The extended window detects cadential functions (authentic, half, deceptive, plagal), modulation events and secondary dominant tonicisations.

Modal analysis draws on the chord-scale theory taught at Berklee College of Music (Nettles and Graf Reference Nettles and Graf1997), which identifies a characteristic tone for each mode. The system operationalises this with weighted character notes: for D Dorian, the major sixth (B natural), the tone that distinguishes Dorian from the natural minor, receives 1.5–3.0x emphasis, and chromatic notes are penalised 2.0x; these weightings are the system’s own heuristic, not part of the source theory. The system calculates modal character strength and reports it alongside key detection confidence. Tonal tension is modelled using Lerdahl’s pitch-space distance metric (Lerdahl Reference Lerdahl2001), producing a continuous tension curve that tracks the distance between the current pitch-class distribution and the local tonic.

Melodic and intervallic analysis. The module maintains rolling pitch-class histograms across each window and tracks interval recurrence by semitone degree: how often a minor third, a major sixth or a tritone appears within a given time frame. Bass register input is separated and weighted independently, enabling the system to distinguish low-register pedal tones and bass motion from upper-register melodic activity. Key history is tracked across windows, recording the sequence of detected keys and the duration spent in each.

Rhythmic analysis. The instant and brief windows extract inter-onset intervals (IOI), onset density (events per second), duration ratios between successive notes and simultaneity counts (how many notes sound at once). Onset clustering detects whether note events arrive in bursts or are evenly distributed. These rhythmic features contribute directly to the emotional recipe: high onset density with clustered attacks produces a different parameter profile than the same pitch content played sparsely.

The module outputs ten simultaneous data streams: pitch-class histograms, HPCP, IOI, duration ratios, simultaneity analysis, pattern detection, key history, progression analysis, tonal tension values and detection confidence scores. Processing latency for this analysis stage is approximately 20 ms.

A concrete example illustrates how these layers interact in composition. During a D Dorian piano improvisation, the instant window registers increasing onset density (3–11 events per second) with clustered attacks. The brief window identifies recurring minor-third intervals (D–F, A–C, E–G). The sustained window confirms D Dorian via Krumhansl–Schmuckler correlation at 78 per cent confidence. The extended window detects cadential motion as the flattened seventh (C natural) resolves to the modal centre. The composer has assigned arbitrary emotional statistics to these patterns: lingering minor-third intervals map to a ‘sad’ coefficient; recurrence of the major sixth in a Dorian context maps to ‘light’ (since B natural, the raised sixth, is the bright character note that distinguishes D Dorian from the natural minor, and its persistent presence signals modal commitment); rising onset density maps to ‘intensity’. These named parameters are compositional assignments, not psychological measurements. A different composer could assign minor-third recurrence to ‘exuberance’. The mapping is part of the piece, not part of the algorithm.

3.3 Gestural analysis and cross-modal fusion

Alongside the musical analysis, TEAL ingests body and hand tracking data via MediaPipe (Lugaresi et al. Reference Lugaresi, Tang, Nash, McClanahan, Uboweja and Hays2019), running within TouchDesigner. The MediaPipe skeleton provides continuous position data for hands, arms and upper body. Three gestural features are derived from this data: rate of motion (velocity of tracked points between frames), spatial extent (the bounding volume of the performer’s gesture field, distinguishing expansive from contracted movement) and directional contour (whether motion trends upward, outward or inward). These features are transmitted to Max via Open Sound Control (OSC) at frame rate.

The primary derived metric is energy level, computed as a weighted combination of rate of motion and spatial extent. Rapid, expansive gestures produce high energy values; slow, contracted movements produce low values. This gestural energy is fused with the musical analysis trajectories in Max. The fusion is a weighted combination: the harmonic, melodic and rhythmic analysis provides affective content (which intervals, which mode, which rhythmic density), while gestural energy modulates intensity and urgency. The weights are part of the piece-specific calibration and are set during rehearsal.

The resulting composite, the emotional recipe, is a structured set of named parameters with numerical values, updated continuously. The inference method is entirely rule-based: correlation-weighted key detection outputs feed into composer-defined mapping rules that assign named emotional values to specific feature combinations. No regression model or labelled emotion dataset is involved. The recipe represents the system’s compositional reading of the current musical and gestural material, according to rules defined by the composer, not a scientifically measured affect.

3.4 Language model: Emotional inference and Synthetic Dream

The emotional recipe is passed as structured input to a locally hosted Qwen2:7B-Instruct language model, running via Ollama for local inference with no network dependency. A Node.js proxy relays the recipe to the model and returns its output to Max.

The model receives a single structured prompt. The input is the emotional recipe rendered as a tagged list: eight named affect terms (happy, sad, delicate, powerful, soaring, light, dark and tense), three macro inferences (intensity, energy and harmonic tension), a certainty value and a dominant-emotion summary, each on a 0–100 scale, alongside the detected key, mode and dominant interval profile. The prompt instructs the model to return two linked texts. The first, the emotional inference, is terse and clinical: it opens by restating the certainty value, combines the affect terms into paired adjectives (with ‘melancholy’ reserved for passages where the sad and happy values converge) and reports intensity, energy and tension as macro descriptors of the passage’s trajectory. The second, the Synthetic Dream, translates the same recipe into oneiric imagery, opening from one of five suggested incipits, which the model paraphrases freely, and drawing on a constrained vocabulary of cosmic and microscopic figures: constellations, microbial life, deep-ocean creatures, the resonant metal plates, the silence of mountains and the light of a star. The prompt is given below, with its two runtime insertions bracketed.

[prior emotional inferences, when present, are prepended here with an instruction to estimate certainty near their running average]

Here is a real-time emotional analysis of Gadi Sassoon’s piano improvisation:

[emotional recipe]

Your task:

• – Write a paragraph with 3 sentences, 5 words each, starting with: ‘Gadi Sassoon emotional inference certainty X%’, where X is the Certainty value from the input above.

• – sentence 2, express the following in 10 words or less: Use all available data metrics to interpret the emotional character of the music. Use dry clinical terminology, machine-like. happy, sad, delicate, powerful, soaring, light, dark, tense must combine in two adjectives. intensity, energy and tension are macro inferences. For similar ranges of sad and happy use the word melancholy.

• – sentence 3, in 5 words maximum Analyse development of the music from prior analysis block.

• – Be extremely brief and data-like. The whole block is 1 continuous paragraph. Phrase your description as if you’re listening to the music, not reading values.

Then:

1. – Add a second short paragraph of 15 words about your synthetic dream based on my musical emotions.

2. – Begin with a choice of one of these, equal probability:

   1. – ‘As I, a synthetic being, listen to the resonance of a human soul, I dream a synthetic dream of…’

   2. – ‘Human emotion detected. Initiating synthetic dream sequence…’

   3. – ‘Synthetic dreamscape:’

   4. – ‘Emotional resonance engaged; machine hallucinating as follows:’

   5. – ‘Emotional vibration detected – elaborating oneiric signifier’

3. – Use the emotional input to imagine a synthetic being’s dream of cosmic or microscopic scope, going into imaginative detail about plates, constellations, cosmic phenomena, microbial life, deep-ocean creatures, the silence of mountains, the shimmering of the light of a star.

The two texts serve distinct roles. The emotional inference is a transparent read-out, projected during performance so the audience sees the system’s compositional reading in its own terms. The Synthetic Dream is the interpretative layer. In neither case does the model hear the music. It receives symbolic descriptors (pitch-class distributions, modal character, tension curves), not audio. This is an explicit design choice: the model interprets a compositional abstraction, not a perceptual experience.

Two representative outputs, generated by the system on recipes spanning the affective range of Modes of Vibration AV, illustrate the result. The first is from a luminous, modally grounded passage (78 per cent certainty):

The second is from a turbulent, atonal passage in which the system reports its own diminished confidence (41 per cent certainty):

Both texts run longer and more lyrical than the prompt’s terse word limits specify; the seven-billion-parameter model honours the format loosely, the same productive non-compliance evident in the generated music. A buffered memory of the previous five inferences gives the output temporal coherence across a performance, averaging recent certainty values and accumulating references to earlier states so the texts develop a narrative arc rather than resetting at each frame. The Synthetic Dream is not a label applied to the music; it is a generated text with its own compositional trajectory, conditioned by but not reducible to the harmonic analysis that feeds it.

3.5 Bidirectional actuator: Closing the loop

Two output pathways reverse the analysis-to-text process, closing the compositional loop: the emotional parameters drive MIDI generation, and the Synthetic Dream conditions the visuals.

MIDI generation. A dedicated generator module (teal_generator.js, v2.7, 1,407 lines) receives the emotional parameters and produces MIDI note events. The module grew from an earlier text-to-melody prototype that read the Synthetic Dream directly, matching the narrative’s affective keywords to register and dynamics and treating the stated certainty value as a control on rest density and pitch stability; the present generator works from the structured emotional parameters instead, trading literal text-parsing for fuller harmonic control while preserving the same text-to-music reversal in principle. The generator operates within four harmonic contexts, selected by the current modal analysis state, and each defines its own vocabulary of progressions and voicings. The functional context draws on tonal progressions ranging from ii–V–I and its tritone substitution to Coltrane changes; the modal context selects mode-specific progressions for each of the seven diatonic modes, such as a Phrygian cadence or Mixolydian flat-seven motion; the parallel context draws individual diatonic chords from the current mode; and the avant-garde context generates non-functional material, including quartal and quintal voicings, whole-tone and symmetric-diminished cycles, polychords and Messiaen mode-3 collections. Voicings are realised from a library of chord templates spanning triads to extended and altered jazz harmonies. The seven rhythm styles (steady, subdivided, syncopated, swing, phrase, groove and rubato) govern onset placement and subdivision, driven by an internal metronome (40–240 BPM); the rubato engine applies phrase-level tempo curves (arch-shaped phrasing, rallentando and accelerando) together with small agogic deviations, producing expressive micro-timing rather than mechanical quantisation. Because the generator emits standard MIDI, the same stream can drive electronic synthesis, an acoustic instrument or other embodied actuators. In Modes of Vibration AV, it is routed to Ableton Live, where it drives layers of synthesis shaped by custom timbral models trained on approximately eight hours of the composer’s recorded corpus; in the Montreal sessions (Section 4.2, Figure 2), it is sent directly to an acoustic Yamaha Disklavier. These models interpolate between the generated MIDI and the sonic vocabulary of the compositional practice, ensuring timbral coherence even when the generated material diverges from the performer’s live input. On an aesthetic level, the resulting MIDI output has an ‘uncanny valley’ quality, weaving between mechanical, repetitive phrases and surprisingly organic articulations.

Figure 2.

TEAL recording session at the Université de Montréal studios, 2025. Yamaha Disklavier grand piano with TEAL system visible on adjacent monitors. Photo: the author.

The reversal is the architecturally distinctive feature. Music generates an emotional recipe; the recipe generates a Synthetic Dream and, through the same parameters, new music. The emotional parameters function as a compositional interface between analysis and generation, not as a recognition target but as a transduction layer.

Visual and physical actuation. The Dream is simultaneously truncated to a short textual prompt, the brief phrasing on which the diffusion model performs best, and sent via OSC to TouchDesigner. There, the prompt conditions real-time generative visuals, including StreamDiffusion-based animations and parametric particle systems. The visuals are grafted onto the performer’s body via the same MediaPipe skeleton data used in gestural analysis, closing a visual feedback loop.

The result is a closed loop across multiple domains. The emotional parameters circulate between music, text, image and motion, functioning as the connective tissue of the system. It is this multi-domain temporal structuring, not the harmonic analysis alone, that constitutes the dramaturgy: the system organises relationships between performer, generated sound, projected text and visual environment across the duration of a performance.

3.6 Piece-specific calibration

TEAL is not a portable engine. It is calibrated to each work. In Modes of Vibration AV, calibration drew on the composer’s idiomatic language: extended modal passages in D Dorian at the piano, violin glissandi traversing consonance and microtonal tension and the resonant properties of four slashed metal plates designed in collaboration with the NEMUS lab at the University of Bologna.

Calibration proceeds in two stages. First, the analysis windows are configured with interval templates and statistical thresholds matched to the harmonic vocabulary of the work. The extended window is tuned to detect the cadential pull of flattened sevenths; the sustained window tracks the persistence of modal centres characteristic of the piano improvisation style. Second, the mapping functions, the arbitrary emotional statistics, are iteratively rehearsed, aligning the composite output with the timbral models and the physical resonances of the plates.

If the same system were applied without modification to a performance in maqam, raga or gamelan, the interval templates and chord models would fail to register the modal logic, timbral hierarchies and rhythmic cycles of those idioms. This constraint is not a flaw but a structural consequence of the design: the emotional parameters are part of the composition, and the composition is culturally situated.

3.7 Sound sculpture resonance

In Modes of Vibration AV, TEAL actuates four large metal plates fitted with transducers and pickups, which together form an analogue feedback loop. The emotional recipe maps directly to resonant excitation patterns. High-energy, high-tension values increase amplitude and low-frequency excitation, producing dense metallic drones. Low-energy, high-confidence values emphasise higher partials and produce shimmering textures. Variations in the ‘confidence’ parameter determine how stable the resonances remain. Confidence is derived from the key-detection confidence score, the correlation strength of the Krumhansl–Schmuckler match, and thus reflects the harmonic clarity of the current musical passage, not a psychological property of the performer. The composer maps this technical metric to an aesthetic function: high confidence stabilises resonance; low confidence produces instability.

The plates extend the system into the physical domain. Vibration transmits through the performance space and through the performer’s body. The performer feels the resonant energy and adjusts phrasing or dynamics in response, and the analysis windows register the change. The plates are both output and input.

4.1 UN AI For Good summit, Geneva, 2025

The world premiere at the UN AI For Good summit (Sassoon Reference Sassoon2025) provided a context in which the system’s transparency and the performer’s emotional sovereignty could be observed across a diverse audience of approximately 2,000 delegates, researchers and policymakers. The 15-minute performance comprised three sections. The first (six minutes) featured solo piano improvisation with real-time TEAL emotional inference and body tracking (Figure 3). The system’s analysis was visible through projected confidence values and textual inferences. Throughout the performance, the emotional inference and Synthetic Dream texts were composited live in dynamic projection layouts, directed by Stefano Polli of Sugo Design. The improvisation moved through tonal and atonal territories; the system’s varying confidence levels and its explicit acknowledgement when Western tonal frameworks proved insufficient were visible to the audience throughout.

Figure 3.

Sassoon performing on piano with TEAL at the UN AI For Good summit in Geneva, 2025. Visible on the projection screens are several layers of hand tracking, the emotional inference metrics and the LLM’s Synthetic Dream elicited by the performance. Photo credit: ITU/AI For Good.

The second section (three minutes) shifted to pre-composed sound sculptures manipulated live. This demonstrated a different mode of embedded agency: responsive manipulation of algorithmically generated materials rather than real-time inference.

The third section (six minutes) featured violin improvisation with body tracking and emotional inference driving live visuals, again directed by Polli. The electric violin explored extended techniques and real-time resynthesis; the visuals reflected the system’s responses, including explicit uncertainty, through animations grafted onto the performer’s skeletal tracking.

4.2 Disklavier sessions, Montreal, 2025

During a residency with the Wilding AI research-creation lab at the Université de Montréal studios in summer 2025, the author recorded a series of duets with TEAL on a Yamaha Disklavier grand piano (Figure 2). The Disklavier is a full acoustic piano that can simultaneously receive and execute MIDI input. In these sessions, the author improvised on the instrument while TEAL’s generator module produced MIDI in real time. The Disklavier played both parts on the same keyboard: the performer’s notes and the system’s generated response sounded through the same strings and hammers.

This configuration physically instantiates the bidirectional compositional loop described in Section 3.5. The performer hears TEAL’s response as acoustic piano, not as synthesised sound. The timbral consistency is total: both voices share the same instrument. The performer responds to the system’s notes as to another musician. The system analyses the combined output, including its own contributions, and generates new material. The loop is audible, physical and contained within a single resonating body. In this configuration, the dramaturgy is purely acoustic: no visuals, no text projection and no external actuation. The temporal structuring of the performer-system relationship unfolds entirely through sound.

Four completed pieces from these sessions form the recording TEAL Suite 1, Montreal: Starwhisp I. Voyage, Starwhisp II. Awaken The Sun, Starwhisp III. Cosmic Relay and Starwhisp IV. Rebirth (submitted as supplementary sound examples 1–4). A second recording, TEAL Suite 2, Vienna, is in pre-production at the Vienna Synchron Stage, where TEAL will generate Disklavier parts alongside a live chamber orchestra.

4.3 Observations

Across both performance contexts, several informal responses recurred. At the UN summit, audience members noted the significance of seeing the system’s confidence values and textual outputs. The visibility of algorithmic reasoning, particularly when the system admitted uncertainty, was among the responses most often remarked upon, distinguishing this approach from opaque recognition systems.

In the Montreal sessions, the compositional dimension was more apparent. The system amplified the performer’s compositional decisions into material that the performer alone would not have produced. The generated MIDI introduced interval patterns and rhythmic configurations outside the performer’s habitual vocabulary, while the performer’s responses steered the system’s subsequent output. The recordings capture this interaction as fixed compositions.

A recurring observation across both contexts concerned the staged withdrawal at the end of each performance, where the musical performance continued as the system’s outputs gradually faded. Several observers read this as an enactment of human autonomy: meaning-making can continue without algorithmic mediation. This withdrawal is a compositional decision. It enacts sovereignty rather than claiming it.