1. Introduction
The idea that technology can extend embodied human capacity is not new – McLuhan (Reference McLuhan1964) described media as ‘extensions of man’ – but how such extension is achieved, and when it is experienced as seamless rather than alienating, remains open. Embedding algorithms into tools for sound making now touches most aspects of electroacoustic composition and performance, raising questions of agency, autonomy and authenticity. This article examines these questions through the lens of Kinesthesis 2.0,Footnote 1 an interactive vocal augmentation system that embeds facial recognition algorithms, gesture tracking and real-time audio processing into theatrical performance – a domain whose particular demands on interactive technology Saltz (Reference Saltz2001) sets out – creating what we term ‘player–instrument–algorithm assemblages’.
Our discussion focuses on the space and mode of participation opened up by embedded algorithmic processes, specifically within the context of three theatrical productions where Kinesthesis 2.0 was deployed: Kolokotronis Memoirs (2021), Rayman Scream (2024) and Antigone’s Lament (2023), the latter featuring human–AI co-improvisation with Somax2, a distributed co-creative system for real-time human–machine musical improvisation developed at IRCAM (Fiorini and Malt Reference Fiorini and Malt2023). The research emerges from over two decades of composition for theatre, dance and film across international institutions including Stanford University’s Center for Computer Research in Music and Acoustics (CCRMA), IRCAM Centre Pompidou, Brooklyn Academy of Music and the National Theatre of Greece.
We combine two modes of thinking: one informed by discourse on embedded algorithms in electroacoustic music, the other by reflection on our own experience of implementing and performing with the system. The approach is qualitative and practice-based rather than experimental, combining technical measurement with artistic observation and theoretical interpretation, and maintaining throughout a distinction between measured specifications, performance-based observations and theoretical framing. The performances function not merely as illustrative examples but as sites of inquiry through which artistic practice generates knowledge.
Unlike traditional vocal effects processing, which typically requires manual parameter adjustment, Kinesthesis 2.0 creates a direct morphological relationship between bodily movement and sonic transformation. The system captures facial expressions and hand gestures using Google’s MediaPipe machine-learning framework (Lugaresi et al. Reference Lugaresi, Tang, Nash, McClanahan, Uboweja, Hays, Zhang, Chang, Yong, Lee, Chang, Hua, Georg and Grundmann2019), translating these into Open Sound Control (OSC) messages (Wright Reference Wright2005) that modulate a four-voice vocoder, pitch shifters, delays and reverb effects in real time.
A recurring concern is that algorithmic mediation severs the audience’s perception of cause and effect between what the performer does and what is heard. Schloss (Reference Schloss2003) argues that once computers interpret abstract gesture to generate sound, the age-old physical link between gesture and result can become imperceptible; Emerson and Egermann (Reference Emerson and Egermann2018) confirm this empirically, reducing perceptible gesture–sound causality significantly lowering both perceived causality and the positive dimension of audience response. A related concern, focused on the performer’s own experience rather than the audience’s, holds that algorithmic mediation risks reducing the body to a control interface and the voice to raw data subordinated to computational processing. Kim and Seifert (Reference Kim and Seifert2007) frame this as a problem of disembodiment in computer and interactive music, arguing that embodiment is a necessary condition of genuine interactivity rather than an optional feature of it. Nor is the concern hypothetical: in Torpey’s (Reference Torpey2009) ‘disembodied performance’, expressive intent is abstracted from the body and redistributed across projected media. We argue that Kinesthesis need not fall into either dilemma, provided the gesture vocabulary is grounded in what the voice is already doing rather than imposed as an arbitrary control layer on top of it.
When a performer raises her hand while singing an ascending phrase, the gesture does not embody the vocal-technical mechanics producing that pitch – breath pressure, laryngeal tension, resonance adjustment – which remain interoceptive and untracked. It embodies the performer’s real-time perception of the sound’s trajectory: rising hand height corresponds to rising perceived pitch, a correspondence long exploited in gestural instrument design. In Kleinberger et al.’s (Reference Kleinberger, Singh, Xiao and van Troyer2022) taxonomy, this positions the gesture as tracking the voice in its ‘embodied characteristics of movement, gesture or motor control’ – the heard dimension of vocal sound – rather than as a trace of interoceptive production (Section 2.3). When the algorithm modulates harmonic processing in response, it augments this perceptually grounded layer rather than controlling vocal production itself.
The ‘embodied algorithmic instrument’ – extending Wessel’s (Reference Wessel2006) enactive account, in which sensory-motor engagement rather than symbolic control defines the performer’s relation to the system – emerges from collaboration between embodied vocal knowledge, the gesture expressing its perception in sound, and the algorithm’s responsive augmentation.
Such embedding raises fundamental questions: when algorithms mediate between gesture and sound, how does performer agency transform and what forms of co-creation emerge when human intention negotiates with algorithmic interpretation?
2. Theoretical foundations: embodied cognition and gestural expression
Our work is grounded in embodied cognition: cognition is not the representation of a pregiven world but the enactment of world and mind through lived action (Varela et al. Reference Varela, Thompson and Rosch1991; Thompson Reference Thompson2007), a position Leman (Reference Leman2007, Reference Leman2010) applies to music, shifting understanding from disembodied processing to bodily engagement as the ground of meaning-making.
For vocal performance, embodiment is especially intimate: the voice emerges from integrated systems of respiration, laryngeal control and resonance, and trained singers develop refined proprioceptive awareness of breath support, pitch intention and micro-coordination – knowledge that is felt rather than calculated. As Afsar et al. (Reference Afsar, Luft, Cotton, Stepanova, Núñez-Pacheco, Kleinberger, Ben Abdesslem, Ishii and Höök2023) show with the Corsetto haptic garment, interoceptive awareness is central to vocal practice and can be supported through somatic design. Kinesthesis 2.0 builds on this foundation: it does not replace vocal agency but augments embodied vocal knowledge through gestural and algorithmic extension.
2.1. Gesture as expression of embodied vocal perception
Within embodied cognition, gesture is integral to expression (Leman Reference Leman2007) – though the register at which it expresses is not uniform. A conductor’s raised hand is continuous with the conductor’s own gesture vocabulary for shaping phrasing. In Kinesthesis 2.0, by contrast, a raised hand during an ascending vocal line expresses the performer’s embodied perception of that pitch trajectory, not the respiratory and laryngeal work that technically produces it (Section 2.3): the gesture is embodied in relation to the heard dimension of the voice’s sound, not a direct trace of interoceptive vocal-technical knowledge.
When the algorithm modulates processing in response, it amplifies this perceptually grounded gesture rather than imposing external control – Deleuze and Guattari’s (Reference Deleuze and Guattari1987) striated and ‘smooth space’: computational systems inevitably structure gesture into discrete mappings, but culturally grounded vocabularies let performers inhabit those structures as meaningful expressive spaces.
2.2. Player–instrument–algorithm interaction
Rather than treating Kinesthesis 2.0 as a tool controlled by a performer, or as an autonomous system operating independently, we understand player, voice, gesture and algorithm as co-constitutive participants in vocal performance. Agency emerges relationally, through interaction, rather than residing in any single agent.
Latour’s actor-network theory (Reference Latour2005) grounds this: agency emerges from ‘associations’, the concrete interactions through which heterogeneous human and technological actors form functioning networks. Performer, gesture vocabulary, algorithmic interpreter, effects processor and audience form an assemblage in which no single component creates musical meaning independently; the audience’s part in it is treated here theoretically rather than investigated empirically (Section 7.2). We return to this assemblage in Section 6, where Jane Bennett’s (Reference Bennett2010) new-materialist account of ‘vibrant matter’ allows its non-human components to be described not simply as nodes in a network but as materials with their own capacities to affect and be affected.
Through rehearsal, gesture–augmentation mappings become what we term embodied fluency: executing gesture-to-sound mappings without conscious deliberation – an instance of Wessel’s (Reference Wessel2006) enactive, sensory-motor basis of skilled computer music performance. We assess it through performers’ rehearsal reports and our own observation of reduced hesitation and increased expressive variation across rehearsals – a qualitative, practice-based judgement, not a controlled measure. These observations were recorded informally during rehearsals and in discussion afterwards; they were not coded as behavioural data. The performer does not learn to ‘use’ the system as an external tool but internalises it as somatic knowledge, as performers internalise any gesture vocabulary. The system does not assume that performers arrive with training in the source tradition; what is at stake is whether a vocabulary grounded in an inherited tradition can become embodied through rehearsal.
This perspective challenges both the fantasy of total human control and algorithmic determinism, proposing collaborative emergence grounded in embodied practice.
2.3. Voice as embodied practice: beyond signal processing
A central premise is that the voice is not a signal source but an embodied instrument, integrating respiration, laryngeal positioning and resonance shaping (Foster Reference Foster2008; Lotze Reference Lotze2013); vocal training involves micro-coordination developed through kinesthetic awareness rather than verbal instruction alone (Afsar et al. Reference Afsar, Luft, Cotton, Stepanova, Núñez-Pacheco, Kleinberger, Ben Abdesslem, Ishii and Höök2023).
Cultivated kinesthetic awareness fosters the interdependence of the breathing and phonatory mechanisms, and nurtures mastery and dependability of vocal skill (Bauer Reference Bauer2016); vocal artistry stands in relation to mimesis that Murdoch (Reference Murdoch2012), reading Plato, ascribes to the performing arts.
This embodied account should not be taken to imply that voice is a natural given. Eidsheim (Reference Eidsheim2019: 9) argues that voice is not singular but collective, not innate but cultural – a claim she develops through the racialisation of vocal timbre, but whose core bears on any account of vocality. Crease and Lutterbie (Reference Crease, Lutterbie, Krasner and Saltz2006) argue in parallel that technique is unavoidable: there is no purely natural behaviour, so the question is not whether a performer works within an inherited scheme but which scheme, and how well it fits what the performer already does. Kinesthesis takes both points seriously: vocal sound is produced through the trained body, and what the body has been trained into is culturally specific. This is why the system’s gesture vocabulary is grounded in inherited theatrical traditions rather than designed from first principles (Section 4.1). Understanding voice as embodied and culturally formed knowledge – rather than as technical mechanism – shapes the design philosophy of both Kinesthesis iterations: algorithmic integration extends embodied vocal practice instead of substituting it.
3. Gesture, communication and embodied knowledge
Ancient Greek tragedy is generally understood to have drawn on formalised acting techniques in which gesture, alongside the mask, indicated emotional intensity through signs rather than words (Rush Reference Rush2017) – anticipating Agamben’s gesture as ‘communication of a communicability’, the body embodying language until it becomes language itself (Agamben Reference Agamben2000; Hallensleben Reference Hallensleben2010). Kinesthesis draws on the same principle.
Contemporary performance likewise draws on accumulated aural gesture, meaningful through its relation to prior performances (Strachan Reference Strachan2013); gesture thus both expresses meaning and teaches embodied knowledge – a dual function essential to Kinesthesis. Kleinberger et al. (Reference Kleinberger, Singh, Xiao and van Troyer2022) distinguish the voice ‘after leaving the body as a sound signal’, ‘in its embodied characteristics of movement, gesture or motor control’ and ‘in its pre-embodied form in the purely internal signal of an inner voice’; Kinesthesis operates across all three levels, positioning gesture as an extension of vocal embodiment; on Krauss’s (Reference Krauss1998) account, gestures accompanying speech are implicated in its production rather than merely illustrative of it. Culturally grounded vocabularies integrated into computational systems create what we might call ‘pedagogical smooth spaces’ (after Deleuze and Guattari): zones where performers learn as dancers and actors have always learned – through repeated embodied practice with gestures carrying cultural and emotional meaning, consistent with Wessel’s (Reference Wessel2006) account of enactive, sensorimotor learning as against symbolic instruction. A mudra is not merely a control input: it embodies centuries of theatrical tradition, and algorithmic systems can serve as scaffolds through which performers deepen embodied practice.
The computational mediation becomes invisible – in Latour’s sense of a technology so well integrated into practice that its machinery recedes from awareness – not through technical simplification but through cultural depth. This is the paper’s central claim: we propose that by rooting gesture in meaningful cultural tradition rather than by minimising the technology, performers can internalise the system into their embodied knowledge as they would any inherited gestural vocabulary. Cultural depth, as we use the term, is a relational property rather than a property of the gesture, the tradition or the interface taken separately: it obtains when an interface draws on a culturally meaningful principle of gestural organisation instead of assigning arbitrary significance to bodily configurations.
Kinesthesia – from the Greek for movement and sensation, and the system’s namesake – denotes the experience of one’s own movement through the sensations it generates; theatrical spectatorship is itself sensorimotor in grounding, audiences inhabiting performers’ movements in a bodily register shared between stage and auditorium (Garner Reference Garner2018).
Two perceptual mechanisms would account for this recession from awareness. The first is gestalt integration across modalities: Vines et al. (Reference Vines, Krumhansl, Wanderley and Levitin2006: 106) show that a performer’s expressive movements and the accompanying sound are not perceived as parallel streams but form a gestalt from which observers extract musical structure. Where gesture and augmented vocal sound are congruent, the algorithmic layer is absorbed into a single expressive event – the perceptual counterpart of Schloss’s concern, and, given Emerson and Egermann’s finding, a condition of such performances being valued.
The second is content-driven immersion. Machon (Reference Machon2013) grounds immersion in the body’s felt response to its medium – being made ‘utterly aware… of our body and its instinctive response’ – rather than in the apparatus producing it; Biggin (Reference Biggin2017: 61) adds that immersion is never guaranteed by a technology and that any sense of force breaks it. The same would hold for the performer: we interpret the lament, the mask and the mudra’s semantic charge, rather than the gesture recognition, as what sustains that engagement. The mediation recedes not because it has been made small but because attention is occupied by material carrying its own significance.
4. System architecture: evolution from Kinesthesis 1.0 to Kinesthesis 2.0
The system has evolved across two iterations, each representing design choices intended to better support embodied vocal practice and performer agency. Both ground gesture in cultural traditions – ancient Greek theatrical schemata and Kathakali mudra philosophy – and engage with contemporary electroacoustic vocal practice (Collins and d’Escrivan Reference Collins and d’Escrivan2017; Emmerson Reference Emmerson2018).
4.1. Cultural grounding: Greek theatrical schemata and Kathakali mudras
Ancient Greek theatre employed stylised systems of gesture, movement and vocal delivery to communicate emotional and narrative content (Roisman Reference Roisman2014) – a field of non-verbal codes now consolidated within Classical Studies (Capponi Reference Capponi2021; De Santis Reference De Santis2024). This tradition shaped ancient Greek music theory, particularly prosody (Georgaki et al. Reference Georgaki, Psaroudakis, Carlé and Tzevelekos2009; Probert Reference Probert, Montanari, Matthaios and Rengakos2015), and vocal delivery was inseparable from the actor’s embodiment.
Kathakali, the classical dance-theatre form of Kerala, employs a codified system of hand gestures whose scale – eleven ways of casting a glance, sixty-four motions of the limbs, mudras as ‘signs’ constituting the alphabet of the acting language – is documented by Barba and Sanzenbach (Reference Barba and Sanzenbach1967). The 24 foundational Hasta Mudras (see Figure 1) each carry specific semantic significance.
The 24 foundational Hasta Mudras.
Drawing on a highly developed non-western tradition raises Emmerson’s (Reference Emmerson and Emmerson2000) question: whether technology can cross cultural boundaries without reducing the borrowed tradition to raw material, composers having too often ‘been allowed to get away with cultural murder’. Two relations should be distinguished. In one, the tradition is itself the object of analysis – classifiers now identify Kathakali mudras from video (Mulerikkal et al. Reference Mulerikkal, Janani, Lakshmi, Sreelakshmi and Sajanraj2024) and the nine navarasas from made-up faces (Iyer et al. Reference Iyer, Saengarunvong and Le2024), for documentation and access (Ng et al. Reference Ng, Kang and Iyer2026). In the other, which is ours, a principle is drawn from the tradition into an instrument that is not about that tradition.
Our position is deliberately narrow, a considered choice rather than a technical limitation: both mudra and navarasa classification are demonstrably achievable, and we attempt neither. Kinesthesis does not recognise or reproduce traditional mudra forms, does not present itself as an aid to the preservation of Kathakali and does not ask performers to execute Kathakali technique; its facial tracking registers geometric quantities, not expressive categories (Section 4.3). What is drawn is a principle of gestural organisation – gesture as discrete, meaningful, embodied expressive unit (Mainsbridge Reference Mainsbridge2022) – not a repertoire of codified signs; the claim to cultural depth (Section 3) rests on that principle being operative in performance, not on traditional authenticity. Such adaptation has its own history: Barba studied Kathakali precisely to assess adapting its technique for European actors.
Throughout what follows, ‘mudra’ denotes a small set of operational gesture states informed by the organising principle of mudra, not the computational recognition of canonical Kathakali forms. The simplicity of the technical implementation paradoxically enables deeper embodied engagement: performers develop embodied fluency through rehearsal (Section 2.2), focusing creative energy on expression rather than system mastery.
4.2. Kinesthesis 1.0: spatial gesture-to-parameter mapping (2021)
Kinesthesis 1.0 employs a Microsoft Kinect V1 depth sensor for hand tracking at 2 metres’ distance, generating continuous data mapped directly to audio parameters (Kean et al. Reference Kean, Hall and Perry2011); a Processing sketch extracts hand coordinates (x, y, z) and transmits them via OSC over UDP to Max/MSP (see Figure 2). The Kinect was already established as a musical controller (Yoo et al. Reference Yoo, Beak and Lee2011; Bellona Reference Bellona2012); Kinesthesis 1.0 differs in tracking not a skeleton of joints mapped to discrete musical events but a singer’s hand as continuous modifier of the singer’s own live vocal signal. The operation of the Kinesthesis 1.0 system is illustrated in Movie Example 1, Kinesthesis 1.0 demonstration.
Max/MSP and sensor integration.
Hand position mapped to three continuous parameters: x-axis to volume (0–127), y-axis to pitch shifting (±4 octaves, transposer or gizmo mode), z-axis to reverse-delay percentage.
Additional controls – pitch-shifting mode and optional MIDI instrument modes – supported multiple augmentation approaches: singing, Sprechstimme, speech in vague tonal heights, verbal recitation and combinations enabling mode switching (Taylor Reference Taylor2014).
4.2.1. Deployment (Kolokotronis Memoirs, 2021)
The system was deployed in 21 performances at the History Museum of the National and Kapodistrian University of Athens (examined as a case study in Section 5.1).
4.2.2. Limitations
While 1.0 enabled real-time gestural augmentation, its continuous spatial mapping proved disconnected from vocal embodiment in a specific sense: raising the hand to raise pitch requires learning an arbitrary convention rather than drawing on any pre-existing embodied connection between hand height and vocal production. What the gesture tracked was the performer’s perception of the pitch contour (Section 2.1), not the vocal-technical mechanics that produce it (Section 2.3). This motivated the redesign toward a discrete, culturally grounded vocabulary.
4.3. Kinesthesis 2.0: three-tier embodied harmonic augmentation (2024–2025)
Kinesthesis 2.0 reconceptualises gesture-to-voice mapping, moving from the continuous spatial metaphors of 1.0 (Section 4.2) to a discrete, culturally grounded vocabulary combined with real-time four-voice polyphonic performance and facial expression integration.
4.3.1. Hardware
The system uses MediaPipe’s Holistic model (468 facial landmarks, 21 hand keypoints per hand) from standard webcam input; the markerless design eliminates dedicated sensors, enabling unencumbered expressivity (Sonami Reference Sonami2006). Its operation is illustrated in Movie Example 2, Kinesthesis 2.0 demonstration.
The system organises gesture control into three tiers, moving from discrete state selection, to continuous multi-dimensional shaping, to facial expression-driven modulation.
4.3.2. Tier 1 – Mudra classification (discrete harmonic state selection)
A binary logic system checks each finger against a threshold, producing a bitmask that identifies the hand configuration. The number of configurations is a compositional decision rather than a fixed property of the system: MediaPipe’s landmark data support whatever set the mapping defines, and each configuration in use sends its own gesture ID over OSC. On recognition, the gesture ID instantly re-maps the four-voice vocoder array’s pitch quantisation or base frequency, changing scales, chords or the entire effect chain – gesture as a meaningful unit independent of spatial position.
4.3.3. Tier 2 – Multi-dimensional harmonic shaping (continuous polyphonic performance)
Once the Mudra establishes the harmonic context – the active scale or chord palette – each of four fingers controls one vocoder voice independently:
-
• X -axis (interval engine): four fingertip positions are tracked independently, enabling non-parallel harmonies – a pedal note against a glissando, moving clusters, counterpoint within the vocoder’s spectral output (Momeni and Henry Reference Momeni and Henry2006; Feugère et al. Reference Feugère, Le Beux and d’Alessandro2011).
-
• Y -axis (timbral expression): raising a finger increases that voice’s filter cutoff – ‘opening’ its frequency response – for independent timbre and volume per voice.
-
• Z -axis (spatialisation and time-effects): distance from the camera controls reverb and delay, moving the hand toward it sending the sound into deeper acoustic space.
4.3.4. Tier 3 – Embodied timbral modulation (facial expression layer)
The system tracks head angle, the degree of openness of the mouth and the openness of the eyes via MediaPipe’s face landmarks (see Figure 3). While the hands control structure, the face controls texture: mouth opening modulates the formant shift according to the singer’s own vocal effort, head angle modulates filter cutoff and timbral grain, and eye openness adds further nuance – an embodied connection between internal vocal expression and external sound texture (Hewitt Reference Hewitt2006).
Facial expression and gesture control.
4.3.5. Technical performance
Measured end-to-end latency is 11.05 ms (±2.1 ms, n = 100 trials, MacBook Pro M2). Of this, 4.65 ms is accounted for by the four identifiable pipeline stages – MediaPipe detection (2.10 ms), Python gesture classification (0.90 ms), OSC transmission (0.20 ms) and Max/MSP audio processing (1.45 ms buffer) – the remaining 6.40 ms being system overhead and jitter (see Figure 4). Wessel and Wright (Reference Wessel and Wright2002) propose 10 ms as an acceptable upper bound on a computer’s audible reaction to gesture, with timing variation held below 1 ms. Our figures sit marginally above both, and we report them as measured rather than claim the criterion is met. Performers in all three deployments described the response as immediate, and the perceptual integration discussed in Section 3 was not reported to break down; but the system operates at the edge of the Wessel and Wright criterion rather than comfortably within it, and tightening the remaining overhead is the clearest technical priority for further development.
Latency analysis.
This also means the environmental variability discussed in Section 7.1 has little margin to absorb before responsiveness is affected.
4.3.6. Design philosophy
The evolution from 1.0 to 2.0 refines how gesture is grounded, from spatial metaphor toward a vocabulary with direct cultural and semantic meaning (Section 4.2). The two are not mutually exclusive. A spatial metaphor maps a continuous dimension onto a parameter through learned convention – hand height stands for pitch height only because the performer has learned it – whereas a discrete gesture signifies through configuration, its meaning independent of where the hand is held. Kinesthesis 2.0 uses both in distinct roles: configuration selects semantic and harmonic state, and position modulates continuously within it (Tier 2). Kinesthesis 1.0 had only position, and so carried semantic weight on a dimension that signifies only by convention.
Performers reported reaching embodied fluency (Section 2.2) within 3–5 rehearsals in the two deployments for which we have a precise count; we hypothesise, but do not establish, that the combination of discrete vocabulary, four-voice polyphonic control and facial modulation contributes to this learning while preserving artistic agency (Section 7). The performer becomes simultaneously performer and composer, manipulating voice and its harmonic augmentation as unified material in real time (Wu Reference Wu2015) – an ‘augmented’ or ‘hyper-instrument’, in which sensors let gestural activity control augmentations of instrumental sound (Bowers and Archer Reference Bowers and Archer2005).
5. Case studies: embodied vocal practice in performance
The case studies span the evolution from Kinesthesis 1.0 (Kolokotronis Memoirs, 2021) to 2.0 (Rayman Scream, 2024; Antigone’s Lament, 2023); where technical set-up matches Sections 4.2–4.3, we cross-reference rather than repeat it.
5.1. Kolokotronis Memoirs (2021): spatial gesture control in historical drama
Kolokotronis Memoirs, an interactive theatrical documentary directed by Marianna Lampiri, ran for 21 nights (27 July–16 August 2021) at the History Museum of the National and Kapodistrian University of Athens, as a research project of the Laboratory of Music Acoustics and Technology (LabMAT) under the scientific direction of Anastasia Georgaki, funded by the Hellenic Foundation for Research and Innovation (project 4_150, 2020–22) (National and Kapodistrian University of Athens 2021; Hellenic Foundation for Research and Innovation n.d.) – developed within a funded research programme rather than as an isolated commission, which bears on the methodological status of the observations below.
5.1.1. Artistic context
The performance placed Kolokotronis – a symbol of the Greek Revolution – as a living ‘exhibit’ within an imaginary historical museum, framed by soundscapes, live music, video art and live filming. Three actors and a clarinettist performed live; Manos Vakousis created vocal soundscapes through voice and hand gesture in what we term a ‘heterotopic pursuit of history’ (see Figure 5; CultureNow.gr 2021).
Scene from Kolokotronis Memoirs.
5.1.2. Technical implementation
Kinesthesis 1.0 was employed as described in Section 4.2, the actor’s voice captured via lavalier microphone and processed in real time in Max/MSP.
5.1.3. Performer observations
Vakousis reported that only a few rehearsals were needed to feel comfortable – one instance of the embodied learning discussed in Section 2.2. Initially unsure about the reverse delay, he enjoyed improvising with pitch shifting and volume dynamics, composing what he called aleatoric soundscapes from his own hand-gesture choreography (see Figure 6); notably, he selected part of that movement vocabulary himself in rehearsal, locating the compositional decision with the performer rather than the designers (Section 6). Georgaki (Reference Georgaki2021) described the techniques as rendering the performer ‘an executor of the acoustic memories’ of the narrative (our translation), and the performance was nominated for the Grand Prize at the 2020/21 Athens Theatre and Music Critics’ Awards (kolokotronisonstage.gr 2022).
Kolokotronis Memoirs (2021): Gesture-driven harmonic shifts in historical drama.
5.1.4. Artistic outcome
The system let the performer manipulate his voice as material in real time, simultaneously performer and composer; Diamantakou and Fanouraki (Reference Diamantakou and Fanouraki2021), theatre scholars serving as scientific advisors to the production, describe its tools as ‘extensions of the protagonist’s thought and action’ (our translation). In our own assessment, the gesture-driven modulation deepened the narrative’s emotional resonance, though we did not measure audience response; this remains a practitioner observation.
5.2. Rayman scream (2024): Kinesthesis 2.0 in contemporary theatre
Rayman Scream (March 2024, Theatre ‘Sfendoni’, Athens) marked the first theatrical deployment of Kinesthesis 2.0. An independent production by the company Askisi, it was awarded €30,000 and the auspices of the Greek Ministry of Culture in an open call selecting 91 of 219 proposals (Greek Ministry of Culture 2023).
The ensemble used Kinesthesis 2.0 to modulate the soprano’s and the main actor’s vocal output, creating a blend that simulated a digital choir and synchronised vocal elements with dramatic expression; Roubis (Reference Roubis2024) described the music and lighting as creating ‘a mysterious, timeless, undefined landscape’ (our translation). The three-tier system (Section 4.3) tracked both performers’ hands and faces simultaneously; rehearsals focused on developing embodied fluency (Section 2.2), with performers discovering that specific mudras triggered coherent harmonic environments within which individual finger movements created meaningful polyphonic variation.
5.2.1. Artistic outcome
The system extended real-time vocal textures and harmonies beyond what vocal technique alone achieves; facial tracking mapped visible facial movement to sonic texture – not a capture of emotional state, which cannot be reduced to facial position (see Figure 7).
Rayman Scream (2024): Soprano and actor – real-time choir simulation.
Reviewing for Αυγή, Polenakis (Reference Polenakis2024) described the music as ‘a somatisation of sound – an organic element and autonomous material of the performance’ (our translation) – a response to the sound design as a whole rather than to Kinesthesis specifically, but consistent with our aim of positioning augmented vocal sound as compositional material. Four further reviews converge on the same pattern (Logothetis Reference Logothetis2024; Papagiannakis Reference Papagiannakis2024; Tsatsoulis Reference Tsatsoulis2024; Toli Reference Toli2024), and the music was shortlisted for the Music/Music Design award of the Hellenic Association of Theatre and Performing Arts Critics (Baketea Reference Baketea2024).
Performers again reported reaching embodied fluency (Section 2.2) within the same 3–5-rehearsal range as in Kolokotronis Memoirs and reported the three-tier system feeling more directly tied to their own vocal choices than 1.0’s spatial mapping had (Section 4.3).
5.3. Antigone’s Lament (2023): Human–AI co-improvisation with Kinesthesis 2.0 and Somax2
Antigone’s Lament is the most complex deployment of Kinesthesis 2.0, integrating it with Somax2 (Fiorini and Malt Reference Fiorini and Malt2023) for real-time co-improvisation, and unifying embodied vocal practice, gestural augmentation and algorithmic co-creation within a single performance – continuing an older lineage of vocally mediating Greek tragedy through worn apparatus. The ancient mask is the closest precedent: Wiles (Reference Wiles2000: 151) records that the classical head mask concentrated and directed the actor’s voice – ‘the mask as a whole works as a resonator for the head’, in Vovolis’s formulation – apparatus reshaping vocal output without displacing vocal production, the relation Kinesthesis seeks by other means (see also Wiles Reference Wiles2007: 153–79; Vovolis Reference Vovolis2012).
In the contrasting ‘data-driven voice-body’ of AI voice conversion (Reus Reference Reus2026), an absent body’s voice is animated by a present one, reopening Eidsheim’s (Reference Eidsheim2019) acousmatic question – ‘Who is this?’ Kinesthesis works in the opposite direction: the voice is the performer’s own, produced live, its augmentation tied visibly to the gesture shaping it.
5.3.1. Artistic context and concept
The performance drew on Sophocles’ Antigone, exploring human–AI interaction through Antigone’s final lament: Kinesthesis 2.0 for voice processing, Somax2 for generative clarinet responses, and a custom graphic score guiding real-time gesture–voice interaction (see Figure 8).
Graphic score.
Figure 8. Long description
Panel A: The score begins with both hands in a specific gesture, indicating a starting point for the performance. The musical notation shows a series of notes with instructions to move up and closer to the camera for reverb and lower for a lower pitch. Panel B: The score continues with various hand gestures and corresponding musical notations. Instructions include moving up for reverse delay and reverb, and lowering for a lower pitch. Panel C: The score features a flowing gesture for glissando, with instructions to move up and closer to the camera for more reverb. Panel D: The final section of the score includes a hand gesture and musical notations, with instructions to move up for more delay, reverb, and higher pitch.
5.3.2. Kinesthesis 2.0 implementation
Kinesthesis 2.0 processed the performer’s voice in real time, transforming the lament lyrics into a lament aria through dynamic vocal effects; Table 1 sets out the correspondence between text, mudra and semantic association (see also Figure 8).
Mudra–text correspondence in the Antigone’s Lament graphic score
Table 1. Long description
A table with 8 rows and 4 columns. The columns are labeled Line, Greek text/translation, Mudra, and Semantic association. Row 1: Line a, Greek text/translation ‘ὁρᾴτ᾽ ἐμέ, ὦ γῆς πατρίης πολ.’, Mudra Pataka, Semantic association Addressing attention. Row 2: Line b, Greek text/translation ‘τὰν νέαταν ὁδὸν στϵίχουσαν’, Mudra Karthareemukham, Semantic association Journey/road. Row 3: Line c, Greek text/translation ‘νέατον δὲ φέγγος λέυσσουσαν ἡλίου’, Mudra Tripataka, Semantic association Celestial objects. Row 4: Line d, Greek text/translation ‘ἀλλὰ μὸ παίκοιτας Ἀΐδας ζῶσαν ἄγϵι’, Mudra Shikhara, Semantic association Divine power. Row 5: Line e, Greek text/translation ‘τὰν Ἀχέροντος ἀκτάν’, Mudra Sarpasirassu, Semantic association Flowing water/rivers. Row 6: Line f, Greek text/translation ‘οὐθὸς ἡμϵνίων ἐγκλίρων’, Mudra Hamsasya, Semantic association Delicate concepts. Row 7: Line g, Greek text/translation ‘οὐθὸς ἐπ’, Mudra Kapitham, Semantic association Sorrow. Row 8: Line h, Greek text/translation ‘ἀλλὰ Ἀχέροντι νυμφϵίσω’, Mudra Ardhachandra, Semantic association Union/connection.
Hand-gesture mapping provided continuous modulation of pitch, reverb and delay while facial expression modulated timbre, mapping the performer’s own vocal and facial articulation onto the augmented sound.
5.3.3. Somax2 integration and human–AI dialogue
Somax2, trained on a corpus of Epirotic dirge clarinet recordings, generated real-time responses to the performer’s voice: onsets segmented it into phrases, melodic and rhythmic features were extracted and mapped generatively onto Epirotic lament patterns, updated through real-time listening.
The interaction topology is a nested feedback loop: Kinesthesis 2.0 acts as primary interface, its four-part harmonic output routed into Somax2, which – analysing dense harmony rather than a monophonic signal – identifies melodic trajectories and answers with clarinet lines from the Epirotic corpus, in effect conducted by the performer’s gestural harmonic shifts (see Figure 9). This corresponds to Emmerson’s (Reference Emmerson2015) model of listening with machines: a partner that ‘enhances and does not offload’ the performer’s skills.
Interaction topology of the nested player–instrument–algorithm assemblage.
Figure 9. Long description
Panel 1: [PERFORMER ACTION] This panel shows a singer providing vocal input through a real-time microphone capture and gestural/facial input through video capture of hand mudras and facial expressions. Panel 2: [KINESTHESIS 2.0: HARMONIZATION LAYER] This panel depicts the processing of MediaPipe extracting 468 facial landmarks and 21 hand keypoints. Gestures control the four-voice vocoder and harmony, resulting in a dense, real-time four-part vocal harmony output. Panel 3: [SOMAX2: AI IMPROVISATION LAYER] This panel illustrates machine listening that analyzes the processed four-part harmony rather than raw voice. Generative clarinet improvisations are in dialogue with the harmony, using melodic motifs mapped to the Epirotic Clarinet Corpus. Feature extraction involves low-latency spectral flux and Yin-based pitch tracking. Panel 4: [DISTRIBUTED AGENCY LOOP] This panel shows feedback where the AI output reshapes the performer’s subsequent expressive intentions.
The human–AI co-improvisation process and the resulting vocal transformations are demonstrated in Sound Example 1, Antigone’s Lament.
5.3.4. Embodied learning and performer agency
The eight mudras defined for this production, integrated with the graphic score, created a structured yet improvisational framework: the score indicated which mudra corresponded to which lyrics, and the performer developed fluency with both the vocabulary and the dialogue with Somax2 through rehearsal (Section 2.2).
The performer maintained substantial creative agency: mudra selection determined the harmonic context, finger positions enabled real-time harmonic and timbral variation, and facial modulation refined texture. Somax2’s responses, while algorithmically generated, were shaped by the performer’s vocal input rather than being predetermined.
Claiming that agency is maintained is only meaningful if its loss is specifiable, and the literature supplies concrete forms: Torpey’s (Reference Torpey2009) disembodied performance, where expressive intent is redistributed away from the body; Emmerson’s (Reference Emmerson2007) sublimation of the body in acousmatic practice; Emmerson’s (Reference Emmerson2015) partnership that offloads a skill rather than enhancing it; and Reus’s (Reference Reus2026) voice-body in which a present body animates an absent one. In each case, what is lost is not control over parameters but the performer’s position as origin of the expressive decision. Three features locate that position with the performer here: the harmonic context is selected by gesture, not proposed by the system; the corpus was chosen for its relation to the dramatic material; and the performer’s own live voice, not a model of it, is the material processed. The design could forfeit agency in specifiable ways – autonomous selection of harmonic context, or a designer-fixed mapping – and we state this to make the claim testable rather than rhetorical.
5.3.5. Artistic outcome
In our own assessment, the performance suggests that human–AI co-creation need not diminish human creativity: Kinesthesis 2.0 and Somax2 functioned as partners in a dialogue between vocal practice, culturally rooted gesture and algorithmic reinterpretation of tradition. We did not measure audience reception; this is artistic goal, not demonstrated effect.
5.3.6. Implications for voice embodiment and algorithmic integration
Antigone’s Lament directly addresses the limitation of Kinesthesis 1.0 (Sections 4.2, 5.1): there, spatial position required a learned metaphorical mapping to pitch; here, facial expression directly modulates the vocoded texture, the mudra vocabulary is drawn from inherited theatrical practice rather than devised as arbitrary control, and Somax2’s responses are generated from the performer’s own vocal output.
This performance suggests that algorithmic augmentation can extend rather than displace vocal expression, provided three conditions hold: the performer remains the origin of the expressive decision rather than its executor; the augmentation enlarges the range of sound the performer can act upon rather than substituting for a skill – in Emmerson’s (Reference Emmerson2015) terms enhancing, not offloading; and the gesture vocabulary is continuous with a practice the performer already inhabits. Where ‘deepen’ is used below, it means this enlargement of actionable expressive range under performer control, not an intensification of feeling, which we did not measure.
6. Implications: toward embodied algorithmic co-creation
The evolution from 1.0 to 2.0 across three deployments suggests insights extending beyond technical implementation to performer agency, cultural grounding and embodied co-creation. The three deployments should be read as a design trajectory rather than as a controlled comparison of interface conditions: they differ in production, performer and dramatic material as well as in system design.
6.1. Voice as primary agent
The case studies indicate that the voice remains the primary expressive source: algorithms extend its harmonic and timbral possibilities rather than replacing it. A distinction the paper has so far left implicit is needed here. By vocal intention we mean intention directed at the sounding result – pitch trajectory, dynamic shape, resonance quality – as distinct from dramatic intention, directed at the character’s situation and realised through vocal means without the two being identical. Kinesthesis tracks neither directly: it tracks gesture indexing the performer’s perception of the vocal result (Section 2.1), which is why augmentation follows vocal decisions rather than dramatic ones, and interpretation of the role remains entirely with the performer.
In Antigone’s Lament, the performer’s facial expression – a component of embodied perception of the vocal line, not of the mechanics that produce it – directly modulates the augmented timbre; in Rayman Scream, independent finger control enables polyphonic augmentation responsive to the performer’s vocal choices. This distinguishes Kinesthesis from systems of gesture-based synthesis, where the sounding result need not originate in a voice at all: here the voice remains the expressive source and the algorithms serve it.
6.2. Gesture as embodied meaning
The vocabulary’s cultural grounding appears to have supported embodied agency, and the direction of support matters. Gestures do not acquire meaning by being classified: a mudra carries its emotional or narrative quality within its own tradition, independently of computational analysis. What the cultural context does is situate the technical layer – the performer enters a mode of signifying with the hands that carries meaning independently of the system, and the classifier attaches to a fluency that develops through rehearsal rather than conferring it, the opposite of a system that assigns semantic weight to arbitrary configurations and asks performers to learn the assignment, as 1.0’s spatial mapping effectively did. This also keeps the borrowing defensible in Emmerson’s (Reference Emmerson and Emmerson2000) terms (Section 4.1): the tradition is not mined for control tokens, because the meaning the gestures carry is nothing the system produces.
6.3. Embodied fluency
Performers reported reaching embodied fluency within a few rehearsals – within the 3–5 range in Rayman Scream and Kolokotronis Memoirs; we lack an equivalent count for Antigone’s Lament and note the gap. This suggests that systems designed with embodied principles – clear vocabulary, immediate feedback, culturally meaningful mappings – are incorporated into practice intuitively rather than as external technical knowledge.
The system’s growing transparency is consistent with Latour’s ‘black box’ (Reference Latour1987; see also Latour Reference Latour1999): as a system functions smoothly, its internal complexity recedes from awareness. Here, that black-boxing appears to support rather than obscure embodied agency – what recedes is the operation of the apparatus, not the exercise of the skill, unlike Emmerson’s (Reference Emmerson2015) offloading, in which the system absorbs the skill itself. The system becomes what Latour calls a mediation rather than an intermediary – an intermediary ‘is fully defined by what causes it, a mediation always exceeds its condition’ (Latour Reference Latour1999: 307) – transforming performer intention rather than transmitting it.
6.4. Distributed agency and assemblage
The integration of voice, gesture, facial expression, algorithmic processing and (in Antigone’s Lament) machine listening creates what we term a distributed assemblage of creative agency: no single agent controls the outcome, and the performance emerges from the integrated activity of all components – without, we suggest, diminishing human creativity.
Recent instrument-design research treats such distribution as a working premise: camera-based instrument design (Butt et al. Reference Butt, Gaster, Renney and Palmer2025), ultrasonic installation practice (Robson et al. Reference Robson, McPherson and Bryan-Kinns2025) and feedback-based improvisation aiming at ‘distributing control beyond the agency of one individual actor (human, or otherwise)’ (Stapleton and Speers Reference Stapleton and Speers2024) converge on an assemblage account from quite different premises. These studies are also methodologically instructive: like ours, they proceed through research in and on the authors’ own practice rather than controlled experiment.
Bennett’s (Reference Bennett2010) ‘vibrant matter’ extends this beyond human–algorithmic binaries. Her ‘thing-power’ names ‘the strange ability of ordinary, man-made items to exceed their status as objects and to manifest traces of independence or aliveness’ (Reference Bennett2010: xvi) – applying here to the made components of the assemblage (classifier, vocoder, room acoustics) rather than to the performer’s body, which this article treats as the site of trained skill. When the harmonic processor responds to facial expression with unexpected resonances, prompting new gestural exploration, it operates as what Bennett, adopting Latour’s term, calls an actant: ‘that which has efficacy, can do things, has sufficient coherence to make a difference’ (Reference Bennett2010: viii). A comparable intuition comes from artistic practice: Eliasson (Reference Eliasson, Eliasson, Calvino, Blom, Birnbaum and Wigley2006) treats sound and vibration not as properties an observer registers but as co-constitutive of the space in which perception occurs, the voice itself being waves that leave the mouth ‘as condensed air, spreading radiantly, entering your ears’.
6.5. Implications for vocal performance and extended voice
The work sits within traditions of extended vocal technique (Emmerson and Smalley Reference Emmerson and Smalley2001; Warren Reference Warren2017), most directly beside Lewis’s (Reference Lewis2007) account of Pamela Z, whose gesture-controlled processing of her own voice is a direct precedent: augmentation as embodied extension of vocal capability rather than external imposition.
6.6. Toward synergistic partnership
Our case studies point toward the synergistic partnership Emmerson (Reference Emmerson2015) proposes – working with rather than through machine systems – via culturally grounded gesture, responsiveness to vocal practice and commitment to performer agency.
7. Challenges and future directions
7.1. Technical challenges
7.1.1. Synchronisation
Environmental factors (lighting, camera distance, system load) affect responsiveness, and the measured latency leaves little margin against the criterion discussed in Section 4.3; reducing system overhead is accordingly a priority.
7.1.2. System complexity and performer training
Three-tier control demands rehearsal: performers reported reaching embodied fluency within 3–5 rehearsals where we hold a count (Section 6), but the initial learning curve remains steeper than for simpler systems. Adaptive designs introducing complexity progressively merit exploration.
7.1.3. Machine-learning Integration
MediaPipe’s performance varies with lighting, distance and background, requiring recalibration across venues. Somax2’s integration supported human–AI co-improvisation, but how machine-learning systems can remain responsive to embodied expression without imposing predetermined behaviour remains open.
7.2. Opportunities for innovation and expansion
7.2.1. Expanded cultural vocabularies
The framework could be extended to other codified traditions (Noh, Bharatanatyam), but such extension must confront Emmerson’s (Reference Emmerson and Emmerson2000) difficulty directly (Section 4.1): the narrow scope adopted here – an organising principle, not a repertoire of signs – would need arguing afresh for each tradition, in collaboration with practitioners trained within it.
7.2.2. Audience interaction and responsive environments
Future systems responsive to audience movement or sound could make performance more collaborative, raising questions of venue design (Emmerson Reference Emmerson2001) – and would make audience response measurable rather than inferred: Pelowski et al. (Reference Pelowski, Leder, Mitschke, Specker, Gerger, Tinio, Vaporova, Bieg and Husslein-Arco2018) demonstrate applicable methods, combining questionnaires with mobile eye tracking in situ, which could test claims we have here offered only as practitioner observation.
7.2.3. Application beyond theatre and in pedagogy
Kinesthesis principles could also be applied to dance, to multimedia installations responsive to visitor movement, and to experimental music. They could support music pedagogy: the embodied learning observed here (Section 6) suggests value for vocal practice and for teaching voice augmentation, and applications for students with special needs and visual impairments, who might benefit from kinesthetic and haptic feedback, warrant exploration.
7.2.4. System development: personalisation and additional modalities
Machine learning could enable systems that learn performer-specific vocabularies rather than imposing standard ones. Future systems might integrate further modalities – respiration, torso or leg movement, electroencephalographic signals – though such expansion risks overwhelming performers with control parameters.
8. Conclusions
Kinesthesis 2.0 shows how interactive media can connect rich traditions of cultural expression – ancient Greek theatrical schemata and Kathakali mudras – with contemporary technology, allowing performers to shape soundscapes through physical and vocal expression, composing and improvising with their own augmented voice as it sounds. It contributes to conversations about embodied cognition, vocal performance and algorithmic agency in four ways.
First, it specifies conditions under which voice augmentation preserves the performer’s position as origin of expressive decision: where harmonic and expressive state is selected by the performer’s own gesture rather than proposed by the system; where augmentation enlarges the range of sound the performer can act upon rather than substituting for a skill the performer would otherwise exercise; and where the gesture vocabulary is continuous with a practice the performer already inhabits. We have also stated what would forfeit this position – autonomous selection of harmonic context, or a designer-fixed mapping, as in Kinesthesis 1.0 (Sections 4.2, 5.1) – so that the claim can be tested rather than assumed.
Second, it proposes cultural depth as a possible mechanism of experiential seamlessness – one of the article’s principal contributions. We suggest that computational mediation recedes from awareness less because the technology is simplified than because the gesture vocabulary is rooted in an inherited tradition the performer can inhabit (Section 3). Two perceptual processes would account for this: gestalt integration of congruent gesture and sound (Vines et al. Reference Vines, Krumhansl, Wanderley and Levitin2006; Emerson and Egermann Reference Emerson and Egermann2018) and immersion driven by content rather than apparatus (Machon Reference Machon2013; Biggin Reference Biggin2017). The proposal is consistent with what we observed across the three deployments, but we have not tested it experimentally, and we offer it as an account to be examined rather than as a demonstrated causal mechanism.
Third, it contributes a vocal case to assemblage accounts of musical agency, and clarifies what such accounts settle. That agency in performance is distributed is not itself in dispute – current instrument-design research treats it as a premise (Stapleton and Speers Reference Stapleton and Speers2024; Butt et al. Reference Butt, Gaster, Renney and Palmer2025; Robson et al. Reference Robson, McPherson and Bryan-Kinns2025). The open question is how agency is distributed in a given configuration; our three deployments describe three distributions, and the voice is a demanding case, being the one component of the assemblage the performer cannot set down.
Fourth, it engages concerns that are specific and attributable: Schloss’s (Reference Schloss2003) severing of perceived cause and effect; Kim and Seifert’s (Reference Kim and Seifert2007) problem of disembodiment in interactive music; Emmerson’s (Reference Emmerson2007) question of whether the body is sublimated or celebrated; Emmerson’s (Reference Emmerson2015) distinction between machine partnerships that enhance a skill and those that offload it; and Reus’s (Reference Reus2026) disembodying tendencies of AI voice systems. By keeping the live voice primary, tying augmentation visibly to the gesture that shapes it, and declining to classify canonical mudras or codified expressive categories (Section 4.1), Kinesthesis addresses these in their own terms.
As algorithmic systems become more prevalent, preserving embodied agency and cultural specificity grows more pressing. Kinesthesis 2.0 offers one model – grounded in embodied cognition, rooted in cultural traditions, committed to distributed rather than hierarchical agency – which remained consistent across the three contexts discussed here, though broader applicability remains to be tested (Section 7.2). The work therefore points toward electroacoustic practices in which algorithmic systems function not as replacements for embodied expertise but as partners within it – working, in Emmerson’s (Reference Emmerson2015) formulation, with machine systems.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1355771826101654
Open access
