2. The COMACROB system

The technical infrastructure of COMACROB consists of three interdependent components: a real-time live coding environment, an AI-generated score and a computer-controlled mechanical acoustic instrument. The Yamaha Disklavier or a custom MIDI-controlled pipe organ serves as the system’s primary sound-producing agent.Footnote ¹ These acoustic instruments are equipped with electro-mechanical actuators (e.g., solenoids and key-strike motors) that can be controlled through MIDI. As such, they occupy a hybrid status between mechanical instrument and programmable automaton. The system incorporates an AI model trained on symbolic music corpora. In the COMACROB Disklavier setup, the AI model employed is a bespoke format (Drymonitis and Koutsomichalis Reference Drymonitis and Koutsomichalis2025: 8) based on the Classical Piano Composer. ^{Footnote 2} In the MIDI-controlled pipe organ setup, the model employed is Notochord (Shepardson et al. Reference Shepardson, Armitage and Magnusson2022). These models are capable of generating polyphonic, musical sequences in specific styles and idioms based on the corpora on which they have been trained. These processes relate to those employed in current machine learning music models such as Google Magenta’s Music Transformer (Huang et al. Reference Huang, Vaswani and Uszkoreit2018) and OpenAI’s MuseNet, in which trained neural networks produce statistically informed and therefore music-style coherent outputs that are reconfigurable in real-time.

In contrast to these models, COMACROB, in its current state, utilises original music composed for it or on specific existing datasets, such as the Lakh MIDI dataset (Raffel Reference Raffel2016), rather than employing vast amounts of data that can be sourced online. The Lakh MIDI dataset comprises 176,581 unique MIDI files, designed to facilitate large-scale music information retrieval. This dataset is freely available online,Footnote ³ as is a list of metadata that gives credit to the creators of these files.Footnote ⁴

Furthermore, COMACROB’s AI technology relies on Recurrent Neural Networks (RNNs) (Medsker and Jain Reference Medsker and Jain1999), rather than the Transformer (Vaswani et al., Reference Vaswani2017), which is typically used in Large Language Models (LLMs). The Transformer is a network architecture that relies on an attention mechanism, in contrast to RNN, which relies on recurrence. An RNN is less complex than the Transformer, but in the case of symbolic music generation, it is sufficient and requires less infrastructure to be trained. Individuals can train RNNs by sourcing material ethically with the consent of the data owners and authors, thus avoiding AI model training practices that in certain cases have been used by certain tech companies and have been criticised for copyright infringement (Buick Reference Buick2025). Finally, the COMACROB project incorporates LiveLily (Drymonitis Reference Drymonitis2023), a live coding environment inspired by LilyPond notation. This tool enables coders to structure, loop, recompose and curate AI-generated symbolic content live. Figure 1 is a snapshot of a COMACROB performance with the MIDI-controlled pipe organ.

Figure 1.

COMACROB performance snapshot, with the LiveLily system in the foreground and AI model output in the background.

Figure 1. Long description

The image shows a computer screen divided into three sections. The left section displays a list of numerical values and code snippets related to musical events. The middle section contains lines of code with specific functions and commands highlighted in different colors, including references to ‘pchange’, ‘osc pd’, ‘function sendbeat’, ‘barcount’, and ‘beatcount’. The right section shows musical notation with various notes and symbols, indicating a musical score. The text in the image mentions ‘LiveLily’, ‘COMACROB performance’, and ‘AI model output’, suggesting a focus on a performance system that integrates artificial intelligence and live music notation.

The system facilitates a network between a human coder, a generative model and a digitally and mechanically augmented acoustic instrument. The system is essentially based on approaches native to the live coding community, where live coding practice can be situated within the wider context of generative music systems (Brown and Sorensen Reference Brown and Sorensen2009: 28). Live coding in this case is not a mere act of instruction, but it is rather integral to the performance itself, a creative terrain for ‘laptop musicians attempting to work with scripting languages in live concerts’ (Collins et al. Reference Collins, McLean, Rohrhuber and Ward2003: 329).

2.1. From operation to ontology

In COMACROB, each part of the system constitutes an integral node of the instrument’s function and identity. The instrument is defined by what it does rather than what each of its components is, and it foregrounds the specificity of the relationships between performers and digital instruments (Gurevich and Fyans Reference Gurevich and Fyans2011). In line with contemporary discourses in digital lutherie, COMACROB is the outcome of new techniques and technologies incorporated in digital music practice that can be deemed both compositional and instrumental (Jordà Reference Jordà2005: 410). Traditionally, instruments are defined by their tactile interfaces, mechanics and acoustic capabilities, all of which are subject to the human body’s expressive abilities. In COMACROB, however, the AI’s generative processes, the code, MIDI and its actuators all form the instrument and define its expressivity. What emerges is a performance ecosystem (Waters Reference Waters2007) in which each element of the system behaves as a generative participant in a compositional-performative event while constituting the instrumental system as a whole.

MIDI and OSCFootnote ⁵  provide precise physical agency over the instruments’ performance parameters, such as pitch, register, note velocity, duration, articulation and pedal use, by translating the algorithms for the mechanical parts of the instrument. This establishes a low-latency conduit through which symbolic instructions are rendered into acoustic events. Such causal efficacy opens space for dynamic affordances that widen the expressive range in sound organisation. The physical interface of the Yamaha Disklavier, for example, becomes a reactive, sensorimotor body that responds to algorithmic precision at levels unattainable by human touch alone. The system is not an interface for controlling sound but a system for musical expression (Dobrian and Koppelman Reference Dobrian and Koppelman2006) resulting from its internal affordances.

In COMACROB, the AI-generated scores can be edited in real-time by the coder through the LiveLily interface, introducing variability and stochastic operations. The AI implementation eliminates issues of mapping limitations (Chadabe Reference Chadabe2002), while introducing the possibility for computer-interactive improvisation (Dean Reference Dean2003). This process functions as an expressive filter and curatorial agent of the musical data stream, responding to the coder’s taste. The live coding, notation and AI processes produce a network that facilitates the entanglement of the human element in the generative process. Figure 2 depicts the looping workflow of a COMACROB performance.

Figure 2.

Looping workflow of a COMACROB performance.

Figure 2. Long description

The flowchart illustrates the looping workflow of a COMACROB performance. The process begins with an AI model prompted with a part of an existing composition. The AI model then generates a few bars of music based on its training and the prompt. A live coder selects which bars will be played by a computer-controlled instrument and the performer. The live coder then chooses to use an AI-generated bar or another fixed composition bar as a prompt for additional music generation, creating a loop.

Therefore, COMACROB’s behaviour is the expression of the interaction of both human and non-human processes. The instrument is not merely played; rather, it performs itself due to its embedded ability to evaluate and weigh conditions, simulating a musical logic. It therefore appears to acquire a distinct musical personality. Such a model challenges conventional taxonomies of instrumentality. It is not an object awaiting to be animated by human or machine control; instead, it is a mediated, dynamic and composite entity whose existence is co-authored by machine learning and music-acoustic actions. As ‘any new musical instrument is intimately tied to the process through which it was designed’ (Fiebrink Reference Fiebrink2016: 1), the stratified aspect of COMACROB was conceived and designed to be the fostering ground for human and non-human negotiations.

3. Sonic writing: a layered framework

Musical instruments can be considered as social actants that contribute actively to shaping interhuman relationships (Bates Reference Bates2012). Furthermore, Magnusson suggests that musical instruments are co-authors of musical ideas (Magnusson Reference Magnusson2019), serving as cognitive interfaces that shape musical thought while facilitating music-making. The development of digital organology, the incorporation of new media processes and Internet of Things practices in instrument design, has prompted studies on the specific ontologies of digital instruments (Paine and Drummond Reference Paine and Drummond2009; Turchet et al. Reference Turchet, Antoniazzi, Viola, Giunchiglia and Fazekas2020; Turchet et al. Reference Turchet, Bouquet, Molinari and Fazekas2022), including the scalable ontologies of digital music instruments (Armitage et al. Reference Armitage, Magnusson and McPherson2023). To theorise the ontology of COMACORB within the realm of speculative realism, Magnusson’s tripartite model of material, symbolic and signal inscriptions offers an analytical lens for understanding how agency is layered and taxonomised in such a system.

The term material inscription (ibid) refers to the physical aspect of the instrument as a crafted artefact: its affordances, limitations and potentials inscribed in its shape and materiality. In traditional instrument design, material inscription is directly experienced through haptic engagement: the friction of strings under fingers, the breath resistance in a reed or the inertia of the key. In COMACROB, the physical affordances and acoustic characteristics of the Yamaha Disklavier and MIDI pipe organ are delegated to a negotiation between their mechanisms and the embedded algorithmic systems. Humans are removed from the chair of the performer, and they only interact with the material inscription of the Disklavier through mediation.

The original materiality of the Disklavier and the pipe organ still exists: one can hear the wooden hammer striking the string or the resonance of the pipes, but sound is shaped by software, not the direct contact of a body. In COMACROB, the human body as an actuator is replaced by MIDI commands, which translate algorithmic processes into dynamics, articulations and rhythms that are not mere emulations of human gestures. They are distinct forms of actuation, with decimal-specific rhythmic precision, dynamic consistency and temporal behaviour. COMACROB’s material inscription thus becomes a cyber-physical hybrid: it persists in the physical properties of the sound-producing end of the system (the piano and the pipe organ), while being augmented by digital agency. Overall, the acoustic materiality and behaviour of the piano and organ are re-inscribed and re-shaped by the algorithms that may push them to behave in ways that are both familiar and alien.

Symbolic inscription (ibid.) encompasses the representational structures of music: notation, compositional logic and algorithmic procedures. Symbolic inscriptions do not make sound directly; they are methods of abstraction, formal languages, and as such, they shape thought. COMACROB employs layers of symbolic representation (Drymonitis Reference Drymonitis2025) that emerge from the interaction between LiveLily and AI. LiveLily is a live coding language inspired by LilyPond (Nienhuys and Nieuwenhuizen Reference Nienhuys and Nieuwenhuizen2003), designed to express musical form in simplified, score-like syntax. It enables real-time manipulation of musical loops and facilitates instant synthesis through its live coding interface. LiveLily is not merely an interface; it is a score-writing system that performs itself. Meanwhile, the AI in COMACROB is trained on a bespoke format or MIDI corpora, which produces symbolic outputs in the form of chord progressions, voice leadings and rhythmic contours. It encodes stylistic norms, harmonic grammars and phrase structures. The model becomes a musical memory inscribed symbolically: it is a generator of style curated by the learning abilities of a non-human agent. The AI outputs are parsed in real time by LiveLily, which selectively renders, edits or loops them. In COMACROB, symbolic inscription is not static; it is generative and iterative, producing musical possibilities rather than fixed forms. Hence, the symbolic inscription of COMACROB does not reflect performance; it rather generates it: it mediates cultural memory while encoding aesthetics. This approach challenges the classical model of notation (from score to performance) by enhancing it with interaction and negotiation between the performer/coder, corpora and AI. In COMACROB, symbolic inscription stands in for both compositional intent and the non-human agents’ performative spontaneity.

With signal inscription, Magnusson refers to the electronic and computational processes that turn symbolic inscription into sound. Signal inscription includes MIDI messages, digital signal processing and control voltages. These do not structure musical ideas directly, but they mediate how these ideas are materialised in time. In COMACROB, MIDI messages act as translators between LiveLily’s and the AI’s symbolic outputs and the Disklavier’s and the pipe organ’s mechanical actions. Each parameter, note on/off, velocity and control change, is an instance of signal inscription. The real-time nature of LiveLily also means that signal inscription can be dynamically managed by the performer/coder. This, in turn, enables the emergence of synchronicity between performance and composition, as signal flow can potentially become a compositional action. When the precision of signal timing shapes the nuances and characteristics of real-time composition, latency, signal jitter and mechanical lags might emerge. Such ‘bugs’ could be seen as aesthetic affordances in the process of sound organisation that are specific to COMACROB’s hybrid cyber-physical nature.

Typically, symbolic (score), material (instrument) and signal (performance) inscriptions exist in different domains. In COMACROB, the algorithmic processes, the robotically driven acoustic instruments and the live coding coexist in a compositional network: they define the instrument’s expressive capacity. This coexistence is conceptual and ontological in nature: the COMACROB instrument is agential and ontogenetically transformed by the algorithms that form it. The embedded mediation of the algorithmic processes that act as bridges between the three layers of symbolic, material and signal inscriptions allows the system to evolve by internalising new functions, thus manifesting technical individuation (Simondon 1958/Reference Simondon2017). The embedded algorithms enable the instrument to imagine, revise and execute music in real time: the instrument thinks of and for herself.

4. Toward a posthuman organology

The analysis of COMACROB in this article contributes to the current discourse on what constitutes a musical instrument in the age of algorithm-driven robotics and AI. COMACROB challenges traditional ideas of organology by merging live coding, notation, mechanical actuation and artificial intelligence into a single cyber-physical hybrid. Moreover, it does not merely focus on the autonomy provided by the mechanised attributes of actuated musical instruments (Overholt et al. Reference Overholt, Berdahl and Hamilton2011; Ângelo et al. Reference Ângelo, Penha, Gomes and Rebelo2018), such as the Disklavier or the MIDI organ, but it attempts to join a group of research projects that embrace virtual agency in music creation (Thelle Reference Thelle2022; Armitage and Magnusson Reference Armitage, Magnusson and Ritis2023; Stefánsdóttir and Magnusson Reference Stefánsdóttir and Magnusson2025). With the recent research developments in AI, interactive music systems are ever more observed and studied for their agencial role in music creation. Such studies might be initiated by the computational advancements employed in music making; nonetheless, a significant part of their contribution is their theoretical and phenomenological perspectives (Moore, Reference Moore2024). Such perspectives tend to challenge and reshape the current views and understanding of human-computer interaction.

COMACROB is not the only musical system that manifests such qualities; on the contrary, it belongs to an ever-growing group of systems that bear properties and characteristics of autonomous agency in music creation. Tatar and Pasquier (Reference Tatar and Pasquier2019) describe this ever-expanding field as musical metacreation, a continuum of partially or fully autonomous musical systems. Examples of such systems may include Hutchings’ and McCormack’s (Reference Hutchings and McCormack2017) multi-agent improvisation framework, which combines machine learning and rule-based systems to simulate collaborative decision-making and Collins’ (Reference Collins2006) real-time machine listening and adaptive behaviour systems for live performance contexts. Other practice-based studies point out that even simple systems can support duet-like creative partnerships, with the notion of agency arising from rudimental reciprocal interactions (Brown et al. Reference Brown, Gifford and Voltz2017). These analyses present musical systems as agents, with musical agency in this context intended not merely by the technological attributes of the system but by the affordances of the system to participate in a collaborative manner in the music-making processes. Therefore, COMACROB is far from being unique in what will be presented further in this article but rather a case study observed under a theoretical lens, a musical system with agential affordances that combines mechanically produced sound, traditional Western music scores and AI technology.

5. ANT: COMACROB as an assemblage of interdependent relations

Actor-Network Theory (ANT) blurs the sharp boundaries of distinction between human subjects and objects, proposing a horizontal ontology in which all entities, whether people, software or instruments, are seen as actants within a system of relations. Actants are defined by what they do: their capacity to act, influence and transform other elements in a network. Agency is not a fixed attribute but a distributed and emergent effect of the relations between heterogeneous components (Callon Reference Callon and Law1986; Latour Reference Latour2005).

COMACROB can be viewed as a network of actants inhabiting nodes of symbolic, material and signal inscription. This relational framing is a dynamic assemblage composed of the Yamaha Disklavier/MIDI organ, the AI model trained on musical corpora, the LiveLily live coding interface, the human coders and the score data. After the initial human instigation, none of the elements in this network is privileged by the primacy of agency. Instead, agency arises from their interactions and negotiations. The Disklavier does not execute commands; it interprets and expresses them through its material constraints. Similarly, LiveLily is a live interpreter that restructures performance in real time. The AI-generated material shapes the musical flow while being shaped by training data and performance context. These functions are distributed, challenging authorship models based on singular intention or control. COMACROB’s operation is a manifestation of metacontrol (De Campo Reference De Campo2014), as the loss of the primacy of human agency opens the field of aesthetic possibilities.

Every node in this music-making network acts both as a transmitter and a translator in the process of sound organisation. The coder translates ideas into code; the AI translates stylistic data into scores; LiveLily translates code into musical structures; the MIDI translates symbolic data into actuator movements; the instruments translate those movements into sound. In each step, there is transformation.

Where other frameworks may see embedding as the internalisation of one layer into another (e.g., symbolic logic embedded into physical instruments), ANT observes embedding as an event of distributed agency, a product of many actants working together. The algorithm’s presence is enacted through a chain of translations that link code, performer, interface and acoustic hardware: ‘the fluid circuit of translations’ (Born Reference Born2005: 30). Embedding, then, is not a stable state but a relational process, and the notion of embedding essentially constitutes the instrument itself.

In observing the act of playing, ANT foregrounds the aspect of negotiation. The coder may reconfigure a loop in LiveLily based on what the AI has produced; the AI’s next output is in turn bound to that structure; the organ reacts differently to the reiteration depending on its tuning and mechanics; and so forth. Playing becomes an ongoing negotiation between actants, each contributing to the emergent shape of the musical output. Action in COMACROB is always mediated, situated and co-produced. COMACROB exemplifies the distributed model of creativity, where technological systems, human decisions and acoustic phenomena converge in real time to produce music that is dynamically negotiated at every step. Musical meaning and performance emerge from the assemblage of interdependent relations.

6. An OOO-based organology

While Actor-Network Theory offers a lens for understanding COMACROB as a distributed model of sound organisation based on translations between networked actants, it privileges relations as the primary subject of observation. Every actant in ANT is reduced by its effects within a network; there is no essence outside its interactions. This relational approach cannot fully depict the profound ontological opacity that emerges in systems like COMACROB. For Object-Oriented Ontology (OOO), the robotic instruments, algorithmic processes and live-coded scripts all exhibit behaviours that escape full knowledge, use or representation.

OOO asserts the autonomy of objects, whether human, physical, conceptual or digital. It challenges anthropocentric and relationist paradigms by suggesting that objects exist independently of their use, human-centred perception or function. They withdraw from total access, retain autonomous being and interact only indirectly with each other: every object has an interiority, a surplus of being that is never exhausted by its actions or relations (Harman Reference Harman2018).

OOO views all of COMACROB’s nodes as objects: the robotic instrument, the AI model, the algorithmic process, the MIDI signal flow and the live coder.Footnote ⁶ Objects embody a richness and complexity that cannot be entirely captured or reduced to their observable qualities or relational interactions; ‘objects for OOO have a depth beneath any relation’ (Harman Reference Harman2018: 99). They are never fully present to each other; they always retain a degree of withdrawn essence or reality, as ‘for OOO the real object always withdraws from direct access’ (ibid: 87). The Disklavier, for instance, is more than its MIDI specifications or acoustic properties; it is an object that withdraws both from the performer and from the algorithms that control it. Equally, an AI, even when trained and queried, contains latent capacities and emergent behaviours that resist total instrumentalisation: objects have their own ‘logic’; they do things, think in their own way and exist independently of what one wants or expects from them (Bogost Reference Bogost2012). This is not a limitation but an ontological feature: the system possesses a reality that transcends access, and this depth is constitutive of its artistic potential.

In ANT, embedding suggests integration and primarily translation: something becoming part of a larger whole, a node within a relational mesh with the ability to translate inscriptions into a new form. But for OOO, embedding is not absorption; it is co-habitation. When an algorithm is embedded in a piano, it does not dissolve into it; it dwells beside it, touching it obliquely through interfaces like MIDI or OSC, but never fully fusing. Their causal relationships are always vicarious (Harman Reference Harman2007). The inherent ontological affordances of OOO manifest in COMACORB when the instrument does not react as one has programmed it to, when it is not as musical as it is meant to be or even when its output is differently musical to how humans would appreciate it.

COMACROB’s behaviour reveals the negotiations between the different categories and the ontological statuses of the entangled entities. The system reacts in unexpected ways, and it may even glitch or crash. Those are expressions of objectal autonomy (Harman Reference Harman2011). The beauty and challenge of composing with such systems is their resistance to mastery: they behave, they co-compose, but they never become fully transparent. COMACROB, in this light, is a theatre of objectal negotiations, where autonomous agents coexist without full disclosure.

Optimisation of a system such as COMACROB can thus be seen as an act of domestication, a move to align objects with human intent, devoiding them of their autonomy. Optimisation aims to relegate the object to its utility, limiting its depth. To resist optimisation is to honour such systems as collaborators rather than passive tools. Glitches become acts of objectal resistance: the objects’ refusal to be fully domesticated. OOO suggests that such errors open a space for posthuman ethics: embedded instruments are not to be tamed but to be accepted with their unpredictability.