1. Introduction
The second half of the 20th century witnessed a significant transformation in compositional practices, as notation expanded beyond its traditional prescriptive function to explicitly incorporate openness and indeterminacy. This shift mirrored a broader evolution in the arts: as artists increasingly embraced relationality and emergence, the conventional notion of the work as a self-contained object representing the direct expression of the artist’s inner intuition underwent a crisis. In the wake of this shift, the artwork became an increasingly open, indeterminate and interconnected system.
This transformation reflects a much broader, overdetermined cultural one. The rise of new media technologies, information theory and cybernetics (Wiener Reference Wiener1948) fostered systemic thinking; postmodernism’s rupture with tradition and scepticism toward grand narratives (Lyotard Reference Lyotard1984), alongside theories prioritising the social construction of knowledge and the intersubjective nature of the aesthetic experience (Bourriaud Reference Bourriaud1998), contributed to the destabilisation of hierarchical structures and traditional modes of artistic enjoyment. These converging factors encouraged fluidity in interpretation, endowed audiences with an active role in shaping the work, and in music contributed to redefining the traditional roles of composers and performers.
Articulating this shift in opposition to Croce’s aesthetics, which treats intuition and expression as indistinguishable, Umberto Eco introduced the notion of the open work (Eco Reference Eco and Cancogni1989), an epistemological metaphor reflecting the new ways in which culture informs the interpretation of signs. In Eco’s theory, two complementary degrees of openness emerge: openness of the first degree, a universal cognitive process by which any work, including formally closed ones, generates new meanings through a continuous germination of internal relationships, and openness of the second degree, a deliberate poetics where indeterminacy and emergence are programmed within the very structure of the work (Eco Reference Eco and Cancogni1989).
This latter dimension has been central to musical experimentation in the second half of the 20th century, with scores becoming ‘junctures of composer and performer creativity, or locations that [enable] the emergence of new and unforeseen musical results’ (Magnusson Reference Magnusson2019). Artists realise this openness through diverse strategies: some focus on the emergent dimension of the inscription (Oliveros Reference Oliveros2013), others problematise the scores’ situatedness in relation to the instrument (Tomás and Kaltenbrunner Reference Tomás and Kaltenbrunner2014), blur the role of composer, performer and audience (Einarsson Reference Einarsson2017), investigate live coding as a new path in the generation of the score (Magnusson Reference Magnusson2011), or through data mining techniques incorporate social forces into the creative process (Privato and Novello Reference Privato and Novello2022).
In Eco’s framing of the open work, interaction and entropy play a crucial role. The programmed interactivity requiring the auditor ‘to do some of [the] organizing and structuring of the musical discourse’ defines the formally open, constantly evolving works that Eco describes with the phrase works in movement (Eco Reference Eco and Cancogni1989: 12) . Examples in this category are Cage’s 4’33’’, wherein the work incorporates the sonic environment in which it is enacted, and Calder’s mobiles, whose form is constantly reconfigured by shifting air currents. To discuss entropy, Eco applies information theory to semiotics, describing it as the degree of informational disorder inherent to the work and arguing that ‘the richest form of communication (richest because most open) requires a delicate balance, permitting the merest order within the maximum disorder’ (Eco Reference Eco and Cancogni1989: 98). This view entrusts the artist with the role of defining, through a careful and deliberate use of structural openness and intention, the space within which poetic meaning arises.
While Eco primarily explores human and environmental factors as sources of openness, his drawing from information theory and cybernetics proves prescient. From the writing of Opera Aperta in 1962, wherein the metaphor of the open work was first introduced, generative systems have indeed evolved beyond mere recombination machines such as those imagined by Calvino (Calvino Reference Calvino1986) or employed by Xenakis in his stochastic works (Luque Reference Luque2009), into machinic agents that artists often view as co-creative partners. Within this technical and aesthetic evolution, the open work’s focus on emergence, interaction and indeterminacy provides an ideal framework to investigate the shifting role of generative technologies in the arts. By viewing the artist as the designer of complex systems where internal and external forces enable the emergence of poetic meaning, the open work also informs a local model of human and machinic co-creation that maintains an ontological distinction between the artist-creator and the generative system.
Building upon this theoretical foundation, in this paper we combine creative AI with two open notational forms that challenge normative ways of thinking around musical notation: inherent scores and text scores. The former category describes the merge of the musical inscription and the interface, and the latter explores the use of natural language to convey performative instructions. We use inherent scores and text scores as probes into real-time and text-to-audio neural audio synthesis (NAS) algorithms (Božić and Horvat Reference Božić and Horvat2024), investigating (i) how NAS alters notions of indeterminacy and openness within these traditions and more broadly in relation to Eco’s framework and (ii) how, in turn, inherent and text scores can mediate, streamline and support our engagement with AI. Through practice research, we introduce novel notational strategies with real-time and text-to-audio NAS, and conclude by extending Eco’s framework through a novel epistemological metaphor that we term haunted work.
2. Inherent scores and NAS
Musical instruments function as tools for sound production as well as mnemonic supports, materially inscribing the practices and theoretical systems that shape them as concretised forms of music theory (Magnusson Reference Magnusson2019). This implies a certain overlap between the instrument and the notation, which in digital musical instruments (DMIs) becomes particularly evident (Magnusson Reference Magnusson2009). The notion of composed instruments, for instance, postulates a decoupling between sound and gesture, and the interweaving of the instrument with the musical work (Schnell and Battier Reference Schnell and Battier2002), thus extending the notational space beyond the physical constraints of the score and incorporating all the structural, tangible and virtual components that support the inscription.
As a theoretical probe to investigate the merge between the instrument and the score, Tomás and Kaltenbrunner propose the notion of inherent scores (Tomás and Kaltenbrunner Reference Tomás and Kaltenbrunner2014), tracing the origin of this concept back to Alvin Lucier, who, in describing the practices of the Sonic Arts Union, stated that the scores were ‘inherent in the circuitry’ developed by the members of the collective (Lucier Reference Lucier1998). This merger can be traced even further back in time, at least to the 1960s, when Gordon Mumma discussed how ‘designing and building of circuits is really ‘composing’’ (Mumma in Nyman Reference Nyman1974: 91). Though Mumma and Lucier articulated this idea explicitly, their positions reflect a broader artistic tendency among contemporaries (including artists such as David Tudor, Suzanne Ciani, Morton Subotnik and Erkki Kurenniemi) who were similarly invested in renegotiating the loci of production and inscription of the musical experience.
To operationalise the notion of inherent score through artistic practice, Tomás develops a series of tangible scores, wooden boards featuring touch-sensitive, engraved inscriptions that the performer plays through tangible interaction (Tomás and Kaltenbrunner Reference Tomás and Kaltenbrunner2014). Similarly, magnetic scores (Figure 1) employ magnetically inscribed surfaces that the performer engages with at a distance, via the attraction and repulsion between permanent magnets and magnetic controllers (Privato et al. Reference Privato, Magnusson and Einarsson2023). These types of notation introduce openness through non-linear navigation and discovery, endowing the score with an intrinsically exploratory character. In the following sections, we examine whether and how this exploratory dimension may be leveraged to facilitate notation and performance using a real-time NAS model.
Detail of a magnetic score with controllers. Under the circular drawings, the wooden score embeds magnets of different dimensions and polarities (Privato et al. Reference Privato, Magnusson and Einarsson2023).
2.1. Composing with real-time NAS
RAVE (Real-time Audio Variational autoEncoder) is a deep learning model enabling real-time NAS (Caillon and Esling Reference Caillon and Esling2021). Trained on relatively small datasets of raw audio, it learns to reconstruct sounds by compressing them into a continuous latent space, a multidimensional representation of learnt sonic features located between an encoding function compressing the audio stream and a decoding function reconstructing the sound. Unlike traditional autoencoders, which map sound features as points, RAVE introduces a continuous latent space through probability distribution.
Once trained, RAVE models can be interacted with in two ways: first, by using a sound source as input, with the model transferring the timbral features of the dataset into it, or, second, by bypassing the encoding function and forwarding control signals directly into its latent dimensions. These two methods may also be combined, for instance, by using one’s voice as input and adding control signals to offset the encoder’s latent input values.
While the direct manipulation of the model’s latent space opens up significant creative potential, it also presents a series of compositional and performative challenges that uniquely shape artistic practice with NAS compared to other synthesis methods. Among these, three have stood out through collaborations with artists from the Intelligent Instruments Lab’s (IIL) extended community and in our artistic practice:
Arbitrary Encoding. The representation and distribution of sound features from the dataset within the latent space is autonomously determined by the model during training and thus varies across different training runs and datasets. The composer is responsible for the selection of the material, but leaves its organisation in the latent space to the model. Once a new model has been trained, the latent space has to be explored a posteriori.Footnote 1 In this sense, the composer approaches every new model similarly to how a practising musician learns a new instrument. Consequently, composing with NAS greatly benefits from notational strategies that accommodate discovery through embodied exploration.
Feature Entanglement. Individual latent dimensions typically encode multiple, intertwined sound parameters. For instance, the first latent dimension of a given model might correlate with amplitude and brightness, while the second might affect both frequency and noise level. This entanglement prevents independent access to, and thus representation of, individual sound features. Prescribing gestural instructions on the interface is thus often more practical than abstracting and notating individual parameters.
Dimensional Entanglement. Latent dimensions are highly correlated with one another. This means that by changing the state of one latent, the response of all the others is affected. This second type of entanglement makes purely one-to-one mapping strategies less effective. Furthermore, the latent space is usually high-dimensional (typically eight to thirty-two dimensions), which makes it impractical to control all latents individually and simultaneously. A synthetic mapping approach is thus frequently preferred, with one gesture on the interface controlling multiple latents at once.Footnote 2
Arbitrary encoding, dimensional entanglement and feature entanglement position the interface at the heart of the compositional process with real-time NAS. Rather than resorting to notational systems abstracting independent sound parameters and independent control mappings, composers benefit from dynamic, exploratory and entangled approaches to the navigation of the model’s latent space. As discussed above, inherent scores adopt a similarly exploratory approach. This parallel between algorithmic and material exploration motivates our investigation into inherent scores as a way of streamlining notation with real-time NAS. To this end, we introduce embodied sketching, a notational strategy that we apply to Stacco (Figure 2), one of the NAS interfaces designed at the Intelligent Instruments Lab (Privato et al. Reference Privato, Lepri, Magnusson and Einarsson2024a).Footnote 3
Embodied sketch on Stacco. Since the magnets do not need to physically touch the board, it is possible to insert a notational layer (embodied sketch) and thus compose directly on the instrument (Privato et al. Reference Privato, Shepardson, Lepri and Magnusson2024b).
2.2. Embodied sketching
Stacco combines four magnetic attractors capable of reading three-dimensional variations in their magnetic field. The attractors, each featuring a permanent magnet and a magnetometer, are symmetrically mounted under a wooden board, on top of which the performer throws, displaces and combines magnetic spheres of different dimensions. With four spheres, one on each attractor, the performer can simultaneously control up to twelve continuous parameters, which are typically mapped to as many latent dimensions in a RAVE model’s decoder.
As with all inherent scores, Stacco builds on the coupling of the interface with the notation. However, such coupling is not as tight as in other inherent scores, in which changing the notational layer often implies a redefinition of the interface. Indeed, Stacco’s magnets do not need to touch the instrument to influence the attractors’ readings: leveraging this action-at-a-distance, we designed a series of oval cardboards and tracing paper sheets to be placed and drawn upon the instrument’s surface while playing. Composers and performers use these embodied sketches (Figures 2 and 3) to describe, prescribe, recall or navigate meaningful areas in the latent space (Privato et al. Reference Privato, Lepri, Magnusson and Einarsson2024a).
Mouja+’s embodied sketch. Relevant areas in the latent space are drawn around the attractors; the timeline is distributed clockwise around the external circle, starting from the top.
As discussed above, one of the consequences of arbitrary encoding is that composers learn how the model represents sounds in the latent space a posteriori and through direct engagement. Thanks to the compositional affordances of embodied sketches, relevant areas in the latents can be at once discovered, incorporated into the score and recalled. At the same time, scores can be dynamically replaced, overlapped and reworked into articulated pieces both on and away from the instrument depending on the requirements of the model in use.
Embodied sketching’s gestural affordances also align with the feature entanglement constraints, which hinder abstracting and notating sound parameters individually. Finally, the model’s dimensional entanglement finds a parallel in Stacco’s ergodynamics (Magnusson Reference Magnusson2019), since each attractor on Stacco can read three spatial parameters, allowing the user to control as many latent dimensions with one single magnet. This synthetic approach to latent manipulation further extends through the reciprocal entanglements of the different attractors, strategically positioned to magnetically influence each other’s marbles. This entanglement reflects in the notation of the sketches, with fewer signs defining strategies for a large number of latents (Privato et al. Reference Privato, Lepri, Magnusson and Einarsson2024a).
Having shown how inherent scores can streamline notation with real-time NAS by accommodating the model’s technical constraints to latent navigation through an approach that blends exploration and composition, we can now turn our attention to a second notational strategy. Whereas embodied sketching engages NAS through the affordances of the interface, text scores engage NAS through natural language, introducing new compositional possibilities but also novel challenges in how meaning is constructed when the ‘who’ interpreting the text conflates with a ‘what.’
3. Text scores and NAS
Despite their relatively recent emergence as a self-standing, open notational method, textual instructions have long complemented traditional notation, often conveying supplementary interpretative and expressive nuances. At the dawn of modernism, Satie began introducing surreal and humorous textual cues into his scores, such as ‘de clairvoyance’ or ‘ouvrez la tête’, whimsically engaging the performers in interpretative riddles (Satie Reference Satie and Klemm1986). This use of natural language, while of secondary relevance compared to the staff notation, shifts the function of the textual cues, complicating interpretation by introducing elements of indeterminacy and openness.
The potential of natural language in sound art was further investigated by Russolo, who championed an art of noises that incorporates the ‘sounds of language’ (Russolo Reference Russolo1913), and by the Dada movement, experimenting with poetry as sound (Erickson Reference Erickson1984). However, it was Cage, both as a composer and as a mentor for a new generation of artists throughout the 1960s, who marked a turning point in the evolution of text scores as a self-standing, open form of notation. Cage’s composition classes at the New School for Social Research indeed served as the incubator for what would emerge as the diverse Fluxus collective, whose action notation and provocative event scores achieved wide circulation and influence in the years to come (Friedman Reference Friedman1998).
While Fluxus works have often been described as event scores, Pauline Oliveros’ works using natural language to convey performative instructions are typically categorised under the broader term text scores. This distinction reflects Oliveros’ unique approach, in which the core of the performative experience centres on the stillness of listening rather than on action. Oliveros describes her scores as ‘algorithmic improvisations or compositions’, as ‘recipes that allow musicians to create music without reading notes’ (Oliveros Reference Oliveros2013). Text serves here as the means to enter the deep listening experience, and from there to access the situated and open network of actors wherein information emerges.
3.1. Text scores and AI
Among contemporary artists experimenting with deep learning and text scores, Jennifer Walshe stands out as one of the most prolific. From 2017 to 2021, Walshe developed a massive dataset of approximately 3,000 text scores, which she uses to train AI algorithms and generate new works (Walshe Reference Walshe2021). This growing dataset contains scores from Pauline Oliveros, George Brecht, Mieko Shiomi and others, but is also open to the contribution of any artist willing to send their work.
Whereas Walshe’s project focuses on the algorithmic generation of text scores, leaving to the human performers the role of enacting them, the recent emergence of NAS techniques in combination with natural language processing (NLP) has introduced new creative possibilities. Tools such as Udio,Footnote 4 Suno AIFootnote 5 and Stable Audio (Evans et al. Reference Evans, Parker, Carr, Zukowski, Taylor and Pons2024) are now capable of synthesising sounds and coherent musical structures from textual descriptors, allowing composers to reverse the paradigm and design text scores to be interpreted by generative algorithms.
An example of a performative use of text-to-audio NAS is represented by Dadabots’ prompt jockeying live sets.Footnote 6 In prompt jockeying, music is generated on stage by the algorithm based on live-prompted textual cues. According to Dadabots, ‘if a DJ must be fluent in their Rekordbox collection, a PJ must be fluent in their PyTorch model’.Footnote 7 Such fluidity presupposes high programming skills, but also many hours of experience in prompting on a particular model. Working with textual instructions on a generative system is indeed very different from DJing, and, what interests us most, from composing text scores for human performers, since our use of natural language needs to change and adapt.
3.2. Text scores and text-to-audio NAS
We can look at how text-to-audio AI introduces new interpretative constraints at least from two perspectives: first, from the angle of the implicit, culturally dependent and contextual rules upon which communication is based, and how these rules shift as we address a non-human agent. This aligns with Wittgenstein’s notion of language games, wherein meaning is not fixed but emerges through a situated use of language (Wittgenstein Reference Wittgenstein and Anscombe1953). Second, we can look at how language’s inherent potential for expression changes when it no longer instructs action, turning instead bodies and events into pure representations.
Among Fluxus scores, the enactment of Yoko Ono’s Stone Piece offers a fitting example of the former case, as it plays with language’s indeterminacy by inviting the performer to ‘take the sound of a stone ageing’. This is an ambiguous statement. What is the sound of ageing? How does a stone age? And as a consequence, what does ageing mean here? Like a haiku, this score introduces an interpretative openness that calls for reflection and introspection. Yet, to an existential question requiring a certain awareness of the context and the language game being played, the model typically defaults to a simplistic interpretation: the sound of a stone being scratched.
While Yoko Ono’s piece introduces a problem of interpretation, Jed Curtis’ provocative Music for Wise Men prompts reflections on disembodiment and signification. The score instructs performers to ‘commit suicide’. These instructions inevitably confront a human performer, with a physical body they are bound to, with the limits of a literal execution of the score. However, when an AI model interprets the same instruction, its disembodied nature inevitably dilutes the intention of Curtis’ piece, rendering its foundational paradox less accessible to the audience. At the same time, from this shift in context new and unforeseen meanings might emerge.
These initial considerations prompted us to investigate through artistic practice the potential and limitations of composing and performing with text scores and text-to-audio NAS. In Mouja+, the piece introduced in the next section, we focus on more Fluxus scores: by experimenting with pieces originally meant for human performers, in which sound is often provocatively viewed as integral to a certain performativity, we could bring to the forefront the model’s interpretative limitations and use the findings to inform compositional strategies and reflections concerning text-to-audio NAS.
4. Mouja+
Mouja+ is a piece complementing a text-to-audio generative model with an inherent score using embodied sketching (Figure 3). Through the combination of the strategies introduced in Sections 2 and 3, this piece extends the practice of prompt jockeying described in Section 3.1 by introducing real-time, NAS-based manipulation of the sound files live-generated by the text-to-audio model. The piece was recently performed by the first author at the SWRL 2024 festival, in Riga (Latvia).Footnote 8
4.1. Pipeline
Mouja+ leverages Stable Audio Open (Evans et al. Reference Evans, Parker, Carr, Zukowski, Taylor and Pons2024), a text-to-audio model partially trained on the Freesound archive.Footnote 9 On the client side, a graphical user interface (GUI) (Figure 4) communicates with a remote server through an open sound control (OSC) application exposing Stable Audio’s API endpoints.Footnote 10 By live-processing the prompt on a remote server with a high-performing GPU, I could sensibly reduce the time required for the generation of the sound files (around 20 seconds for a 47-second sample) compared with most consumer machines, which was a critical aspect to the live performance.
Mouja+ GUI. The top section allows one to prompt Stable Audio. The buffers with the generated files are shown on the left, and the right section routes the playback to the resynthesis section with RAVE and Stacco.
On the client GUI, one writes positive and negative text prompts and defines output length in seconds, number of steps, temperature of the generation and buffer for the download. The client forwards the prompt to the server, which generates an audio file. Once ready, the file uploads into the specified buffer on the client computer. The GUI allows one to view, play, process and mix up to eight audio files at the same time, that can be routed either to the speakers or to the real-time NAS model (RAVE) for timbre transfer.
When timbre transfer is applied, Stacco’s control signals are added to RAVE’s latents feeding into the decoder, allowing one to influence the resynthesis by acting on the interface. The models were collected from open source material or with the explicit consent of all the parts involved; they are for the most part free to download and useFootnote 11 and comprise foley sounds, liquids flowing, guitars, human voices, percussions, magnets bouncing, birds singing and mixed choirs.
Mouja+’s pipeline includes Ambisonics spatialisation, as the piece was premiered on a 36 + 2 speakers dome (Figure 5).Footnote 12 The readings from each of the four magnetic attractors in Stacco were mapped to the spherical coordinates of one sound source within the Ambisonics space using ICST Ambisonics (Schacher Reference Schacher and Kocher2006). The movement of the four sound sources was automatised, but at all times it was possible to selectively switch to manual control with Stacco.
Mouja+. SWRL’s 36 + 2-Speakers Ambisonics Dome.
4.2. Mouja+
Mouja+ builds upon a pre-existing set titled Mouja, based on embodied sketching and Stacco (Privato Reference Privato2024), which the text-to-audio pipeline extends. The sketch displays a series of relevant areas and gestures around Stacco’s attractors (Figure 3), that I converge toward when I use the RAVE model for timbre transfer. Around the score is a circular timeline used in a solo section on Stacco towards the end of the piece.
The score consists of a sequence of 10 event scores adapted from the Fluxus Performance Workbook (Friedman et al. Reference Friedman, Smith and Sawchyn2002). The scores are displayed on a large screen for the audience, along with information on whether the synthesis from each successive prompt is in process or complete. The structure is the result of a careful selection and readaptation of the ten scores, overlapped and juxtaposed into a tripartite form.
The first section consists of layers of concrete sounds moving concentrically around the audience and begins with the sound of a heavy resonating body (a piano) being dragged into the stage, whose position in the Ambisonics space is controlled using Stacco. The text scores are three: ‘Let piano movers carry the piano on stage’, ‘Wash the piano, wax it and polish it well’ (adapted from Nam June Paik), and ‘Eat a juicy apple’ (adapted from Bob Lens). As the files are generated, I use Stacco to control transitions and dynamics.
In the second section, the concrete sounds become pointillistic. The scores I use are ‘Hammer nails into the piano keys’ (adapted from Tomas Schmit), ‘Dropping coins’ (excerpt from George Maciunas Solo for Rich Man) and ‘The sound of the stone ageing’ (adapted from Yoko Ono’s Stone Piece). As in the first section, I resynthesise the sounds using Stacco’s embodied sketch. The second section ends with a solo on Stacco, using direct latent navigation and no resynthesis, based on the score in Figure 3.
The third section conjures on stage an open market, with unintelligible voices emerging from the crowd, that I dynamically displace in the Ambisonics space. Finally, the crowd’s noise deflates into that of a piano, this time being dragged out of the stage. The text scores I use in this section are ‘Performer sells the theatre’ (adapted from Ben Vautier), ‘Piano movers carry piano out of stage’ (adapted from Nam June Paik) and once again, as a conclusion, Ono’s ‘The sound of the stone ageing’.
4.3. Reception
After the performance at SWRL (Sound Example 1) (Figure 6), I briefly explained to the audience Stacco’s mechanics and Mouja+’s technical pipeline. While time constraints prevented extended discussion, I gathered informal feedback during the following days of the festival. Among the insights I collected, three are worth mentioning and will be used to ground the discussion in the next section.
Mouja+ at SWRL Festival, Riga, Latvia.
First, the visual emphasis on the model’s real-time generation of the sound files brought to the forefront the text-to-audio model’s agency, effectively incorporating it as part of the performance. Second, the visual display of the text scores along with the dynamic spatialisation of the sounds in the Ambisonics dome, produced a lingering, eerie impression of presence, materialising the sonic traces of the agents and actions described in the scores.
A third insight concerns the model’s interpretative failures, which resulted in cognitive dissonances between the audience’s expectations from the textual instructions and the sonic outcomes. One participant, for instance, commented on how the model reconstructed the hammering of nails onto the piano keys in the score adapted from Schmit with no consideration of the word ‘hammering’, resulting in dense, minuscule sounds of nails falling on a surface, along with sparse and feeble notes from the piano. This clearly exemplifies the tension identified in Section 3.2 between the human understanding of language through context and AI’s statistical processing of the instructions.
5. Discussion
As I worked on Mouja+, collecting feedback and further reflecting on my practice, I focused (i) on the complementary aspects between real-time NAS with inherent notation and text-to-audio NAS, (ii) on elaborating a series of good practices in composing text scores for NAS and (iii) on the novel compositional affordances introduced by text-to-audio NAS in combination with text scores.
5.1. Complementing real-time and text-to-audio NAS
Once generated by Stable Audio, the audio files cannot be further refined through additional prompts. While this poses minimal constraints for purely generative applications such as prompt jockeying, it represents a significant bottleneck in a system designed for real-time performance, wherein control and responsiveness are critical. By complementing asynchronous NAS with real-time resynthesis through RAVE, I could overcome this limitation. The resynthesis algorithm allowed me to exert control over the unfolding of the piece, and to effectively overlap and transition through the AI-generated files to outline the macro formal structure of the piece.
Moreover, through the notational cues on Stacco’s embodied sketch, I could easily recall relevant areas within each of the models’ latent space. This was particularly useful in compensating for Stable Audio’s relative unpredictability: when the file generated by a certain prompt was not satisfactory (which happened at least in two instances), I would route the whole signal into one or more RAVE models, resynthesising the file’s output to suit my needs whilst incorporating traces from the original file.
The combination of real-time and text-to-audio NAS enhanced the audience’s experience as well. The projection of the text scores on the screen allowed playing with the tensions between the audience’s and the model’s interpretations of the textual information. At the same time, by performing on Stacco I could introduce a gestural and embodied dimension to the performance that contrasted with, and thus amplified, the disembodiment foregrounded by Stable Audio’s interpretation of the scores, an aspect that will be further discussed in Section 5.3.
5.2. Compositional strategies
In working with Fluxus scores, I had an initial idea of what sounds I was looking for, how much variability in the generation of the files I was willing to accept, and how I intended to combine them to shape Mouja+’s macro form. To achieve this, I had to work my way through the model’s interpretative constraints. I thus developed a series of strategies, of which three proved particularly useful:
-
• Dataset exploration. Stable Audio Open is mainly trained on the Freesound archive, which is open and freely accessible online.Footnote 13 I used the Freesound advanced search engine to explore text embeddings, typing keywords and selecting tags based on the scores I was working with. One example is the word ‘exit’, which I experimented with extensively. In Freesound, this word is often used in environmental recordings involving trains arriving at stations, opening and closing their doors, and leaving. This suggests that the statistical likelihood that prompts containing this word might incorporate sounds of moving trains is high, as confirmed through practice.
-
• Strategic use of negative prompts. Once I understood how words were embedded in the original data, I typically decided whether to discard a score or use negative prompting to stir the system towards a certain outcome. Useful negative prompts when prompting the word ‘exit’ were ‘train’ and ‘bus’, ‘station’ or ‘soundscape’, which allowed me to counterbalance the model’s tendency to converge towards sounds of public transportations coming and going, as emerged through dataset exploration. With negative prompts, I could effectively define niches of sounds by exclusion, compensating some of the problems arising from the model’s lack of context awareness. This became a critical strategy in Mouja+: for every score, I included negative prompts such as ‘music’, ‘piano music’ or ‘song’ to prevent the generation of structured musical fragments that would have hardly integrated with the performance.
-
• Prompt fine-tuning. As I tested different Fluxus scores, I realised that by slightly changing the grammar and by inverting or reducing the order of the words, I could guide the generation of the sounds towards a desired outcome or make a certain output more consistent through successive prompts. ‘Let piano movers carry the piano out of the stage’, ‘piano movers carry piano out of stage’ or ‘eat juicy apples during concert’ and ‘eat a juicy apple during concert’ tend to be interpreted in similar ways by human performers. However, by introducing small changes, such as adding ‘a’ and turning ‘apples’ into ‘apple’, I could increase the statistical likelihood of generating an expected result given a certain prompt.
These three strategies were crucial in adapting the use of natural language to Stable Audio. Yet, due to the statistical nature of the system, the possibility of the model generating unexpected and sometimes undesired outputs on stage was non-negligible. For instance, as noted in 4.3, the hammering of nails in Schmit’s piece was reconstructed with no consideration of the word ‘hammering’, which considerably defused the intention and impact of the score.
While effective strategies to tailor textual prompts can be developed and mastered, the risk of diluting the original communicative intent is indeed always present, and should be considered attentively when composing with these systems. On the other hand, their non-deterministic nature introduces new entropic processes into the open work, that artists can learn to govern to meet their artistic goals.
5.3. NAS and spectrality
As the title of this piece suggests by alluding to the word ‘Ouija’, a central interpretative lens in Mouja+ is that of spectrality, here intended as ambiguity of presence. The subtle yet pervasive sense of eeriness that the piece evoked, rather than from an explicit narrative design, was achieved by amplifying the spectral effects inherent to the model’s workings. Through the use of scores that incorporate concrete objects and human actions (such as ‘piano movers’, or the final selling of a theatre, which conjures an entire crowd on stage), reinforced by the active displacement of the sounds through spatialisation techniques (e.g., the sound of the piano dragged across the stage), Mouja+ introduced a palpable tension between presence and absence; a spectrality that manifested in at least three ways.
-
• The model’s spectrality. As discussed in Section 4.3, by displaying the text scores on a screen that also reported on the generation of the audio files, the model became an actor integral to the performance. Yet, its agency on the sounds was never overt. It rather emerged indirectly, as in a sort of photographic negative, through the semantic gaps between the text scores and their sonic outcomes, such as for instance the not-quite-right character of the reconstructed nails, hammers, and voices in relation to the textual instructions. A timbral parallel to a hand with six fingers in a text-to-image model, which indirectly reveals the traces of the model’s presence and agency.
-
• The referents’ spectrality. Because the model’s workings were known by the audience, the sounds of dragging pianos, the chewing of fruits, the clinking of coins or the murmuring from the crowd could not possibly signify pre-existing referents. These sounds existed only ‘here and now’ and thus predated the hammer, the piano and the fruit they evoked. In this indexical inversion, sounds conjured possibilities rather than re-presentations, bearing witness to their referents’ radical absence: certainly, their not being there, but also the ontological paradox of having never been in the first place.
-
• The dataset’s spectrality. As I explored the Freesound archive to adapt the scores to the model’s interpretative constraints (as in the ‘exit’ example discussed above), I became highly aware of the traces from the original sources in the dataset, which I perceived as echoes of bodies and events pulverised and reconfigured into something ‘other‘ and yet eerily familiar. Because this third dimension of spectrality depended on my understanding of the dataset, it did not extend to the audience in the context of the performance.
These meaning-making processes opened up new interpretative dimensions in the Fluxus scores. The act of chewing an apple, for instance, inevitably provoked an eerie effect, as the chewing seemed to come from a spectre walking through the audience. Similarly, the piano dragged on stage felt alive (or unalive) as much as the piano movers that dragged it, and the crowd conjured on stage felt strangely tangible while bearing, in the unintelligibility of the voices, the traces of the model that made it, and along with them those of the sources reassembled in a sort of atomic montage.
Crucially, while through these dynamics one can access new creative potentials, certain interpretative stances are at the same time rendered inaccessible. This is particularly true when meaning depends on a ‘necessity of presence’. My adaptation of Schmit’s piece, for instance, prescribes the hammering of nails on a piano, a politically charged act given this instrument’s symbolic tie with Western musical history. Importantly, the power of this act lies precisely in its materiality and irreversibility. What remains of this radicality when no physical nail is hammered, when only the spectre of a piano is brought on stage? Even though by conjuring its ghosts we can extend a score’s expressive potential, this process carries concrete risks of sanitisation of certain artistic stances that one should be aware of in working with these systems.
5.4. Haunted works
The spectral effects observed in Mouja+ operate in the interstices between signifier and signified, wherein Derrida situates his notion of hauntology (Derrida Reference Derrida1994). Derrida frames hauntology within his broader critique of Western thought’s tendency to focus on presence and stability of meaning (presence intended here as being here as well as being tout court), suggesting that every ontology is a conjuration, one that is haunted by the seeds of its undoing.
Within this expansion of his deconstructive framework, Derrida describes media technologies as inherently spectral in at least two ways. First, they introduce processes of temporal fragmentation, causal inversion and ‘out of jointness’ that resemble the way ghosts fracture time and inhabit multiple temporal planes at once. Second, they destabilise the binary of presence and absence, conflating agencies, and revealing the deferred nature of language and the spectral quality of representation.
As Mouja+ showed, these hauntological dynamics, while present in various media forms, become particularly pronounced in AI-mediated artistic practices. Indeed, the models introduced temporal disjunctures and agential ambiguities akin to those described by Derrida, conjuring spectral referents whose sounds marked the name and took the place (Derrida Reference Derrida1994: 9), as if haunting the void of the stage from the future. At the same time, an element of haunting from the past was always already present, as the sounds constructed by the model bore the traces of the dataset’s sources, the original, schizoid referents, pulverised and recombined by the model’s algorithmic pastiche.
In our work with AI, this hauntological tendency has emerged far beyond Mouja+. Through artistic works, user studies, workshops and collaborations with artists engaging with these systems, we have observed how spectrality and haunting increasingly infiltrate the discourse and the cultural production with this technology (Privato and Magnusson Reference Privato and Magnusson2024c, Reference Privato and Magnusson2024d). These hauntological tendencies, which Mouja+ brought to the fore, signal the emergence of a new metaphor within the epistemological space opened by the encounter of Eco’s framework with generative AI, one that may be described as the haunted work.
The haunted work does not reject Eco’s far-seeing observations on openness and indeterminacy in modern cultural production, it rather focuses on how, with the introduction of generative technologies, entropy and indeterminacy foreground a deferring, elusive sense of presence from which poetic meaning arises. In haunted works, the liminal becomes a generative force, and poetic meaning arises from the traces of what is, intentionally or not, removed, excluded, deferred, hidden or repressed (e.g., traces of the model, of the dataset, of the prompts’ referents).Footnote 14 Whereas in the open work notation contributes to increasing entropy and indeterminacy, in haunted works, whether in the form of a score, whether inherent or distributed, notation works as a spell, the conjuration by which ghosts within and around the machine are brought on stage. This was, as a matter of fact, the role played by Fluxus scores in Mouja+.
6. Conclusions
We can now return to our initial questions and reflect on how inherent and text scores might mediate, streamline and support our engagement with AI, and how NAS might shift the dynamics of openness and indeterminacy discussed by Eco.
Real-time generative models such as RAVE centre co-creation in the compositional phase, involving the curation of the data and the training of the model. With real-time NAS, due to arbitrary encoding, dimensional and feature entanglement, notational approaches leveraging the interface are often preferable. This makes inherent scores ideal to streamline composition and performance with these systems, as the practice of embodied sketching exemplifies.
Text-to-audio models encourage co-creation within the performance. The statistical variability in the generation of the outputs introduces an element of unpredictability, which also reflects on the compositional phase, where one develops and sharpens the textual instructions to limit or direct the entropy. Here, text-to-audio models introduce openness through their interpretative limitations, requiring composers to familiarise with their algorithms and to adapt the use of language accordingly. The methods discussed in 5.2 suggest a few practical steps in this direction.
Mouja+, the piece introduced in this paper, combines real-time NAS using inherent scores and embodied sketching with text-to-audio NAS, extending the latter beyond the limitations of prompt jockeying and into performance practice. This piece reflects on how AI tends to conflate presence and absence through different dynamics of signification involving the model, the sounds it generates and the referents they conjure, a process that may be interpreted through Derrida’s notion of hauntology.
Building on Derrida’s spectres, on their temporal discontinuities and agential ambiguities, we can tentatively describe Mouja+ as a haunted work. This notion may be extended beyond this piece, to include other works in which generative technologies foreground a deferring and elusive sense of presence upon which poetic meaning arises, summoning, to extend Walshe’s reflections on AI and spectrality (Walshe Reference Walshe2023), ghosts within, around, but also before and after the hidden layer.
The haunted work aligns Eco’s epistemological metaphor of the open work, its attention to ambiguity, uncertainty and emergence, to the new processes of mediation introduced by generative technologies. As the theoretical ground upon which the notion of haunted works has emerged, hauntology might provide the foundation for a general framework to investigate how AI shapes meaning-making within and beyond the arts. As Derrida reminds us, ghosts are ‘always there, even if they do not exist, even if they are no longer, even if they are not yet’ (Derrida Reference Derrida1994). It is by following their wanderings, that we can begin to understand our technological ‘there’.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/S135577182510085X
