2.1. Agency and creativity

Agency and creativity are both elusive concepts; they take on different definitions depending on the context in which they are applied. Agency, in its most simple conception, refers to the capacity of an entity to act (Schlosser Reference Schlosser and Zalta2019) and can be ascribed to both human and non-human actors. The extent of agency varies among different entities depending on their ability to influence the course of events (Latour Reference Latour1996). This definition of agency can be considered causal, distinguishing it from intentional agency, which implies that the agent initiating an action has a specific objective (Davidson Reference Davidson1963). In software, agency is associated with a programme’s intentionality and autonomy of operation. In this article, we define a software agent as a computer programme situated within an environment, capable of sensing its properties and acting within it over time. This allows the agent to modify the environment’s state according to its own internal objectives (Franklin and Graesser Reference Franklin and Graesser1996).

The meaning of creativity depends heavily on the context in which it is applied. Creativity has several interpretations in psychology and computer science, each representative of distinct yet intersecting ideological and epistemic stances (Hsueh et al. Reference Hsueh, Ciolfi Felice, Alaoui and Mackay2024). Generally, an artefact is considered creative when it is novel while stylistically coherent to its domain (Boden Reference Boden2004). Creativity can be associated with the discovery of novel solutions to clearly defined problems (Kaufman and Sternberg Reference Kaufman and Sternberg2010) or with the generation of novel ideas through free association (Kerne et al. Reference Kerne, Koh, Smith, Webb and Dworaczyk2008). In first-person experiences, creativity is associated with flow states (Csikszentmihalyi Reference Csikszentmihalyi1990), exploration and play (Boden Reference Boden2004). Ultimately, as noted by Cherry and Latulipe (Reference Cherry and Latulipe2014), notions of creativity are not one-size-fits-all; they should be situated within a specific practice (Remy et al. Reference Remy, MacDonald Vermeulen, Frich, Biskjaer and Dalsgaard2020).

In this article, we study creativity within the context of non-idiomatic music improvisation (Bailey Reference Bailey1993). Improvisation is the art of decision-making in time, and a creative ‘artifact’ in this context is a decision made by a (human or machine) agent which affects other agents’ behaviour. In an improvised performance, an agent’s decision is evaluated by how it shifts the course of the performance, either supporting or contrasting the specific behaviour of other agents. Consequently, we define a creative decision as one that fosters novel and surprising outcomes while maintaining stylistic coherence within the improvised context (Blackwell et al. Reference Blackwell, Bown, Young, McCormack and d’Inverno2012). We do not define creativity (or intelligence) as an inherent property of individual agents; rather, we view it as a phenomenon that emerges from the interactions of (human and/or machine) agents within a specific context (Haraway Reference Haraway1985; Saunders Reference Saunders, Veale and Cardoso2019).

2.2. Empirical evaluations of co-improvisation systems

Musical tools embody notions of the music theories they are designed for, carrying these assumptions through their affordances and constraints (Magnusson Reference Magnusson2010). Co-improvisation systems are performance-driven interactive music systems (Drummond Reference Drummond2009) in which the algorithm’s decision-making is based solely on analysis of the material played by a musician during a performance, without any predetermined temporal structure. In co-improvisation systems, the music-theoretical notions are expressed by the aspects of a musician’s sound the system listens to and by the way it responds to them, encoding stylistic expectations (Dubnov Reference Dubnov and Miranda2021). When interacting with semi-autonomous systems, musicians tend to describe their experience in terms of the artificial system’s behaviour. In previous studies, such behaviours have been interpreted by users across several categories, including communication and control (Fiorini Reference Fiorini2024), influential agency (Dahlstedt Reference Dahlstedt and Miranda2021), contextual relevance (Van Nort et al. Reference Van Nort, Oliveros and Braasch2013), emergence (Eldridge et al. Reference Eldridge, Dorin and McCormack2008; Eigenfeldt Reference Eigenfeldt, Vear and Poltronieri2022), co-adaptation (Young and Bown Reference Young and Bown2010; Erdem et al. Reference Erdem, Wallace, Glette and Jensenius2022), domain competence, active involvement, originality, ability to handle uncertainty (Jordanous Reference Jordanous2012), and exploration and stylistic coherence (Blackwell et al. Reference Blackwell, Bown, Young, McCormack and d’Inverno2012). In this article, we refer to behaviour as ‘the internally coordinated responses (actions or inactions) of an agent to external stimuli’, adapting a biological characterisation from Levitis et al. (Reference Levitis, Lidicker and Freund2009) to the definition of software agent discussed above. This definition can be applied to consider individual behaviour of agents, collective emergent behaviour of multi-agent software systems (Heylighen Reference Heylighen2008) and of human-machine co-creative systems. Consequently, in this article adapting and personalising behaviours imply modifying how agents respond to different stimuli to achieve a desired aesthetic objective.

The outputs of a co-improvisation system depend – at least in part – on the musician’s live input. For this reason, these systems can be described as in between completely reactive and completely autonomous in the continuum of autonomy proposed by Tatar and Pasquier (Reference Tatar and Pasquier2019). Similarly, Erdem et al. (Reference Erdem, Wallace, Glette and Jensenius2022) show that their co-improvisation system, coadaptive audiovisual instrument compostion, was interpreted by performers as either a tool or an actor, depending on the degree of control they felt they could exert over it, with this perception shifting throughout a performance. This echoes previous studies on feedback instruments and multi-agent systems where the musician can be a part of the system (Eldridge Reference Eldridge2007) or a manager of the system’s complexity as a curator of a sound-generating process (Wiggins Reference Wiggins, Veale and Cardoso2019; Sanfilippo Reference Sanfilippo2023). Dahlstedt (Reference Dahlstedt and Miranda2021) highlights that a tool’s influence on the final aesthetic outcome – its influential agency – is related to its complexity, which is determined by the constraints built into it by the tool’s creator. The more generative an instrument is, the more influential agency shifts from the player towards the tool designer and maker, reflecting their conception of music. Jordanous (Reference Jordanous2012) proposed a standardised procedure to evaluate co-creative systems by identifying domain-specific definitions of creativity, establishing standards based on these definitions and testing for compliance.

2.3. Personalising co-improvisation systems

Designing novel musical tools requires both technical knowledge and active musical practice (Pelinski et al. Reference Pelinski, McPherson and Fiebrink2024). Musical tools are always adapted to the practice of specific musicians, ideally allowing for subtlety, detail (Armitage Reference Armitage2022), personalised expression and virtuosity (Pachet Reference Pachet, McCormack and d’Inverno2012). This is also true when designing tools capable of making autonomous decisions, such as co-improvisation systems; in this context, the tool’s increased agency raises questions about how its behaviour should be adapted to reflect a musician’s unique expression.

According to Blackwell et al. (Reference Blackwell, Bown, Young, McCormack and d’Inverno2012), co-improvisation systems – or live algorithms – typically consist of a listening/analysis module, a patterning/reasoning module and a sound synthesis module, with each of these modules having different degrees of personalisation depending on how the system is designed. Systems based on concatenative synthesis, like Somax (Assayag et al. Reference Assayag, Bonnasse-Gahot and Borg2022), Dicy2 (Nika et al. Reference Nika, Schwarz and Muller2025) and Musical Agent based on Self-Organizing Maps (MASOM) (Tatar and Pasquier Reference Tatar and Pasquier2017), can be personalised by enabling users to define the corpus of sounds utilised for synthesis and by allowing them to select values of control parameters to adapt the system’s behaviour. Thelle and Pasquier (Reference Thelle and Pasquier2021) later expanded MASOM’s reasoning module, incorporating four discrete control behaviours for users to select from, thereby personalising the patterning/reasoning module. In both systems, there are also a few parameters available for personalising the sensing module. Similarly, systems based on neural synthesis, such as the Living Looper (Shepardson and Magnusson Reference Shepardson and Magnusson2023), can be personalised by training their models on a user-selected sound corpus. Other methods to personalise patterning/reasoning modules incorporate human judgements of model outputs in addition to providing training data. These approaches, such as human-in-the-loop reinforcement learning (Justus Reference Justus2023) and interactive evolution using genetic algorithms (Farzaneh and Mahdian Toroghi Reference Farzaneh and Mahdian Toroghi2019), offer greater control over the aesthetics of the resulting system; however, they require more effort from users to generate the necessary data and/or to contribute to the training configuration and process.

Vigliensoni and Fiebrink (Reference Vigliensoni and Fiebrink2024) argue that sustained musical practice necessitates the continuous development of an instrument and propose an Interactive Machine Learning (IML) approach (Fiebrink, Reference Fiebrink, Till, de Campo, Egermann, Hardjowirogo and Weinzierl2016) in which a model can be iteratively personalised by changing the training examples collected by the user. This approach entails a ’small data mindset’ (Vigliensoni et al. Reference Vigliensoni, Perry and Fiebrink2022) and encourages a conceptual shift from viewing data as ground truth to seeing it as a medium for communicating intention. While IML is an established technique for learning personalised reactive mappings, it has not yet been employed to fine-tune musical tools that exhibit greater autonomy, such as co-improvisation systems. The approach proposed in this article builds on the concept of using data to communicate intentions and applies it to the personalisation of a dynamic, responsive music co-improvisation system.

2.4. Evolutionary algorithms in music

Genetic algorithms (GAs) are a class of stochastic optimisation methods inspired by abstract biological models of Darwinian evolution. They are a class of optimisation methods in which the algorithm searches for optimal solutions in a search space by iteratively generating new samples. Defining a problem in GAs requires a genetic representation and a fitness function. The optimisation process starts by generating a population composed of a fixed number of candidate solutions. Each candidate solution is evaluated against the fitness function, yielding a fitness score. A small number of solutions with the largest fitness score is kept, while the remaining solutions are discarded. A new population is then generated by computing several mutations of the retained candidates, called an offspring, which is then tested on the fitness function, starting another iteration. This process is repeated until an optimal solution is found or a maximum number of iterations is reached.

GAs have been employed in several musical scenarios (Husbands et al. Reference Husbands, Copley, Eldridge and Mandelis2007). Applications in algorithmic composition include the generation of symbolic music respecting specific rules (Gartland-Jones Reference Gartland-Jones2003); sound design applications include sound matching (Shier et al. Reference Shier, Constanzo, Saitis, Robertson and McPherson2025) and sound discovery (Jónsson et al. Reference Jónsson, Erdem and Glette2024b). Jónsson et al. (Reference Jónsson, Erdem, Fasciani and Glette2024a) point out that, in contrast to machine learning’s tendency to output the most likely (and therefore least creative) outcome in response to an input, GAs have the potential to promote novel discoveries through techniques like novelty search (Jónsson et al. Reference Jónsson, Erdem, Fasciani and Glette2024a) and interactive evolution (Johnson Reference Johnson1999).

3.1. Multi-agent autonomous looper

The MAAL is based on a computational model of a looper utilised in live performances, typically consisting of one or more audio buffers where musicians can record sound. Once recorded, segments usually begin playing in a loop, developing harmonic and macro-temporal musical structures over time. In the MAAL, artificial agents decide which segments of a musician’s live performance to loop using custom rules derived from machine listening. The MAAL can be configured to operate with an arbitrary number of loop tracks, each controlled by a distinct agent. Agents decide which segment to loop according to predefined rules which utilise comparison metrics and thresholds that interpret the musician’s sound at the meso-level. For example, an agent equipped with rules about rhythmic compatibility can choose to loop a segment of the musician’s performance if its rhythm is sufficiently similar (or dissimilar) to the rhythm found in the sum of the other concurrent loops. Comparison metrics assess various aspects of harmony, melody, dynamics, timbre and rhythm, and agents’ logic can be defined by combinations of multiple rules that must be satisfied concurrently, enabling complex musical reasoning. This makes each agent’s decisions dependent on other agents’ configurations, as well as the human musician’s performance, establishing a network of relations whose global characteristics exhibit emergent, self-organising behaviour or stigmergy (Theraulaz and Bonabeau Reference Theraulaz and Bonabeau1999; Heylighen Reference Heylighen2008). This makes the configuration of the system complex because of the overall global unpredictability about how different agents’ rules affect each other in a specific context. The interaction between the musician and agents in the MAAL is illustrated in Figure 1. A deeper technical description of the MAAL’s design is available in the previously published article (Madaghiele et al. Reference Madaghiele, Fasciani, Kelkar and Erdem2025).

Figure 1.

Diagram illustrating the interaction model and data flow of a MAAL with three agent-controlled loop tracks. The dashed blue lines show how musicians would normally control a manual multi-track looper by explicitly starting and stopping the recording. When interacting with the MAAL, musicians instead control only the instrument they play, producing a continuous stream of sound that the agents listen to.

Figure 1. Long description

A diagram illustrating the interaction model and data flow of a multi-agent audio looper with three agent-controlled loop tracks. The diagram includes a legend explaining the symbols used: artificial agent, control of instrument, control of manual looper, and sound. A musician plays an instrument, producing a continuous stream of sound that the agents listen to. The sound flows into three agent-controlled loop tracks, each containing an artificial agent and a gear symbol representing the processing of sound. The dashed blue lines indicate how musicians would normally control a manual multi-track looper by explicitly starting and stopping the recording. In this setup, musicians control only the instrument they play. The solid black lines represent the flow of sound from the musician to the agents and between the agents. The dashed lines in various colors (purple, green, orange) represent the control signals between the agents and the instruments. The diagram shows the complex interaction and data flow between the musician and the artificial agents, highlighting the nuanced expression and enhanced personalization in the interaction model.

The MAAL continuously analyses non-overlapping segments s[m] of a live musician’s performance x(t), each with a predefined duration T _u , which typically corresponds to meso-level musical duration (Roads Reference Roads2002). This is defined in Equation (1), where m is a discrete-time index that increments with time intervals T _u .

(1) $$s\left[ m \right] = x\left( t \right),{\rm{\;\ \ \ \ \ \ with\;}}t \in \left[ {t + \left( {m - 1} \right){T_u},{\rm{\;}}t + m{T_u}} \right)$$

The MAAL operates under the assumption that both the musician and the agents follow a fixed known common tempo. The predefined system parameters, which users can select, include the maximum buffer length T _u , expressed in number of beats; the number of loop tracks N; optional maximum and minimum durations for which a selected segment is repeated in a loop; and the tempo. The agent controlling each loop track n utilises a combination of rules ρ _n (), which includes an arbitrary number J _n of individual rules r _j,n (). Each rule consists of a comparison metric c _j,n () chosen from a pool {c ₀(),c ₁(),.c ₁₉()} of nineteen distinct comparison functions, a threshold τ _j,n ∈[0,1] and a comparison coefficient ξ _j,n ∈{−1,1}. A rule set for loop track n can be expressed as in Equation (2).

(2) $${\rho _n}() = \{ {r_{j,n}}()|{r_{j,n}}() = \{ {c_{j,n}}(),{r_{j,n}},{\xi _{j,n}}\}, {\rm{\;}}j \in \left[ {1,{\rm{\;}}{J_n}} \right]\} $$

These elements determine the behaviour of the agent controlling the loop track based on the similarity or dissimilarity computed according to a specific combination of rules. The comparison metrics c _j,n () yield a coefficient between 0 and 1, as in Equation (3).

(3) $$r\left( {s\left[ m \right],\;\overline {{l_n}\left[ m \right]} } \right) = \left\{{ {\matrix{ {0,} \hfill \cr\quad \quad {\rm\;if} \quad {\xi \cdot\left( {c\left( {s\left[ m \right],\;\overline {{l_n}\left[ m \right]} } \right) - \tau } \right) \lt 0}\cr {\xi \cdot\left( {c\left( {s\left[ m \right],\;\overline {{l_n}\left[ m \right]} } \right) - \tau } \right),} \hfill \cr \quad \quad {\rm\;if} \quad {\xi \cdot\left( {c\left( {s\left[ m \right],\;\overline {{l_n}\left[ m \right]} } \right) - \tau } \right) \ge 0\;} \cr } }}\right.$$

Comparison metrics are based on machine-listening principles that analyse sonic characteristics at the meso-level derived from low-level acoustic features. The complete list of the rules and a deeper technical explanation of their operations are presented in Madaghiele et al. (Reference Madaghiele, Fasciani, Kelkar and Erdem2025). An example of MAAL configuration, detailing the rule sets utilised by agents in a four-loop-track setting, is shown in Table 1. If an agent utilises a set with more than one rule, they are combined using ‘and-logic’, meaning the candidate segment is selected for looping only if all the rules are concurrently satisfied.

Table 1.

Example of a MAAL configuration. A rule set ρ _n() is associated with each loop track n. Rule sets can contain more than one rule, each consisting of a comparison criterion c _j,n(), a coefficient ξ _j,n and a threshold τ _j,n. In this configuration, ρ ₁ and ρ ₂ select rhythmically contrasting segments with different timbres, ρ ₃ creates a harmonic accompaniment, and ρ ₄ selects segments with contrasting density. The effect of rules depends on the musical context, and they should be interpreted as relative to each other

Table 1. Long description

A table detailing loop track rules and their associated criteria, coefficients, and thresholds. The table has four rows and five columns. The columns are labeled ‘Loop track rules ρn’, ‘Rule rj,n()’, ‘Comparison criterion cj,n()’, ‘Coefficient ξj,n’, and ‘Threshold τj,n’. The rows are labeled with different loop track rules (ρ1, ρ2, ρ3, ρ4) and their respective rules, criteria, coefficients, and thresholds. For example, under ρ1, the rules are ‘r1,1()’ and ‘r2,1()’ with criteria ‘Rhythmic similarity’ and ‘Timbral similarity’, coefficients ‘-1’ and ‘-1’, and thresholds ‘0.6’ and ‘0.5’. Similarly, ρ2 includes ‘r1,2()’ and ‘r2,2()’ with criteria ‘Rhythmic similarity’ and ‘Timbral similarity’, coefficients ‘-1’ and ‘1’, and thresholds ‘0.9’ and ‘0.6’. ρ3 includes ‘r1,3()’, ‘r2,3()’, and ‘r3,3()’ with criteria ‘Harmonic similarity’, ‘Frequency range overlap’, and ‘Rhythmic similarity’, coefficients ‘1’, ‘-1’, and ‘1’, and thresholds ‘0.65’, ‘0.75’, and ‘0.35’. ρ4 includes ‘r1,4()’ and ‘r2,4()’ with criteria ‘Rhythmic density’ and ‘Harmonic similarity’, coefficients ‘-1’ and ‘1’, and thresholds ‘0.5’ and ‘0.65’.

For each segment s[m] played during the musician’s performance, all N agents compare it against the sum of the concurrently playing segments in the other loop tracks, represented as $$\;\;\overline {{l_n}\left[ m \right]} $$ . If all rules associated with agent n are satisfied, s[m] is selected for looping in track n. When s[m] satisfies the rules of multiple agents, the segment is selected for looping in the track that satisfies the rules to the greatest extent.

The MAAL has been implemented as both an online and offline system. The online version is a mixed implementation of Pure Data (PD) and Python, while the offline version is fully implemented in Python. In both versions, the system settings are defined in a text-based configuration file using JavaScript Object Notation (JSON) format. In the online MAAL, loop agents dynamically respond to a continuous stream of sound generated by a live musician. The sound of each loop track can be processed separately. The PD Graphical User Interface (GUI), shown in Figure 2, visualises the live input analysis and the content of each looping buffer. Settings such as the agent’s rules, tempo and loop duration are defined in the configuration file. The offline version simulates the agents’ behaviour during the musician-MAAL co-performance by using a pre-recorded audio file simulating the musician’s input to the looper. It generates a separate output audio file for each loop track, with each file containing the sequence of segments selected by the MAAL throughout the entire duration of the input file. Additionally, it produces a JSON file that includes a list of co-performance annotations D, reporting whether segments s[m] are selected or not selected for looping.

Figure 2.

GUI of the online MAAL. This implementation provides a maximum of 12 agent-controlled loop tracks.

3.2. Personalisation based on genetic algorithms

The personalisation method we propose, when applied to the MAAL specifically, configures the agents’ rules according to the user’s preferences. We developed an approach that derives these rules from a small corpus of co-performance examples indicating the collective behaviour the MAAL’s agents should produce in response to specific user inputs, identifying which segments of the input should be selected for looping and, consequently, which decisions the agents should make. The objectives of corpus-based automated rule discovery are providing an overall more accurate, efficient and accessible way to obtain desired behaviours compared to manual rule definition. Automated rule discovery could make the system more robust to diverse sounds in the musician’s palette, enhance the system’s autonomy and enable discovery of novel configurations inspiring the musician’s creativity. These aspects are discussed among the results of the user study in Section 6. A summary of the GA optimisation process is illustrated in Figure 3.

Figure 3.

Dataflow chart illustrating the GA process. At each iteration, configurations ϕ _p are used to simulate performances using the sound recordings x _i(t) from the co-performance corpus and the offline MAAL. Agents’ decisions D_i,p in the simulated co-performances are then compared to the target decisions D_target,i from the corpus. The configurations obtaining the best fitness score across all co-performances in the corpus are selected and used to produce a new population through cross-combination and mutation. The iteration stops when one or more configurations achieve a fitness score above a threshold or when the maximum number of iterations is reached.

Co-performance corpus. The GA search is based on a corpus C composed of I co-performance examples provided by the musician. Each example i is an annotated co-performance, which includes an audio file of the musician’s performance x _i(t) – representative of the musician’s style and instrument – along with annotations D_i-describing the decisions the co-improvisation system should make at any point in time; in the case of the MAAL, decisions annotations report which segments s[m] should be chosen for looping at each time step m of the sound file. These annotations can be produced in various ways, as detailed below in Section 3.3. The annotation is a binary array D_target,i = {D _i[0],D _i[1],…,D _i[M _i]}, where D[m] ∈ {0,1} indicates which segments s[m] of the performance x _i(t) are selected for looping. Each element of the array corresponds to a segment s[m] in x _i(t), with a value of 1 if the segment at time step m is selected for looping and 0 if it is not selected. Examples can have varying total durations M _i. Each entry in the co-performance corpus C is composed of a sound segment x _i(t) and its corresponding annotation D_target,i, as in Equation (4):

(4) $$C = \left\{ {\left( {{x_i}\left( t \right),{{\bf{D}}_{{\bf target},i}}} \right){{|}}i = 0, \ldots, I} \right\}$$

Annotations do not specify on which MAAL track the segment is intended to be looped. This enables the learning of a set of rules that can be applied to different numbers of agent-controlled loop tracks, simplifies the annotation process and gives the MAAL freedom to discover diverse rule populations. These rule sets, while satisfying the binary annotations, can yield varied segment – track allocations, leading to diverse layerings and macro-structures in performance and, in turn, providing the system with collective creative agency. To compute the input segmentation, the fixed tempo and loop duration T _u must be provided.

Population. The GA process consists of a search over a population Φ = {ϕ ₁,ϕ ₂,…,ϕ _P} of configurations, where a configuration ϕ _p = {ρ ₀(),ρ ₁(),…,ρ _N()} is composed of a rule set ρ _n() for each track n of a MAAL with N loop tracks, and P is the population size. The initial population of configurations is generated randomly, with each loop track associated with only one rule r() = {c(),τ,ξ}, with c(),τ,ξ extracted randomly. An individual configuration ϕ _p is a set of rules for each track, such as in the example of Table 1. Each individual ϕ _p is used to simulate the behaviour of the MAAL for each sound example in the corpus, utilising the offline implementation, which records the looping decisions D_i,p made for each x _i(t) in the corpus in an annotation file. These decisions are made by the agents in accordance with the rules defined by the configuration ϕ _p. In GA terminology, ϕ _p represents the genotype, producing a phenotype D_i,p as in Equation (4). The decision annotation D_i,p contains all the information needed to replicate the behaviour of the MAAL from the input sound x _i(t).

(5) $$MAAL\left( {{\phi _p},{\rm{\;}}{x_i}\left( t \right)} \right) = {{\bf{D}}_{i,p}}$$

Fitness function. The fitness function L is computed by comparing the decision annotation D_i,p produced by the offline MAAL against the annotation D_target,i of the corresponding corpus track. The fitness function measures the similarity between the two annotations, computing a score between 0 and 1, with a score of 1 indicating identical co-performances. The objective of the GA search is to find, for each loop track, the rule set that maximises similarity across the entire corpus of examples, meaning the system learns rule combinations that select the same segments as the user-selected annotations. This is done by computing a logic ‘and’ operation, as in Equation (5).

(6) $${L_p}\left( {{\bf{D}}_{{\bf{target}}},{{\bf{D}}_p}} \right) = \frac{1}{I}\mathop \sum \nolimits_i^I \frac{1}{{{M_i}}}\mathop \sum \nolimits_m^{{M_i}} {{\bf{D}}_{{\bf{target}},\;i}}\left[ m \right] \wedge {{\bf{D}}_{i,p}}\left[ m \right]$$

In contrast to directly comparing audio, we compare system decisions resulting in equivalent changes to the audio. Decision annotations are more lightweight and computationally efficient than audio comparison, less error-prone, and they allow for the generation of more diverse solutions that satisfy the same requirements.

Cross-combination and mutations. After computing the fitness of each individual, the K individuals with the highest fitness score are selected, and an offsprings of P new individuals with new configurations ϕ _p are generated by cross-combining two randomly selected elements from the configurations of the K best individuals and computing R mutations over the resulting configuration. Cross-combination consists of substituting the rule set associated with a random track of a randomly selected agent in a configuration with the rules associated with a randomly selected agent in a different configuration. Six distinct mutations are possible, which are randomly selected with different probabilities. More than one mutation per iteration is possible for each individual. The possible mutations are (1) add a new random rule to a random track, (2) remove a rule randomly from a random track, (3) substitute a random rule in a random track with a new one randomly extracted, (4) increase the threshold of a random rule in a random track by 0.1, (5) decrease the threshold of a random rule in a random track by 0.1 and (6) reverse the comparison coefficient of a random rule in a random track. Mutations are computed, ensuring that there is at least one rule for each agent and that threshold values remain between 0 and 1.

Optimisation loop. The search process is repeated using the offspring as a new population. The search ends either when the fitness score of the best configuration exceeds a threshold or when the maximum number of iterations is reached. In most cases, depending on how the corpus has been generated, rule combinations that reach the maximum fitness score – making exactly the same decisions found in all examples of the user-provided corpus – are not possible; therefore, a sub-optimal solution must be accepted.

Search parameters. Optimal parameters for the search process depend on the variety of the materials in the corpus and on the cumulative number $$M = \mathop \sum \nolimits_i^{} {M_i}$$ of segments s[m] of each example track. The global parameters of the GA are the maximum number of iterations, the threshold for the fitness function and the probabilities of each mutation type. Heuristic experiments suggest that, with a corpus size of around M ≈ 50, appropriate parameters seem to be a threshold of 0.9, a maximum of 60 iterations, with a population P = 16 individuals and K = 2 best parent configurations kept, and R = 8 mutations at each iteration, with a higher probability of mutations (4) and (5) with respect to the others, to favour adjustment of thresholds to the musician’s sound materials. K = 2 parent configuration can be considered a low value for a typical genetic search; however, with a relatively large number of mutations, it tends to reach satisfactory values of fitness in relatively fast search times (3–4 hours). GA results obtained with these hyperparameters can more easily get stranded in a local minimum of the vast search space, a sub-optimal result; increasing K would make the GA result more robust; however, it would significantly increase the time needed for convergence. The search for optimal agent rules using the described genetic algorithm is implemented in Python, utilising multithreading to accelerate computation by enabling the simultaneous execution of multiple offline MAAL simulations.

3.3. Corpus collection

Looping decisions in a live context depends on the specific moment at which they are made, making it challenging to predict how segments would sound when looped together, whether those decisions will make musical sense in context and how the musician would respond to them during a performance. We developed two separate data collection methods that allow us to record decision-making during live looping performances and then select sections of these performances that represent behaviours the user would like to replicate.

The first data collection method employs a custom-made, manually controlled looper implemented in PD. The custom non-autonomous looper creates an annotation of the looping decisions D_i as the musician plays and controls the manual looper, while simultaneously recording the live sound generated by the musician. This results in a pair consisting of a sound file and a decision annotation that can be used to mimic an ideal MAAL co-performance. In this way, musicians can explicitly indicate which materials to loop. Another way to collect data is by producing an annotation D_i while using the online MAAL. The online MAAL produces an annotation of the looping decisions while the musician is playing, in a format identical to that produced by the custom non-autonomous looper.

These two data collection methods produce identical D_i formats, which can both be used as part of the corpus for GA optimisation. After collection, users can visualise each co-performance as shown in Figure 4, listen back to it and select subsections of each co-performance in which they find the global musical structure most interesting by indicating start and end timestamps. The software then computes the closest m to the user-provided start and end timestamps and slices x _i(t) and D_i accordingly. The newly obtained subsection of the co-performance is added to the search corpus. In this way, users can select behaviours they want to promote and remove behaviours they don’t appreciate.

Figure 4.

Example of visualisation of a co-performance. The red box indicates the section of the co-performance the musician chooses to be included in the corpus used for GA optimisation. x(t) visualises the live performance of the musician, while loop tracks L _n portray the content of each loop track during performance. D shows the content of the annotation vector, which is 1 for segments s[m] selected for looping and 0 for segments not selected.

Figure 4. Long description

Panel A: A line graph visualizing a co-performance. The horizontal axis represents time in minutes and seconds, ranging from 0:00 to 3:20. The vertical axis represents different data lines. The graph includes multiple data lines labeled as x(t), L1, L2, and L3. The x(t) line visualizes the live performance of the musician, while L1, L2, and L3 portray the content of each loop track during performance. The red box indicates the section of the co-performance the musician chooses to be included in the corpus used for GA optimisation. The content of the annotation vector is shown as a series of 1s and 0s, where 1 indicates segments selected for looping and 0 indicates segments not selected. The graph shows various trends and patterns in the data lines, with the red box highlighting a specific section of interest.

4.1. Participants

Our study aims to highlight the co-adaptation of musicians and the MAAL. For this reason, we opted to conduct a user study with a relatively small number of participants, emphasising each participant’s uniqueness rather than generalising results across many standardised studies. Five professional musicians were selected to participate in this study. All are expert improvisers and active performers in the music scenes of Norway and Sweden. We chose expert improvisers because they have a defined long-term practice, with well-defined styles, instrumentation and specific aesthetic preferences. The study was conducted by the first author at the Department of Musicology at the University of Oslo. All participants voluntarily agreed to have their identities disclosed in this article. Participants’ instrumentation and style are summarised in Table 2.

Table 2.

Information about user study participants

Table 2. Long description

A table summarizing musicians’ instruments, sound materials, and improvisational styles. The table has five rows and four columns. The columns are labeled ‘Musician name’, ‘Instrument’, ‘Sound materials’, and ‘Improvisational style’. Row 1: Musician name, Kelsey Cotton; Instrument, Voice; Sound materials, Dynamic patterns of alternating vocal elements with extended techniques; Improvisational style, Non-idiomatic. Row 2: Musician name, Victoria Johnson; Instrument, Violin and distortion effect pedals; Sound materials, Sustained melodies, pizzicato, extended techniques; Improvisational style, Non-idiomatic. Row 3: Musician name, Giuseppe Pisano-Riise; Instrument, Moog Mother and Soma Laboratory Lyra-8 synthesizers; Sound materials, Frequency-modulated sound bursts and sustained feedback; Improvisational style, Non-idiomatic. Row 4: Musician name, Aleksander Tidemann; Instrument, Drum set played with drumsticks and brushes; Sound materials, Repetitive sequences on drums, slow dynamics and resonant cymbals; Improvisational style, Non-idiomatic. Row 5: Musician name, Kristian Wentzel; Instrument, Nord Electro 4D keyboard synthesizer; Sound materials, Piano and organ timbres, melodies, chords, arpeggios; Improvisational style, Soul and jazz/fusion.

4.2. Study procedure

The study was structured into two separate sessions, each lasting three to four hours, conducted on different days with a maximum interval of one week between them. The methods included a combination of semi-structured interviews (Adams Reference Adams2015), stimulated recall (Dempsey Reference Dempsey2010) and think-aloud protocols (Charters Reference Charters2003).

At the beginning of the first session, participants were asked to describe their musical experience, focusing on their approach to improvisation and their instrumental setup.Footnote ² After, musicians were introduced to the MAAL and the custom non-autonomous looper used for data collection. They were then asked to perform an improvised performance using the PD custom non-autonomous live looper, configured with four looping tracks and operated via a Musical Instrument Digital Interface controller. Subsequently, musicians were asked to perform two improvised performances with the MAAL. The MAAL had two different baseline configurations in these two sessions, equal for all participants, designed by the authors as a baseline adaptable to many instruments and styles. The first configuration was oriented to capturing melodic and harmonic relations, while the second one was focused on rhythmic and timbral relations. The two baseline configurations are available in the supplementary materials of this article. Multi-track recordings of these performances, including the musicians’ sound and co-performance decision annotations, were collected and stored. Finally, musicians were asked to listen back to their two co-performances with the MAAL and to their performance with the non-autonomous looper, visualising the annotations as in Figure 4 and selecting subsections of them to create their personal corpus for the GA to determine the MAAL’s rule set.

The co-performance corpus selected by the musician during the first session was used to run the GA, resulting in two personalised MAAL configurations of agent rules, ϕ ₁ and ϕ ₂. During the second session, musicians were asked to perform with the two personalised configurations of the MAAL and to compare their experiences. Musicians were asked to choose the configuration they preferred among the two and use it to film two co-performances, this time with the option of adding more loop tracks duplicating some of the existing rules. Videos of the preferred co-performance of each musician are included in the supplementary material of this article.

Throughout the whole study, all participants were limited to four loop tracks and a T _u = 8 beats at a tempo of 120 beats per minute. All performances had an approximate duration of 10 minutes. The complete MAAL implementation allows selecting different loop durations per track through parameter T _l; for comparability, in all the user studies we set T _u = T _l. The sound from each of the four loop tracks was panned at equal intervals from left to right between two stereo speakers, and participants did not have the option to process the sound of the MAAL in any way, including volume and panning.