1.1. Background

Although the ideas for this project do not stem directly from the discussions about interactivity in the context of music and sonic arts (Paine Reference Paine2002; Bown, Eldridge and McCormack Reference Bown, Eldridge and McCormack2009; Drummond Reference Drummond2009), and the system itself is not closely related to the interactive systems presented at the New Interfaces for Musical Expression conferences (NIME, since 2002), ENACTIV may be considered as an exploration of interactivity along with the discussions mentioned above. Its mapping algorithms, empowered by the iterative equations which generate chaotic attractors, take their interactive potentials in the direction of the investigations in the sense of ‘human and machine agency’ (Paine Reference Paine2002) and ‘behavioural objects’ (Bown et al. Reference Bown, Eldridge and McCormack2009).

In his article entitled ‘Interactivity, Where to from Here?’ (Paine Reference Paine2002), Garth Paine brings forth the problem of interactivity and calls for ‘interaction that responds more directly and uniquely to those who engage in the work’ and ‘human and machine agency’. This new approach to interactivity involves performance by people through the dynamic of their movement. Paine criticises the use of the term ‘interaction’ or ‘interactive system’ used to describe systems which involve some kind of ‘reaction’ or response but exhibit very little (if any) agency on their own. The description of interactivity in these instances refers to a user-controlled system that reacts in a more or less prescribed way. Brian Belet (Belet Reference Belet2003) emphasises the importance of the change from ‘static’ to ‘dynamic’ performance interaction that has been under way since the early 1990s, referring to the role and agency of the software used in an interactive performance. Further, Jon Drummond (Drummond Reference Drummond2009) defines the interactive system as one that has ‘the potential for variation and unpredictability in its response’.

Similar ideas about developing an interactive software with a quality of a ‘behavioural object’ with ‘capacity to act as the musical and social focus of interaction in digital systems’ are explained by the authors of ‘Understanding Interaction in Contemporary Digital Music: From Instruments to Behavioural Objects’ in their recent article (Bown et al. Reference Bown, Eldridge and McCormack2009). They explore the potential of interactive systems to facilitate the creation of dynamic compositional sonic architectures through performance and improvisation.

Simon Waters introduces ‘Ecological Approaches to Musical Interaction’ with the idea to ‘suppress cultural and historical factors when looking at the distinctions between performer, instrument and environment’. He emphasises the role of the body and the physicality of the performer (Waters Reference Waters2007: 2).

Ian Whalley's ideas about interaction involve ‘software-based agents’ for creating music and sound art in both ‘linear’ and ‘non-linear’ (non-predetermined form and/or content) idioms (Whalley Reference Whalley2005, Reference Whalley2009). This ‘generative’ approach allows agent technology to create content and even structure based on the dynamic interplay of parts. This terminology is also used by Dorin and Miranda (Dorin Reference Dorin2001; Miranda Reference Miranda2003).

1.2. Interactive systems

Many present systems exhibit some of the forms of interaction mentioned above. A few recent interactive systems have been developed for non-expert musicians. For example, reactable (Kaltenbrunner, Jorda, Geiger and Alonso Reference Kaltenbrunner, Jordà, Geiger and Alonso2006), Audiopad (Patten, Recht and Ishii Reference Patten, Recht and Ishii2002), Composition on the Table (Blaine and Fels Reference Blaine and Fels2003), Blaine's collaborative performance environment Jam-O-Drum (Blaine and Perkis Reference Blaine and Perkis2000), Voyager (Lewis Reference Lewis1993), GO/Max – an agent programming language for real-time control of sound/music generation written in the Max/MSP environment (Bottoni, Faralli, Labella and Pierro Reference Bottoni, Faralli, Labella and Pierro2006) and others. The last two are examples of interactive systems in which a performer can work with other performers and develop musical ideas in a collaborative way.

Di Scipio's Audible EcoSystemic Interface (AESI) system generates music through interactions between software elements with strong performative agency. Using a System Dynamics modelling system (Whalley Reference Whalley2000), Whalley investigates the nature of creativity and communication and employs modelling of narratives providing metaphorical means of music composition based on archetypes in a non-real-time environment. Whalley's Public Interactive Web-based Composition System PIWeCS (Whalley Reference Whalley2004) is a multi-agent system that allows for dialogue between performer and the software agency programmed in MaxMSP. Miranda and Gimenes (Miranda and Gimenes Reference Miranda and Gimenes2008) have introduced iMe (Interactive Music Environments), based on adaptive learning agents interacting with a performer's input and generating music in real-time.

1.3. ENACTIV's interactive potentials

In a link with the ideas about autonomous agency-exhibiting systems, ENACTIV's interactive nature relies on its mapping system's agency, empowered by the unpredictable nature of the attractors-generating objects. By varying one or more parameters of the iterative equations, the system develops variations with an additional self-organising quality, which provides for the system's agency and generative potentials. This is due to the chaotic system's nature, which does not produce random outputs but allows for self-organisation with a certain degree of unpredictability.

ENACTIV's expressive potentials are driven by the following two elements:

1. 1. Attractors’ self-organising quality, which allows for a unique agency of the system during the interactive performance and thus approaches the ideas of ‘behavioural object’, which offers an alternative to mappings based on a neural network or a simple linear method.

2. 2. The multi-layered mapping subsystem through which the performer's input data is mapped onto the parameters controlling the sound synthesis and processing algorithms.

2.1. Emergent properties

In the early 1920s the philosopher C.D. Broad (Broad Reference Broad1925) coined the term ‘emergent properties’ for those properties that emerge at a certain level of complexity but do not exist at lower levels; that are properties of the whole, but which none of the parts have within themselves – ‘as the taste of sugar is not present in the carbon, hydrogen, and oxygen atoms that constitute its components but in the sugar as a whole’, as the physicist and philosopher Fritjof Capra writes in his seminal book The Web of Life (Capra Reference Capra1996). They arise from the interactions and relationships among the parts and are destroyed when the system is dissected into isolated elements. In the context of ENACTIV the several modes of expression together – sonic/vocal, visual and gestural – bring forth emergent properties that are absent in each of them when used separately. The ‘enactive’ approach to composition initially named by Wessel (Wessel Reference Wessel2006) can be regarded as a process that translates or maps a set of composer/performer's metaphoric associations, which are closely related to his or her mental imprints or archetypes, into patterns of another kind that constitute the ‘structure’ of the composition. The process includes real-time gesture capturing, mapping algorithms, controlling sound synthesis and processing parameters, and involves a considerable sensory-motor activity. The emergent quality also derives from the system's non-linear response based on strange attractors employed in the mapping processes.

2.2. Creative process and archetypes

ENACTIV's multi-modal and structural-coupling-dependent creative potentials are closely related to the archetypes as our deepest mental imprints. The archetypes, a term used initially by Jung (Jung Reference Jung1958) and furthered by Neumann (Neumann Reference Neumann1954), were studied by numerous psychologists, philosophers and sociologists (Chomsky Reference Chomsky1972; Dawkins Reference Dawkins1989; Gray Reference Gray1996; Hawkins Reference Hawkins1997; Ramachandran and Blakeslee Reference Ramachandran and Blakeslee1998; Koch Reference Koch2004; and others). In general they exist in a very deep layer of the brain giving rise to images, bodily movements and sonic patterns, and act as a bridge between the non-symbolic cognitive processes and the complex social and cultural exchanges that take place in human social groups based on symbolic communication such as language, signs, music, visual shapes, and so on.

Archetypes’ meaning lies in their cognitive and emotional structuring patterns, whose manifestations may be found in art, rituals, imagery, dreams, philosophy and other human aspects. Impelling to action, they present themselves as ideas (unity, the number one, contrasting opposites), images (triangular shapes, spirals, circles), as well as kinetic and sonic patterns in dancing and chanting.

These archetypes generate a network of abstractions which are embodied in metaphors that can potentially serve as drivers for creation evolving into sonically articulated structure via the process of enactive embodiment. This process is shown in figure 1.

Figure 1

Process of creation.

3.1. Enaction

Santiago Theory (Maturana and Varela Reference Maturana and Varela1980, Reference Maturana and Varela1987) served as an inspiration to the Italian composer Leonardo Di Scipio in the context of the system for interactive composition he developed at the turn of this century (Di Scipio Reference Di Scipio2003). He focused on the notion of interaction in a search for answers to the questions, ‘What kind of systems are interactive computer music systems?’ and ‘What paradigm of interaction do they implement and make socially available?’. Briefly, his idea was to create a system that will exhibit adaptive behaviour to the surrounding external conditions and will be capable of interfering with the external conditions themselves – and instead of ‘composing interactive music’, to paraphrase Winkler's Composing Interactive Music: Techniques and Ideas Using Max (Winkler Reference Winkler2001), Di Scipio worked toward ‘composing musical interactions’ (Di Scipio Reference Di Scipio2003). He designed, implemented and maintained a network of connected components whose emergent behaviour in sound ‘one calls music’. This is a strong element of self-regulation of the system which can be related to the key words of Santiago's theory of living beings – that of autopoiesis.

Although ENACTIV was also inspired by Santiago Theory it differs from Di Scipio's interactive system in terms of the performer's role. In this project I have been led by the notions of enaction, structural coupling, and embodiment (embodying abstractions) as terms related to cognitive, multi-modal and movement-based thinking. The interaction occurs between the software's agent output and the new actions by the performer in a continuous feedback loop during the live performance, and the role of the performer/user/composer is essential. To a certain extent the performer's actions or feedback will decide how the system will behave in the future.

In terms of its creative function, ENACTIV can be defined as both a conceptual and a material framework for composing. It relies on a mapping system for translating mental structures into sonic articulations via a set of algorithms for non-linear mapping based on dynamically controlled strange attractors. The ‘mental narrative’ can be obtained from reciting a poem (sounds to be captured and mapped), dialogue, monologue, narrative on any topic (sound and gestures to be mapped), non-language based vocalisations, gestures, and so on. The mapping model (see figure 12) involves gestural and sonic patterns that evolve into composer/performer parameters’ patterns, which in turn are mapped onto patterns for sound synthesis.

3.2. Gestures

A considerable body of evidence confirms that people unwittingly produce gestures along with speech in various situations (Kendon Reference Kendon1972; McNeill Reference McNeill1992; Cassell and Scott Reference Cassell and Scott1996; Cassell and Vilhjálmsson Reference Cassell and Vilhjálmsson1999 and others). Cassell (Cassell Reference Cassell1998) defines a general framework for generation and interpretation of spontaneous gesture and describes how the gestures’ morphological features (hand shape, trajectory, etc.) develop along with the spoken language. They are characterised by a smooth, continuously changing relationship of the hand to the body where each movement is made up of many small variations and adjustments. Jensenius, Godøy and Wanderley have led one of the most comprehensive studies of gestures in Music (Jensenius, Godøy and Wanderley Reference Jensenius, Godøy and Wanderley2005). The research evolved into Musical Gestures Toolbox – a collection of Max/MSP/Jitter modules for analysis of spontaneous gesture–sound associations.

According to McNeill (McNeill Reference McNeill1992), gestures are symbolic and present the thought in action supplying the idiosyncratic, personal and context-specific aspects of thought. Since gestural discourse is not a coded system, according to McNeill, no specific gestures or movements can be considered to denote particular meaning. Instead, they create homology between concepts and images or between sonic and kinetic patterns, which leaves room for individuality. It is this quality of the gestures that makes them powerful and a potential source for a unique way of communication, and for this reason they were considered as one of the crucial inputs in ENACTIV. The meaning emerges from the expressive metaphoric patterns – either single- or multi-modal ones – that ‘tell the story’ and can be captured and translated into patterns of data streams driving the sound synthesis and processing algorithms.

4.1. Context

A number of technically sophisticated audiovisual systems that produce music from body movements have been developed since the end of the twentieth century: STEIM's BigEye, Rokeby's Very Nervous System, EyesWeb (Camurri, Ricchetti and Trocca 2000), SICIB (Morales-Manzanares, Morales and Dannenberg Reference Morales-Manzanares, Morales and Dannenberg2001) and others. Until the early 2000s little attention was paid to the mapping process ‘by which data from an input device is associated with synthesis parameters for real-time control’ (Cadoz and Wanderley Reference Cadoz and Wanderley2000). In his article ‘Gestural Control of Music’, Wanderley discusses the problem associated with the lack of mapping strategies in general – ‘there seems to exist a huge unexplored potential on the definition of mapping strategies for different contexts’ (Wanderley Reference Wanderley2000: 11).

Several years of investigations into the complexity of mapping procedures at IRCAM in Paris resulted in the development of MnM toolbox in 2005 – a first systematic collection of ‘practical tools to implement complex mapping in a relatively intuitive manner’ and in real-time that were presented, discussed at NIME and published in the conference's proceedings (Bevilacqua, Müller and Schnell Reference Bevilacqua, Müller and Schnell2005). MnM's most recent non-linear mapping modules were released at the beginning of 2010 and are an invaluable practical mapping resource for ENACTIV. The MnM toolbox uses vectors and matrices to represent mappings and provides methods to build them iteratively. MnM's mapping will be discussed later in this article.

4.2. Definition

Mapping can be defined as connecting gestures to processing, and processing to response. ENACTIV benefits from the developments in the area of mapping strategies based on multi-layered mapping in the early 2000s (Hunt and Kirk Reference Hunt and Kirk2000; Hunt and Wanderley Reference Hunt and Wanderley2002; Van Nort, Wanderley and Depalle Reference Van Nort, Wanderley and Depalle2004).

In the specific context of a digital musical instrument (Miranda and Wanderley Reference Miranda and Wanderley2006: 3), mapping defines the connections between the outputs of a gestural controller and the inputs of a sound generator. In EyesWeb (Camurri et al. Reference Camurri, Ricchetti and Trocca2000) the authors approached mapping from a cognitive perspective, taking into consideration the structure of creation and emotion modelling (KANSEI project). They investigated how users can communicate by means of different modalities, such as body movement, gesture and voice, in real-time, by extracting a small number of parameters and by sending them as inputs to a self-organising neural network for mapping. Cont, Coduys and Henry (Reference Cont, Coduys and Henry2004) have implemented an ‘adaptive approach to gesture mapping’ developed in the Pure Data environment.

In both cases the mapping procedure is based on neural networks, which have the potential to simulate complex natural processes.

4.3. Toward a non-linear mapping

ENACTIV's contribution to interactive creation and multi-modal expression lies in the expressive potentials based on capturing sounds and gestures and organising the modalities into several performer-side parameters, which in turn are translated into a system's parameters. These qualities are based on the following two points:

1. 1. The introduction of equations that generate strange attractors employed as non-linear mapping tools, both as independent mapping algorithms and in conjunction with IRCAM's MnM mapping library, whose self-organising and non-linear nature is applied to the many-to-many mapping algorithms, thus producing agency or ‘behavioural’ characteristics at the system's output and allowing for a genuine interaction between the performer and the system.

2. 2. The strong emphasis on the multi-layered mapping scheme, inspired by the developments in the area of mapping strategies since 2000 (Wanderley and Battier Reference Wanderley and Battier2000; Hunt and Wanderley Reference Hunt and Wanderley2002), as a comprehensive way of associating processing and synthesis parameters to the performer-generated control data, translating the performer's multi-modal expression into a sonically rich output.

4.4. Vectors and matrices

Vectors – from the Latin word ‘vector’, which translates into a ‘carrier’ – are defined by both a magnitude and direction, and play an important role in physics, representing quantities such as the velocity, speed and acceleration of a moving object. A number of physical quantities can be represented by vectors. In physics and mathematics, they change in time and can be identified as lists of numbers, which depend on some coordinate system. The MnM's vectors are systematically arranged numbers in matrices consisting of rows and columns. A ‘matrix’ can be described as a rectangular array of numbers. A matrix consisting of only one column or row represents a vector. Matrices are very convenient constructs for representing multi-dimensional linear transformations and functions, and therefore there is very good potential for building multi-dimensional mapping procedures using matrix operations. For example, one of the simpler mapping operations in MnM can be described as Y = A*X, where X is a vector from the controller parameter space and Y a vector from the sound parameter space. The mapping operation is represented with the matrix multiplication, where A is defined as n × m (for both n ⩾ m and n < m), which reads n(size)-to-m(size) linear mapping. The integer values of X and Y are subjected to a complex single value decomposition (SVD) and principle component analysis (PCA) and then multiplied by the floating point matrix A (figure 2).

Figure 2

MnM's matrix multiplication A = m × n, for m = 8, n = 3, Y = A*X.

4.5. Strange attractors and non-linear mapping

‘Strange attractors’ are systems associated with the Chaos Theory, a field of physics that deals with irregularities and nonlinearities found in many natural phenomena such as the weather, population cycles of living beings, biological systems such as rates of heartbeat and blood pressure, human physical behaviour and others. These systems are deterministic; however, their future behaviour cannot be predicted in the long run. These nonlinear dynamic systems are based on mathematical equations whose evolution can show both organised and random conditions depending on the values of initial parameters. They create self-similar patterns and are distinct from randomness resembling patterns of change found elsewhere in nature.

4.5.1. Iteration

The driving force of these formulas is iteration. As the chosen function is iterated, its values create a set of points toward which the value of the function tends after a number of iterations, filling the delimited space with points. This space is referred to as ‘strange attractor’. It is particularly important to note that nonlinear dynamic systems are extremely sensitive to initial conditions; the initial conditions often produce very large qualitative differences in the output. In the current version, ENACTIV's mapping is based on the logistic ‘population growth’ equation (figure 3).

Figure 3

Baker's map based on the logistic ‘population growth’ equation Xn = K * X * (1.–X).

In the future, different attractors from the Attractors Library will be tested and implemented.

4.6. The moment of mapping

4.6.1. Attractors-only mapping

Here is an example indicating how the population growth mapping equation is employed in practical situations: a common ‘waving–saluting’ gesture consisting of lifting the arm, waving and returning the arm into the resting position can be mapped to the ‘k’ parameter of the logistic equation in the baker's object, which controls the character of the growth and the level of stability of the system by choosing the moment of bifurcation that translates into various degrees of disorder (figure 4).

Figure 4

Baker's map (population growth) equation's ‘k’ parameter fluctuates between 1.1 and 4.

If ‘k’ is greater than 1 but less than 3, the long-term behaviour of the logistic map is to fall into a fixed point. If one sets ‘k’ greater than 2.8 but less than 3.0 then the values of the attracting fixed point will increase as one makes ‘k’ larger. When ‘k’ is set just above 3.0–3.2, for example – the system will never converge to any fixed point. Instead it will settle into a period-2 limit cycle, which means that it will oscillate between two values. The moment when the system moves from a fixed point to a period-2 limit cycle occurs at exactly k = 3.0. This is called a bifurcation. When the ‘k’ is set further to higher values, for example just higher than 3.44, the logistic map bifurcates a second time, giving a period-4 cycle. With further increases the system will enter into period-8, period-16, and so forth, cycles. At a special critical value, k = 4.0, the dynamical system will fall into an infinite-period limit cycle, which is called chaos.

4.6.2. Mapping based on synergy of attractors and MnM's matrices

This is an example of one of the five subsystems which currently do the mapping in ENACTIV, showing how the eight parameters derived from the gestures’ acceleration and locus, which altogether define the intermediary performer-side parameter entitled ‘character’, are mapped to three parameters, which in turn are further mapped to parameters as components controlling the pitch, amplitude and durations of the ‘instruments’. The attractor-generating Max external object bakers modifies the data stored in the MnM's matrix-based objects. The idea is to ‘delinearise’ the linear ‘many-to-few’ matrix-based mapping (8 to 3 in this case) in real-time, consisting of eight-dimensional input and three-dimensional output vectors (figure 5). The x from the input vector changes the ‘k’-parameter of the logistic equation. The y from the input vector changes the ‘x’-parameter of the logistic equation. The randomised x re-scales MnM's three-dimensional output vector. In the bottom-left part of figure 5 one can compare the ‘normal’ and the delinearised output of the system.

Figure 5

MnM's many-to-few (8-to-3) linear matrix-based mapping delinearised by the logistic equation's output.

5.1. General remarks

ENACTIV (v1.0) (figure 6) is an integrated open-ended interactive multi-modal composing system, designed for composers, audiovisual artists and amateurs who can express themselves by converting their sonic (instruments, singing, speech) and bodily kinetic expressive patterns or ‘narratives’ into streaming data, which in turn can control sound synthesis and processing algorithms. The data coming from both video and sound capturing device(s) is first conveniently scaled and converted into streams of variables, and then routed to a ‘mapping field’ where they are subjected to non-linear mapping into control parameters for sound synthesis, processing and multi-channel spatialisation algorithms in real-time. The mapping itself is based on a subsystem that combines the following two libraries:

Figure 6

ENACTIV v.1.0.

1. 1. Attractors Library (Spasov Reference Spasov2009–10a and b) – a collection of Max external objects that generate strange attractors in real-time (mentioned above).

2. 2. IRCAM's MnM, a matrix-based mapping library (Bevilacqua, Müller and Schnell Reference Bevilacqua, Müller and Schnell2005) that allows for a multi-dimensional vector handling, which has already been introduced earlier in the article.

The project that resulted in a software application integrated the following components:

• • Studying human interactive situations, gestural and vocal communication and identifying patterns to be implemented in ENACTIV. This includes analysing the combination of body language and speech inflection in conversations and discussions, public speeches, recitations, children's intuitive vocal and gestural expression and body posture, and so on.

• • Designing software algorithms that ‘translate’ or map movement into parameters for sound synthesis and processing in live performance. The system captures gestural and sonic patterns using a camcorder and microphone, then converts the data into parameters, which in turn generate variables for controlling sound and video synthesis and transformation.

• • Exploring the creative potential of the system through action – compositions and improvisations – which is the crucial component of this project. Hence the title derived from the idea of enaction – cognition via action.

5.2. Structural coupling and strange attractors

The study and implementation of chaotic systems – the strange attractors embodied into the Attractors Library (Spasov Reference Spasov2009–10a and b) (figure 7) in this case – is one of the most important parts of the system, which has already been noted in this text. These attractors define ENACTIV's character regardless of the sound synthesis and processing algorithms used. The library consists of a collection of MaxMSP externals organised into modules (MAX/MSP/Jitter abstractions) based on iterative mathematical equations that generate ‘strange attractors’, which are based on mathematical equations whose evolution can show both organised and random conditions depending on their parameters. They create self-similar patterns and are distinct from randomness resembling patterns of change found elsewhere in nature. It was this pattern-generating feature of the strange attractors in combination with some variations caused by the slight changes in the equation's parameters in real-time that made them a very convenient nonlinear mapping tool in ENACTIV.

Figure 7

Attractors Library.

5.3. Data acquisition

ENACTIV implements both direct and indirect data acquisition. The direct acquisition includes a camcorder(s) that is used to monitor the following actions of the performer:

• • distance between the hands;

• • physical position of the performer's hands relative to the screen (monitor);

• • direction of the movement radiating away and toward the body;

• • curvature, torsion of the movement (locus-curvature and magnitude); and

• • velocity and acceleration of the movement.

Tactile or haptic devices were abandoned in ENACTIV due to complexity related to their implementation by amateurs or children in practical performance situations. Also they might be too invasive for capturing unwitting gestures.

The indirect acquisition includes capturing the sounds that instrumentalists, singers, actors or the installation participants produce. The sound's frequency, amplitude and spectral envelope generated by utterances, prosodic properties, vocalisations, singing, mouth noises and playing an instrument are analysed and converted into data. Then the data coming from both the video and sound capturing device(s) is converted into continuous variables that are subjected to multi-layered non-linear mapping, resulting in real-time control parameters for sound synthesis and processing.

5.4. Sound capturing

The sound generated either by a musical instrument or a human voice – speech's prosodic properties (including the semantics), singing, or other less articulated sonic inflections – is picked up by a microphone and converted into frequency and amplitude data streams using the FTM Gabor object gbr.yin (IRCAM). Then the data is conveniently scaled and fed to the mapping algorithms.

5.5. Gestures capturing

The physical gestures are captured by an optical motion capture device – a camcorder, computer camera, or similar. The captured video is routed to a pair of objects, jit.qt.grab and jit.qt.movie, which allow for certain adjustments of the captured video such as the hue, luminosity, saturation and some others to be made before the gestures tracking begins.

The stream is then connected to two objects that analyse the input data and track the movement – Cyclops's jit.cyclops (Eric Singer Reference Singer2005) object, which tracks and reports the momentary position, and the jit.findbounds object, which tracks the dynamic properties of the tracked part within the window (see figure 8). There are two monitors and two jit.findbounds objects, allowing for two separate parts (colours) to be tracked at once. On the one hand, if the two jit.findbounds objects are connected to a single jit.qt.grab and jit.qt.movie pair then two different moving ‘objects’ or colours can be tracked at the same time. For example, in the case of a single person (a monologue or recitation) the movement of both performer's hands can be tracked independently. On the other hand, each of the jit.findbounds objects can be connected to a separate jit.qt.grab and jit.qt.movie pair, allowing for two video capturing devices to be used at the same time. In this case the cameras can be positioned under the angle of 90 degrees, which will allow the movement to be captured in 3D.

Figure 8

Movement tracking and the tracked region windows. The picture shows both the tracking window – the tip of the pen is being tracked – and the tracked region, showing the magnified tip of the pen in the small window.

5.6. Tracking

The user stands in front of the camera that captures the gestures. The position of the torso, the limbs and the head is checked against the predefined zones, and variables are generated each time a hand, the head or an object with a specified colour and luminosity appears or disappears in a zone or moves within a zone. Further, the direction, momentary position, angular displacement, velocity, acceleration and locus (angle and magnitude) of the tracked part's movement are calculated and conveniently scaled for further use. The motion capture set of objects jit.findbounds (Jitter's standard object) and Cyclops capture and route the variables to the mapping modules.

5.6.2. Distance

The distance is calculated as continuously streaming moving mean value calculated from the momentary mean value (between the maximum and minimum) coming out from the hundred blocks of the jit.cyclops object as an input vector. The entire calculation is done by the MnM's matrix-based vector-calculating set of objects. Mean or moving mean operates on a vector or element by element on a stream. The distance itself is calculated as a moving mean of the vector coming from the Cyclops blocks. The result translates into the following: the higher the moving mean value, the greater the momentary distance between the tracked objects (figure 9).

Figure 9

Calculating the position of the tracked objects (hands) in the predefined zones as well as the momentary distance between the tracked parts.

5.6.3. Direction, velocity, acceleration and locus

The direction, velocity, acceleration and locus (angle and magnitude) of the two tracked moving objects (hands) in the left and right windows independently are calculated from the data reported by the jit.findbounds object (figure 10). Direction for both tracking windows – the horizontal (0–320 pixels) and vertical (0–240 pixels) – are scaled from 0 to 127. The movement in a direction away from the resting position results in values increasing from 0 to 320 for the horizontal and 0 to 240 for the vertical movement. Both the velocity and acceleration are calculated using the ‘timer’ object, which reports the elapsed time between two events – number of pixels per millisecond. The locus or curvature of the movement is calculated using a simple patch and ‘poltocar’ and ‘cartopol’ objects doing polar to Cartesian (and the reverse) transformation of the data coming from the jit.findbounds object that reports the momentary position of the tracked objects (angle and magnitude).

Figure 10

Patches that calculate the direction, velocity, acceleration and locus of the moving object from the data acquired by the jit.findbounds object.

5.7. Mapping strategy

It has already been mentioned in the article that only very recently has attention been given to the process by which data from an input device has been associated with synthesis parameters for real-time control. Bevilaqua's idea is that the mapping system has to be an integral part of the compositional intentions, and there is a strong emphasis on the relation between the input data to the parameters for sound synthesis and processing as an integral part of the compositional process. Hunt and Wanderley (Hunt and Wanderley Reference Hunt and Wanderley2002) have shown that explicit and multi-layered mapping gives the best results.

ENACTIV's current mapping philosophy takes the following directions:

• • Translating gestural/kinetic and sonic patterns into other patterns by means of basin-of-attraction systems (strange attractors) to provide for a non-linear mapping, which in turn can be used for ‘translating’ natural movement. Inputs are sent through attractors’ coefficients, which in turn change the system's behaviour ranging between order and disorder. For example, the baker's map nature is characterised by a continuous evolution into bifurcations.

• • De-linearisation of IRCAM's MnM matrix-based many-to-many systems by applying the iterative equations producing strange attractors.

The mapping area consists of a collection of objects and relationships among these objects based on equations and algorithms that are used as drivers to the set of mapping strategies, which evolve into control parameters for audiovisual synthesis and processing. The defined parameters are arranged in specific cognitive tasks or class of tasks, described as patterns of organisation, and presented as ‘meaningful’ parameters. At the moment these are: potential energy, kinetic energy, syntax and character, defined through the movement's curvature.

The data is routed to algorithms based on both linear (MnM's matrix-based ‘many-to-many’ abstractions) and non-linear iterative mapping functions employing strange attractors’ equations, which in turn produce parameters for audiovisual synthesis and processing.

The general mapping structure is divided into pathways representing the user's choice, each with its own expressive level (mapping sensitivity) and a degree of randomicity. These pathways direct the cross-mapping between the video and audio data and define how the incoming values will be evaluated and executed. At the moment there are three mapping ‘pathways’ related to the sensitivity of the non-linear equations, to which a certain level of randomicity can be added. For example, the data obtained from aberrant vocalisations (known as slurring, snorts, etc.) or vocal expressions integrated into various communicative situations (oh, aaah, wee!), which contain predominantly inharmonic and noisy spectra can be routed to the mapping sensitivity level ‘1’ if the user wishes a ‘mild’ processing to be applied to the sound input. However, if a more complex, drastic and somewhat random processing is expected, the mapping sensitivity ‘2’ with higher value for randomicity can be used.

In the case of articulated sonic inflections that are inseparable parts of the speech (known as prosodic properties), such as rhythmic expressions (clapping hands, for example) and singing, the mapping sensitivity ‘3’ combined with a moderate randomicity will result in polyphonic or heterophonic chorus-like effects.

5.8. Mapping sequence

Experiences from direct one-to-one mapping – individual controls to individual synthesis parameters – have shown lack of expressivity, as noted in several papers (Rovan, Wanderley, Dubnov and Depalle Reference Rovan, Wanderley, Dubnov and Depalle1997; Garnett and Goudeseune Reference Garnett and Goudeseune1999). Similarly ‘divergent mapping’, where one gestural output is used to control more than one simultaneous synthesis parameter, and ‘convergent mapping’, where many gestures are coupled to produce one synthesis control parameter, have shown limitations in terms of accessing the internal micro-features of the sound object and are harder to master. Considering the shortcomings of the mapping strategies noted above, ENACTIV's mapping sequences, consisting of many-to-many multi-layered mapping algorithms (Wanderley Reference Wanderley2000), are the most flexible and promising way of relating the processing and synthesis parameters to the performer-generated control data and have great potential to ‘translate’ the performer's expression into an interesting, sonically rich and visually attractive output. ENACTIV's mapping sequence is shown in figures 11 and 12.

Figure 11

Multi-level mapping: the variables from the sound and gesture acquisition modules are routed to the performer-related parameters (potential energy, kinetic energy, syntax and character), then mapped onto the music-related or instrument-related parameters (range, pitch, amplitude, texture and inter-onset interval) and finally routed to the sound synthesis and processing algorithms.

Figure 12

ENACTIV's multi-layer mapping scheme.

This is the outline of the mapping scheme:

1. 1. Gathering data from a capturing device and forming variables:

   • • Sound: frequency, amplitude

   • • Movement: direction, velocity, acceleration and locus

2. 2. Converting these variables into ‘performer-meaningful’ parameters:

   • • potential energy

   • • kinetic energy

   • • syntax

   • • character (movement's curvature)

3. 3. Mapping the ‘performer-meaningful’ parameters into intermediate ‘sound synthesis’-relevant (or musical) parameters. At this stage several mapping modules can be employed simultaneously:

   • • range

   • • pitch

   • • amplitude

   • • texture

   • • inter-onset interval

4. 4. Mapping the intermediary parameters onto the specific sound synthesis and processing parameters.

5.9. Synthesis and processing

There are six ‘instruments’ – MaxMSP objects and abstractions – whose parameters are controlled via the mapping process:

• • ‘FOG’ granular/textural transformation of sound developed by Michael Clarke with the following parameters: transposition, speed, duration, density, left–right and front–back spatial movement (panning).

• • stharms1∼ from the ‘Jimmies’ library by Zack Settel. Its parameters are transposition, window, delay and feedback.

• • Bennies Library:

  • ○ Choros∼ – randomicity-controlled chorus granular synthesis

  • ○ vdb∼ – variable reverb object.

• • FTM's Gabor Library (granular synthesis on a real-time audio stream):

  • ○ Simple Granular Synthesis;

  • ○ Stretch, Stick and Stumble.

5.10. Spatialisation and output

ENACTIV's default spatialisation is based on a simple stereo panning using objects from MaxMSP/Jitter Tutorials.

One can easily connect IRCAM's SPAT for multi-channel spatialisation via the simple patch in Spatialisation_Ctrl_Data sub-patcher, which ‘translates’ the gestures tracked by the jit.findbounds object into trajectories driving the movement of the sound being sent out.