1. Introduction
The goal of the Musical Interface Technology Design Space, MITDS, is to expose a framework and approach for developing expressively powerful gestural interfaces for music performance and composition. We begin by proposing a set of guidelines for evaluating and iteratively developing interactivity in musical instruments. Complete musical instruments are not reached until physical interfaces and synthesis techniques are connected via mapping algorithms for realtime expressive control. Any new musical instrument must consider these three primary concerns in the MITDS: the human interface, the sound synthesis, and the mapping of data between these input/output systems. We uncover the issues that arise when designing, developing, and performing with digital musical instruments, and explore their impact on the MITDS.
Inspired by David A. Jaffe’s article ‘Ten Criteria for Evaluating Synthesis and Processing Techniques’ (Jaffe Reference Jaffe1995) a set of seven MITDS principles is revealed for developing expressively powerful interactive musical interfaces. We assume here that the purpose of a new musical instrument is that of performance and/or composition, though other contexts for their use such as personal enjoyment or music therapy are possible as well. Other researchers have explored different methods of evaluation, from a human–computer interaction perspective using tools from the HCI domain adapted for musical interfaces (Wanderley and Orio Reference Wanderley and Orio2002), in a comparison of instrument features such as expressivity, immersion and feedback (Piringer Reference Piringer2001), and by examining characteristics of interfaces with respect to their possible applications (Birnbaum, Fiebrink, Malloch and Wanderley Reference Birnbaum, Fiebrink, Malloch and Wanderley2005). The recommendations discussed here can be seen as a set of criteria for how the overall goal of the MITDS may ideally be met.
Elementary considerations such as the resolution of sensors and analogue-to-digital conversion, communication protocols and wireless capabilities have significant impacts on interaction, but we focus here predominantly on higher-level characteristics. Basic technical issues include things such as the sampling rate, latency and jitter of an interface, and protocols for data transmission, storage and retrieval such as open sound control (OSC),Footnote 1 the gesture description interchange format (GDIF)Footnote 2 or the gesture and motion signal format (GMS).Footnote 3 The higher-level characteristics we will examine include the gestures enabled by an interface and how they allow a performer to extend or enhance the playability of a musical interface. The author’s dissertation (Overholt Reference Overholt2007) further elucidates both the fundamental yet important technical issues and the higher-level attributes discussed here. The seven evaluation measures used in the MITDS are as follows.
1.1. How intuitive are the gestures?
When designing a new interface, one must decide what type of gestures should be captured; the answer can fall into two broad categories. While some musicians prefer using the gestures they have developed through years of practice on traditional instruments, others are interested in developing new gestures and techniques. In either category, the intuitiveness – a consideration of how ‘natural’ a gesture feels – is important when designing the human interface and sensor system. For more traditional musicians, sensors should be used to capture previously learned gestures with enough accuracy and precision to match the nuances they are accustomed to with a traditional instrument. In the case of a less traditional interface one can either invent entirely new gestural grammars and performance techniques, or capture gestures that are somehow similar or related to techniques with a traditional instrument. More imaginative techniques can be interesting, but an interface may become less optimal for performance if the causality relationship is broken. Deterministic behaviour is desirable in the MITDS – this provides intuitive learning and interaction. With non-traditional interfaces, musicians should expect to spend time learning a new set of gestures if the instrument is to have a long-term impact. It helps the process of learning if new gestures are put forth in the interface in a clear, consistent and intuitive manner.
1.2. How perceptible are the gestures?
Gestures should cause an understandable change in the resulting sound for a musician to grasp an instrument’s playability. A gesture that causes unpredictable results may be interesting at first, but can be extremely frustrating while performing in front of a live audience. Therefore, interactive musical instruments should have action–sound couplings clear enough to be perceived by the audience, while gestural actions should preserve some sense of complexity, uncertainty or mystery in order to maintain the interest of the audience. This may be accomplished in the design of the instrument itself and the mastery of its performance techniques, or inherently via a particular composition or improvisation. If a new interface looks like or is played like a traditional instrument, it can lead observers to anticipate certain types of sounds, breaking their expectations by surprising them with previously unheard sounds. Interfaces using completely non-traditional gestures have no accumulated common knowledge or reference as a key to comprehension for the audience; this puts the responsibility of helping an audience understand what is happening on the developer and performer of such instruments. Any interface in the MITDS should carefully consider the perceptibility of its gestures both to the performer and to the audience.
1.3. How physical/powerful are the gestures?
Making an obvious physical gesture should have a significant audible effect. Electronic technology allows even a tiny motion to have a huge outcome, but it can be important to consider the dramatic effect of a gesture in the design of a new instrument. A performance interface should attempt to provide a vehicle for expressive communication with an audience, so it can be wise to incorporate extra human effort into the interaction design for meaningful gestures. This creates an intrinsic relationship between the performer and the instrument as an extension of the body. This consideration of effort will have an impact on the music, lending it a ‘human feeling’, as more exertion is required for some musical ideas than others. Both coarse- and fine-grained sensitivity to gestures is crucial in the development of a truly expressive interface. If an interface incorporates elements of traditional instruments, then conventional techniques can be made more powerful by gesturally controlling digitally generated sounds and/or parameters of effects processing algorithms. Such augmented or hybrid instruments, including the author’s Overtone Violin (Overholt Reference Overholt2005) are powerful because they provide a way of enhancing traditional gestures, bridging the gap between the world of traditional instruments and interactive interfaces. See Movie example 1 for a video showing the author performing with the Overtone Violin.
1.4. How well-behaved is the controller and synthesis algorithm?
Audio synthesis, analysis and processing techniques such as phase vocoding, pitch-tracking and pitch-shifting should avoid algorithmic glitches, as poor implementations can produce undesirable audible artifacts. Such problems can also occur in mapping algorithms or sensor electronics. This does not imply, however, that one should ‘clean up’ sound analysis data or sensor inputs excessively, as this can remove nuances from a musician’s inputs. For example, traits such as ‘fuzzy’ note-starts and noisy indeterminate pitches can be problematic for pitch-tracking algorithms, but can be extremely expressive in the trained hands of an instrumentalist. Many of these problems are best avoided through the use of interfaces, synthesis/effects and mapping algorithms that employ continuously variable inputs rather than discrete quantisation of the gestures of a musician. This approach can yield well-behaved, repeatable performances that still allow inflective modulations within a note rather than limiting the potential expressivity to simple triggers (e.g., fixed sample playback). Taking these concerns into account when designing a new instrument will help to refine the system through iterative development cycles.
1.5. How unique is an instrument’s identity?
Successful instruments in the MITDS must withstand the test of time. While the seemingly limitless potential of standard computers as ‘musical instruments’ is evident in many ways, the generic nature of the interaction does not give a true identity to their musical use. On the other hand, when the interaction is combined with an embodied interface providing rich control inputs with well-designed mappings to a synthesis algorithm, one can produce systems with visceral ‘character’, resulting in distinctive action–sound couplings from instantaneous human inflections. The challenge of blending human skills and machine capabilities can result in expressiveness that can be near or even surpass that of traditional acoustic instruments. Many musicians would like to take advantage of sounds that have a unique identity, which encourages the development of synthesis techniques producing ‘striking’ resonances. In the MITDS, the appropriateness of a given interface depends on the musical task at hand, and the call for musical interface technologies to have a unique identity should not be confused with an unadulterated endorsement of personally idiosyncratic instruments. Indeed, for an instrument to take hold and develop a true performance practice (where the focus is on what humans are able to master with a certain instrument), gestural vocabularies must develop, and more than just one prototype interface must be available to performers.
1.6. How rich is the mapping methodology?
This evaluation measure concerns the controller inputs and synthesis parameters, and how they are mapped to musical attributes such as dynamics and articulation. Parameters of the signal-processing algorithm in use should not be treated simply as mathematical variables, as they have a strong correlation to the resulting perceptual musical experience. Mapping is dependent on sensor inputs and synthesis parameters because of the interconnected limiting and enabling factors arising from both domains. For example, an interface with five sensors cannot simultaneously and independently control a synthesis algorithm that expects ten input parameters, and a simple sample- and playback-based synthesiser will not respond in complex ways to multiple expressive sensor inputs. Every instrument in the MITDS uses some type of mapping methodology in order to connect performer inputs to sonic outputs, and there can be many different levels of richness and variety in treating the problem. A rich mapping methodology will lead to more expressive instruments, allowing higher levels of skill to be developed.
1.7. What is the widest range of expression?
Musical technologies in the MITDS should focus on deepening the sensitivity and expressivity within a few mappings and synthesis algorithms to the precision control that a virtuoso is capable of (rather than implementing too many of the possibilities in shallow ways). When extremely detailed inflections are obtainable, musicians are able to explore the nuances ‘inside’ a sound. Accomplished performers can portray a wide range of emotive meanings with very few synthesis mappings by making the most of the available sensitivity and dynamic range of a given physical interface, and using different gestural inflections in each performance. Physical interfaces in the MITDS should also incorporate high-resolution sensors and tactile feedback, allowing such meticulous performance techniques to be developed; this places the responsibility for expressivity on the human in live performance instruments.
The MITDS framework is a combination of the three areas of music performance, human–computer interaction and digital technology (figure 1). These are highly interrelated within the framework, due to the system-wide integration necessary for a viable musical instrument to be developed. The goal is to allow humans to be musically expressive through the use of advanced technologies. This can also be described as the emancipation of expressivity in computer music through the incorporation of multiple levels of human inflection. Our approach to the problem encompasses multimodal interaction design as matched to human perception. By looking at different design patterns that emerge from the MITDS, systems can be developed with attention to physical and cognitive affordances that connect multimodal sensor arrays to digital signal processing algorithms through interfaces and mappings that permit fluency of interaction and rich gestural expressivity. The remainder of this article examines some of the design patterns that can be used in the MITDS.
The Musical Interface Technology Design Space (* indicates the area represented) is a conceptual framework for the development of new musical instruments and their corresponding performance techniques, as informed by iterative and reflective analyses as well as inspiration from existing instruments via a taxonomy of musical design patterns regarding interfaces, mappings, synthesis algorithms and interaction techniques.
2. The balance of power between musician and machine
The two extreme ends of a scale weighing the balance between human control and machine control in a musical instrument are:
a. Algorithmic compositions (pre-arranged generative processes) with a performer’s input controlling only the start time and overall volume once at the beginning of a ‘performance’.
b. Improvisatory group performances with several musicians playing new electronic or electroacoustic instruments that allow many details of their sounds to be manipulated.
In example b), musical interface technologies can be used to gain better access to the inner workings of realtime signal processing algorithms. In this way, new musical interfaces can control a sound’s timbre with fewer restrictions. One example of this is the author’s MATRIX interface (Overholt Reference Overholt2000). Theoretically, any imaginable timbre can be rendered through digital sound-generating techniques such as additive or subtractive synthesis – interactive sounds can change abruptly or evolve continuously with human control input (e.g., Grey Reference Grey1975; Wessel Reference Wessel1979; Wessel, Wright and Schott Reference Wessel, Wright and Schott2002). The ambition of the MITDS is to simultaneously give the human performer as much expressive control over a sound as they can physically and cognitively master, while simultaneously utilising the machine’s capacity for realtime sound generation and processing to as full an extent as possible.
The middle-regions shown in figure 2 give composers and performers powerful and exciting capabilities never before possible, but also lead to many aesthetic and philosophical questions of what levels of control should be utilised within artistic performances. The success of a performance (surprising, engaging, stimulating, etc.) using these multiple levels of control is clearly an individual, subjective judgement, but certain limits are determined by the characteristics and processing power of human perceptual and cognitive systems, as well as cultural influences. Reliance on a computer’s processor/memory to enhance a human performer’s brain/memory is a double-edged sword; it reduces the requirements for explicit human control input, thus allowing possibilities for higher-level control over musical processes (such as mixing, controlling sonic density rather than individual notes), while concurrently hindering realtime expressivity at the level of subtle inflections and nuance.
The balance of power between human and machine – the MITDS is concerned with the augmentation of human capabilities through the incorporation of high-, mid- and low-level controls.
Given the continuum of possibilities for a system’s balance of control between musician and machine, a primary musical question that arises is the level of determinism desired in a performance. If a composer prefers a largely deterministic work, allowing only small changes in the outcome from one performance to the next, then an interface focused on high-level musical attributes such as tempo, volume levels, or spatialisation may be appropriate. Control of more detailed musical events, however, requires a performer to use an interface suitable for discrete elements within a phrase, such as musical notes, timbre, and even finer-grained musical attributes. Experienced musicians tend to prefer these types of inflective interfaces even though they can be difficult to master, because they allow for the control of sonic details in a more fluid manner than higher-level interfaces.
A good musical interface should not be limited to either high- or low-level control paradigms if it is to be useful beyond a small repertoire; with suitable design and mappings, there is no reason that a fine-grained interface cannot be used to control both high-level and low-level musical attributes. Control over high-level attributes is almost always needed in a performance (excitation, dynamics and pitch), and can be mandatory in certain instances (e.g., controlling the rate of a stream of sonic particles in granular synthesis). The MITDS advocates provisions for multiple levels of control, incorporating high-, mid- and low-level mappings within a single musical interface. This can be accommodated by capturing sufficient multimodal gestural input to manipulate the multidimensional parameter spaces of electronic music performance at a wide range of levels. Clearly, one of the most important issues in the MITDS is the placement of responsibility for musical details executed by the machine (semi-automated) versus contingent upon human inputs.
3. Experimental performance – the evolution of performance using technology
It is significant to note that computers have turned the initial dichotomy between recordings (fully automated) and traditional musical instruments (fully human-controlled) into a continuous scale, and that new forms of performance have been emerging from the midpoints of this continuum. Viewed from a Darwinian perspective, these new performances are essentially ‘speciations’ of performance techniques that may either be viable, or headed towards extinction (figure 3). Viable musical instruments will undergo a period of thriving within a culture. Let us speculate here (without making any aesthetic judgements) that live coding performances (Collins Reference Collins2003) may be an attempt to recoup some of the instantaneous expressiveness inherent in traditional instruments (in addition to simply exploring new ideas for performance), and that circuit bending (Nies Reference Nies1999; Ghazala Reference Ghazala2005) may have at its core a desire to recapture the subtleties and nuances lost in digital synthesisers (but inherent in traditional instruments) – aspects so important to emotional expressivity. Whether or not live coding and circuit bending end up being ‘viable species’ of performance techniques in the long term, they are both near the midpoint of the MITDS continuum between recordings and live performances, making them interesting experiments in new genres of machine-enhanced human expression.
The Darwinian perspective can be used as an analogy describing the evolution of new musical instruments.
The combination of human skills with digital algorithms was made possible only in fairly recent times, and as such is still being explored. While some of the current performance trends may or may not seem entirely convincing from an evolutionary perspective, interactive performance does have vast potential if the strengths of humans and machines are suitably leveraged within each musical context. One of the primary factors in the evolution of acoustic instruments promoted instruments that were better at expressing human emotion, and it seems quite likely that this factor will continue to push the natural selection of future musical instruments.
There are instrument sounds which are often used to portray particular emotional states.Footnote 4 For instance smooth, quiet tones for tranquility, loud, rich tones for boldness. A biting violin onset sound suggests determination, a wide unsteady vibrato, intensity and so on. All these effects are natural to the instrument and not contrived by the player. It is as if these are emotional states of the instrument itself. This can be explained by an evolutionary process in instrument design, with positive selection for instruments which can be used to better express emotional states. (Menzies Reference Menzies1998: 52)
4. Game design – similarities and differences with musical interaction design
Interactivity in musical instruments has some parallels to game design theory (Crawford Reference Crawford2002), but goes well beyond simple musical games like ‘Guitar Hero’ or ‘Dance Dance Revolution’. Nonetheless, game-design theory can lend some insights into the development of interactive musical interfaces in several significant ways. First, games that are too easy to win are only fun for a short period of time, so both positive and negative outcomes are needed. Likewise, good instruments should be able to make bad sounds. It is important to design an instrument that is not too effortless to play (nor too difficult), or it may end up being more like an interactive installation (which are more than likely meant to be enjoyed in a single visit) rather than a truly expressive musical instrument. If the musician does not find an instrument or mapping rewarding enough, it is not conducive to achieving a deep level of control. On the other hand, if the mapping encourages a performer to spend a significant amount of time exploring the space of possibilities and learning its capabilities (Oore Reference Oore2005), they can eventually develop techniques for controlling the multidimensional parameter spaces of music, with the end result reflecting this in the quality and virtuosity of the performance.
Counter to game design methodology, however, it is not productive to simply ‘add a new level’ in order to increase the difficulty level of a musical instrument. Instead, musical interface designers should make every effort to find interfaces and sound-generating methods that by their nature have so much richness of interaction that the space can never truly be fully mastered, in essence leading to a never-ending game of musical practice. If this is done well, a musician can spend years working with the same instrument (just as with acoustic instruments), exploring a wide range of musical ideas, and developing new skills and new mappings with the interface. As philosopher James P. Carse notes, there are two basic types of games: finite and infinite. A finite game is played with the purpose of winning (thus ending the game), while an infinite game is played with the purpose of continuing the play (Carse Reference Carse1986). In the MITDS, the aim is to build interfaces that have the potential to provide musicians with infinite play. If an interactive musical instrument has only 100 possible states (or even 100,000), the musician will eventually realise that they have exhausted all possibilities the game has to offer, and give up on the instrument.
5. Language and Musical expressivity
To understand human musical expressivity, we look at traditional musical instruments to try and comprehend what makes them compelling enough for humans to spend years mastering the skills needed to go beyond mundane beginning steps – to truly explore the possibilities they offer, and discover personal expressivity through them. Some have called this process to develop a voice or to speak through an instrument, both of which are analogies for expressive communication, which is what talented performers have developed so highly. While we all learn to communicate through the use of words, many argue that being musically expressive with our vocal chords came before the development of formal language in human pre-verbal utterances; the voice is unarguably our most intimate musical instrument.Footnote 5 In The Singing Neanderthals: The Origins of Music, Language, Mind and Body (Reference Mithen2006) Mithen posits that ‘musicality is a fundamental part of being human, that this capacity is of great antiquity, and that a holistic protolanguage of musical emotive expression predates language and was an essential precursor to it’.
Language can in fact be seen as a way of simplifying, standardising and giving meaning to our vocal sounds (Subotnick Reference Subotnick2007), when considering the wide range of expressions the voice offers as a communicative interface. It seems that we have always entwined spoken language with musical (in a broad sense) intonations and inflections in order to give our words a unique sentiment each and every time we say them, thereby bringing extra meaningfulness to our communication. We know from experience that it is easier to understand implied meaning when listening to someone talk than it is to ‘read between the lines’ when browsing a manuscript, due to the modulations that are present in speech. Sarcasm is one obvious example. Although Mithen discusses these ideas mainly as they relate to the earliest days of humanity (when homonid vocal calls were non-symbolic and non-representative), such things are still ubiquitous in our language today. The inflections heard in our words gives them implicit meanings, beyond the simple semantics. There are still many instances of vocalisations that are more explicitly musical than linguistic (one example being infant-directed speech). The relationship between music and language is further discussed by Dobrian (Reference Dobrian1992), including how it relates to semiotics, information theory, generative grammar, and modern music theory.
The MITDS contends that this capacity for sonic inflections in a manner similar to our vocalisations is a crucial component of any musical instrument. An interface should provide the means to impart delicate nuances or rough emphasis while expressing a musical idea to an audience, and to do it instantaneously as we do while speaking, since the nature of time provides us only with fleeting moments to make the most of during a performance. Philosophically, this leads to an appreciation of how musical inflections are an important part of human life (both inside and outside of an overtly musical context), and also posits the tremendous importance of music in our lives.
6. Influences of ergonomics and mappings
There is no question that instrument design has had a huge influence on the history of music, even though this is sometimes overlooked in the history books. Meaningful performances depend on the capabilities of an instrument to impart musical ideas, and convincingly reveal emotive themes to an audience. In addition, simple physical constraints have had lasting effects on music and harmony. For example, playing the same piece on the piano in B-flat and in F-sharp feels completely different under the hand due to the key layout (note that isomorphic keyboard layouts do not pose this hurdle). Different instruments allow different methods of expressivity, and a successful phrase on one instrument can rarely be played on a different instrument while accurately retaining the same musical sense. This is not a negative aspect, rather it gives each instrument its own natural character. Thus, a pianist may accentuate a certain note by playing it with more force and elongating the rhythmic pattern around it, while a saxophone player might make a note ‘growl’ by humming through the embouchure at the same time as playing.
These considerations are related to the ergonomics of an interface and its mapping to musical output, both of which are important fulcrums within the MITDS. However, the customary approach to ergonomics (which specifies ease of use as being of prime importance) is not directly applicable to all aspects of musical interfaces. In the MITDS, one must also consider the role effort plays in the development of meaningful performance interfaces. The relationship between our bodies and musical instruments has always been at the centre of performance and compositional activities. A crucial capability of any musical instrument is therefore the means to capture the inflections of a performer’s physical articulations to impart a powerful and moving experience; musical interfaces should have enough dynamic range to reveal not only small nuances in human gestures, but also more emphatic efforts portraying emotional intensity and calling for empathy from the audience.
Human expectations about an instrument’s response (excitation) to an action (gesture) and the resulting sound (resonance) also play a large role in the MITDS, including the motor-mimetic cognition of the performer (Godøy Reference Godøy2001), and the corresponding perception of the audience. The coordination of many muscular actions is required in playing an instrument. Fingers can be controlled through small distances well, and are particularly sensitive to small contact forces. Arms and other limbs require larger movements in order to achieve a similar degree of control resolution (such as bowing motions), and their free dynamics are felt by the musician in the proprioceptive tensions and inertial stresses in the joints and tendons.
7. Categorisation of gestures as inputs to mapping systems
There are several types of gestural inputs we can use in the MITDS, and many different kinds of musical outputs we can map them to. Cadoz (Reference Cadoz1994) broadly categorises the various functions of hand gestures into three areas:
a. Semiotic gestures: those used to communicate meaningful information (such as ‘thumbs up’).
b. Ergotic gestures: those used to manipulate physical objects.
c. Epistemic gestures: exploratory movements to acquire haptic or tactile information.
Ergotic gestures are the most useful in the MITDS, as their functions are those used to control a musical interface. Epistemic gestures will mostly be used while learning a new interface, and are always necessary at first. Semiotic gestures can also be used within the MITDS through techniques such as computer vision and gesture recognition. This can be seen in one example of the author’s work with the Multimodal Music Stand (Overholt, Bell, Kleban, Putnam, Thompson and Kuchera-Morin Reference Overholt, Bell, Kleban, Putnam, Thompson and Kuchera-Morin2007). A musician performing with the Multimodal Music Stand can semiotically cue the software mapping system to trigger a new section of a composition through the multimodal combination of computer vision, audio analysis, and electric-field sensor inputs.
Within the ergotic gesture category, Cadoz (Reference Cadoz1988) provides us with three basic functions that are commonly used in traditional musical instruments, controlling elements such as pitch, dynamics and timbre. These are:
a. selection (picking from a range of discrete values);
b. excitation (putting energy into the system); and
c. modulation (shaping a control parameter).
In the MITDS, these can be seen as elements of a rudimentary design pattern for gestural inputs, which become more complex when incorporated into interactive interfaces through physical sensors and software mappings that graft multimodal sensor arrays into a combinatorial unified system for a particular instrument. When compared to the world of traditional acoustic instruments, however, it is clear that extrapolating such design patterns from the categorisation of gestures is vastly oversimplified. For example, many actions incur more than one type of output in traditional instruments (excitation also involves modulation, etc.). Acoustic instruments can be excited in four possible ways: they can be blown, struck, plucked or rubbed (without even considering nuances such as scraping, shaking, sliding, ruffling, crunching, caressing, etc.) (Pressing Reference Pressing1990), and all of these gestures can affect sonic outputs in multiple dimensions.
An acoustic instrument’s range of musical output can be summarised at a first approximation by the three perceptual areas of pitch, amplitude dynamics, and timbre (again, oversimplified for many instruments). The variety with which these attributes are controllable in traditional instruments ranges from continuous glissandos across an instrument’s full range of pitches (slide whistle) to discrete note selection (piano), from attack-only amplitude control (percussion) to continuously variable dynamics during a note (wind and bowed string instruments). Connections need to be made between the various excitations, selections and modulations to the domains of pitch, amplitude and timbre for new instruments in the MITDS, as well as the lower-level sub-note-level inflections, and higher-level decisions contemplating the possibilities offered by control over procedural and algorithmic sound-pattern generation.
8. Approaches to developing mapping systems
Mapping systems can include many different types of connections between gestural inputs and synthesis parameters – one-to-one, one-to-many, many-to-one, many-to-many, few-to-many, many-to-few, and so on – as well as modal mappings, in which the user can select alternate functionality of some controls (via a modifier key, for example). Although there is no hard and fast rule, many mappings tend to allow players to have simultaneous control over several parameters of the sound with a single gesture (one-to-many or few-to-many). One reason that these types of mappings tend to be more common is that they are consistent with the way traditional musical instruments behave, and thus with our pre-conditioned psychological expectations. Also, because timbre space is multidimensional (Grey Reference Grey1975) yet we perceive these correlated dimensions as a whole, it feels intuitive to control multiple musical dimensions with one motion. This is, after all, what we do with acoustic musical instruments (Wessel Reference Wessel1979).
Regarding mapping in hardware, it is clearly optimal to utilise sensors that correlate directly to human response curves. This is generally logarithmic, as it is related to our perception of properties such as frequencies and loudness. Also, the use of dedicated rather than re-usable electronics will help increase a system’s overall stability in the long term. For a performer the stability of both hardware and software mapping is important, as these need to be learned and practised without an instrument’s functionality changing radically under their hands. The MITDS makes use of the CREATE USB Interface (Overholt Reference Overholt2006) (or similar devices) in order to implement this approach, as it is both small enough and inexpensive enough to put inside various custom controllers. Certain mappings can be contained in the firmware of the device, and the use of standard communication protocols should extend the useful lifetime of the instruments. Whether implemented in firmware or host software, a fundamental set of mapping operations can be described as follows:
Shaping response controller data can be shifted or inverted (addition), compressed and expanded (multiplication), limited, segmented or quantized (thresholding). Methods which keep track of the history of a signal allow measurement of rates of change, smoothing and other types of filtering to amplify specific features in the signal or to add delays and hysteresis in the response (differencing, integration, convolution). The rates of data transmitted can be reduced and expanded (decimation and interpolation). Linear and nonlinear transforms allow the shaping or distortion of the signals to any desired response (functional and arbitrary mappings). (Ryan Reference Ryan1991: 8)
9. Transparency of mappings
A performance interface should attempt to provide a vehicle for expressive communication with the audience. As it is stage-based, it should take into account both audible and visual effects of the gestures it utilises. Transparency of mappings is an important consideration in the MITDS, wherein the roles of the performer and the computer should be clearly defined, and the actions of the performer should have clear consequences in order for the interaction to be perceived and understood by the audience. However, the range of the performer’s actions should not be overly simple or predictable, since if the interaction process is too obvious the audience is not given any sense of complexity, uncertainty or ‘marvel’ through the music or the performer’s skills, and the piece is in danger of becoming a simple technology demonstration.
Because musical interface technology allows isometric sensors to be used for control inputs, there is some debate as to whether new interfaces should purposefully make use of visually appreciable gestures (stage presence), and what relevance this has to the music. Many would say that watching a musician interact with his or her instrument adds to the expressiveness of the performance, providing an important element of the music. Others argue that the sound of music by itself is inherently expressive, and need not rely on any external visual stimulation. With careful consideration, there are decisions that can be made by the designer of new musical interface technologies that provide a good level of audience understanding. The development of new performance practices that provide the desired level of transparency of the mappings is placed on the shoulders of the inventor/composer/performer with entirely new sensor-based instruments in the MITDS.
10. Complexity of the mappings
The musical responsibilities that must be managed by a performer include those of a physical and cognitive nature (Fitts Reference Fitts1954; Cook Reference Cook2001) as required by the interface, its mapping, and the music to be performed. A developer working in the MITDS must take all of these things into account. With traditional instruments, polyphonic musical lines are more difficult to play than single pitch melodies or rhythms, and the cognitive and physical limitations of humans can be reached easily in the simultaneous control of many expressive elements. Indeed, playing the violin can be somewhat akin to ambidextrous drawing, with each hand simultaneously performing a complex task. Many interfaces such as wind or bowed string instruments involve two interdependent motor skills; one for precisely timed musical events and another allowing subtle continuous control. With the clarinet, for example, the player moves the fingers and tongue to change notes, and modifies his or her embouchure and breath to change tone and dynamics continuously. This division of efforts is generalised in a description of bimanual actions in the kinematic-chain model (Guiard Reference Guiard1987), and has been explored explicitly in new musical interfaces by Kessous and Arfib (Reference Kessous and Arfib2003).
The attempt to approach and even surpass the expressive capabilities of traditional instruments is what we are primarily striving for in the MITDS. This may necessitate the migration of certain low-level controls up to higher-level ‘set and momentarily forget’ mappings that reduce physical and cognitive loads until the parameter needs updating. It also requires the development of new performance skills and techniques through many hours of practice; successful new instruments will need to incorporate both of these elements – advanced multimodal musical interface technologies and mappings, along with advanced human proficiency and talent.
11. Human-centred interface design in the MITDS
Along with our physical limitations, human cognitive limitations such as speed of thought and the number of elements we can hold in our brains at one time can be mitigated to a certain degree through both practice and the integration of interactive digital technology – thereby partially circumventing the natural restrictions placed on our performance abilities. By allowing our subconscious and/or an interactive musical instrument to handle skills learned through discipline and aspiration, we can devote our conscious attention to more elaborate mind–body fluencies, potentially reaching another whole level of expressivity. When such a performance goes well, the audience will always realise that something extraordinary is happening, even if they do not fully understand what exactly it is. It is the author’s belief that striving to develop such capabilities will always be important, because social and personal expression are vital elements of human life. This conviction is also influenced by personal experiences showing the ‘magic’ that can happen in such moments; not a ‘how did they do that?’ type of magic, but a deeper feeling that somehow expresses real sentiments, dreams and desires through a musical performance. In order to develop personal expression, we must build up a relationship with an instrument, eventually mastering it and finding a rewarding interaction akin to what some have called a musical Zen state (Nachmanovitch Reference Nachmanovitch1990). If an instrument is to become a communicative vehicle for musicians, it must allow us to attain enough fluency to be able to transcend the instrument and connect with other humans through the music, and it is both a technical and artistic challenge to create systems that provide enough power and nuance to capture the passion, intuition, and joy of music that is the inheritance of humankind.
12. Conclusion
This article has put forth some of the analysis guidelines, principles and design patterns in a framework for the development of new interactive musical instruments called the Musical Interface Technology Design Space. The field of interactivity in musical instruments is a complex, evolving and exciting research area, still in its infancy, with much work yet to be done. The practical approach of the author’s research is exposed in some of the examples given, indicating possible directions for future research in the field. The long-term process of conceiving, designing, performing with, listening to, evaluating, and iteratively developing new and evolving musical instruments is merely outlined in the MITDS framework described here. The author looks forward to seeing what comes in the future, as well as helping shape the course of interactivity in musical instruments through the worldwide community of research, development, composition and performance.
