1. Introduction
In this article we discuss recent developments of actuated musical instruments and the implications they may hold for the performative ecosystem surrounding the instruments themselves, the performer and the environment in which they are used. Figure 1 represents some of the authors’ developments, in which electronic signal processing techniques are applied to actuators physically embedded into the instruments themselves. For example, instead of using audio compression algorithms to simulate the prolonged sustain of a plucked string, actuation of the strings and/or body of the instrument enables physically palpable real-world behaviours (e.g., with strings that are magnetically actuated even infinite sustain is possible). All of these instruments are unique – each is imbued with its own form of actuation. However, due to their tight integration of human (gestural) and electronic (transducer) actuation, we feel that these instruments all share the need for an ecosystemic approach to performance technique and interaction.
Clockwise from top left: the Electromagnetically Prepared Piano, the Haptic Drum, the Feedback Resonance Guitar, a teleoperated acoustic shaker held by the Falcon force-feedback device, and the Overtone Fiddle.
Actuated instruments invite and even require the performer to engage with programmed models of musical interaction that involve one or more layers of interactive feedback systems. We seek to incorporate actuation into new musical instrument designs in order to explore alternative methods of leveraging a performer's (slower) consciously controlled gestures and (faster) pre-learned gestures, by sharing some of the control with the machine while still providing proactive and intimate human control of the resulting musical sounds. In our experience, effectively employed actuation can
• free up some human cognitive bandwidth in order to promote concentration on other facets of musical endeavour;
• promote combined human and technological capabilities that can enable and even compel the performer to interact in new ways via systemic feedback; for example, an actuated musical instrument can enable a performer to make gestures that would otherwise be difficult or impossible;
• allow the performer to physically attend to other aspects of playing; instead of always having to inject energy into an instrument, the performer can steer external sources of (usually) electrical energy by controlling when and how they are applied;
• combine musical gestures from multiple musical performers into a networked ecosystem of co-controlled instruments, allowing performers to directly control or affect the behaviours and capabilities of other performers; and
• endow a physical instrument with virtual qualities that are adjustable by computer in real-time but which are nevertheless tangible, facilitating more intuitive and possibly even intimate interactions.
Moreover, there exists a continuum of possible instrument designs, between real (non-augmented, always requiring human energy input) and virtual (augmented with external energy sources, possibly even fully automated) instruments. We define actuated musical instruments as those which produce sound via vibrating element(s) that are co-manipulated by humans and electromechanical systems. These instruments include the capability for control to be exerted by a simple or complex system of external agency in addition to existent methods of control through traditional instrumental performance technique with the instrument.
1.1. Model: performance ecosystems
Musicians have been playing musical instruments for thousands of years. In most live musical performance paradigms, a musician employs the natural feedback provided by an instrument to help him or her control a performance. For example, consider the model shown in Figure 2 describing a musician playing a musical instrument. The model is adapted from Bill Verplank's ‘Interaction Design Sketchbook’ (Verplank Reference Verplank2003), and we assume here that the musician uses his or her hand to provide mechanical excitation to the instrument. We acknowledge that a wide range of other excitations and/or modifications could of course be provided using the mouth, the remainder of the body, and so on, but we do not explicitly depict them here for the sake of brevity.
Musician interacting with a musical instrument providing auditory, visual and haptic force feedback.
While it is clear that musicians rely heavily on auditory feedback for ecosystemic performance, as discussed by Di Scipio (Di Scipio Reference Di Scipio2003), visual and haptic feedback also play an important role. For example, musicians can employ visual and haptic feedback to help orient their hands in relation to an interface. Performers may wish to use their sense of vision for looking at a score or looking at the audience; in contrast, haptic feedback provides localised information at any point on the human body. Furthermore, in comparison with other feedback modalities, haptic feedback enables the performer to respond the fastest to the behaviour of the musical instrument. For this reason, we believe that haptic feedback is essential for a performer who wishes to quickly and intimately interact with an ecosystemic musical instrument.
Without yet considering environmental or audience intervention, Figure 2 attempts to represent all the necessary parts to describe a musician's performance ecosystem (Berdahl Reference Berdahl2009: 2). We include the brain as the primary link in the composite feedback system where the visual, auditory and haptic feedback modalities are combined. The brain and the hand are internally interconnected by the human motor-system control loop. In order to understand the complex behaviours enabled by the entire system, all of the parts must be considered. In this sense, the model is an ecological model. In a well-designed system, it can be shown that the whole can become more than the sum of its parts, leading to emergent behaviours. However, computer musicians are well aware that achieving this wholeness can be challenging (Waters Reference Waters2007: 1–20). Nevertheless, based on our experience we believe that actuated musical systems can exhibit emergence if they leverage the concepts of tangibility, energy steering, enabling new kinds of performance interactions and freeing up cognitive bandwidth as outlined in Section 1. In addition to these discoveries, we hypothesise that composers can obtain a whole that is larger than the sum of the parts either by trial and error, or by considering which elements of the system the performer controls consciously or unconsciously.
Moving beyond traditional musical performance ecologies, wherein performers interact with one another through anticipated or improvised musical gesture, there exists a novel ecosystemic model for musical performance with actuated instruments in which physical control of musical instruments can be shared between musical performers, based upon intentional or derived musical gesture or analysis. Actuated instruments capable of external physical excitation allow musicians to interact at an entirely new level, both with one another as well as with rule-based or reactive computer systems. By linking musical instruments to one another, routing musical output from one as a control signal for another, new musical patterns and performance behaviours can emerge. As bi-directional enactive connections are created between performers, ensembles of actuated instruments truly become active musical ecosystems in and of themselves. In this manner, reactive instruments form the basis for a complex control schema, directly enabling co-control and shared haptic interaction between musicians and instruments themselves.
Sayer introduces the concept of a continuum describing how a performer interacts with a musical feedback system while improvising (Sayer Reference Sayer2007: 1–5). He argues that the continuum spans between extremes of reflex-based and volitional control. However, we prefer to expand this by describing the continuum as spanning between conscious and deliberate control by the performer, and involuntary control, where the performer's response is dominated by motor control system reflexes, muscle stiffness, body mass, and so forth. Based on knowledge of the human motor control system, we can argue that the performer should be able to consciously control events happening sufficiently slowly, whereas the musical instrument could evade deliberate control of the performer if it takes unexpected actions faster than the conscious reaction time of the human motor control system, which is about 120–180 ms (Berdahl Reference Berdahl2009: 147–9). In other words, if an actuated musical instrument reacts irregularly within this initial time window, it will surprise the performer and be difficult to control, although this could in fact be desirable in some circumstances! Programmed instrument behaviours lasting longer than this time window could theoretically be compensated for by the performer, if he or she is able to act quickly and decisively enough to master the haptic force feedback. The gestures produced by the total ecosystem can be remarkably complex, and the composer should consider the question: who is really in control anyway?
1.3. Background: prior research in actuated musical instruments
1.3.1. Feedback damping of acoustic musical instruments
Force feedback can be applied to an acoustic musical instrument in such a rigorous way that it is possible to actively damp acoustical vibrations. Early work in Paris provided some insight into this area. Laboratory experiments were performed on feedback control emulation of a virtual bow (Weinreich and Caussé Reference Weinreich and Caussé1986), feedback control of the end of a column of air (Gùerard and Boutillon Reference Gùerard and Boutillon1997), and feedback control emulation of virtual springs, virtual dampers and virtual masses (Besnainou Reference Besnainou1999). More recent work has been carried out in Paris on controlling a xylophone bar in a laboratory (Boutin and Besnainou Reference Boutin and Besnainou2008). From these experiments and others, it has become clear that it is especially challenging to damp vibrations or cancel sound. However, damping by feedback control can be achieved by delivering feedback as fast as possible, employing sensors and actuators that are collocated (i.e. physically matched and placed at the same point) and programming the feedback controller to behave physically like a viscous damper (Berdahl Reference Berdahl2009: 30, 27–9 and 34–71). In some previous experiments, piezoelectric actuators were employed, which required dangerously large voltage levels for sufficient actuation. While electromagnetic actuation can solve the safety problem, these challenges combined explain why there has been no long history of performance practice with acoustic musical instruments subject to feedback damping.
1.3.2. Inducing self-sustaining oscillations by feedback control
In contrast, inducing self-sustaining oscillations by feedback control is technically much easier – typically any kind of acoustic feedback with an acoustic delay causing the actuation and sensing signals to be out of phase results in self-oscillation for sufficiently large loop gains. This is the kind of feedback that most performers are familiar with as it causes the howling or squealing effect that affects the performance of public address systems and electric guitars. In addition to its ease of implementation, performers are often excited by the idea of self-oscillation in that musical instruments can be caused to vibrate by themselves or exhibit infinite sustain. In this case, performers no longer need to supply the energy to make the instrument produce sound, rather they can steer energy from external sources. As a consequence of all of these facts, there is a performance history of this kind of feedback control, albeit non-subtle in nature. Undoubtedly it will not be possible to note all prior work, as there are now even commercial products for sustaining strings. The E-Bow is a small, portable and electromagnetic virtual bow that can be coupled to a real vibrating guitar string when held directly over the string (Heet Reference Heet1978), and the Sustainiac is a virtual bow that is mounted in place of one of the pickups of an electric guitar and constantly electromagnetically bows the strings to set them into self-oscillation unless damped by the hand of the musician (Hoover Reference Hoover2000). Finally, the Moog Guitar,Footnote 1 released in 2008, incorporates a magnetic feedback system to sustain or damp the strings while playing.
Here we describe some of the notable more academically oriented compositions involving feedback self-oscillations. As early as 1974, Nic Collins created feedback loops in a more generalised ‘musical instrument’ using a microphone and a loudspeaker in a room. He writes of Pea Soup that ‘a self-stabilizing network of phase shifters nudges the pitch of audio feedback to a different resonant frequency every time feedback starts to build, replacing the familiar shriek with unstable patterns of hollow tones – a site-specific raga reflecting the acoustical personality of the room’ (Collins Reference Collins2004). Also employing complicated acoustic feedback, Agostino Di Scipio regulated the gain of a room acoustic feedback system in the Feedback Study portion of Audible Ecosystemics (Anderson Reference Anderson2005), and Steve Reich's Pendulum Music was a compelling example of the use of acoustic feedback in performance, where metrically phasing tones were created through swinging microphones hung above loudspeakers (Reich Reference Reich1974: 12–13).
More recently, the elaborate Magnetic Resonator Piano allowed up to fifteen strings to be actuated simultaneously. Although only one sensor was employed, banks of bandpass filters allowed for multiple sensing signals to be estimated from the single sensor measurement. Consequently, fifteen strings could be electromagnetically sustained somewhat independently of one another. The pieces Secrets of Antikythera for a single pianist and d'Amore, in which piano string vibrations were sustained to create a ‘harmonic glow’ surrounding the pitches of a viola solo, were performed using the Magnetic Resonator Piano in 2009 (McPherson and Kim Reference McPherson and Kim2010). These works helped extend previous work by Per Bloland for the more portable Electromagnetically Prepared Piano (Berdahl et al. Reference Berdahl, Backer and Smith2005; Bloland Reference Bloland2007). Indeed, piano design has evolved over centuries into an instrument that many people believe sounds good, and we believe that sound synthesised by electromagnetically actuated piano strings consequently also tends to sound good. In addition, the strings’ resonant character promotes new performer interactions as any energy injected into the strings can require multiple seconds to decay back to silence.
Feedback systems for inducing self-sustaining oscillations can either be relatively simple or remarkably complex. As with formally chaotic systems, small changes in the parameters can result in large changes in the sound. Work with more complex acoustic feedback systems includes Simon Waters’ concert flute modified for feedback incorporating DSP, Stef Edwards’ Radio Pieces, which involves feedback to a live radio show through listeners’ telephones placed near their radios, and John Bowers’ contributions to drum, long string, electric guitar and room-based feedback instruments, which are all described by Waters (Waters Reference Waters2007: 1–20).
1.4. Postulations on performative ecosystems
Symbiosism (van Driem Reference van Driem2004) is a theory that looks at language as an organism residing in the human brain – language is then recognised as a memetic entity. As humans, we form and propagate language through our speech and writing, while language itself furnishes the conceptual universe that informs and shapes our thinking. Given that many aspects of musical expression can be viewed in this same manner, it is interesting to view the characteristics of music itself, along with the tools we use to make music (instruments), as being in a similar symbiotic relationship. This parallel can be useful in a conceptual exploration of how every musical instrument – from the stone age drum to the laptop computer – is a piece of technology that simultaneously drives and limits the creative process. Feedback systems have thus always played a role at many different levels and time spans, and music performance has adapted to dynamic cultural environments, showing emergent qualities stemming from the interpenetration of performer's ideas, tools, audiences and environments – the entire ecology of musical creation. Actuated musical instruments bring to the table programmable physical behaviours and direct or immediate tactile feedback from things that were previously virtual-only musical interactions.
Music lives through our minds and bodies, and is affected by the technologies we use to make and share it, as well as the environments in which we play it. Hence, it is subject to the strengths and weaknesses of any technologies in use, including sound-producing mechanisms of all types, notation and compositional systems, musical software algorithms, the physical interfaces and the real or virtual elements of electronically enhanced instruments. It is the intermingling of all of these that are used to create music within a performative ecosystem. While some may conjecture that digital technologies today provide such flexibility that there is no further need for advancements in traditional or physical musical-instrument designs, it would surely be unwise to let digital technologies be the only focus of future developments. They certainly have a strong appeal due to their abstract nature and programmability, but other technologies (electromechanical, materials, acoustics, haptics, etc.) undeniably offer immediate or tangible benefits that computers alone cannot. The combination of electronic and acoustic elements in a single instrument, some of which are exemplified in the authors’ work here, show the field rich with possibilities. As it is not yet fully explored, the area is still ripe for exploration, and it is this motivation that has led to the examples of new actuated musical instruments seen in the remainder of this article – all of which allow gestural control over dynamic performance ecologies. This is not to say that the entire musical performance ecosystem (the interpenetration of a musician's ideas, tools, audiences and environments) can be embodied in any single instrument, but, rather, that the technology-enhanced instruments discussed herein have built-in performative feedback loops that we hope may encourage musicians to engage with an ecological approach to music-making.
2. Actuated instruments and compositions
Actuated musical instruments inherit a pre-existing tradition of musical performance practice as well as pioneer a novel and extensible practice of extended and augmented performance technique. The musical possibilities of such instruments inhabit a space both familiar and challenging, building on historically validated traditions while simultaneously breaking new sonic ground and new paradigms of interactivity.
The classification of ‘actuated musical instrument’ thus implies an idiomatic extension of performance technique, as the physical instrumental system itself serves as the starting point for technology-aided modification: the set of performance techniques and interaction methods traditionally associated with a given instrument are generally extended without reduction. The adoption of adaptive algorithms could even, for example, lead to instruments that change their response to player inputs over time according to usage patterns.
2.1. The Overtone Fiddle
An extension of the prior Overtone Violin (Overholt Reference Overholt2005), the Overtone Fiddle by Dan Overholt is an electronic violin that incorporates electronic sensors, integrated DSP and feedback actuation. An embedded tactile sound transducer (see Figure 3, lower left) vibrates the body of the Overtone Fiddle, allowing performer control and sensation via traditional violin technique, as well as extended playing techniques incorporating shared man/machine control. A magnetic pickup system is mounted to the end of the fiddle's fingerboard in order to detect only the signals from the vibrating strings, avoiding the vibrations from the body of the instrument. This focused sensing approach allows less restrained use of DSP-generated feedback signals, because there is less direct leakage from the actuator to the sensor.
Top: Overtone Fiddle with optional carbon-fibre lower resonating body; bottom: detail of embedded electromagnetic actuator and string pickups.
Signals can be injected directly into the main acoustic body of the instrument via a voice-coil type of tactile transducer, as well as into a second acoustic body that hangs below the instrument. This lower resonating body is made of extremely thin carbon-fibre and balsa wood – materials that would not be strong enough to support the full string tension of strings on the main body – thus allowing extremely efficient transfer of acoustic energy from another embedded tactile transducer into this box (which is designed as a Helmholtz-resonator). The second body of the Overtone Fiddle can also be driven with DSP-generated feedback signals, or indeed any audio signal the performer desires. The tactile transducers inside both the top and bottom resonating bodies are driven by a battery-powered Class-T amplifier, and the production of the audio signals with which to drive them is accomplished via real-time software on an iPod Touch mounted on the fiddle (the main body of the instrument is designed to accommodate this, as shown in Figure 1). This makes the entire instrument self-contained, not including the bow and its corresponding electronic circuits.
The bow used with the Overtone Fiddle is custom made from a carbon-fibre rod that is lighter in weight (and longer) than a normal violin bow, in order to accommodate the added mass of a small battery-powered sensor circuit based on the CUI32Footnote 2 (an electronics prototyping board), along with a wireless 802.11 g radio module, and an absolute orientation sensor. A simple BASIC-language program written on the CUI32 retrieves the orientation data from the sensor, and translates it into the Open Sound Control (OSC) protocol in order to send it to the iPod Touch on the instrument. The 802.11 g radio module is configured to broadcast its own ‘AdHoc’ base station, which is chosen in the setup of the network preferences in the WiFi settings of the iPod Touch, allowing the devices to communicate easily thereafter.
Software used with the Overtone Fiddle is generally written in SuperColliderFootnote 3 (McCartney Reference McCartney2002), PureData (using libpd or RjDjFootnote 4), or MoMuFootnote 5 the Mobile Music Toolkit (Bryan, Herrera, Oh and Wang Reference Bryan, Herrera, Oh and Wang2010), all of which can run in real-time on the iPod Touch. The instrument thereby incorporates all of the flexibility of modern digital signal processing techniques – for example, altering the timbral structure of the sound being produced in response to player input. Since these algorithms can be controlled through gestural interaction using both the motion sensors in the iPod (accelerometers, gyroscopes, etc.) and the orientation sensor on the bow, the Overtone Fiddle promotes new performer interactions beyond those supported by traditional acoustic violins. For instance, in an improvisational performance by the authors (seen in the video accompanying this article: Movie example 1),Footnote 6 the timbre of the instrument's sound is made to change dramatically when bowed rapidly.
While the sound quality of a traditional acoustic instrument is fixed by its physical design, actuated musical instruments can malleably simulate even physically impossible acoustic properties, such as a violin with hundreds or thousands of sympathetic strings. For the performer and the audience, however, the complete sound produced by an actuated musical instrument such as the Overtone Fiddle is perceptually contiguous, as both the natural acoustic and the digitally processed waveforms are ‘folded together’ – both produced locally. The music is generated through immediate performative gestures on the instrument, and musicians can use the embedded technology to produce interesting timbres that might never have been heard before. Nonetheless, the internal actuation is caused by excitation from digitally processed signals, so this aspect of this sound is due to virtual computerised agents. The instrument can also make use of wireless networks to incorporate itself into a wider ecology of musical endeavours, enabling shared instrumental control schemes, as previously mentioned in section 1.1.
Algorithms are used to adjust the body vibrations of the acoustic part of the instrument, actively changing the harmonics (overtones) heard in the musical tone quality. Consequently, the acoustic properties of the instruments are adjustable, instead of being permanently defined by the woodworking skills of a luthier. In other words, the force feedback from the actuator can cause the wooden body to exhibit new dynamic behaviours, and it is this programmable force feedback that distinguishes the Overtone Fiddle from prior instruments such as the Variax from Line 6 (Celi, Doidic, Fruehling and Ryle Reference Celi, Doidic, Fruehling and Ryle2004) and the Chameleon Guitar (Zoran and Maes Reference Zoran and Maes2008). This feedback actuation promotes the continuing evolution of musical instruments today, and aims to see what might result from the combination of some of today's most advanced technologies with traditional instrument-making and musical skill and practice. It is hoped that such combinations, by enabling performative feedback loops, will encourage performers more often to engage with the entire musical performance ecosystem when incorporating such technology-enhanced instruments into their musical practices.
2.2. The Feedback Resonance Guitar
In the case of the Feedback Resonance Guitar by Edgar Berdahl (see Figure 4), audio signals routed into the instrument's dual embedded electromagnets drive the activation and subsequent resonance of given frequencies on each of the six strings of the guitar. By varying the source of input – using sources such as pre-recorded or computer-mediated electronic signals, live third-party instrumental audio streams, or the guitar's own output – significantly different practices of instrumental performative practice become possible, including self-sustaining oscillations using acoustic feedback, exciting instrument resonances by a computer-mediated system, and the control of and communication between a third-party instrument and the guitar itself.
The Feedback Resonance Guitar, played by Robert Hamilton.
2.2.1. Inducing self-sustaining oscillations
As described in Section 1.3.2., the artificial extension of a guitar string's resonance or the sustain of a given note on the instrument can be achieved by feeding output from the guitar's pickups into the electromagnetic resonators driving the excitation of the instrument. Through gating and limiting of the excitation signal, a performer can control the amount of sustain generated by the instrument, evoking responses that can range from a subtle lengthening of a given plucked note to a fast and powerful self-oscillating response. A note may be sustained for an arbitrarily long period of time, enabling the performer to steer the actuated energy – a musical interaction that would otherwise require some other external energy source.
Highly amplified self-oscillating ‘feedback’ tones hold an idiomatic place in the historical performance practice of electric guitars, with rock-based examples ranging from early Beatles recordings (Emerick and Massey 2006) to the iconic controlled-chaos of Jimi Hendrix's live performance techniques. In traditional electric guitar systems, the unique sonic qualities of these tones are achievable through the completion of a complicated analogue feedback loop, most commonly created with high-volume loudspeaker outputs being transmitted through the air, intro the instrument's body, through the strings and subsequently into the pickups. As such, a low-volume recreation of quickly growing self-oscillation tones in live performance has always proven difficult to achieve for live electric guitar performance.
Feedback loops created by routing output from various combinations of the Feedback Resonance Guitar's seven acoustic pickups into its actuators allow for the generation of compelling feedback timbres with no correlation to the performance system's output volume. In this fashion, the levels of input signal driving the feedback system can be throttled through the use of digital or analogue limiters in the signal path, creating a highly controllable feedback performance system capable of subtle performance control (Berdahl and Smith Reference Berdahl and Smith2006).
In live performance, the Feedback Resonance Guitar has been used as a self-supplying feedback system in performances of …of giants by Robert Hamilton (Sound example 1), The Metaphysics of Notation by Mark Applebaum and Siren Cloud by Chris Chafe. In these compositions, the combination of induced self-oscillations in combination with multi-channel output and spatialisation has been used to create rich timbres and textures that are at the same time idiomatic to the guitar and sufficiently novel to provoke questions and commentary from guitarists and audience members alike.
2.2.2. Computer-mediated actuation
Audio signals generated by interactive and reactive real-time computer software systems under algorithmic or performer-mediated control enhance active physical instrumental systems by granting them access to properties and functionalities normally associated with computer systems. Since actuated musical instruments transform the actuation signal into a physical response through the vibration of notes on a set of strings, membranes or the instrument's body, a tangible interaction exists between the performer and the ‘virtual’ computer system. For example, audio output from a computer system fed into the actuated fiddle or guitar can cause the strings to vibrate, a physical reaction clearly felt by the performer's hands. Without controlling the performer's instrument or diminishing his or her inherent choices in adjusting the length of the strings, such a haptic interaction instead indicates through a tangible interface to the performer that a given pitch location is at that moment a region of interest for the virtual computer system – the performer hears the actuated notes; however, the actuated notes will be quiet if they do not correspond to the performer's currently selected string lengths. The choice of whether or not to follow this suggestion is left to the performer, and the communication occurs at a much more intimate level between the performer and the computer-mediated system than simply hearing normal audio output from loudspeakers attached to the system.
Once the computer-mediated actuation becomes sufficiently complex, it can be helpful to have a convenient interface such as an iPod touch to adjust its parameters. As a brief aside, let us consider the Overtone Fiddle, for which soundscape actuation signals have been implemented. One useful iPod mapping allows the performer to press the screen to capture a sound sample from the strings. This sound sample is then applied as an actuation signal, in short harmonising drone segments as a background over which the performer can continue to play the fiddle. Finally, another mapping generates semi-autonomous percussion sounds, while simultaneously processing the sound of the strings in real-time using a band-pass filter with a cut-off frequency that is dependent upon the tilt angle of the violin.
Another use of computer-mediated sound as input can be found in an interactive iPhone-driven control patch used for improvisation with the Feedback Resonance Guitar, wherein a bank of five oscillators is routed into the guitar, the pitch and relative amplitude of each being controlled in real-time by an iPhone Touch interface mounted on the guitar body itself (see Figure 5). In this scenario, touch locations on the lengthwise-axis of the multitouch display are mapped to oscillator frequency, while motion on the widthwise axis is mapped to oscillator amplitude. Individual oscillators can be enabled or disabled through a bank of toggle controls on the touch screen itself. In this manner, a performer can easily and idiomatically sweep through harmonic frequencies of the guitar with one hand, while fretting desired pitches with the other hand.
iPhone control interface mounted on the Feedback Resonance Guitar.
2.2.3. Live audio signals
In Sections 2.2 and 2.3 the instrument designs fit the augmented reality paradigm because the musical instruments are real, but their capabilities are augmented by coupling to some virtual elements via feedback control. Now we present a new application of the Feedback Resonance Guitar, which subscribes to the augmented virtuality paradigm. As before, the actuators on the Feedback Resonator Guitar serve as portholes to a virtual environment; however, in this application, elements in the virtual environment are avatars representing other real objects.
Consider employing musical output from live acoustic or electronic instrumental performers as actuation signals to an acoustic musical instrument. This kind of acoustical connection can create a new level of haptic ensemble communication during a musical performance. This actuated sound made tangible occurs in parallel to traditional methods of intra-ensemble communication (visual and aural cues) or replaces traditional communications in situations where such communication is difficult or impossible such as telematic or networked musical performance. In this manner, rich musical ecosystems based on shared instrumental control are created, allowing musical communication based on haptic excitation to drive novel emergent musical ideas and behaviours. This kind of haptic communication through tangible sound enables musicians to interact through the haptic modality, which we believe has so far been underutilised in the field.
For instance, in …of giants for double bass and Feedback Resonance Guitar by Robert Hamilton (Sound example 1), cues for half-step key-switches in an improvisational structure are triggered by the playing of low cue notes on the double bass, which are in turn routed into the Feedback Resonance Guitar. When a cue is received by the guitar, not only are nodes representing the root (i.e. first octave overtone) suddenly expressive and heavily weighted areas on which to play the guitar, but each node representing an overtone of that cue pitch too suddenly becomes heavily weighted. While only one instrument in this duo is currently an actuated instrument, even such one-sided co-control of an actuated instrument creates an exciting ecosystemic musical space, wherein the performers are connected through their shared control of the Feedback Resonance guitar.
In the work six in one hand for seven guitars and six network performers, also by Robert Hamilton, live audio signals are routed into the Feedback Resonance Guitar from six guitarists located in various locations around the world (Figure 6), sending individual audio outputs into the guitar across low-latency high-bandwidth network connections (Cáceres and Chafe 2008). Output from remote guitars is visualised within a fully rendered three-dimensional environment and manipulated in real-time by virtual performers interacting with 3D artefacts in the virtual space (Hamilton Reference Hamilton2008, Reference Hamilton2010). By essentially filtering audio input and musical gestures from seven performers through the musical lens of the Feedback Resonance Guitar, with subsequent manipulation by virtual performers in a virtual space, the complex musical ecosystem realised within the instrument itself is made visible and tangible. Such feedback between performers would be practically impossible or certainly impractical without actuation, yet it enables the performers to engage in new kinds of musical interactions.
Conceptual schematic of Robert Hamilton's six in one hand.
2.3. Teleoperation with Robothands
The prior section demonstrated how the Feedback Resonance Guitar can serve as a porthole into a virtual environment, for both augmented reality and augmented virtuality applications. Here we consider constructing a porthole into a virtual musical environment that is so accurate, it is like a hand. For this, we use a general haptic force-feedback device, such as the three degree-of-freedom (3-DOF) NovInt Falcon shown in Figure 7. However, the virtual world is a big place! In other words, the three-dimensional space is expansive, and the performer must explore it in order to find virtual objects with which to interact. Creating high-fidelity virtual worlds with virtual objects to facilitate efficient exploration requires great attention to programming detail (Cadoz, Luciani and Florens Reference Cadoz, Luciani and Florens1981 and Reference Cadoz, Luciani and Florens1993). For this reason, we should study how to place objects in the virtual world with reduced programming effort.
Left, a slave robothand holding an acoustic musical shaker, and right, a performer interacting with the master NovInt Falcon force-feedback device.
We therefore suggest creating avatars in virtual environments that are connected to real objects instead of programmed. For instance, see the slave ‘robothand’ shown in Figure 7 (left) that is holding an acoustic shaker. When a (stiff) virtual spring is connected between the master and a slave robot in virtual space, the performer can remotely play the shaker through the robothand while receiving force feedback from the shaker (Berdahl, Smith and Niemeyer Reference Berdahl, Smith and Niemeyer2010).
We find it striking to observe percussion instruments controlled by robots that move as if they were being played by a real human performer making essential as well as ancillary gestures. The sound is lifelike, and perhaps even more importantly it is tangible. Furthermore, while the percussion instruments are indeed actuated by robots, the performer can touch the instruments to interact with them directly if they are placed next to the performer. For instance, by holding onto one of the percussion instruments, the human performer can change the mechanical load seen by the robot, so the robot will cause the percussion instrument to move in a filtered but still somehow lifelike manner.
One intriguing question is, ‘How can one performer control distinct musical parts on multiple slave robothands simultaneously?’ One could employ artificial intelligence algorithms to cause the slave robothands to behave semi-autonomously, but we are more interested in intimate control musical instruments. Therefore, in the piece Edgar Robothands: When The Robots Get Loose, Berdahl employs a technique that he terms mechanical looping to control the tambourine, snare drum and two shakers by way of four slave robots, as shown in Figure 8. During teleoperation, the trajectory of the gestures made by Berdahl into the master robot are recorded into wavetable loops in Max/MSPFootnote 7 using the HSP toolbox (Berdahl, Kontogeorgakopoulos and Overholt Reference Berdahl, Kontogeorgakopoulos and Overholt2010). At later times during the piece, these loops are played back into the slaves through virtual springs, providing for mechanical (real world) synthesis with an especially human feel, even when the loops are played back irregularly or at superhuman speeds. An audio recording of the piece as performed on 17 February 2010 for the CCRMA Winter Concert (Sound example 2) demonstrates that robots can perform actions with repeatability, accuracy and speed that may be unparalleled by human users, creating physical and lifelike sounds that are nevertheless new and that were not possible before. The net result is co-control of the percussion instruments via ecosystemic performance that would not be the same without either the performer or the Robothands.
Edgar Robothands: When The Robots Get Loose.
2.4. Haptic Drum
The last musical instrument in this paper fits within the augmented reality paradigm, and is a percussion instrument that applies feedback control to change drum-roll gestures. When a drumstick is dropped upon a normal drum, it typically bounces two to several times before coming to rest due to frictional forces. Percussionists use this bouncing action to play drum rolls on traditional percussion instruments at rates of up to 30 Hz (Hajian, Sanchez and Howe Reference Hajian, Sanchez and Howe1997: 2294–9), even though humans cannot move the hand back and forth unassisted at such fast rates. The Haptic Drum explores the possibilities that arise when friction is partially counteracted by actuation. For example, the Haptic Drum prototype v1 being played in Figure 9 consists primarily of a drum pad attached to the cone of a woofer by a lightweight, cardboard cylinder. Every time that a stick strikes the drum pad, a pulse is sent to the loudspeaker, which adds energy to the drumstick vibrations. As a consequence, the Haptic Drum can enable the performer to make drum-roll gestures that would otherwise be difficult or impossible, such as drum rolls at fast rates such as 70 Hz with a single drumstick (Berdahl Reference Berdahl2009: 153–74). These rates are so fast that the drum roll begins to sound pitched at the drum-roll rate.
Berdahl playing Haptic Drum prototype v1 (left) using two drumsticks simultaneously.
The pulse height, which is constant, is selected by the position of the top knob on the Haptic Drum (see Figure 9). The knob in the middle controls the shape of the loudspeaker pulses, which is only of minor importance, and the third knob controls the debounce threshold for the impact detector. By reducing the debounce threshold far enough, it is possible to cause the Haptic Drum to emit multiple pulses for each drumstick impact, or the system can even become unstable, depending on how the performer manipulates the drumstick. However, the energy in the system does not grow without bounds because the pulse height is constant (Berdahl Reference Berdahl2009: 153–74). This feature enables the performer to steer the drumstick vibrational energy provided by the Haptic Drum power source to create new gestures and patterns. In particular, when the performer employs two drumsticks simultaneously, it becomes possible to obtain complex bifurcation phenomena as the performer modulates the stiffness and angle of his or her hands. An online video demonstration is available that illustrates the most basic new gestures and musical interactions that are enabled by the Haptic Drum.Footnote 8
A sound synthesiser can be connected so that a drum sample is triggered each time the drumstick strikes the Haptic Drum. In our opinion, the resulting synthesised sound is perceived as physical because each triggered sound sample is due to a real drumstick striking a real drum pad. However, a snare drum sample played back at 70 Hz may sound bizarre to the Western ear – it may even be perceived as a machine gun sound. For this reason, Berdahl often employs non-Western drum samples.
Multiple pieces have been written for the Haptic Drum. For example, It's Like A Car Footnote 9 (excerpt in Sound example 3) is a recording of a live improvisation by Edgar Berdahl and Frankie Siragusa involving the Haptic Drum and traditional percussion instruments including drum set, maracas, xylophone, egg and FM radio. It was premiered at the CCRMA Transitions Concert at Stanford University in September 2009. A similar approach is taken in the piece Sike!,Footnote 10 which incorporates the Haptic Drum, the Feedback Resonator Guitar and NovInt Falcon-controlled harmony synthesiser voices, as well as an acoustic drum set.
3. Conclusion
Actuated musical instruments produce sound via vibrating element(s) that are co-manipulated by humans and electromechanical systems. We believe that effectively employed actuation can enable the performer to interact in new ways. For instance, actuation can provide energy from an external source that the performer can feel and steer. More generally, actuation can make virtual qualities tangible in physical instruments, allowing designs that can free up some cognitive bandwidth. During musical performance, shared control of actuated instruments allows for the emergence of novel musical behaviours, as rich polyvalent control schema evolve into complex musical ecosystems, generated and sustained by the complex interactions between interconnected performers and instrumental systems alike.
Towards these ends we have described some of our ongoing explorations into the world of actuated musical instruments, including the development of new instruments and the performative ecosystems they bring about. While some of these instruments are so new (especially the Overtone Fiddle) that they have not been used in a large number of performances as yet, we have portrayed them in a recent ensemble performance An Evening of Actuated Musical Instruments,Footnote 11 a trio performance with the Feedback Resonance Guitar, the Haptic Drum and the Overtone Fiddle (see Figure 10). We have also started an online presence ‘Actuated Musical Instruments Guild’ at www.actuatedinstruments.com that we hope will serve as a community hub where related work can be shared.
Berdahl, Overholt and Hamilton playing the Haptic Drum, Overtone Fiddle and Feedback Resonance Guitar in An Evening of Actuated Musical Instruments.
We postulate that the two most promising directions coming out of this research might be:
1) a revival in the fairly stagnant evolution of traditional instruments; heading towards a ‘new renaissance’ of instrument development focused on actuated instruments and their related discourse; and
2) further advances in the area of teleoperation of acoustic instruments, telematic performance and the expanded performance ecosystems these types of approaches enable.
We feel that it is especially advantageous to have the sound localised with the action, and also good for the instrument to be untethered while performing and easily portable. Future actuated musical instruments will tightly incorporate sound elements coming from embedded signal processing, and enable more intimate control of the combination of digital algorithms with the natural physics of interaction. They will accomplish this by bringing digital behaviours out into the real world, allowing them to be used as musical inflections via gestural interaction. In so doing, they will enhance the expressivity of electronic and acoustic instruments alike, opening up new directions in music.
