2. The Mastery of a Musical Instrument: A Sub-symbolic Skill Acquisition

Twentieth-century cognitivism has not been successful in portraying human intelligence, and various approaches have emerged that propose different views of cognition where typically the body and the environment enter the equation.Footnote 2 One of the most musically relevant approaches is enactivism, as developed by Varela, Thompson and Rosch (Reference Varela, Thompson and Rosch1991). This theory, with roots in biology but inspired by phenomenology and Eastern thought, depicts the mind as necessarily embodied in an external environment. In enactivism there is no distinction between perception and action; both co-emerge through the agent’s active involvement with the world. This involvement is primarily non-representational – in other words, it is at a level where the cognition is subconscious, pre-conceptual, distributive and emergent. Varela et al. define the term:

Importantly, this does not mean that humans are not symbolic cognisers. That fact is obvious by looking at how humans constantly create and use symbolic systems such as language, mathematics and music, or games such as chess. Stressing that they ‘see symbols as a higher-level description of properties that are ultimately embedded in an underlying distributed system’ (Varela et al. Reference Varela, Thompson and Rosch1991: 101), Varela et al. find it more appropriate to define symbols (or cognitive representations) as ‘approximate macrolevel descriptions of operations whose governing principles reside at a subsymbolic level’ (Varela et al. Reference Varela, Thompson and Rosch1991: 102).

Varela et al. describe embodiment as an enactive stance towards the world where we create our (not ‘the’) world through an active engagement with it. Enactivism is resourceful in explanations of embodiment, something we should bear in mind when analysing how acoustic musicians incorporate knowledge of their instruments through repeated practice. It also explains how symbolic systems are higher-level descriptions of phenomena understood by the agent through bodily perceptions. There is a general consensus in cognitive science that musicians (or athletes for that matter) learn their skills gradually through persistent practice and a minimum of verbal instructions (Dreyfus and Dreyfus Reference Dreyfus and Dreyfus1986). Such bodily incorporation transforms our tools into ready-at-hand phenomena (Heidegger Reference Heidegger1962) that are not based on symbolic or theoretical relations any more. The focus becomes the act and not the object, or, as Winograd and Flores explain in a Heideggerian manner, ‘[m]y ability to act comes from my familiarity with hammering, not my knowledge of a hammer’ (Winograd and Flores Reference Winograd and Flores1986: 33).

According to the enactive view, the skill acquisition related to learning an acoustic instrument is highly embodied, non-symbolic and perceptuo-motor based.Footnote 3 It explains how Ihde’s ‘embodiment relations’ to the world are established, and provides a description of this process of incorporationFootnote 4 from the level of biology. An important research question here becomes to explore, together with the enquiry into the epistemological or music theoretical nature of digital tools, how the digital musical instrument manifests a different relationship between the human body and the body of the instrument itself through its characteristic split between the interface and the sound engine.

3. The Extended Mind

In the 1990s, working in another strand of cognitive science from that of Varela et al., Andy Clark developed his theory of the ‘extended mind’. Clark illustrates how people use props in the environment to extend their cognitive capacity and ease cognitive load. Sticky notes, notebooks, diagrams, models, and so on all serve as scaffoldings onto which we ‘offload’ our cognition. It should be noted that this is not some kind of mystical panpsychism, as Clark and Chalmers adamantly point out that they do not equate the cognitive process with consciousness (Clark and Chalmers Reference Clark and Chalmers1998). The cognitive process happens both inside and outside the skull, an observation reminiscent of Wittgenstein’s account of thinking as physical activity (Wittgenstein Reference Wittgenstein1969: 6).

Objects and artefacts serve as an external playground for thinking. We calculate on paper, draw graphs, build models, make reminders of all kinds, and, as Clark illustrates, we reorganise the pieces when playing Scrabble. According to this view, it does not matter whether information is stored and processed in the brain or outside in the environment, what matters is how the data is retrieved and taken into use (Clark Reference Clark2003).

While the theory of extended mind has much potential for interesting discussions in cognitive science, it is pertinent, in the context of musical technology, to probe deeper. Indeed, the picture is more intricate since we are surrounded by objects and technologies (or ‘thrown’ into world of equipment as Heidegger would describe it) that contain complex material properties, scripts of their usage (Latour Reference Latour1994), and even politics (Winner Reference Winner1980). The objects around us – the technologies that serve as props in our thinking and music making – are stuffed with people and their ideas; in them we find programmes of action, manuals of behaviour, and political and sociocultural constructions, including aesthetic tendencies. Technological objects are therefore never neutral, they contain scripts that we subscribe to or reject according to our ideological constitution. The problem is that the scripts are often well hidden and concealed, which can result in an uncritical use of creative technologies – technologies inherent with ideological content.

To illustrate this point in the field of musical instruments, we see that the piano keyboard ‘tells us’ that microtonality is of little importance (and much Western music theory has wholly subscribed to that script); the drum-sequencer that 4/4 rhythms and semiquavers are more natural than other types; and the digital audio workstation, through its affordances of copying, pasting and looping, assures us that it is perfectly normal to repeat the same short performance over and over in the same track. It is therefore germane to ask ourselves how much theory is inscribed into our new tools, particularly with regard to how automated, structurally complex and aiding (in composition and performance) our new musical instruments have become.

4. Material Epistemologies

Things are impregnated with thoughts; they are embedded with ideologies that have ethical, political and aesthetical implications. Furthermore, we use those artefacts in our daily tasks as extensions of our cognitive mechanism. But how do things ‘contain’ knowledge? How do we write our knowledge into artefacts, and how do we read that knowledge from them? By the same token, how does this relate to digital musical instruments?

Davis Baird (Reference Baird2004) shows how material objects can have a different epistemological status to statements of language. Technological objects may contain functions which their users understand, but would not be able to describe in language as ‘[t]he material products are constitutive of scientific knowledge in a manner different from theory, and not simply “instrumental to” theory’ (Baird Reference Baird2004: 1). Much like Heidegger and Ihde, Baird shows that technological objects can precede science, and afford scientific discoveries through their physical structure and functionality. The point here is not what precedes what, but rather that the instrument becomes an expression in itself, an externalisation of knowledge in a form that is not symbolic but material. Additionally, it is not only designed objects that inhere knowledge, but natural objects as well, by means of the physical and mechanistic properties of their material.

Baird is well aware of Bruno Latour’s ideas of concretisation: namely, that when many elements combine into one actor and start to operate as a unity (with either human or non-human agency), they gradually become a black box. When the black box works, its origins are forgotten and thus ‘paradoxically, the more science and technology succeed, the more opaque and obscure they become’ (Latour Reference Latour1999: 304). Baird agrees with Latour on the nature of scientific blackboxing, but highlights another and perhaps more epistemologically active function of the black box itself: while talking about a particular instrument, called Spectromet, Baird says

Here (and spectacularly exemplified in the digital musical system) we see how the blackboxed instrument contains the knowledge of its inventors, which means that the users of the instrument do not need to have a deep knowledge of its internal functions. If we assume that both the designers and the users of the instrument have an understanding of it, this understanding is very different and attained from distinct origins. The former creates the instrument from a conceptual understanding of the domain encapsulated by it, whereas the latter gains operational knowledge that emerges through use (or habituation) and not from abstract understanding of the internal functionality. This picture is particularly complex in today’s new musical interfaces as typically their designers are also the performers. This implies a continuous oscillation between a mode of conceptual (system design) engagement with the instrument and embodied (performative) relationship with it. Again, we are reminded of Waisvisz’s stance introduced above, where we might talk of two modes: that of the instrument designer and the instrument player.

5. The Acoustic, the Electric and the Digital: Interfaces of a Different Kind

When the technological artefact is made of material substrata (as opposed to symbolic), it can contain knowledge that precedes the scientific understanding of its functioning. The acoustic instrument is a good example. The sophisticated sound of the clarinet or the cello was developed over an extended period of time, but this sound was mature long before Fourier or Helmholtz brought forth their theories of sinusoidal functions and timbre. Strings, wood and brass tacitly encompass the theories of sound in their materiality. The observation here is that acoustic instruments inhere knowledge that has to be explicitly understood and stated in the design of digital instruments; without this explicit knowledge no code could ever be written. We explore and discover the sonic properties of wood and strings, but a solid theoretical knowledge of sound is required in order to create digital musical systems.

Let us briefly explore some main differences in the design of acoustic, electric and digital instruments. The acoustic instrument builders will acquire their skills through the embodied practice of making the instruments. These skills, including the intuition of material properties and sound physics, are largely in the form of tacit knowledge (Polanyi Reference Polanyi1966) acquired through apprenticeship. It is a predominantly embodied and non-theoretical knowledge derived from discovery, exploration and refinement. The development of the acoustic instrument involves iterations of designing, building and testing. The instrument makers work with what physical materials afford in terms of acoustics; a change in curve, in width, or in material means a change in sonic timbre. The interface is necessarily conditioned by the properties of the materials used, and various solutions are therefore introduced to adapt to or extend the scope of the human body with the use of levers, keys and other mechanisms. Acoustic instruments are therefore firmly grounded and conditioned by human physique in their design.

The makers of electronic instruments, such as the early synthesisers, work with different materials. Instead of millennia of tradition, they start with exploring the material properties of electricity, magnetic waves, oscillators, capacitors, inductors and transistors. As with the acoustic instrument, the creative process is one of designing, building and testing through a cycle of iterations, actively working with the materials. However, the materials of the electronic instrument makers come with instructions and schematic diagrams that describe their behaviour. There is an increased logic of calculation, science and engineering. Fourier’s and Helmholtz’ theories are now well known, and the instrument makers can draw from that knowledge in their designs of oscillators and filters. However, some of the characteristic sound in electronic instruments depends on the chaotic or entropic properties of the materials used.Footnote 5 The user interfaces in electronic musical instruments can be built in any shape and form. In terms of ergonomics, we are still constrained by physical mapping, so when the instrument has been wired up, its fundamental functionality is not easily changed. This gives the machine an instrumental quality, a character that affords in-depth explorations.

The digital instrument makers are in a different world altogether. The ‘workshop’ is the meta-machine of the computer, and inspecting them at work might not suggest any associations with musical activities.Footnote 6 Code as material is not musical; it does not vibrate; it is merely a set of instructions turned into binary information converted to an analogue electronic current in the computer’s soundcard. The materials of the digital instrument are many: a computer, a monitor, a sound card, an amplifier, speakers, and tangible user interfaces. Behind this material surface lie other materials: audio programming languages, digital signal processing, operating systems, mapping mechanisms between gestures and sound engines, and so on. From the perspective of Latour’s actor-network theory, the networks enrolled in the production of digital instruments are practically infinite. There is an impenetrable increase in complexity, which means that the inventors have to constantly rely on black boxes. Furthermore, the materials used in the digital instrument originate from technoscientific knowledge. There are relatively few physical material properties at play (although of course at the machine level we find matter) compared to the amount of code that constitutes its internal (and symbolic) machinery. The inventors have knowledge about digital signal processing, sound physics, audio synthesis, gesture recognition, human–machine interaction, and the culture of musical performance. In general, the digital instrument is based on the knowledge of symbolic systems and their essence in the form of code. From a design perspective, any interface can be designed for any sound. There is no natural mapping between gesture and sound in digital systems. In acoustic instruments the performer yields physical force to drive the instrument, whereas in electric instruments there can be mixture of both physical and electric force. In digital instruments, the physical force becomes virtual force; it can be mapped from force-sensitive input devices to parameters in the sound engine, but that mapping is always arbitrary (and on a continuous scale of complexity), as opposed to what happens in physical mechanisms.

Therefore, from the perspective of embodiment relations, there is a characteristic diversity in the way we work with the materials that constitute our musical instruments. Digital music systems, whose foundations are essentially of a symbolic nature, are more likely to establish hermeneutic relations to the world than acoustic instruments, where understanding of expressive affordances are incorporated into the performer’s motor memory. To work with symbolic tools means that one has to continually switch modes from focusing on the world to focusing on the tool with regular intervals and to a more pronounced degree than in acoustic instruments. This detachment from the terminus of our activities could be paraphrased as a disruption in flow and is present in the majority of existing digital music systems.

6. Epistemic Tools

Anyone who speaks more than one language, in particular if those languages are of different linguistic families, knows how differently each language portrays the world. A language is a world-view. This observation is the basis of a theory called the Sapir–Whorf hypothesis, which states that different languages portray both space and time differently.Footnote 7 The Sapir–Whorf hypothesis claims that grammatical categories of the language determine the speaker’s behaviour: ‘We cut nature up, organize it into concepts, and ascribe significances as we do, largely because we are parties to an agreement to organize it in this way – an agreement that holds throughout our speech community and is codified in the patterns of our language’ (Whorf Reference Whorf1956: 213). Seventy years earlier, Nietzsche had argued that physics were an interpretation of the world and not its explanation. The sciences reject phenomenological experience in favour of objective language and they ‘do this by means of the pale, cool, gray, conceptual nets which they threw over the colourful confusion of sense, the rabble of the senses’ (Nietzsche Reference Nietzsche2004: §14). Rather more recently, Bowker and Star (Reference Bowker and Star2000) have argued that there are certain problems in the design of computer systems that relate to the act of classification. When we classify and categorise our external world, in order to represent it through the machinery of information technology, we are inevitably performing acts of reduction. Such reductions are bound to be contingent and messy, with abundance of ethical and political implications. Human actions are displaced into representation, thus establishing strata of complexities and interdependencies that limit the agent.

This nature of categorisation and abstraction of human knowledge (both know-that and know-how) and actions is the essence of computer software, a fact that is vividly apparent in the realm of musical software tools. Software is, as Bowker and Star point out, a ‘contingent’ and ‘messy’ classification that has diverse implications and, in our case, aesthetic, cultural and culture-political effects. As Latour (Reference Latour1987) so elegantly demonstrates, objects establish themselves as black boxes through repeated use, and in that process their origins disappear. The object is naturalisedFootnote 8 through heavy use, a fact that makes Bowker and Star observe that ‘[t]he more naturalized the object becomes, the more unquestioning the relationship of the community to it; the more invisible the contingent and historical circumstances of its birth, the more it sinks into the community’s routinely forgotten memory’ (Bowker and Star Reference Bowker and Star2000: 299).

Writing digital musical interfaces therefore necessarily entails the encapsulation of a specific musical outlook. It is a (sub-)culturally conditioned systematisation of musical material. Musical patterns, musical styles and musical aesthetics become blackboxed in software. Most people do not know why the standards, implementations, patterns or solutions in musical software are there. At times, they manifestly limit the musical expression, but at other times limitations are concealed by the rhetoric, streamlined functionality and slick interface design of the software tool. Here we encounter yet another difference between the worlds of acoustic and digital instrument making. The acoustic instrument maker can freely re-invent or improve the instrument at any time with new materials, changing structures or adding/deleting features (Eldredge and Temkin Reference Eldredge and Temkin2007). The software instrument maker is limited by the complex infrastructure of operating systems, programming languages, protocols and interface limitations. In digital instruments, systems of classification form an organisation of musical language, a clarification and explicit elucidation of the musical language and its rules. However, and well known since Wittgenstein’s later writings, languages are not easily formalised as systems. At the core of all languages are dynamic, emergent and adaptive sets of rules that change through time and differ in geographical locations or in sub-cultures. The act of formalising is therefore always an act of fossilisation. As opposed to the acoustic instrument maker, the designer of the composed digital instrument frames affordances through symbolic design, thereby creating a snapshot of musical theory, freezing musical culture in time. The digital instrument is thus more likely to contain an expressive closure as contrasted with the explorative openings of the acoustic instrument.

From our analysis of the epistemic tools, we can now crudely generalise (and therefore exaggerate for the sake of an argument) a core difference between the acoustic and the digital. The acoustic instrument is material and developed from bottom-up exploration of the acoustic properties of the materials used. Here, the sound generation (and the required knowledge of it) is given to us for free by nature. We had complex instruments in the form of lutes, flutes and organs long before we ever had a mathematical understanding of sound. The physics of wood, strings and vibrating membranes were there to be explored and not invented. However, using the acoustic instrument we develop an intuitive understanding of the instrument, the music theory, and the tradition in which we are playing. As the interaction with the instrument is primarily embodied, we have to train our body to be in rhythm with the instrument – in other words to incorporate the musical knowledge in sync (or mutual resonance) with the particular instrument of choice. As opposed to the generic explicitness of the digital instrument, the acoustic instrument contains a boundless scope for exploration as its material character contains a myriad ways for instrumental entropy, or ‘chaotic’, non-linear behaviour that cannot be mapped and often differs even in the same type (brand and model) of instruments.

The digital instrument is theoretical and developed from a largely top-down methodology. In order to make the instrument, we need to know precisely the programming language, the DSP theory, the synthesis theory, generative algorithms and musical theory, and have a good knowledge of Human Computer Interaction. To use the instrument, on the other hand, we learn the tool through working with it and habituate ourselves with its functions, progressively building an understanding of its workings. As users we often do not need to know as much about synthesis or music theory, as the black box is intended to be used through its simple and user-friendly interface. Ergonomically, the interaction happens primarily through a symbolic channel, which gradually teaches the user to operate with technical terms (such as ‘low-pass’), but this happens from the habituation of the model (what we call the epistemic tool). The predefined quality of the digital instrument means that its functionality can be exhaustively described in a user manual; all is supposed to be explicit, nothing covert. Where the digital instrument exhibits any chaotic or entropical behaviour, it tends to be due to a failure in design, a bug in the code or loose wiring in the hardware.

Let us now reverse out of the epistemological investigations and revisit the phenomenological point: it has been explored above how people learn acoustic instruments through an enactive and embodied practice. Digital music systems or instruments (such as Logic, Pure Data, The Hands, or the reacTable) pose a difficulty here. They do, of course, provide for a certain virtual embodiment, but the locus of their real nature is fundamentally in the symbolic realm. Although we interact in an embodied manner with the computer using physical interfaces (moving our mouse on a two-dimensional plane, touching screens, or swinging Wiimotes) the interaction always takes place through symbolic channels of varied bandwidths. The interaction is primarily with graphical representations such as words, menus, cables or icons on the one hand, or abstract representations such as variables, parameters and functions on the other. This symbolic communication is based on a designed construction, which is inherently an approximation and an ad hoc representation of the task/world/music. What characterises the player of the digital instrument is a mental representation of the instrument’s parameters, and a strong awareness of how easily and arbitrarily those can be changed. As such, the digital instrumentalist is always a luthier (Jordà Reference Jordà2005) as well, someone that consciously engages with the instrument as a dynamic and fluid tool of a contingent nature.

The constraints in acoustic instruments are of a different kind: they have naturally inbuilt affordances, but those are mechanistic and physical (typically the domain of frequency range and timbre), not musicotheoretical and symbolic (typically the domain of notes and form). The acoustic instrument presents itself as a coherent whole where the interface and the sound engine are one and the same. Although the digital instrument can be conceived of as an integrated whole (Bongers Reference Bongers2000; Jordà Reference Jordà2005), this is often done for the sake of practicality and is not the only approach that should be taken when they are analysed. Particularly in intelligent instruments we find that the expressive design and the determinant of performance experience is to be located at the symbolic computational level. This area, abounding with culture-specific models of music, has been largely neglected in the analysis of digital musical systems.

The abstract characteristic of all system design constitutes the epistemic nature of digital tools. The tool is designed, its affordances and constraints are outlined, and the user’s actions are predicted and delineated into the instrument’s interface and interaction design. It is the designer who decides with clear rational arguments what is revealed and what is concealed in the use of the system. Whereas the body of the acoustic instrument is physical, the body of the digital musical instrument is intrinsically theoretical. Skill acquisition, the path to mastery, and the nature of virtuosity are all features that are transformed with the digital musical instrument. From this perspective, when computers are seen as mediation, the role of the system designer is that of outlining a system that directs the energy from the physical interface to the work (or the terminus – the music) through the symbolic engine of the epistemic tool. This activity of blackboxing, of creating abstractions of activities where bodily movements and thoughts are represented as discrete chunks in time, grounds the complexity and the non-transparency of digital tools. Therefore, if there is a normative message in this paper, it would be an encouragement to acknowledge the theoretical, cultural and social context in which all tools are designed – and this implies an awareness by designers of the responsibilities they have in terms of aesthetic and cultural influence.

8. Acknowledgements

The author would like to thank Enrike Hurtado Mendieta for collaborating in the generation of these ideas over the years. Nick Collins, Chris Thornton, Alex McLean and Tom Ottway read the paper and made important suggestions. Finally, the author expresses his gratitude to the three anonymous reviewers and the editors of Organised Sound whose comments made this text much better than it otherwise would have been.