1. IntroductionFootnote 1
The term sonification is used today in various contexts ranging from scientific applications to sound art and music composition. An early definition by Kramer (Reference Kramer1994) as ‘representing data with non-speech sound’ states the primary role of sonification is as a scientific display. As simple as this definition seems, it hides various challenges in designing and implementing sonifications. The sonification community still struggles to define its relationship to music and sound-art practices. This is mostly because the sound arts are concerned with issues other than scientific display. A sonification definition focusing on scientific display aspects that was proposed by Hermann (Reference Hermann2008) along with a taxonomy of techniques and interactions, emphasised the needs to be systematic, reproducible, objective and transparent with respect to the transformation of data into sound. Even though, as Hermann noted, this definition constitutes the necessary requirements, it does not guarantee that an information-revealing sonification will result.
While sonification must be rooted in the data, which are algorithmically translated into a sound signal, what is perceived through listening remains, in essence, just sound; extracting information from it is a second crucial level of transformation. In this context, information is less a notion from the domain of informatics (see Shannon Reference Shannon1948), but rather refers to the etymological root of the word (from Latin in-formare), which literally means putting into form, as in an act of interpretation. The kind of information that we receive is not equal to what we perceive, but depends on how we perceive it – in other words, which listening modes we engage and how these modes influence the capacity of sonifications to represent data structures. It could be said that the information content of a sonification needs to be found in the ear of the listener and not only in the signal into which the data are converted.
Listening modes and how they shape the information content of the perceived sound also help to differentiate questions of aesthetics in sonification, which have so far been mostly discussed along the lines of information aesthetics (see Fishwick Reference Fishwick2006), a notion applied to sonification by Barrass and Vickers (Reference Barrass and Vickers2011) emphasising functionality over representation. The functionality of a sonification can partly be assessed by how it helps to fulfil a task in which well-defined information needs to be displayed. Here pleasantness or user appeal can be useful concepts during design cycles. However, the concept of information aesthetics becomes increasingly less applicable when the task for which sonification is used has an exploratory character. In this case it is necessary to turn towards the phenomenology of listening and to analyse which listening modes offer the possibility to discover data structures in the sound of sonifications. In the conclusion, we will propose how a listening-mode based discourse can help to differentiate the notion of aesthetics for sonification according to specific display purposes.
In the beginning of auditory display research, sounds were conceptualised on a continuum between a symbolic and an analogic pole by Kramer (Reference Kramer1994: 21). In this continuum, symbolic stands for sounds that coincide with existing schemas and are more denotative, and analogic stands for sounds that are informative through their connotative aspects (see Worrall Reference Worrall2009: 315). Along with the terms symbolic and analogic, listening modes have been discussed in auditory display literature, usually in dichotomic terms; either identified with the words listening and hearing or understood as musical listening and everyday listening (Gaver Reference Gaver1993). Almost three decades earlier, Schaeffer (Reference Schaeffer1966) introduced four direct listening modes together with the fifth synthetic mode of reduced listening. Unfortunately, Schaeffer's theoretical work remained largely unnoticed by the auditory display community. In particular, the notion of reduced listening justifies a recasting of the conceptual organisation of sonifications according to descriptive or normative display purposes reaching beyond the connotative and denotative poles proposed by Kramer. The renewed interest in listening modes is also due to a recent contribution to this field by Vickers (Reference Vickers2012), who discusses how this field of study can contribute to sonification and auditory display, and particularly the contribution by Tuuri and Eerola (Reference Tuuri and Eerola2012), who bring forth convincing arguments to rethink listening and auditory display from an embodied cognition perspective.
Hunt and Hermann (Reference Hunt and Hermann2011: 295) discuss multi-modal and interactive aspects of sonification as options to expand and advance the field, and point out that there is a pressing need for a better understanding how these aspects can be integrated in auditory information displays. An awareness of the variety of listening modes can help to enunciate how sound might refer to other modes of perception and action and how they influence the connection between data and sound. In light of the recently increased interest in listening modes by the auditory display community, we provide in this article an approach which complements existing design guidelines by Barrass (Reference Barrass1997), deCampo (Reference deCampo2007) and Frauenberger (Reference Frauenberger2009) and, for product sounds by Jekosch (Reference Jekosch2005) and Hug (Reference Hug2009).
2. Listening modes, perception and intention
The existence of different listening intentions is reflected in most languages through various synonyms for the act of listening, describing different ways of perceiving the world through sound. Understanding listening modes can help us to identify if displays tend to favour one or another mode and can thereby help to create conditions that support the purpose for which a sonification is made. This is of particular interest for sonifications in multi-modal and interactive contexts, where sonification design needs to take into account how the other senses and actions mutually influence the information content of the sound. As a starting point for the discussion of listening modes and sonification we briefly recapitulate those proposed by Schaeffer (Reference Schaeffer1966) and discuss how they apply to examples of sonification and auditory display.
2.1. Direct listening in sonification and auditory display
Écouter, to listen, is to use sound in order to identify its source, event or cause. In auditory display this is often understood as the physics of processes and energetic attributes of events – in other words, where and with what intensity things happen. This mode is exploited in audification in the case of data substrates, whose oscillations originate from physical phenomena. Audification does not always provide a sonic substrate that lends itself to indexical listening, so it is particularly important not to confuse the directedness of the method with listening for indices (Grond and Hermann Reference Grond and Hermann2012). Model-based sonifications introduced by Hermann and Ritter (Reference Hermann and Ritter2005), also, attempt to exploit this listening mode by rooting the cause of the sound in invariant dynamical physical laws with respect to global data properties. Auditory icons were equally conceived with respect to the causal aspects of listening. In multi-modal sonifications, this mode helps us to analyse how sonic indexes points towards other perceptual modalities.
Ouïr, to perceive, refers to the sense of hearing as a whole, and first and foremost informs the listener that something is audibly present which can be distinguished from the background. All sonifications are at first perceived on this level, after which sound sources, sonic values, or meaning can be discovered and related to data properties. By reducing the sound to the raw outline of its audible presence and a very coarse categorisation, Ouïr plays a role in monitoring situations.
Entendre, to hear, pays attention to deliberately selected sonic values of the sonification (pitch, timbre, envelope, duration) depending on the context. If a sonification consists of several audible percepts, this reduction is inevitable and makes us perceive a sonification as more or less musical, which illustrates a specific challenge in sonification. Being structured, this listening mode has the potential to reveal more information than just signalling an audible event (like ouïr). However, since the choice of sound properties that are abstracted from it is subjective, it is not possible to tell if the information has been decoded, and to identify which unaddressed elements of embodied cognition the listener exploits in order to make sense of the sound. In order to attack this uncertainty, Worrall (Reference Worrall2010) proposed a gesture-encoded sound model for parameter-mapping sonifications.
Comprendre, to comprehend, refers to extra-sonorous influences such as natural language or culturally dependent musical motifs. For sonification, comprehending reflects the fact that semantic categories influence auditory perception. In sonification, and auditory display more generally, being able to comprehend the meaning of sounds is the stabilising basis for auditory icons as auditory signs and enables a transitioning from listening to comprehending. As musical signs, earcons (the short sounds that alert us to events in computer programmes such as an error being made or the programme shutting down) are situated in the transition from hearing to comprehending. Here we can see again that methods and concepts from applications can be blurred by appealing to various listening modes or transitions amongst them at the same time.
2.2. Circling through modes, reduced listening and sonification
A central point that Schaeffer makes is that we tend to cycle through the direct listening modes, and that information emerges through changing between them. Reflecting on a learning process associated with sonification, or more generally speaking when acquiring auditory expertise, our experience suggests that this is correct for the learning processes involved in using sonifications and auditory displays. The doctor who is in training and who diagnoses a patient's heartbeat through (acousmatic) auscultation perceives, first, that there is an extraordinary sound and hears that this sound has a recognisable structure. Then the doctor starts listening for the possible cause and will later recognise it by its name (comprehending that it is a specific heart murmur). After having reached a certain level of expertise, the doctor might first comprehend that it is a certain disease and than have to make an effort to hear and listen to it in order to differentiate the stage of a medical condition, (see Chion Reference Chion1983: 22).
Similarly model-based sonifications, although conceived mostly for their indexical link to the data, require some conceptualisation of the data space and the virtual physics of the sound-producing dynamics. Therefore, hearing and also comprehending are involved. An example would be to hear and comprehend two pitch-separated clusters of data-points in a data-sonogram. This illustrates that, despite their universality, the perception of causalities in a physical sense requires expertise through learning processes and does not happen naturally.
In fact, according to Schaeffer (Reference Schaeffer1966: 343), ‘it is the swirl of intentions that creates connections or exchanges of information’. In this sense, Schaeffer's reduced listening mode – and its correlate, the sound object – is an attempt to listen to and describe the sound for its own sake and is, if understood as consciously circling through listening modes, a prerequisite for auditory skill acquisition. The sound object has been critiqued as an ahistorical essence of sonic material by Kane (Reference Kane2007), and Schaeffer's phenomenological method and his use of the term ‘intentional’ by Kim (Reference Kim2010). However, for sonification, reduced listening as an attempt to lift habituation and conditioning offers the possibility to free the sound from a first degree of signification. Its significance for descriptive display purposes – for instance, exploratory data analysis – lies in its attempt to circumvent the complexity-reducing mechanisms of our perceptual and cognitive apparatus, thereby potentially lifting cultural and experiential biases in the process of linking sonic structures to data properties, which corresponds to the oscillation between description and identification when perceiving and interpreting sounds.
2.3. Musical versus everyday listening and sound making
In auditory display and sonification research, listening modes have been mostly discussed in relation to the ecological approach introduced by Gaver (Reference Gaver1993), who suggests everyday listening as a complementary research agenda to musical listening. Interestingly, Gaver makes no reference to Schaeffer's theoretical work, which precedes his own by almost three decades. He seems not to have crossed the discipline's language barriers. The difference in their approach is to some degree of a practical nature: while Schaeffer's focus is listening modes with respect to sound objects as potential material for composition, Gaver discusses sound as an information carrier in auditory display and is interested in a bottom-up algorithmic synthesis of everyday sounds with the goal to gain access to salient and meaningful synthesis parameters. Compared with Schaeffer's listening modes, everyday listening does at first not differentiate between listening for indices and causes and the interference from the comprehension of extra-sonorous signs.
For interactive sonifications, it is interesting to consider that listening to sounds with Schaeffer's phenomenological approaches is strongly tied to making sounds. Schaeffer introduces the term musicianly listening (see Chion Reference Chion1983: 39), which must not be confused with Gaver's musical listening. Although musicianly listening makes one think of music, it is a listening intention towards any audible action or interaction, fostered through an attitude of preparedness to focus on sonic details (see Chion Reference Chion1983: 36).
Musicianly listening can be understood as a form of ergo audition (see Chion Reference Chion1998: 84–5), which describes the perception of sounds while making them. In ergo audition, Chion differentiates both the causal listening of the closed loop experience which has also been elaborated by Hunt and Hermann (Reference Hunt and Hermann2011) as well as the need to dissociate listening from making in learning processes such as in instruments similar to the musicianly listening mode.
With respect to interactive sonifications, the distinction between the abstract instrument's timbre and the timbre resulting from the manner in which it is played (see Chion Reference Chion1983: 53) also needs to be considered. Through the process of musicianly listening, the instrumentalist – or for our purpose the user – acquires her skills by learning to distinguish between both. The relevance of this distinction for interactive sonification lays in the fact that, depending on the display purpose, the emphasis is either (inspired by cybernetics) on closing a feedback loop (i.e. on the player's timbre and thereby on focusing on interaction) or on exploring the data, which corresponds to engaging with the instrument's timbre. The latter requires being able to abstract from the audible effects of interaction.
2.4. Recent contributions to listening modes
Since the beginning of auditory display research, listening modes have arisen in only few contributions; concepts from the field of concrete music have barely been integrated. However, interest in this topic has increased recently. One recent contribution by Vickers (Reference Vickers2012) proposes an extension of the direct listening modes in order to accommodate complementing modalities and interaction. The organisation of these additional listening modes is inspired by Schaeffer's objective/subjective and abstract/concrete criteria.
Tuuri and Eerola (Reference Tuuri and Eerola2012) give an alternative embodied cognition-centred conceptualisation of listening modes. Their approach is based on preceding publications (see Tuuri, Pirhonen and Hoggan Reference Tuuri, Pirhonen and Hoggan2009) which critique a simplistic idea of multi-modality based on the number of sensory input channels, each contributing to a detached symbolic representation of the perceived. Instead, the authors argue for a non-symbolic a-modal completion of uni-modal stimuli, which challenges the idea that sound in auditory display is a direct signification. The embodied cognition approach, where sounds are perceived for their action potential, incorporates the concept of the gestural sonorous object by Godøy (Reference Godøy2006).
Despite the hierarchical organisation of the listening modes, Tuuri et al. emphasise that each mode can be activated independently and simultaneously: the experiential mode contains the most basic listening modes, reflexive and kinaesthetic, as well as connotative modes (action–sound–object, action–sound–intersubjectivity, action–sound–habit). The denotative mode encapsulates the indexical aspect of listening, which is subdivided in categories that span from source orientation (causal binding) to the orientation towards the context. The reflective modes include reduced listening, which is a conscious turn towards the sound, as well as critical listening, which as Tuuri et al. argue involves aesthetic judgement of the sound. The reflective modes are, as Tuuri et al. point out, premeditated listening modes and involve higher cognitive structures. This new scheme of listening modes convincingly diversifies connotative and denotative aspects and conceptualises, with the kinaesthetic mode, the bodily basis of meaning creation.Footnote 2
3. Comparing the different theoretical frameworks
The most significant difference between the approaches is that Schaeffer's phenomenological account of listening strictly tries to avoid any acoustic explanation of what we perceive, as his core conviction can be summarised as listening being everything but a passive mode of reception. This leaves the semiotic modality of a sound - its sign nature - tied to the listening intention. Gaver's approach is motivated by the desire to operationalise sound for HCI, to influence perceptually salient features by controlling the physics of the sound source. This is the reason why his – as he states himself – preliminary semiotics of sounds are tied to the mapping in the design process in the HCI context.
This tension between reflections on the signal and the semiotic modality of sounds referring to the data can also be found in the proposed hierarchy of indexicality of sonification methods by Vickers (Reference Vickers2005) who suggests the most indexical being audification followed by Model-based sonification and the least indexical being parameter-mapping sonifications. Whilst it is true that for exploratory data sonifications indexicality needs to be understood as how the sounds refer to the data substrate, indexicality remains first and foremost a listener category and less an attribute of the sonification method.
It is interesting to see how the first accounts of listening modes by Schaeffer and later Gaver relate to each other and where Gaver's modes are located in Schaeffer's abstract/concrete and subjective/objective grid. Vickers identifies everyday listening as being primarily objective and hence equates it with écouter and comprendre. Tuuri and Eerola (Reference Tuuri and Eerola2012) emphasise the similarity of musical listening with reduced listening as turning towards the sound. The subjective abstract listening mode hearing tends to disappear in both the HCI centred scheme of Vickers (Reference Vickers2012) and the embodied cognition-centred approach of Tuuri and Eerola (Reference Tuuri and Eerola2012). Reduced listening being at first a synthesis of hearing (entendre) and perceiving (ouïr), preserves entendre as a more reflective listening mode.
Combinations of the listening modes from Tuuri and Eerola (Reference Tuuri and Eerola2012), can be understood as specific listening intentions or styles described by Chion and Schaeffer. For instance, the kinaesthetic basis projects into the empathetic and the semantic listening, thereby enabling a listening for the messenger's intentions. The action–sound–object, in combination with the functional listening mode, captures well what Chion named ergo audition, which is exploited in auditory display as closed loop feedback – in other words, movement sonifications in training contexts. Also, musicianly listening can be understood as an activation of the connotative action–sound–object, causal and reduced listening.
Vickers (Reference Vickers2012) ironically points out that the spellchecker of the text editor he uses always insists on replacing sonification with signification, taking that as a sign to exclude the non-signifying reduced listening as a functional mode for auditory display. This is somewhat in contradiction with other auditory display literature. For instance, in the sonification design space map by deCampo (Reference deCampo2007), the goal is to find ways to design potential sound objects, which are by definition the correlates of reduced listening. As we will elaborate later, the functionality or efficacy of reduced listening for sonification depends on the purpose for which the display is designed. So, within the context of sonification, we propose thinking of reduced listening as an iterative nested approach, which helps to discover indices in a sonic structure beyond the first level of signification.
What can be found in all conceptualisations about listening modes are more or less explicit dichotomies, either by them proposing only two different listening modes, like Gaver, or by organising the four direct listening modes according to dichotomic criteria such as abstract/concrete and objective/subjective, as proposed by Schaeffer. The new scheme by Tuuri and Eerola (Reference Tuuri and Eerola2012) is the least binary approach in that they provide a rich multi-faceted hierarchy. However, even here, the reflective listening intentions (reduced and critical) seem to be categorically different from the action-oriented embodied modes. The nature of their relationship – and its relevance for interactive sonification – may well be described by the German word aufhören, which illustrates that the focus can be shifted towards perception to the point that action is interrupted. Aufhören means to deliberately discontinue an activity, the root of the word being to hear. The Duden (2001, article: hören) definition suggests ‘aufhorchend von etwas ablassen’ (to discontinue something in order to listen) as the origin of the word. Although this interruption might be due to sounds that do not originate from the interaction, it illustrates the shift from the closed action and perception loop towards the reflective oscillation between description and identification of audible percepts. The effort of musicianly listening in interactive sonifications is hence to maintain or initiate the action, despite the focus on perception, or to keep the focus on perception, despite the need to cause the sound through action.
As far as the embodied cognition-centred approach is concerned, the fact that the reflective modes involve higher cognitive resources can be understood as offering higher levels of abstraction from first level significations. As Tuuri and Eerola (Reference Tuuri and Eerola2012) indicate, reduced listening is a highly premeditated activity. In this sense it more emphasises thinking than doing, and becomes potentially detached from the immediate meanings of the connotative embodied levels, which offers the possibility to engage with the unknown, for instance when exploring the complex sound of sonifications which aim at representing complex data relations.
4. Between Phenomenology and the Signal
When investigating the sense of listening, science focuses on quantifiable signal properties and their behavioural, neurobiological or psycholinguistic correlate. The sonic arts have turned towards the phenomenology of listening from a philosophical perspective. With respect to sonification, both approaches contribute in a complementary way: the main advantage of scientific findings is that they can sometimes be operationalised or they can provide upper and lower limits of some perceptual dimensions within which the auditory display needs to be designed.
As progress is made in mapping the auditory cortex and auditory pathways, listening modes might find their neural correlate and hence a scientific explanation. In fact, the taxonomy of listening modes proposed by Tuuri and Eerola (Reference Tuuri and Eerola2012) is supported with ample references from the neurobiological literature. It is beyond the scope of this article to discuss in detail the neurobiological correlates of listening modes. However, it is worth mentioning that recent studies based on functional magnetic resonance imaging (fMRI) by Lewis, Talkington, Tallaksen and Frum (Reference Lewis, Talkington, Tallaksen and Frum2012) shed some light on the neurological basis of the scene-like or object-like status of sounds in auditory scenes, which might be partly identified with the perceiving listening mode, ouïr. Interestingly, Lewis et al. report a dependence on top-down task demands. This dependence supports the view that while sonic substrates might have the potential to evoke sets of naturally appropriate listening intentions, specific listening intentions among these sets need to be invoked from the top down. Within the taxonomy of Tuuri and Eerola (Reference Tuuri and Eerola2012) the hierarchical order suggests that the higher a listening mode is located in their scheme the more it can and needs to be consciously invoked (witness the premeditated reflective listening modes, for example).
5. Listening modes and display purposesFootnote 3
The integration of sonification in interactive and multi-modal displays confronts the designer of auditory displays with an ever-growing number of elements to consider and integrate during the design process. The necessity to provide guidelines and a better theoretical foundation for integrating interactivity and multi-modality was identified by Hunt and Hermann (Reference Hunt and Hermann2011: 295) and Frauenberger (Reference Frauenberger2009: 167). Here we propose integrated design guidelines for sonification in applications with multi-modal and interactive elements and will focus on the design aspects of interactivity and repetition. These are guidelines, rather than rules, and take a diagrammatic form in Figure 1, which provides an overview of the central issues in the design process.
Design aspects for multi-modal and interactive sonifications adapted from Grond (Reference Grond2013: 150); see Section 5.
When conceptualising an auditory display system, we propose, it is necessary or helpful to first identify how the display's normative or descriptive purpose can connect with the user's listening intentions. The term descriptive is inspired by Grimaldi and Engel (Reference Grimaldi and Engel2007), who argue for the relevance of descriptive science, which creates taxonomies and systems based on the identification of forms and shapes proper to the phenomena under study. In essence, this is the idea of the sound object and its typology and morphology. The descriptive purpose also parallels the oscillation between description and identification in the sound object through reduced listening, which for sonification links sound features with data properties.
Commonly identified advantages of sonification, such as transmitting information on a sub-symbolic level, or the notion of sound as conveying data holistically and having the potential to communicate or represent complex data structures,Footnote 4 can be identified with the descriptive axis. Here the sound of the sonification is meant to make us think. One is reminded of Kepler's wish ‘that the intellect articulates what the ear would naturally have to say’ (Reference Kepler1967: 134).
On the other side of the axis, we find a normative display purpose where the sound is meant to inform the user about well-defined events or actions. Here auditory display has advantages in being eyes-free, having a high temporal resolution and not always requiring a directed focus and orientation. On this axis, the auditory display system has an emphasis on feedback (control and conditioning of actions in an HCI context), and data sonification serves a normative purpose because the focus is less on the sound itself than on whether sound helps to achieve a predefined or desired goal. For a historic overview on sound as a corrective means see Schoon (Reference Schoon2012). Here, the reduction of the totality of the sonic percept to features matters less, as long as they are perceived and help to optimise the action. Sound is meant to help to condition us through augmented perception.
For the associations of listening modes with display purposes the taxonomy proposed by Tuuri and Eerola (Reference Tuuri and Eerola2012) is well suited, as it is hierarchically ordered. Here the most normative purpose would be an auditory alarm, which needs to cater at first to reflexive listening modes. Alarms, however, also illustrate the multiplicity of purposes, since they need to convey information beyond the initial reflexive reaction. A descriptive purpose, such as exploratory data analysis, corresponds to reflective listening modes and here more specifically to reduced listening.
Figure 1 is vertically organised according to the two axes, descriptive on the left and normative on the right. This diagram was conceived to support the reflection on repetition, multi-modality and interactivity and their roles for the two poles of display purposes.
A sonification which serves a descriptive purpose is related to the practice of reduced listening in that it supports reflection on sonic details and structures as a prerequisite to relate them to data properties. Here we also find a possible connection between the notion of aesthetics and the semiotics of sound in a sonification context, where the question of aesthetics extends beyond the mere design for pleasantness – in other words, beyond a balance between beauty and function, or its associative potential. In reduced listening, the sound does not take on at first a referential but rather an ‘aesthetic function’.Footnote 5 Sound does not stand as a sign for something else but is first and foremost perceived for itself. In this sense, designing for a descriptive display purpose understood as facilitating the perception of the sound object adopts an aesthetic dimension. This is not in contradiction to its consistent link to the data substrate, quite the contrary: reduced listening as an aesthetic turn towards the sound is a necessary prerequisite after which less biased references to the data can be discovered.
Since listening intentions are a listener category, they cannot be directly influenced like sound synthesis parameters. In this sense, the evoked potential, understood as the reaction to salient sound properties, needs to be put to use for normative purposes. For descriptive purposes, as in exploratory tasks, the invoked potential needs to be raised, which is to create auditory display systems that allow engagement in premeditated listening intentions. The elements of interactivity and repetition shown in the diagram in Figure 1 are discussed in the next sections.
5.1. Repetition
Perception as a processing of sensory input always happens in time. In this sense even still images or scientific plots change as we study them. However, we want to maintain the distinction between time-based and persistent media, for a number of reasons. Due to the fleeting nature of sounds as signs that evolve in time, they cannot coexist next to each other in the same sense as two static visual signs can. If sounds are played at the same time, they can merge, split, highlight certain aspects of their differences, mask others or even cancel each other. While two co-occurring visual signs will influence each other, their persistent nature allows for a re-investigation and re-evaluation of the aspects in which they differ or where they are the same. For sound, re-investigating and re-evaluating depends inevitably on repetition. Or, as Chion (Reference Chion1994: 30) put it for reduced listening: ‘the descriptive inventory of a sound cannot be compiled in a single hearing. One has to listen may times over, and because of this the sound must be fixed, recorded’.
Grond and Hermann (Reference Grond and Hermann2012) assert that repetition is also an aesthetic choice favouring engagement with sound because it creates more than just one unique event, which, due to its singularity, can only fascinate but cannot be understood. Paradoxically, repetition can induce the perception of both sameness and difference in sound. Sounds can differ in repetition as different listening modes set in, reducing sound to events, objects, indices or features. In this process, direct listening intentions can also become exhausted and as a consequence the formation of a more persistent sound object can emerge. Hence, for a sonifier, in the design cycle one needs to pay attention to which option is more likely to be engaged by the listener through the context – other modalities or interaction – in which sonifications are used.
It is interesting to recall that the sound object was discovered partly through the repetitive experience of the closed groove (see Chion Reference Chion1983: 13). For sonifications it is equally important to analyse how technology can help to unlock the effects of repetition. Repetition can be either rooted in the data structure, through repetitive data features, or realised through repeated playback or through interaction. For the latter, the degree of sameness depends on how the action of the user translates into sound and if the data-driven aspect of the sound remains recognisable, a distinction that we will discuss later in the context of design guidelines for interaction. With respect to repetition, we suggest the efficacy of paying attention to the following points for sonifications, which serve a descriptive purpose:
-
• Data: Identify the potential for repetition through repetitive structures in the data. Evaluate how sonification can lead to recognisable sound objects that exhaust direct listening intentions, but at the same time provide noticeable audible contrast for variations in the data.
-
• Interaction: Identify the potential for repetition through repeated interaction. Does the interaction device allow the sonic experience to be repeated?Footnote 6 Evaluate how the singular instance of a sonic interaction relates to all possible sonic interactions with the data set or subset, and whether stable and comparable sound objects have the potential to emerge.
-
• Playback: The simplest way to repeat is to re-trigger the playback of the sonification. This can offer interesting possibilities together with recorded interaction data. This kind of repetition through playback can offer a more reflective approach towards the sound and its cause for skill-learning situations beyond mere conditioning.
-
• Reproduction: The listener can also achieve repetition through imitation. This applies explicitly for vowel-based sonifications,Footnote 7 but also implicitly for all sounds that evoke forms and gestalts that relate to embodied listening modes. (See the notion of the gestural-sonorous object by Godøy (Reference Godøy2006).) In the latter case, the sound cannot necessarily be reproduced, but the images of effort and chunking can be re-invoked by the listener. Strictly speaking, this does not lead to sound objects, for which the sound needs to be fixed. However, imitation might help to exhaust specific listening intentions, while still taking advantage of the embodied cognition aspects as means of representing and re-evaluating the sound.
5.2. Interaction
As discussed earlier, the concept of the sound object and listening modes were conceived together when considering sound making. In this vein, we propose a listening-centred approach towards interactive sonification, which is inspired by the notions of ergo audition and musicianly listening. We suggest organising interactive sonification according to the following three categories of sonic interaction: sonic exploitation, sonic exploration and sonic expression. All three categories are schematically compiled in Figure 2.
Modes of sonic interaction taken from Grond (Reference Grond2013: 155); see Section 5.2
Sonic exploitation is the use of specific characteristics of sound in order to support a task with a specific goal. This category corresponds to direct listening modes, where sound is used to provide information about interaction objects and causes. In a normative sonification task, sonic exploitation means reducing sound to features that support maintaining the feedback loop, by being (a) informative for the action and (b) extractable from the sound with the smallest possible effort. Although a real-time closed-loop feedback strikes a balance between perceiving and acting, in so far as the interaction data are fed back to sensory input modalities, an emphasis is put on perceiving in order to act. In an auditory closed loop, the distinction of data and interaction can vanish.
Sonic exploration corresponds to the idea of musicianly listening. Here the emphasis is put on acting (conscious and not reflexive) in order to perceive and consequently to reflect on what was perceived. Sonic exploration is like musicianly listening tied to an attitude of preparedness and a conscious turn towards the plethora of audible features of a sound. At first, these features do not signify the interaction but are discovered in the process of asking what is actually perceivable. Only after this process of describing the perceived, sonic features are identified with physical aspects of the interaction and the nature of the sounding object. For musical instruments this corresponds to the exploration of the potential of the instrument for musical expression; here the sonic features need to be referred/directed back to the interaction in order to be able to reproduce the desired sonic result in a controlled way. In sonifications, the perceived sonic features need to be understood as indices to data properties.
Sonic expression is, in our opinion, a musical category, which is relevant for new instrument design. Sonic expression comes after the process of sonic exploration if interaction and sound synthesis correlate in a way that is not only consistent but also musically interesting. (See Hunt and Wanderley (Reference Hunt and Wanderley2002) for considerations concerning the mapping between action and sound.) Expressivity in interaction has an intentional and idiosyncratic element to it and shadows the consistent connection to data, by shifting the focus onto the sound-producing gestures away from sound-producing causes, the link between the sound and the data. Outside the context of interactive sonification, sonic expression can very well be used in auditory display. By representing intentions, it can communicate internal states in the context of HCI, as has been explored for vowel-based sounds by Tuuri (Reference Tuuri2011).
In Figure 2, sonic expression and sonic exploitation are at the bottom of the triangle and dominate data because the resulting sound either communicates an intention or represents external objects and causes related to the interaction. For both cases, the option to record the interaction data for playback, as discussed for repetition, is worth considering in order to move towards the descriptive corner of the triangle on the top. We chose this triangular organisation in the diagram to highlight that there is no trivial distinction that can be made between musical sounds and auditory display. An interactive sonification with a normative purpose seems particularly incompatible with musical goals, but at the same time in both, in sonic expression and in sonic exploitation, sound takes on the role of a medium and closes loops, contrary to sonic exploration, where sound stands at first for itself.
In comparison with closed-loop interactive sonifications, where the interaction becomes data, in sonic exploration the listener tries to distinguish both. This is important because only the instrument's timbre is carrying the information about the object – that is, data properties. The player's timbre belongs to sonic expression. If the timbre cannot be separated from the instrument, exploratory interactive sonification makes the data disappear and leaves the user just with sound making. With respect to interaction, we suggest focusing on the following three points in an auditory display system with a descriptive purpose. The last two points need to be considered in light of the earlier comments on repetition:
-
• Sonic exploration: Analyse how interaction supports sonic exploration, the turn towards all audible features of the sonification, rather than just representing the interaction itself. This point is very general and provokes thought about which audible aspects connect with the interaction as its cause, and if they possibly detract from an attitude of preparedness towards listening. Which aspect of the sound refers to which aspect of interaction (interaction quality, interaction progress, sound localisation and proprioception, etc.)? To which listening mode between the reflexive and reflective hierarchy are they most likely to appeal? The design goal is acting in order to perceive.
-
• Player's and instrument's timbre: Attention needs to be paid to the distinction, borrowed from Schaeffer, between the audible effects of the interaction versus the sound originating from the data. In this context, sonic or other modalities can be considered to introduce the distinction between them, always evaluating whether it enables reflective rather than reflexive listening intentions.
-
• Order of temporal evolution: Focus on how sound objects can be created so that they are retained in memory and comparable. The question of how interaction intervenes with information that is contingent with the temporal order of sounds in the sonification needs to be addressed here. If the interaction is continuous,Footnote 8 like moving along a slider, a discrete sonification is worth considering. If the interaction is discrete, such as with triggers, temporally extending sonifications can be considered.
6. Conclusion
In an attempt to support the theoretical development complementary to sonification methods and application areas, we have presented sonification design guidelines for data exploration, which are rooted in the phenomenology of listening. In order to recast the conceptualisation of sonifications based on listening modes, we proposed to evaluate if the display in which sonifications are used serves a normative or a descriptive purpose. Listening modes that match display purposes can be identified from the recently revised taxonomy by Tuuri and Eerola (Reference Tuuri and Eerola2012). We suggest adopting this taxonomy as it allows describing specific listening styles such as musicianly listening and ergo audition as combinations of listening modes. In the future, finer distinctions of display purposes between the normative and descriptive poles could be identified and described. For monitoring applications, for instance, it might be interesting to look into how attention levels in listening such as listening-in-search, listening-in readiness and background listening (see Truax Reference Truax2001: 21 and 24) relate to listening modes and how they match the monitoring task categories discussed by Vickers (Reference Vickers2011).
Although it is beyond the scope of this paper to discuss the aesthetics of sonifications in detail, we propose to differentiate the notion of aesthetics within the context of sonification by distinguishing its purpose: for normative purposes, the concept of information aesthetics seems useful to address questions related to user experience. For descriptive purposes – for example, sonification for data exploration – aesthetic considerations are more related to phenomenological concerns, and to whether appropriate reflective listening intentions can be invoked when listening to sonifications in interactive and multi-modal auditory displays. By referring to Mukařovský and his notion of the aesthetic function of a sign – an idea which is structurally similar to reduced listening – we propose a bridge to design thinking which is often approached from semiotics. Invoking reflective listening intentions can be compared with the aesthetic attitude and with Bullough's notion of ‘putting the phenomenon, out of gear with our actual practical self’ (see James Reference James2013). We propose both as a design for openness, where the sound of a sonification is not trapped and reduced by immediate signification.
For sonifications in an interactive and multi-modal auditory display, the aesthetic judgement of the designer with respect to the whole display is required in the design cycle so that an aesthetic attitude (i.e. the invocation of reflective listening intentions) becomes feasible for the user. Here, design must not allow the sound to get stuck in simply pointing to other modalities, the action that caused it, or the visual event that happened at the same time. Only then can sonification potentially point beyond the immediate and obvious and help to explore complex data structures.
In order to complete these guidelines, complementing modalities, such as audiovisual displays, also need to be considered in more detail together with the embodied dimension of listening. While we tried to disentangle sonification methods and applications from the phenomenological aspects of listening, it will be interesting to consider how parameter-mapping sonifications can be designed such that they provide a sonic substrate that allows the listener to engage through one or several listening modes appropriate to the given task. Here the gesture-encoded sound model by Worrall (Reference Worrall2010), for instance, suggests a promising approach inspired by embodied cognition.
However, even if some aspects related to listening modes can be operationalised, we believe that listening remains the ultimate activity in the design process to judge the usefulness of the result. With the presented guidelines, we hope to have provided a tool for systematic reasoning in the design cycle for the context of interactive sonification with a descriptive purpose, in order to support the designer's ears in the judgements they need to make.
