2. CHALLENGES

The success of the sonification process in transmitting information depends on the optimal application of the four components – data, mappings, musical language and emotional content. The learning effort required to understand a sonification can be the marker of its success. The most widely used sonification method is Parameter-Mapping Sonification (PMSon), where a data parameter is mapped to a sound parameter. The choice of mappings is the most critical decision in the compositional process because there is ‘a very large set of possible mappings’, but only ‘a notoriously small subset of perceptually, or cognitively, valid mappings’ (Roddy and Furlong Reference Roddy and Furlong2014: 70). We must choose ‘valid’ mappings which require little learning effort from the listener. For instance, an intuitive mapping could produce the hearing of rising data values as rising pitch or increasing tempo. Furthermore, the ear has differing change detection thresholds for each musical parameter, so not every mapping is equally useful in transmitting small changes in data. Sound parameters such as ‘pitch, tremolo rate, rhythm and attack time’ have a low detection threshold while ‘volume … number of harmonics or envelope shape’ have a higher threshold (Ballora Reference Ballora2014: 31). An understanding of psychoacoustics is useful when choosing effective mappings for a dataset.

The choice of musical language can also have a significant impact on the understanding of a sonification as it must be suited to the type of data. For example, the use of a diatonic scale might not be suitable for a dataset with continuous values, and a continuous frequency scale might offer a better representation of the data. Spatialisation is an under-explored musical parameter despite our ability to locate sounds precisely in our surroundings. These two examples point to the potential of electroacoustic music and its associated language for sonification design. Yet, most sonifications still use a limited range of parameters and musical language; they are generally limited to diatonic scales and mappings to pitch.

Recently, sonifications have received a lot of media attention. The discoveries of the Higgs Boson particle and gravitational waves were shortly followed by a sonification promising the ‘sound of…’ each discovery. Unfortunately, most popular examples are poor examples of the method because they do not follow a systematic mapping process or transmit information in an intuitive or clear manner. The 2012 sonification of Higgs Boson data by Dominico Vicinanza is a case in point. It maps the dataset of a particle collision experiment to a diatonic scale (from E3 to C7); lighter to heavier particles are mapped as lower to higher pitches. The Higgs Boson particle (which is heavier) is therefore heard as the highest notes in the second bar (Figure 1). However, the mappings are sometimes arbitrary in order to ‘fit’ continuous values into discrete pitch categories:Footnote ² ‘the … diatonic melody is a pure artefact’ (Pesic and Volmar Reference Pesic and Volmar2014). The sonification lacks musical interest and misses many opportunities to transmit its information more clearly. The listener would hear the Higgs Boson particle better if the data has also been mapped to rhythm and loudness in order to highlight changes.

Figure 1.

On the left, Higgs boson data (credit: ATLAS collaboration; copyright CERN), showing the number of events observed plotted against the invariant mass in GeV. The lower register shows the background as a straight line, highlighting the appearance of the ‘bump’ at 126.5 GeV, interpreted as the invariant mass of the observed Higgs particle (Pesic and Volmar Reference Pesic and Volmar2014). On the right, Sonification by Domenico Vicinanza, from LHC Open Symphony (Pesic and Volmar Reference Pesic and Volmar2014).

The lack of quality in many examples that are featured by media outlets has a number of causes. Most of these sonifications are designed by scientists with little understanding of aesthetics and cognitively valid mapping strategies. Furthermore, the popularity of sonification has led to a ‘conversion hysteria’ (Connor Reference Connor2013: 137), whereby datasets are arbitrarily chosen without considering whether the data, mappings and musical language are suitable to convey the intended message and emotional meaning. The existence of low-quality examples is exacerbated by the willingness of the general public to accept these displays as true representations of the data, because they lack an understanding of the display method. This shows that a learning effort is required – from sonification designers and listeners – in order to raise the quality and knowledge of sonifications.

3. MUSIFICATION

Sonification has been increasingly used by composers and science communicators. It can provide a ‘wow! factor’ and an entry point to scientific and musical disciplines for a general audience (Ballora Reference Ballora2014: 31). Musical practices in sonification are sometimes called musification, although the definition of the term varies. As used here, musification is both a sonification intended for ‘artistic creation’ (Grond and Berger Reference Grond, Berger, Hermann and Neuhoff2011: 392) and a sonification which is ‘subjected to a set of musical constraints’, whether for functional or artistic purposes (Williams Reference Williams2016). A musification should function equally well on a functional and an artistic level by transmitting information clearly while working as a self-contained piece of music. Natasha Barrett’s work on sonification explicitly deals with this concept even if the term of ‘musification’ is not employed. Her piece Aftershock (2011) – written in collaboration with geologist Karen Mair – was created ‘for science (as public outreach) and for music as an exploration of new temporal structures, sound-worlds and as new material for composition’ (Barrett and Mair Reference Barrett and Mair2014: 15). Barrett is one of the few composers who use electroacoustic compositional methods in sonification.

The use of musical aesthetics in sonification design is crucial because functional and aesthetic perceptions are interlinked. Usability and user satisfaction are related to aesthetic attributes. Attributes contributing to user satisfaction might include pleasantness, loudness, noticeability, clarity and integration (Leplâtre and McGregor Reference Leplâtre and McGregor2004). Aesthetics should not be considered for the sake of beauty, but because the ‘simple, that is, the beautiful, brings understanding more readily’ (Barrass and Vickers Reference Barrass, Vickers, Hermann, Hunt and Neuhoff2011: 158). In the context of sonification, the idea of beauty in Nature supports the claim that improved aesthetics supports functionality in the design of auditory displays as it can increase acceptance and reduce ‘display fatigue’ (Kramer Reference Kramer and Kramer1994: 53).

The information transmission process can be supported by using emotional connotations and extra-musical meanings that emerge from the mapping process. While it is debatable whether any musical features and their perception are truly universal, some elements – for example, major and minor chords signifying happy and sad – have widely accepted connotations. These can be used to associate emotions with datasets through specific mappings. Furthermore, it is useful to use sounds or musical elements which can be intuitively related to the dataset. Different types of data should sound differently (McGee Reference McGee2009): it seems intuitive that data related to rivers should be mapped to water sounds, while geological data should be mapped to pebble sounds, for example. An intuitive mapping between data and sound can lessen a sonification’s learning effort because the listener can easily discern which sound is related to which dataset. The use of recorded sounds can be particularly useful in this respect.

3.1. Some solutions to sonification challenges

Some of my work has sought to solve some of the compositional challenges described above. The acousmatic piece Blyth-Eastbourne-Wembury (2015) uses sea temperature data from the British coast. The average sea temperature has risen since the 1970s and the sonification of this information made the difference between the pre- and post-datasets audible (Figure 2). I mapped the temperature values of each month for the two sets to sine waves at frequencies between 200 and 600 Hz. When playing the two values for each month, the mapped frequencies were close enough to create beating dissonances. The use of a continuous frequency scale rather than a diatonic scale, for example, created close dissonances which clearly signalled a difference between the data values. The dissonances add an emotional component to the sonification as we tend to perceive them negatively or even as alarming; this is the intended emotional response to data describing climate change. The different sonifications created from the data are embedded within a maritime soundscape which has an intuitive connection to the nature of the temperature data (Bonet, Kirke and Miranda Reference Bonet, Kirke and Miranda2016).

Figure 2.

Monthly averages of seawater temperature in C° at Eastbourne for the entire data set and for 1971–2000 (Joyce Reference Joyce2006: 64).

Many possible musical mappings are explored in The Sonification of Dark Matter (2016), which combines visualisations and sonifications of dark matter data produced by the SLAC National Accelerator Laboratory at Stanford University (USA). The complex and very large datasets were mapped to a variety of parameters used in electroacoustic music – including pitch, rhythm, tempo, loudness, panning, randomness, granulation and spectral density – to determine a hierarchy of useful parameters. These were categorised as primary and secondary auditory cues depending on their detection threshold. Primary parameters include pitch, rhythm and panning while secondary parameters include loudness and randomness. The hierarchy of musical parameters is a useful way of thinking about sonification design since musical pieces are constituted of musical parameters with different degrees of salience (the melody often stands out over the articulation, for instance).

The issue which has remained difficult to solve during the compositional process of Blyth-Eastbourne-Wembury and The Sonification of Dark Matter is that of the learning effort. Even though the mappings were carefully chosen to transmit a message and convey its emotional content, some explaining and learning was necessary to understand the piece fully. Sonification is an information transmission process, so it can be theorised within a communication theory framework such as the Shannon-Weaver Mathematical Theory of Communication. Once the communication process is defined in terms of interlinked components, we are equipped to optimise the information transmission and reduce the learning effort required.

4. SHANNON-WEAVER MATHEMATICAL THEORY OF COMMUNICATION

The Shannon-Weaver Mathematical Theory of CommunicationFootnote ³ describes the way in which ‘one mind may affect another’; and it includes music as a form of communication (Shannon and Weaver Reference Shannon and Weaver1949: 3). While the model was intended for the improvement of telecommunications, it has been applied to social sciences and can have interesting applications in discussing music. While it is of interest to any composer to understand the process by which a message might be transmitted to a listener, the model also offers important parallels to the sonification process. In fact, it helps demonstrate that the use of shared knowledge in the design of a sonification can improve the information transmission process.

Shannon and Weaver describe the communication process as a system with five components:

1. 1. ‘An information source which produces a message or sequence of messages to be communicated to the receiving terminal’ (Shannon and Weaver Reference Shannon and Weaver1949: 33).

   An information source acts as the Sender of a Message.

2. 2. ‘A transmitter which operates on the message in some way to produce a signal suitable for transmission over the channel’ (ibid.).

   A Transmitter which is the Encoder of the Message that becomes a Signal.

3. 3. ‘The channel is merely the medium used to transmit the signal from transmitter to receiver’ (ibid.: 34).

   A Channel is the Medium which transmits the signal.

4. 4. ‘The receiver ordinarily performs the inverse operation of that done by the transmitter, reconstructing the message from the signal’ (ibid.).

   The Decoder reconstructs the Received Signal as a Message.

5. 5. ‘The destination is the person (or thing) for whom the message is intended’ (ibid.).

   The Receiver gets the Message.

The sonification process can be described by this communication model (Figure 3):

1. 1. The sonification Designer (Sender) sends Information (Message) to a Listener (Receiver).

2. 2. The Mappings (Encoder) produce a Sonification (Signal).

3. 3. The Musical Language (Channel) is the medium through which the Sonification (Signal) is transmitted.

4. 4. The knowledge of the Musical Language and Mappings used (Decoder) decrypt the Sonification (Signal).

5. 5. The Listener (Receiver) receives the Information (Message).

Figure 3.

Shannon–Weaver Mathematical Theory of Communication applied to the sonification process (in italics). The terms in boxes describe the five parts of the communication process, while the annotated arrows describe the transmission between parts. Noise is an external influence on the channel which can distort the signal and the feedback describes the message returned from the Receiver to the Sender.

The original model also includes Noise which is any interference to the signal which creates ‘distortions’ or ‘errors in transmission’ (Shannon and Weaver 1948: 8). In telecommunications, noise is most likely to occur at the channel stage. Feedback denotes the return message from the Receiver to the Sender; it was introduced by later theorists (Chandler Reference Chandler1994).

Information can be defined as data coupled with context (Roddy and Furlong Reference Roddy and Furlong2014: 75); so, the Message consists of Data and Context. The equivalence of the Message as Information needs to be deconstructed further. The Message contains Data but needs context in order to become a Signal or Sonification which can be understood by the Receiver or Listener. The Context of the Information is contained by the Message, Encoder and Channel, all of which contain the Data, Mappings, Language and Emotional Content of the Sonification. In fact, these elements provide context to a dataset.

Let us reconsider the Shannon-Weaver model as applied to sonification by adding the elements of the sonification framework which contain context:

1. 1. The Designer sends a Message which contains Data. A dataset has a Context which needs to be transmitted. The emotional connotation of the dataset determines the Emotional Content of the sonification signal.

2. 2. The Mappings are chosen as to transmit the Data and its Context clearly.

3. 3. The Musical Language is chosen to act as an appropriate medium to transmit the Sonification.

4. 4. The Decoder understands the context provided by the Musical Language and Mappings to decrypt the Sonification.

5. 5. The Listener receives the Sonification and its Context and can understand the Data and its Emotional Content.

The communication process of the Sonification implies that the Sender transmits the Data and the Context of the Information, and the Receiver gets the Sonification and its Context. Where the Sender and the Receiver have a shared knowledge of the Context, the Signal can be decrypted without needing to learn the Context. A shared knowledge of the Mappings, Language end Emotional Content enables a more efficient information transmission process because it reduces the learning effort required. If the Receiver understands the Data, Mappings, Musical Language and Emotional Content, they can decrypt the Received Signal.

Let us now consider musical borrowing as a musical device which can improve the information transmission process by providing shared knowledge of Context, Mappings, Language and Emotional Content.

5. MUSICAL BORROWING

The compositional practice of musical borrowing is rarely explored in sonification or musification. Defined as the ‘deliberate evocation within a composition of a different musical work’ (Bicknell Reference Bicknell2001: 185), it draws on the composer’s and the listener’s reciprocal knowledge of an existing piece. The listener might not be familiar with the borrowed music but can still appreciate the new piece as a self-contained musical work (ibid.: 189). Familiarity with the original work will, however, increase the appreciation of the compositional process and extra-musical meaning of the new work.

Two recent examples of musical borrowing in sonification used the entirety of an original work, which can be defined as ‘modeling’ (Burkholder Reference Burkholder1994: 854). Data parameters are mapped to the existing sound parameters so that the data distorts the borrowed music. The first example is a sonification of the drop of the value of the pound Sterling after Brexit. The currency’s value (from 23 June to 7 October 2016) is mapped to the frequency of a pitch shift applied to a recording of the British national anthem, God Save the Queen (Belfast Telegraph 2016). The second example is a sonification of the predicted number of floods in US cities by 2045 (Keng Reference Keng2015). Positive deviations from current flood values result in a speeding up of the tempo of Vanilla Ice’s song Ice Ice Baby (1990). The two pieces have been featured in mainstream media, showing their potential for engaging the wider public with current issues through sonification. They also share some interesting characteristics: they use existing recordings of well-known songs on which to map data, consequently the data is readily understood by most listeners, and the sonifications produce a comical effect.

Little context is required for listeners to understand the meaning of the pieces because they know the original material and the transformations applied to it are obvious to hear. Their humorous or even grotesque character stems from the perception that the pieces are played ‘wrong’ or ‘badly’; they transgress their original style and performance context. Performances by the now defunct Portsmouth Sinfonia produced a similar effect. Founded in 1970 at the Portsmouth School of Art, it invited musicians with no or little musical or instrumental training to join and play Classical repertoire. It was not intended as a spoof or joke, rather, it invited musicians with no skill in their chosen instrument to perform classical repertoire (Wilsher Reference Wilsher2004). Their recordings have, however, attained cult comedic status because of the juxtaposition of near universal repertoire (e.g., Strauss’ Also sprach Zarathustra (1896) or Beethoven’s 5th Symphony (1808)) and musical incompetence. The link between music and humour is discussed below.

The use of recorded material for sonification is accessible because of the widespread availability of commercial audio software which can map data to sound. Both the God Save the Queen and the Ice Ice Baby examples are quite crude from a technical point of view. Artefacts produced by the audio transformations are clearly audible; God Save the Queen is pitch shifted rather than transposed, for example. This might be a desirable effect for information transmission because the artefacts highlight an unmistakable change in the music. However, this approach can be problematic in a musification where the composer also seeks to write a piece which achieves a professional standard of musicianship. Using recordings as borrowed musical material demands certain skills so that the resulting sonification meets expectations from a functional and artistic point of view.

An alternative is an ‘orchestral sonification’, where the mappings and transformations are applied to the original score rather than a recording. In the following piece, the score was composed ‘manually’ rather than by a computer program. Composers might prefer a manual approach to mapping an orchestral score as they rely on their musical intuition and orchestration knowledge. While a composer might be less systematic than an algorithm in their mapping strategy, they might also work more intuitively. Some of the available orchestration software might in due course democratise access to the creation of orchestral sonifications.Footnote ⁴

6. WASGIISCHWASHÄSCH

Wasgiischwashäsch (2017) is a sonification piece for chamber ensemble (Sound example 1) which maps climate change data to the score of Rossini’s William Tell Overture (1829). It is intended to exploit the piece’s near universal fame to optimise the information transmission process of the musification. It was premiered as an arrangement for chamber ensemble but can be re-orchestrated to suit other performance contexts. The William Tell Overture’s third and fourth movements (‘Ranz des Vaches’ and ‘March of the Swiss Soldiers’) are used and become Wasgiischwashäsch’s first and second movements. Two temperature datasets are mapped to the structure of the original score: the data parameters are mapped to the Overture’s musical parameters. Rossini’s piece was chosen for its popularity and also for its semiotic connection to Switzerland; William Tell is a Swiss folk hero and the ‘Ranz des Vaches’ is often considered Switzerland’s alternative national anthem.

6.1. Data

The data used – temperature temporal trends – shows the extent of climate change in Switzerland. These datasets were published by the MeteoSchweiz (Swiss meteorological agency) in 2015. The first graph (Figure 4) shows the average annual temperature in Switzerland from 1870 to 2010 as yearly values (bars) and as the weighted average (line). Values before 1920 remain below −0.5 °C despite fluctuations; after 1920 they never drop below −0.5 °C again. This signals the beginning of a continuous temperature rise (the slight decline in the 1950s should be disregarded as it follows a dramatic increase in the late 1940s). The values rise by 0.5 °C between 1920 and 1980, with some considerable positive differences in the late 1940s. After 1980, the values rise dramatically with the highest difference of about 1.25 °C in 2010.

Figure 4.

Deviation of average yearly temperature in Switzerland, 0.0 on the y-axis is the overall average temperature value. Years above average appear above the 0.0 and those below average below the 0.0. The black line is the weighted average (MeteoSchweiz 2015a).

The second graph (Figure 5) shows the deviation of the temperature in Switzerland from the global temperature average from 1864 to 2015. It reveals a growing difference of, for instance, more than 1.5 °C in 2015. Climate in Switzerland is changing at a faster rate than the world average (MeteoSchweiz 2016). This is partly due to the country’s alpine nature and could have catastrophic consequences: melting glaciers, avalanches, floods and the consequent effects on human population (MeteoSchweiz 2015a).

Figure 5.

In grey, the temperature deviations from the average in Switzerland. In black, the global temperature deviations from the average (corresponding to the years between 1864 and 1900) (MeteoSchweiz 2015b).

6.2. Mappings

The data is mapped to the score of the third and fourth movements of the William Tell Overture. The year to bar ratio was calculated so that the year 1920 would coincide with the end of the first movement of Wasgiischwashäsch. The average annual temperature rises above −0.5 °C and does not return below it in subsequent years; it marks the beginning of a warming climate. In the first movement, two years in the data are mapped to two bars in the score. In the second movement, two years in the data are mapped to five bars in the score. Each single year has not been mapped to the equivalent number of beats in order to not disturb the flow of the piece excessively by using constant changes in mapping.

For the first graph, the value of −0.5 °C is chosen as the ‘zero’, where the musical parameters remain the same as the original. Any deviation from this value results in changes in the musical parameters of the relevant bars. The parameters affected are: pitch, intervals, harmony (modes, modulation, atonality) and rhythm. Rises in temperature result in rising pitches for high registers and descending pitches for low registers; this eventually results in extreme registers at higher temperatures.

A deviation of 0.2 °C is mapped to a deviation of a semitone. In the first movement where the temperature lies below the mean, this translates as flattened notes, diminished intervals and chromatic runs (Figure 6). In the second movement, this usually results in sharpened notes and augmented intervals. Changes in rhythm occur in ‘Ranz des Vaches’, where some runs are mapped to more complex rhythms when higher temperatures occur. Some of these transformations are not strictly systematic, but rather based on compositional intuition regarding the required effect for the given temperature value. The higher-level meaning of the data, however, is clearly audible.

Figure 6.

Comparison of the beginning of the first movement of Wasgiischwashäsch.

In the second dataset, the difference between the global and Swiss temperature deviations influences the mapping of parameters such as tempo, dynamics and distance between registers. The ‘Ranz des Vaches’ remains at the original speed, as does the beginning of the ‘March of the Swiss Soldiers’. As the temperature difference rises, the piece’s tempo becomes slower. This first happens in bar 101 where ♩ = 120, before returning to ♩ = 152 in bar 137; this corresponds to the rise in temperature during the 1940s. The final tempo is ♩ = 40 which completely alters the character of the piece. The dynamics are mapped similarly, with a rise of temperature translating to quieter music. The end of the piece is pp with a diminuendo al niente. Again, the original is unrecognisable. An ‘incorrect’ mapping appears at bar 283 which has a sudden fff (fortississimo) indication. It serves a dramatic function within the structure of the piece as it introduces a slower tempo which is introduced by a long crescendo.

While clear mapping rules were laid out before the composition of the piece, some artistic liberties have been taken to emphasise some interesting features in the data and create a more interesting musical experience. Such adjustments are reasonable within the context of musification where the artistic quality of the work is important, although one might argue that it does not satisfy the definition of a sonification which should be systematic and repeatable with different data (Hermann Reference Hermann2008: 2). However, there is a case to be made for a certain freedom in mapping decisions where they might support the understanding of data; for example, by highlighting or introducing relevant passages. As described earlier, non-systematic changes in rhythm occur in the piece but they strongly help the transmission of the higher-level meaning of the data; some mappings based on compositional intuition improve the information transmission process.