1. Introduction
The production of non-representational moving images, be they captured on film or video or performed live by using colour organs or any other special instruments, often was and is motivated by the intention to create visual artworks that are as abstract as music, hence the term ‘visual music’ to denote this kind of moving images. In studying the different forms, it seems therefore appropriate to ask about the function of music in the production and aesthetics of visual music. (It should be mentioned beforehand that visual music does not have to be accompanied by music to be called ‘visual music’.)
As in every other time-based art form the central aesthetic feature of these artworks is – in my view – their structuring of reception time, or the way they tie the recipient's attention over the course of time. So, in trying to describe the aesthetics of non-representational moving images and concerning the role of music therein, we should first and foremost ask: how does the music accompanying those moving images contribute to their total – that is, combined – temporal structure?
In principle there are two possible ways of answering this question. We can ask about the role the music played in the production process: was the musical accompaniment chosen first and were the images structured according to it, or was it the other way round? And we can ask about the actual contribution of the accompanying music to the total temporal structure of the resulting piece in the eyes and ears of the recipient.
I will try to present here a taxonomy of the different roles that music can play in the production of non-representational moving images and of the different aesthetic functions it can have. This will also enable me to shed some light on the history of this genre. The guiding question in building this taxonomy will be: how close are the relations between the music and the moving images?
2. Taxonomy
2.1. Translations
2.1.1. (Semi-)automated translations of music into moving images
The closest connection between music and images can be achieved if one tries to translate a given music into moving images, respectively transcoding certain characteristics of the music into certain visual characteristics by some kind of automated or at least semi-automated process. Such translations of music typically lead to non-representational moving images. The earliest attempts to produce non-representational moving images fall into this category:
In his book L'optique des couleurs, published in Paris in 1740, Father Louis-Bertrand Castel envisioned a ‘clavecin oculaire’, a rebuilt harpsichord where each key-stroke would open a little window showing a certain colour (Castel Reference Castel1740). It is unclear if this audiovisual instrument was ever built since the only first-hand reference concerning actual performances is a letter by Castel himself (Jewanski Reference Jewanski2006: 152). The concept of music-translation that Castel tried to realise can be called ‘semi-automated’ because he mapped certain colours to certain pitches so that playing a certain note on the instrument would automatically show a certain colour.
Many of the inventors of so-called colour organs picked up this idea of mapping a certain colour to a certain pitch, which can be traced back to Isaac Newton (Optical Lectures, 1672), Athanasius Kircher (Ars magna lucis et umbrae, 1646) and even Aristotle (On Sense and the Sensible, I/3).
As one example standing for many others I mention Alexander Wallace Rimington (1854–1918), who built various colour organs and tried to give his system of mapping pitches to colours a scientific basis by relating the ‘ether vibrations’ (i.e. the wavelengths) of the different colours to the frequencies of the different musical notes (Rimington Reference Rimington2004: 265).
A prominent example of a semi-automated translation of music into moving images made with the means of (sound)film is Norman McLaren's Synchromy from 1971 (McLaren Reference McLaren2006b). In the 1940s McLaren and Evelyn Lambart developed a system of animated sound based on techniques invented by the German and Russian animators Rudolf Pfenninger and Nikolai Voinov in the 1930s.Footnote 1 They ‘made cards, each of which had a certain frequency of striations on them. Each frequency, when photographed onto the sound-track, was capable of producing a particular pitch. The volume could also be controlled’ (Dobson Reference Dobson2006: 167; see also McLaren Reference McLaren1950: 168). By that means McLaren produced a completely synthetic piece of music for Synchromy and he photographed the same sound cards onto the picture area of the film, only multiplying them at times, giving them different colours and moving them across the screen (Figure 1). But in principle in this film one can see and hear the optical soundtrack at the same time.
Still from Synchromy at 5′26′′, Norman McLaren, CA 1971, ©National Film Board of Canada, all rights reserved.
Of course the invention of the computer made it much easier to automatically translate music into moving images. An early attempt to do this was undertaken by Stephen Malinowski from 1985 on with his Music Animation Machine, a software-player for MIDI-files which also generates visualisations of the played tunes. The visual output of early versions of this software, which was further enhanced in later years, looked much like the graphical editor of music sequencing software like Steinberg's Cubase (Reference MalinowskiMalinowski n.d.).
For today's artists the software Jitter – a set of video processing extensions for the visual programming language Max – is, amongst others, a convenient tool to realise automated music-translations, but the principle of parameter-mapping still is at the core of this process (Jones and Nevile Reference Jones and Nevile2005). Like Castel and others the artist still has to decide which parameters of the chosen music – pitch, volume, tone – should manipulate which parameters of the visuals – colour, form, position and so on – in which way (Frank and LIA Reference Frank2010: 378).
Nowadays every user of a personal computer can easily and automatically translate any piece of music into moving images by starting the ‘visualisation’ feature of Microsoft's media player, the VLC player or the iTunes player. But in this cases the user can only choose from a fixed and limited set of parameter-mappings designed by the programmers of these features.
Concerning modern technological possiblities it is also necessary to stress the difference between production and perception: today's computers and software tools enable the artist to produce translations of music that are not recogniseable as such by defining translational rules that are so complex that the viewer/listener will no longer be able to connect the resulting visual events with the musical events causing them. An example is Chronomops by Austrian video artist Tina Frank from Reference Frank2005 – all visual parameters are manipulated in realtime by the soundtrack fed into 3D software that she has written (Pfaffenbichler Reference Pfaffenbichler2010: 318). But even after having learned this the viewer won't be able to parallel most of the movements in the image with certain events in the accompanying music.
2.1.2. Manual translations of music
Some artists also tried to translate music into moving images by applying traditional animation techniques and without the aid of some specialised machine or process. The purest example of a manual translation of music I have found so far was made by the Polish experimental filmmakers Franciszka and Stefan Themerson. Their last film, titled The Eye & the Ear, made in London in 1945 after their emigration, is an ‘experiment designed to use the medium of the screen to create for the eye an impression comparable to that experienced by the ear’ as the introductory titles of the film state.Footnote 2 It is made entirely with traditional filmmaking and animation procedures. Four songs by the Polish composer Karol Szymanowski are translated into moving images by using four ‘different methods of cinematographic interpretation’. While for the first and the last song representational images are used to illustrate the music and – to a certain degree – also the words of the lyrics, the second song is interpreted through a combination of representational and non-representational images, and the third part of the film is completely non-representational (Figure 2): like in the examples of music translations mentioned above a certain mapping of musical to visual parameters was fixed in advance ‘by assigning to each instrument a geometrical form which changes its shape according to the pitch of the note. For instance, each note of the violin is marked by the varying height of its corresponding symbol – a triangle’ (Themerson and Themerson Reference Themerson and Themerson2007). After that the translation was done according precisely to those rules.
Still from The Eye & the Ear, third song: ‘Rowan Towers’ at 1′11′′, Franciszka and Stefan Themerson, GB 1945, ©Themerson Estate, all rights reserved.
2.1.3. Translation vs. visualisation
Now, what exactly is a translation of music into moving images? In the narrowest sense it can be defined as moving images:
-
• where all visual events can be paralleled with events in the accompanying music, and
-
• where the concrete kind of parallelism, which means the parameter-mapping, stays constant throughout the piece.
This does not mean that all musical events have to be represented visually. There could also be used a translational rule that says, for instance, that only certain changes in pitch and only every second note of the musical piece should be represented on the screen. But the once established rules of translation have to stay constant throughout the piece so that we can talk of a music translation in the pure sense. The more the mapping of musical to visual parameters is changed throughout the piece the less the resulting visuals will represent the structure of the music but show instead their very own structure, consequently the less the resulting visuals can be called a translation of the music.
Therefore the third song of The Eye & the Ear by the Themersons is an example of a pure translation, whereas Synchromy by Norman McLaren is not, because McLaren added colours and multiplied the images of the optical soundtrack in a way that cannot be consistently paralleled with events in the accompanying music – the mapping of acoustic to visual parameters does not stay constant throughout the film.Footnote 3
By formulating this concept I do not make an aesthetic judgement of any kind. Changing the parameter-mapping can result in aesthetically appealing visuals while a ‘pure’ translation of music can – depending on the parameter-mapping chosen – result in very boring moving images. The purpose of my taxonomy is not to aesthetically evaluate but to conceptually differentiate between the different kinds of using music in combination with visual music.
I prefer to use the term ‘translation of music’ instead of the more familiar term ‘music visualisation’ for the simple reason that the latter is widely used: I propose to speak of a ‘translation’ only if the single recognisable intention of the visualist was to consistently represent certain aspects of the accompanying music. So, one might call the Disney film Fantasia (1940; Algar et al. Reference Algar2000) a ‘music visualisation’ but it is not a ‘translation of music’ in the aforementioned sense.
Music translations are typically the artistic achievement of the visualist as well as the composer of the translated music, since one of the most important artistic features of the resulting moving images – their temporal structure – is largely determined by the music. So – unless both parts of the work weren't made by the same person – we can say that in music translations both the visualist and the composer are the ‘authors’ of the moving image-sequence. Of course this aspect of divided artistic responsibility is reduced if the mapping of parameters is changed in the course of the piece or if visual changes do not represent musical changes.
2.1.4. Translations of moving images into music
Technical means also allow a reversal of the process discussed so far: the sonification of moving images which also yields a very close connection between the temporal structure of the soundtrack and the images. If all visual events are represented acoustically and the mapping of visual to aural parameters remains unchanged the resulting impression for the listener/viewer will be that of a music-translation.
Although sonification (of data) is mostly a theme in the scientific realm (Grond and Schubert-Minski Reference Grond and Schubert-Minski2010) there are some examples of artistic applications. See for instance calidri (Evans Reference Evans2006a) and other works by the US-based artist and scholar Brian Evans, who produced the soundtrack for these computer animations by ‘making a time slice … a static image built by extracting a single line from each frame of the animation and stacking these lines sequentially’ and transforming these time slices into music ‘using a variety of mapping approaches and audio synthesis techniques’ (Evans Reference Evans2006b). Again the difference between production and perception is to be stressed here: while in Michael Filimowicz's experiments in ‘video sonification’ (Filimowicz Reference Filimowicz2010) at least some of the visual events can be paralleled with events in the soundtrack this is not possible while viewing/listening to calidri. So, with regard to its production, calidri is a translation of moving images into music but it is not in an aesthetic sense – the translational process is no longer perceptible in the final video.
2.2. Synthetic structures
One could think that the Studien made in the 1930s by the grand master of non-representational film, Oskar Fischinger, are also manual translations of music. Malcolm Le Grice writes in his book Abstract Film and Beyond:
In terms of the aesthetics of Fischinger's films of that period, in spite of his denials that they are not visualized or illustrated music, their formal principles rest heavily on the musical analogy: one kind of shape corresponding to one kind of sound, another to another, one kind of movement corresponding to one development in the melody, and so on, repeats in the music being followed by repeats in the animation. (Le Grice Reference Le Grice1977: 65)
I will review this assertion on the basis of Fischinger's Studie Nr. 6, a two-minute masterpiece from 1930 (Fischinger Reference Fischinger2006a), which William Moritz called ‘the most brilliant of the black-and-white Studien’ (Moritz Reference Moritz2004: 28). We know that ‘Oskar marked out the synchronisation, which he always determined (thanks to his engineering training) by scratching an “X” across a phonograph record, then calculating the exact time a certain sound occurred by using a slide-rule to compensate for the diminishing size of the grooves toward the centre of the disc!’ (Moritz Reference Moritz2004: 36). After that he made precise diagrams for synchronisation purposes (an example was reprinted in Gehr Reference Gehr1993: 25).
So, the role the music played in the production process is relatively clear: the music was chosen first and its structure was the starting point for building the temporal structure of the images. But are the resulting images indeed just mirroring the structure of the music as Le Grice points out?
The structure of the music Fischinger used for this film, a piece called Los verderones by the Spanish composer Jacinto Guerrero (Moritz Reference Moritz2004: 29), is rather simple: the composition consists of five segments or ‘phrases’, roughly setting up an ABACCA structure. At the beginning of the first and the third phrase a motif is repeated exactly (0′09′′–13′′ and 0′44′′–48′′), but the images Fischinger combined with these musical passages look very different.
Figure 3 shows a graph comparing the usage of musical and visual motifs in the film. In the upper area the structure of the music is indicated (rows titled ‘music rough’ and ‘music fine’, where identical shades show identical or nearly identical repetitions of motifs). In the lower area (all rows below ‘music fine’) the usage of different visual motifs (i.e. forms) is shown in a simplified manner. As one can readily see, the usage of visual motifs differs heavily from the usage of musical motifs throughout the film. Studie Nr. 6 is by no means a mere translation of Jacinto Guerreros's Los verderones.
Musical and visual motifs in Studie Nr. 6.
So how can we describe the relationship between music and images Fischinger has constructed here? On the visual layer Fischinger built a temporal structure that sometimes mirrors each note of the music, at other times only some of them. Sometimes the visual motifs change in accordance with the music but in most cases they don't. So, as a whole, the structure of the music and the structure of the images together create a new, even more complex structure. Therefore Fischinger's film is not so much an example for a translation of music into moving images, but a type of audiovisual connection that I propose to name ‘synthetic structure’.Footnote 4
Music and images unite to form a total audiovisual structure, which is the result of both the music and the images. If we recognise a sequence of moving images as a ‘mere’ translation of a musical piece we understand that there is only one temporal structure, provided by the accompanying music, and this structure is doubled by the visuals, or – in less pure forms – certain elements of the musical structure are emphasised but it basically remains the structure of the music that is presented here. In synthetic or real audiovisual structures the perception of the total temporal arrangement is a result of combining different temporal informations transmitted on the auditory and the visual channel.
This kind of audiovisual combination, which is – in my estimation – the most frequent choice in the production of visual music, can be observed best in very slow pieces with only few changes taking place on the visual and the auditory layer. Take for instance the first minutes of the film uarei by Austrian experimental filmmaker Marc Adrian from 1977 (Reference Adrian2004): on the soundtrack we can hear only a few tones sustained for a long time while the images also move and change very slowly, sometimes in accordance with musical changes at other times not. On the whole, and if we try to apprehend the total audiovisual structure of the work, a synthetic structure arises.
Of course a sharp line cannot be drawn between music translations and synthetic structures; rather, there is a continuous spectrum of possiblities between the pure forms of these two categories: the more the parameter-mapping is changed or even cut off in the course of a music translation, the more there will be produced some kind of synthetic structure in the eyes and ears of the beholder.
2.3. Accentuation/emphasis
But we can decribe at least one category that lies more or less exactly in the middle between ‘translations of music’ and ‘synthetic structures’: there are examples of non-representational moving images that cannot be called translations of music, because the parameter-mapping is not consistent throughout the piece, but where the visual changes one can perceive in those movies do rhythmically coincide with certain musical events. The resulting impression is, in my view, an accentuation of these rhythmical events of the music by the images and – vice versa – an emphasis of the temporal structure of the images, because each change on the visual layer is doubled acoustically. Examples for this kind of audiovisual combination in visual music would be Norman McLaren's film Dots from 1940 (McLaren Reference McLaren2006a), the first sequence of Op Hop – Hop Op by Canadian experimental filmmaker Pierre Hébert (Reference Hébert1966), or the computer animation Raindrops by the Spanish audiovisual designer and architect Nico Juárez Latimer-Knowles (aka iconish) from Reference Juárez Latimer-Knowles2007.
2.4. Counterpoint?
Sometimes the term ‘counterpoint’ is used to describe the relationship between sound and images in visual music.Footnote 5 William Moritz, for instance, writes about Walter Ruttmann's first film Lichtspiel opus 1 (1921: Ruttmann Reference Ruttmann2009): ‘Although the images are beautiful and dynamic without the music, the sound counterpoint adds a significant perspective to the imagery’ (Moritz Reference Moritz1997: 223; see also Le Grice Reference Le Grice1977: 96).
But I think we should be careful in transferring this musical concept onto moving images. In music theory and history ‘counterpoint’ denotes a compositional principle. The term originates from the Latin phrase ‘punctus contra punctum’ meaning ‘point against point’, indicating that against each note of the first voice there has to be set a note of the second voice in accordance to certain principles of harmony. Although these rules were loosened over the course of history, the basic principle of first composing a cantus and then setting a counterpart to it according to certain rules remained the same (Sachs and Dahlhaus Reference Sachs and Dahlhaus2001).
Therefore, composing a visual counterpart to a certain piece of music would mean to ‘set’ a corresponding visual element ‘against’ each note of the music, which would essentially result in moving images that would have to be categorised as ‘translations’ or at least ‘emphases’ in the aforementioned sense (which is of course not the case with Max Butting's original music for Lichtspiel opus 1).
In accordance with this conception of ‘counterpoint’ an explanation of the piece by the Themersons that I have categorised as ‘manual translation of music’ reads: ‘Each instrument of the orchestra [is] represented by a simple geometrical form which changes its shape up and down according to the pitch of the note. Shapes representing different instruments were superimposed by multi-exposure, frame by frame. Crescendo, diminuendo, staccato, pizzicato, all had their visual counterpart’ (Anon. 2005).Footnote 6
2.5. Mutual disturbance
There is at least one prominent case in which a filmmaker combined his non-representational films with an accompanying music that not only shows no comprehensible connection with the moving images but – at least in Fred Camper's (Camper Reference Camper1999) and my opinion – therefore even disturbs an understanding of their structure: I'm referring here to the so-called Early Abstractions (Films No. 1–5, 7 and 10) by Harry Smith. According to William Moritz, Smith first wanted to synchronise his Films No. 1, 2 and 3 to certain pieces by Dizzy Gillespie but could not ‘afford the cost of transferring the music to an optical soundtrack’, so they were accompanied by another live jazz band when they were premiered at one of the ‘Art in Cinema’ events in San Francisco in 1950 (Moritz Reference Moritz2010: 65–6). In the 1960s Smith simply played certain records at the screenings, for instance the LP The Village Fugs, which he had produced himself. ‘When a sound version of Early Abstractions was finally being made, Smith reportedly chose the Beatles’ album [Meet the Beatles, US 1964] in the hopes that his film would achieve a comparable popularity. This soundtrack … remains on the distribution prints to this day …’ (Singh Reference Singh2010: 36).
The mutual disturbance of music and image becomes especially apparent in Film No. 7: Color Study (1952; Smith Reference Smith2007): in the first seconds the music fits astonishingly well to the moving images, although the whole album Meet the Beatles is simply played along all of the films of the collection. But when it comes to the refrain of ‘All My Loving’ the disparateness of sound and image becomes more and more evident – culminating in the sudden end of the song after 42 seconds of screen time while the film is still going on. And the next song (‘Don't Bother Me’) seems to have a rhythm completely different from the moving images of Film No. 7.
David Chapman argues that Smith's more or less arbitrary selection of a soundtrack for his Early Abstractions is in accordance with his idea of ‘“automatic synchronization”. This method essentially consisted of taking any music, run it next to any of his films and the rhythms of the images and the music would “automatically” sync-up’ (Chapman Reference Chapman2009: 10). But even the episode recounted by Fred Camper (and cited by Chapman) about Smith demonstrating this ‘method’ at a seminar shows its inadequacy:
At one point he explained automatic synchronization, and, preparing to show one of his films to demonstrate it, pointed at me, as I was sitting on the floor near a stack of records, and said, ‘Hey, you, pick a record, any record.’ Without looking (it's to my eternal regret that I didn't look) I passed the first record on the stack up to him. He looked at it and said, ‘You idiot, not that record.’ I handed another record up to him, and he looked at it, and said, ‘You moron, not that record!’ Finally the third record was acceptable. (Camper Reference Camper1999)
If really any music would have the ability to ‘synchronise’ with his films than it should have made no difference which record Camper had passed over. Different musics obviously synchronise more or less well to a certain non-representational film or a representational one that also heavily relies on visual rhythm. Anyone can confirm this observation by playing different musics with different rhythms and tempi to one and the same film of that kind.
The point we have to make here in favour of Harry Smith is that while perceiving music and abstract images simultaneously we obviously tend to rhythmically connect the impressions received on those two separated sensory channels. If there are enough points of temporal coincidence the viewer/listener will be able to construct a total audiovisual structure, a ‘synthetic structure’ as described above. If there are not enough synchronous musical and visual events the impression of mutual disturbance will arise, especially concerning the rhythmical qualities of the moving images since these are far more difficult to apprehend.
2.6. No sound
I close with an option that should be mentioned here as well: some artists didn't use any music at all to accompany their non-representational moving images, thinking that music would only distract the viewer from concentrating on this autonomous art form. For instance, Thomas Wilfred, famous builder and performer of specialised colour projection systems from the 1920s to the 1960s, envisioned ‘the use of light as an Independent art-medium through the silent visual treatment of form, color and motion in dark space’ (Wilfred Reference Wilfred1947: 16).
Another prominent example for this strategy is American experimental filmmaker Stan Brakhage: most of his films – and nearly all of his non-representational films – don't have any soundtrack. In an interview from 1969 he said: ‘for the art of the film … sound is an aesthetic error … it cuts back sight’ (Fries and Gebhardt Reference Fries and Gebhardt1969).
In contemporary production, using no music or other kind of sound at all is by far the exception. At the 2009 edition of Punto y raya, a spanish film and video festival dedicated exclusively to non-representational moving images, there wasn't a single film or video in the whole competion (consisting of 76 entries, Santos and Palazzo Reference Santos and Palazzo2009) that had no soundtrack.
3. Résumé
To summarise, there are – with regard to the emergence of the total, or the combined temporal structure of the artwork – three basic types of connections between music and non-representational moving images:
-
• ‘Music translations’: certain parameters of the accompanying music are transcoded into certain visual parameters, and as a result all visual events can be paralleled with events in the accompanying music – the accompanying music provides the temporal structure of the audiovisual artwork. In pure forms the parameter-mapping stays constant throughout the piece.
-
• ‘Synthetic structures’: musical and visual events do not always temporally coincide but instead form an audiovisual structure that is a result of different temporal informations transmitted on the auditory and the visual channel.
-
• ‘Mutual disturbance’: there are not enough temporal coincidences between music and images and the viewer/listener is not able to mentally synthesise an audiovisual structure.
All of these categories can be applied with reference to the production or the artistic intention and to the perception of the resulting artwork – these two distinct applications will not necessarily produce the same results. And obviously the relationship between music and images can also change in the course of a work.
