1. Introduction
Construction in Zhuangzi (CiZ) is an audiovisual work based on a modified Lorenz dynamical system (Petržela and Hanus Reference Petržela and Hanus2005: 1) derived from a nonlinear, three-dimensional model of convection (Lorenz Reference Lorenz1963: 130). It is implemented in the programming environment Max/MSP/Jitter, and during performance parameters are modified dynamically: these ‘perturbations’ of the system are indicated by a short colour inversion and an audio ‘glitch’. A set of instructions and suggested presets serve as a rough score with scope for improvisation. All audio and visuals are derived from the Lorenz system and hence it is an example of the audiovisualisation of the same data source in realtime, a category within current audiovisual practice.
2. Integration of sound and image
The objective behind the representation of the same source of data in the audio and visual domains is the integration of the two in the same work. There are several historical precedents for the combined use of moving image and music, such as Alexander Wallace Rimington's colour organ accompanied by a piano, a normal audio-only organ and an orchestra, which was also the instrumentation used in Scriabin's colour-symphony Prometheus: The Poem of Fire in 1911 (Peacock Reference Peacock1988: 401–402). But one of the earliest examples of their integration at a more formal and structural level occurs in John and James Whitney's Five Abstract Film Exercises (1941–44). This work won a prize for sound at the First International Experimental Film Competition in Belgium in 1949 (CVM 2012), and had ‘provided an unequalled opportunity to integrate image and sound … [because] the procedures of frame by frame production were alike for both sound and image. Despite many difficulties, design ideas for image somehow stimulated counterpart sound ideas, and in turn sound pattern was literally mirrored, figure for figure, in an image/sound dialog’ (Whitney Reference Whitney1980: 92).
It was made, however, using ‘two homemade devices – a simple sine wave pendulum array and an optical-printer instrument’: large and cumbersome machines that were extremely time-consuming to use, and John Whitney longed for ‘a perfected facility that would provide music and graphics capabilities unified within one instrument’ (Whitney Reference Whitney1991: 599). He believed this ‘digital instrument [would] have totally revolutionary consequences’ (Whitney Reference Whitney1980: 95). This eventually became a reality for him in the form of a computer, and software that was specifically developed through his collaborations with Jerry Reed (Whitney Reference Whitney1994: 50) and with Bill Alves (Alves Reference Alves2005: 45).
Digital technology has further developed the audiovisual medium through the numerical representation of data as digital code resulting in its separation from the algorithm for its representation, for example as audio or video (Manovich Reference Manovich2001: 60). Firstly, this enables ‘transcoding’ – that is, the translation of something into a different format such as mathematical data into audio, video or any other medium, and in all other permutations (Manovich Reference Manovich2001: 64) – and thus increases the potential for more control over the integration of the audio and the visual domains. Secondly, this allows for ‘variability’, the possibility of multiple versions of the same material and its customisation often at run-time (Manovich Reference Manovich2001: 56), both in the data and the algorithm for its representation, and hence the possibility of generative works that can also respond dynamically to interaction in contrast to the fixed medium of film. Currently, there are many programming environments available that allow for the integration of audio, video and other media through digital technology such as Max/MSP/Jitter used for the creation of CiZ.
3. Musical Texture Metaphors
3.1. Counterpoint
The musical texture that Whitney used as a metaphor to describe his solution for the integration of image and sound towards the end of his career was counterpoint: ‘I aspire to compose both visual and audio “voices” independently, discovering and creating while performing a counterpoint or dialog between these voices’ (Whitney Reference Whitney1994: 50). This is partly a return to the momentary and short-lived ideals of late silent film and early sound filmmakers.
3.2. Dissonant harmony
However, this counterpoint model largely disappeared; as Michel Chion notes: ‘films tend to exclude the possibility of such horizontal-contrapuntal dynamics … [as] harmonic and vertical relations … are more salient – i.e., the relations between a given sound and what is happening at that moment in the image’. For him, most examples of supposed counterpoint are effectively ‘dissonant harmony’ (Chion Reference Chion1994: 35–37). This is perhaps the best description for the practice of juxtaposing unrelated images to dance music that permeates the VJ scene.
3.3. Monophony
Some evidence of integration does, however, appear to be emerging (see, for example, Faulkner Reference Faulkner2006). Nevertheless, most involve the standard procedures of amplitude following or timbral analysis at best: in effect, a form of audio to visual mapping not dissimilar in method to that found in freely available software media players such as iTunes or Windows Media Player. As Whitney stated, ‘this would be a partnership that is valid only if the combinations produce interest greater than the separate contribution of either the aural or the visual member’ (Whitney Reference Whitney1980: 92), which is often not the case in this more commercial end of audiovisual practice. In the more experimental field, a few practitioners producing pioneering work involving deriving the video from the audio, or the audio from the video, or both did emerge from the 1980s onwards (see, for example, Abbado Reference Abbado1988; Ritter Reference Ritter1993; Rudi Reference Rudi1998).
Continuing with Whitney and Chion's musical texture metaphors, audio-to-video and video-to-audio following would constitute unison or monophony, whereby notes are merely doubled or paralleled.
3.4. Homophony
The audiovisualisation of a single source (if successful) would be the equivalent of homophony: much like a melody and its harmonisation where each part by itself would merely be a line whilst its combination would create new depth by producing or implying a chordal structure, the effect of the audio and the video together would ‘combine to make an inseparable whole that is much greater than its parts’ (Whitney Reference Whitney1991: 599).Footnote 2
In John Coulter's categorisation of the possible permutations of the relationship between the audio and the video, it is the last method mentioned in the eighth and final category of isomorphic pair of abstract video and abstract audio. His example of ‘the real time synthesis of audio and video materials from a single data source’ is the ‘use of an electronic device to track the unique eye movements of individual users and translate the data in real time into sounds and moving images’ (Coulter Reference Coulter2010: 29, 32). This is a fast-emerging field that is yet to be explored fully. A few practitioners worth mentioning are Golan Levin (Levin Reference Levin2000), Ernest Edmonds (Edmonds Reference Edmonds2003) and Mick Grierson (Grierson Reference Grierson2005).
3.5. Voicings
Homophony of the audio and the visual produces more besides redundant representation of data. Whereas with audio-to-visuals following, the visuals can only contain data already present in the audio (and vice versa for visuals-to-audio following), in audiovisualising a single source, the audio and the visuals can both potentially contain information not present in the other due to technical or perceptual limitations in the number of possible parameter mappings in each medium. The metaphor here would be that of the presence of different notes of the same chord or different voicings in each medium, perhaps as extended chords. Furthermore, whereas audio to video following results in either a mirroring of what can already be heard or, at best, information on the final product through audio analysis, the audiovisualisation of the same source may produce data in the form in which it is parameter mapped, explaining the process of their construction rather than the end product. Hence it can contribute to its didactic function, but it can also alter perception and contribute to sound affecting vision and vice versa in a misleading manner.
4. Auditory Displays and Perception
A similar practice of the simultaneous visualisation and sonification of the same data source has been studied for the research into auditory displays: for example, a combination of an auditory and a visual display suggested for the analysis of seismic data (Saue and Fjeld Reference Saue and Fjeld1997: 4) and the effectiveness of such systems: see for example (Gröhn, Lokki and Takala Reference Gröhn, Lokki and Takala2003; Bouchara, Katz, Jacquemin and Guastavino Reference Bouchara, Katz, Jacquemin and Guastavino2010). Their raison d’être differs somewhat as their main purpose is the creation of alternative, more effective, efficient, accurate and suitable forms of data representation, and furthermore their findings may be considered preliminary (see for example Walker and Kramer Reference Walker and Kramer2004: 169–170; Ghirardelli and Scharine Reference Ghirardelli and Scharine2009: 602). Nonetheless, important factors are highlighted that should be taken into consideration in understanding the potential of the audiovisual medium.
4.1. Cross-modal phenomena
The homophonic model can contain more than the doubling of data in the audio and the video as discussed previously. But even the redundant presentation of information can cause cross-modal ‘synergy’ or ‘interference’ – that is, the improvement or the deterioration of ability to extract data (Walker and Kramer Reference Walker and Kramer2004: 169–170) – and thus it provides great potential for exploiting in aesthetic contexts.Footnote 3
4.2. Cross-modal synergy
Cross-modal synergy through the audivisualisation of a single source further assists in explaining the underlying structure of the work. Research into auditory displays have demonstrated that certain features can be better detected in different modalities: for example, the ear has a better capacity for monitoring a large number of changing variables and/or temporally complex information simultaneously (Kramer, Walker, Bonebright, Cook, Flowers, Miner and Neuhoff Reference Kramer, Walker, Bonebright, Cook, Flowers, Miner and Neuhoff1997: 5). For instance, a field where sonification may prove useful is seismology, where the direct audification of seismic data – in other words, its transposition to the audible range – can reveal information about earthquakes such as the magnitude and the distance of the earthquake by its relative loudness, the material the ground is composed of by its timbre (e.g. metallic sounds indicating sediments and wooden ones for bedrocks), and even its tectonic source mechanism from its amplitude envelope (e.g. a sharp, hard beat indicating one plate subducting the other and a ‘plop’ for two plates moving apart) (Dombois Reference Dombois2002: 28). The duplication of data in different modes of display therefore offers the possibility of one modality filling in for another in case of data being missed, increasing the likelihood of its detection and comprehension. This is not restricted merely to the domain of research and data representation. An example of its everyday occurrence is in lip reading, or speech reading, used unconsciously in order to clarify ambiguous speech data (Ghirardelli and Scharine Reference Ghirardelli and Scharine2009: 605), though this is not always beneficial as will be explained below.
Synergy therefore enhances the didactic function for the furthering of the aesthetic understanding of the work which may be useful for abstract art of this nature in order to establish its new form. This is apparent in most of the Movie examples, namely 1, 2, 4, 6 and 7. In Movie examples 1, 2 and 6, the visualisation is perhaps the most ‘straightforward’, as curves plotted in three-dimensional space. Audio samples are triggered at points of inflection–in other words, at peaks and troughs when the rate of change equals zero in any of the three dimension. These are marked by crosses and small cubes and their spatial coordinates are mapped to amplitude, speed/pitch, periodicity and panning, resulting in an evident correlation between the audio and the video domains. This direct correspondence serves to confirm their relation to each other as well as to the Lorenz system.
This facilitative effect is perhaps more beneficial in Movie example 4, where the audiovisual relationship is less clear. In the visuals, in place of a curve as in the previous examples is a cube in a mesh structure. There are two instances of the Lorenz system coloured blue and orange that initially follow an almost identical path, combining to become white; the audio is also effectively doubled. The two trajectories later diverge but in an undefined way: in the visuals, the blue and the orange do separate but the Moiré effect makes this far from obvious; in the audio the two parts no longer double each other but their individual voices are difficult to distinguish, the occasional treble-frequency ‘blips’ being the only components whose separate panning is clearly noticeable. The combined presence of the audio and the visuals does, however, enable the divergence of the two trajectories to be more apparent.
4.3. Cross-modal interference
Cross-modal interference is obviously undesirable in the field of auditory and visual display but for aesthetic purposes, depending on the context, consistent clarity might not necessary be effective. Extreme examples of its effects are ‘cross-modal interactions affecting early levels of perception’ (Vroomen and de Gelder Reference Vroomen and de Gelder2004: 142). These can explain visual and auditory ‘capture’ effects (Ghirardelli and Scharine Reference Ghirardelli and Scharine2009: 603–606), the former resulting from sound affecting vision as evident in the ‘freezing’ phenomena (Vroomen and de Gelder Reference Vroomen and de Gelder2004: 146–147), and the latter from vision affecting sound, for example in the famous McGurk effect (McGurk and MacDonald Reference McGurk and MacDonald1976: 746–748), whereby our unconscious dependence on lip reading actually impedes our ability to distinguish speech.
Although the aim of audiovisual art should not necessary be to merely replicate these illusory effects, which are perhaps of a more spectacular rather than aesthetic nature and analogous to special effects evident in blockbuster films, similar phenomena of a more subtle nature can contribute to the audiovisual medium becoming more than the sum of its parts. Movie example 3 is a case in point, where there are two trajectories of the Lorenz system present with the difference between them visualised as a set of straight and white lines connecting the corresponding points of the two curves (see also figure 1). By itself, this could be interpreted either as one integrated whole or two separate trajectories – that is, foregrounding either the lines depicting difference or the two independent curves. In the audio, although the two trajectories are sonified separately, their amplitudes are both controlled by the same visualised difference and heard as one combined, parallel voice rather than two separate parts. This ‘captures’ or influences our perception of the visuals to be seen as one element.
A still image from Part 1 of Construction in Zhuangzi (2011)
4.4. Combined cross-modal interaction
The ventriloquist effect (Radeau and Bertelson Reference Radeau and Bertelson1977: 137) is primarily an auditory rather than a visual ‘capture’ effect, whereby the visual stimulus affects the localisation of the sound. Although to a much smaller degree, studies have shown that the visual localisation can also be drawn towards that of the audio stimulus (Bertelson and Radeau Reference Bertelson and Radeau1981: 578–584; Vroomen and de Gelder Reference Vroomen and de Gelder2004: 142), in other words as audio ‘capture’. It is often difficult to establish an instance of cross-modal interaction as being either synergy or interference exclusively: many would fall under the category of a combination of the two.
5. Added Value
5.1. Synchresis
The phenomena of combined cross-modal interactions are supported by Chion's notion of added value in cinema, of ‘the expressive and informative value with which a sound enriches a given image so as to create the definite impression, in the immediate or remembered experience one has of it, that this information or expression “naturally” comes from what is seen, and is already contained in the image itself’ (Chion Reference Chion1994: 5). He states that ‘added value works reciprocally’, alluding to auditory and visual capture effects described above in research on perception: ‘sound shows us the image differently than what the image shows alone and the image likewise makes us hear sound differently than if the sound were ringing out in the dark’ (Chion Reference Chion1994: 21). Furthermore, as research into auditory displays have suggested that some information is better suited to being analysed aurally, Chion states: ‘What we hear is what we haven't had time to see’ (Chion Reference Chion1994: 61). He also notes how ‘it is not acoustical realism so much as synchrony above all, and secondarily the factor of verisimilitude … that will lead him or her to connect a sound with an event or detail’ (Chion Reference Chion1994: 22–23), this being the basis for his principle of ‘synchresis’: ‘the spontaneous and irresistible weld produced between a particular auditory phenomenon and visual phenomenon when they occur at the same time’ (Chion Reference Chion1994: 63). This is confirmed by findings of studies into cross-modal perception that, with visual capture, the desynchronisation of the auditory and the visual data was found to reduce its effect significantly while the degree of realism of the context of the auditory and the visual pairing did not (Radeau and Bertelson Reference Radeau and Bertelson1977: 137). The effectiveness of this integration of the pairing is further strengthened through the audience paying attention to the stimulus (Talsma and Woldorff Reference Talsma and Woldorff2005: 1098) which should ideally be the case for cinema and audiovisual art.
The following sets of examples are perhaps the most immediately successful in this respect. In Movie examples 1, 2 and 6, only a purely formal correspondence between the triggered audio samples and the visuals exists, the former resembling a conventional graphic plot of the Lorenz system and the latter being its direct audification. The effectiveness of the pairing arises foremost out of their synchronisation and not from their realism, the possible argument of audification being a ‘true’ sonic representation as its oscilloscope-like visual counterpart being irrelevant (however appealing).Footnote 4 In Movie example 7, as parameters become 0, silence is produced and the movement of the mesh completely ceases but for a slow rotation. These absent representations combine through a reciprocal articulation through their simultaneous occurrence: the silence establishes the ‘zero’ state of the visuals as a slow rotation as opposed to other options such as complete stillness or darkness; this in turn signals the ‘zero’ state of the audio as silence instead of, for instance, a sustained tone or a repeating loop.
5.2. Superfield
The combined cross-modal interaction in the ventriloquist effect demonstrates how localisation in the audio and the visual domains does not necessarily have to correspond exactly for them to be perceived as an audiovisual entity through synchresis: depending on the aesthetic context, even a complete left–right inversion can be entirely plausible (Coulter Reference Coulter2010: 32). Furthermore, it is possible for the effect of synchresis to be prolonged even after the removal of one of the stimuli (for example, when an entity moves off screen but can still be heard), thus creating a ‘sort of superfield, a general spatial continuum or tableau … that can issue from loudspeakers outside the physical boundaries of the screen’ (Chion Reference Chion1994: 150). This occurs in Movie example 5: here the non-uniform rational basis spline (NURBS) figure forms a combined audiovisual entity with the sonification partly due to sufficient correlation in their panning. Later, the visual shape moves off-screen with only the paired synthesis instrument remaining perceptible. However, it remains combined and its subsequent reappearance on screen is perceived as the same entity that disappeared previously.Footnote 5
5.3. Stretching the audiovisual canvas
Chion notes the potential for the development of synchresis by moving beyond the realism of the pairing and ‘stretch[ing] to the maximum … the “audiovisual canvas”‘ (Chion Reference Chion1994: 213). Walter Murch has also argued for the necessity of its further exploration (Murch in Chion Reference Chion1994: xix), which could certainly expand the audiovisual medium. Returning to the metaphors of musical textures, this would constitute a further chromaticism or even microtonality, perhaps with ornamentation resulting in heterophony, but still retaining intelligibility and not reverting to dissonant harmony. And one possibility for resolving this apparent dichotomy between the necessity for experimental yet coherent audiovisual pairing lies in the use of the same data source for audiovisualisation. Although it could be argued that sound, moving image or any other art form can benefit from external modes of organisation, the phenomena of combined cross-modal interactions and added value provide the audiovisual medium with unique possibilities by allowing for a wide variety of organisational forms to be perceived through the complementary use of the two domains.
6. Synaesthesia
These are not, however, imitations of mild forms of synaesthesia, a condition in which a blending of two or more senses is experienced, such as seeing the colour blue from hearing the note C sharp played on a piano (Ramachandran and Hubbard Reference Ramachandran and Hubbard2003: 53). Partly due to subjective differences in its effects and a lack of structured ‘mappings’ between the senses, Whitney disapproved of its recreation in audiovisual art (Whitney Reference Whitney1994: 47). Chion is also clear that added value depends on ‘transsensorial’ perception and not an imitation of synaesthesia, which he describes as ‘intersensorial’ (Chion Reference Chion1994: 137). Perhaps it is more accurate to describe audiovisual art as being ‘cross-modal’, and inspired by synaesthesia rather than being synaesthetic art (Cádiz Reference Cádiz2004: 2).
7. Conclusions
A case has been made for audiovisual homophony as being in accordance with Whitney's ideal of integrating sound and image. The embracing of digital technology has allowed for its further control through the possibilities of data transcoding and realtime, generative interactivity. In contrast to monophony such as audio-to-video following, an attempt at a new form of organisation for audiovisual material that does not revert to existing audio or visual forms is possible through the audiovisualisation of a single source of data.Footnote 6 In CiZ, an appropriate form is provided by a chaotic dynamical system, the source of the audiovisualised data. Thus its audiovisual ‘gestures’ are describable by extra-musical and ‘extra-video’ terms such as:
-
• convergence towards periodic attractors in Movie examples 2 and 6 and steady-state attractors at the end of Movie example 1, and divergence of trajectories belonging to different basins of attraction from 0′20″ in Movie example 1 and at the end of Movie example 4 (see also figure 2) (DeLanda Reference DeLanda2002: 15);
Figure 2A still image from Part 2 of Construction in Zhuangzi (2011)
-
• perturbations resulting in ordinary events in most of Movie example 2, or singular events (i.e. phase transitions) in Movie examples 1 and 5 and at 0′44″ in Movie example 2 (DeLanda Reference DeLanda2002: 19, 73);
-
• and heterochrony in Movie example 6 (DeLanda Reference DeLanda2002: 97).
Figure 3A still image from Part 3 of Construction in Zhuangzi (2011)
The aesthetic approach of an audiovisual work can also retain relevance to its audiovisual material by being informed more generally by the system behind the data. The Lorenz system upon which CiZ is based is one of the most recognisable examples of the new science of chaos and complexity developed through ‘trans-intuition’ based in differential geometry (Smith Reference Smith2003: 419). Particularly in the study of such nonlinear systems, ‘Seeing, seeing what happens, has always had a more essential importance than demonstrations’, further facilitated with the aid of computers (Deleuze and Guattari Reference Deleuze and Guattari1994: 128). This ‘trans-intuitive’ research approach involving ‘non-standard’ formulations (Smith Reference Smith2003: 424–425) is akin to the improvisatory and exploratory nature of the performance of CiZ, both being dependent on computational power and approximating to ‘a qualitative characterisation of the interdependency among parameters, as opposed to a quantitative, analytical characterisation … reflect[ing] the “non-integrability” of mathematical models of dynamic systems’ (Di Scipio Reference Di Scipio2001: 251). This similarity serves as more than a metaphor, as aesthetic and scientific areas of interest often overlap, for instance in the vicinity of critical points where entropy is highest (DeLanda Reference DeLanda2002: 78–9) or in a limit cycle around an attractor (DeLanda Reference DeLanda2002: 15) that produce chaotic or repetitive audiovisuals respectively where the most useful information for the understanding of the dynamical system can also be garnered.
Through the audiovisualisation of the same data source, two conflicting effects are possible: cross-modal synergy and interference. The former enhances the didactic potential of the audiovisualisation and the latter can affect perception in an illusory manner. As mentioned, research findings into their cause and their exact nature are often conflicting (Ghirardelli and Scharine Reference Ghirardelli and Scharine2009: 602). Therefore, a systematic or thorough demonstration may not be possible for the time being. Nevertheless, the fact that they occur and to such an extent further validates the audiovisualisation of a single source of data as an artistic practice, especially as there is no reason why they cannot be explored for their aesthetic potential as supported by Chion's theory of added value and the possibilities of stretching the audiovisual canvas.
Due to cross-modal interactions often being combined, a sliding scale ranging from the facilitative to the inhibitory can be conceptualised. As these effects are not mutually exclusive, a topological or ‘fuzzy’ manifold is perhaps a more accurate model. The navigation inside this continuum of the tension between synergy and interference is an important determining factor of the nature of an audiovisual work and its relative aesthetic success. In CiZ, this is evident in the method of audiovisualisation that approximately traverses this manifold from synergy to interference, from Part 1 through to Part 3 (see figure 1, 2 then 3, and Movie examples 2, 6, 1, 3, 4, 7 then 5).
