1. Visual Music: A Historical view from the electroacoustic dress circle
Before we can find a locus for the electroacoustic community in relation to visual music we ought to define visual music itself, and attempt to place its developmental context in a trajectory that is traverse, or at least tangential, to that of the sonic arts. Maura McDonnell (Reference McDonnell2010) identified various etymological paths to describe and define visual music works and practices. Andrew Hill (Reference Hill2010) proposed a summary of various interpretations of ‘electroacoustic audiovisual music’ in the context of setting the boundary conditions of his research on audience reception. For the sake of our discussion, in primis one should note that the label visual music itself has a hybrid provenance, as it was originally used in 1912 by artist and art critic Roger Fry to describe Kandisky's abstract painting. We may also note that rather than (or along with) a concern for some temporal dimensions in the Russian painter's style (Fischman Reference Fischman2011) the term could have been used, back then, merely to draw attention to the shift of focus in contemporary painting from the figurative to the abstract, the latter being the habitual artistic territory of absolute music.
Kandinsky's work can be considered one of the early examples in which abstract painting, hence still images, can be ascribed a temporal dimension which the spectator journeys through as he or she scans the shapes and colours (Fischman Reference Fischman2011), their mutual connections and their relations with the whole. Kandinsky's quest for a visual equivalent of luxuriant orchestral colours, however dissonant, is at the basis of many of his most audacious works such as Fuga (Fugue) (Reference Kandinsky1914). Peculiarly similar in the musical allusion of the subtitle, František Kupka's Disks of Newton (Study for ‘Fugue In Two Colors’) (Reference Kupka1912) typifies the potential temporality, hence musicality, of a still frame. Rather than the pertinence of this painting to musical form, we will be inclined to fathom its gestural essence – that is, a spatio-temporal trajectory in the unfolding of its visual spectromorphologies. For some viewers the concentric spiralling geometries can conjure up a centripetal or centrifugal motion with accelerations linked, directly or inversely, to the brightness of the colours in each curvilinear strand. Another viewer's timeline might gravitate towards the climatic red-purple spot in the centre of the painting, while the arches heading towards or departing from it could be simulacra of leading motifs or cadential releases. Through these paths the viewer can apprehend Kupka's Disks of Newton in a way which is similar to the experience of a sonic or musical gesture.
Undoubtedly, a similar quest for ‘temporality’ can be made for all figurative, as well as abstract, fine arts. Indeed the spectator's assimilation always happen within a time interval, time being the only inescapable dimension of our existence and experience of the world, even one artificially frozen into the immobility of a plastic artwork. Nonetheless, we can agree that the non-representational nature of abstract painting enhances the temporality of the spectator's experience, with the additional democratising benefit that the artist relinquishes control over the way in which such a temporal dimension is experienced by the viewer (Brougher, Strick, Wiseman and Zilczer Reference Brougher, Strick, Wiseman and Zilczer2005: 19).
Interestingly, early manifestations of visual music precede both electroacoustics and cinematography. Examples of colour organs date back to the eighteenth century (Peacock Reference Peacock1988). Although this is yet another construal of the visual music term rooted in the world of visual arts, rather than in music or sonic arts, we owe to the likes of Mary Hallock Greenewalt, Thomas Wilfred and Charles Dockum an innovative approach which strived to disenthrall visible light from the expectations of figurative portrayal.
The latter process was not dissimilar, in its guiding principles, from the emancipation of noises as a prime audible mean of artistic expression separate from, or in extension of, the musical sounds European composers, performers and audiences were familiar with until the beginning of the twentieth century. It is therefore little surprising that among the early examples of visual music, albeit not referred to in these terms at the time, we find the experimental films by Italian futurists Arnaldo Ginna and Bruno Corra who, during the 1910s, produced animations by means of painting abstract shapes directly on film strip (Bendazzi Reference Bendazzi1996). Within the Futurists circles, in the early twentieth century, grew not only a desire for a new anti-musical language, based on the primacy of noise, but also a push towards the technologically aided art of moving images which, in the case of Ginna and Corra's work, was informed by a typically futurist concept of unity between different artistic media (sound, images, music, poetry).
2. Visual Music: A working title
In view of its past and present incarnations, we can say that visual music indicates a form of art in which the combination of moving imagery and sound establishes a temporal architecture in a way similar to absolute music (Evans Reference Evans2005) both through the articulation of the ‘visual things themselves’ and the ‘musical things themselves’, and through the interaction and interplay between them (McDonnell Reference McDonnell2010). Our definition of visual music will typically consider works that are non-narrativeFootnote 1 but not necessarily non-representational. Joseph Hyde's Vanishing Point (Reference Hyde2010), Steve Bird's Time and Tide (Reference Bird2008) and Nick Cope and Tim Howle's In Girum (Reference Cope and Howle2007) will be given as examples in which real-world recognisability of many of the audio and visual materials does not propose a storyline the viewer can latch onto: the artistic context, as well as the treatment of the raw sonic and visible matter, shifts the noesis from the real world onto the visible and audible things themselves. Hence visual music is considered an ‘experience of pure time, without the burden of a story. … Where cinema is audiovisual novel, visual music is audiovisual poetry’ (Piché Reference Piché2003; my translation).
Visual music has become an umbrella-term which encompasses a body of works particularly pertinent to the scope of this journal – that is, the artistic output witnessed since the early 1980s, in which abstract or, more generally, non-narrativic visuals (computer generated, or digitally captured and then computer manipulated) are combined with electroacoustic sounds. Other, similar terms have come to signify essentially the same thing: for example, ‘vidéo-musique’ (video music) (Piché Reference Piché2003), ‘electroacoustic audiovisual’ (Hill Reference Hill2010), ‘computer music video’ (Rudi Reference Rudi2005) or ‘electroacoustic video’ (Garro Reference Garro2005a).
The relationships between electroacoustic arts, in particular acousmatic, and visual music will be discussed in the next section, but for now it will suffice to point out that the latter has become an intrinsic modus vivendi of the former, as evidenced by the increasing number of video presentations featured in the programmes of most electroacoustic presentations.
3. Visual Music Versus the electroacoustic arts
We are cast into these dream-like, organic soundscapes where colors, movements and textures merge together.Footnote 2
3.1. The hybrid nature of the electroacoustic arts
Visual music undoubtedly challenges the electroacoustic credo of the primacy of the ear, but, in the hands of many composers, it does so while treasuring all technical and artistic values nurtured during several decades of sonic endeavours, research and practices. Moreover, the cultural shift introduced by the addition of moving images should be placed into a perspective which considers the peculiarity of the idiom we are dealing with: composers, listeners and analysts will find that the historically accepted addition of recognisable anecdotal sound material to the arsenal of an otherwise absolute sonic language poses an even bigger syntactical challenge than the extension of pure sound spectromorphologies into the audio-plus-visual combined domain. Thus, for example, the sound of machine guns orchestrated against a backdrop of abstract sonic textures is, semantically speaking, at least as problematic as the vista of coloured granular clusters on a projection screen to accompany the same musical material.
The acousmatic is already multi-mediatic. Only in the case of entirely aural acousmatic compositions (Emmerson Reference Emmerson1986) can we speak of a coherent mono-mediatic experience. If an acousmatic composition combines abstract and mimetic materials along with (especially) intelligible speech or musical phrases, then both the listener and the composer are engaged in a relationships with the output that are, to say the least, multi-modal if not plainly multi-mediatic. For example, technically speaking we can mix together a wide variety of audible material: a purely abstract synthesised tone, the recording of a forest ambience, and a voice reading a poem. It would be naïve to regard the resulting composition as mono-mediatic, for instance purely acousmatic, simply on the grounds that we project such a mix through mechanical means in a concert hall. The audience will have no choice but to engage in continuous shifts in decoding strategies, making sense of the narrativic implications of the forest virtual space, establishing links between the spectromorphological properties of the various abstract tones and the whole, and, especially, relating empathically to the message and the emotions conveyed in the spoken word, moving on to consider anything else only when their ineluctable anthropological interest in meaning has been satisfied (Chion Reference Chion1994: 6). The audience may indulge the continuous skips in listening modes required to relate to such work, but it should be noted that these relational shifts are not very different from, and certainly not less problematical than, those solicited by a multi-media work, typically when moving image is added to an electroacoustic soundtrack.
Sound can, and often does, carry a multiplicity of cognitive, cultural and artistic categories. In many of its manifestations, in much of its vast repertoire, electroacoustic music carries within itself the seed of the journey that took it beyond the acousmatic, mono-mediatic paradigm. Visual music composers have, from their viewpoint, done nothing more than going one small step beyond.
Acoustic imagery (Filimowicz and Stockholm Reference Filimowicz and Stockholm2010) and its articulation have been an integral part of the electroacoustic arsenal throughout the history of the genre, its effect being particularly compelling in works constructed with anecdotal material causally rooted in our experience of the real world. In fact, the cliché cinéma pour l'oreille (cinema for the ears), initially coined by Canadian composer and electroacoustic guru Francis Dhomont (Reference Dhomont1996: 23) was, and still is, a popular epithet to subtitle electroacoustic works and concert presentations. Luc Ferrari's phonography (Reference Ferrari1970), Trevor Wishart's cross-breeding sonic metaphors (Reference Wishart2006) or Natasha Barrett's augmented soundscapes (Reference Barrett1999) in their own different ways tell us the same fundamental truth: that most electroacoustic music, even when composed and listened to according to the acousmatic cultural template, is inevitably, often inherently, and always fascinatingly, very ‘visual’.
Yet it is not visual enough, for some. Jean Piché's candid critique of electroacoustic music culture questions, among other things, the purpose of the concert ritual and challenges its validity on the grounds of the missing visual dimension in acousmatic concert presentations (Piché Reference Piché2003). Piché advocates the role of video, clearly not as a surrogate for the gestural and emotional empathy offered by live performers, but as the added domain that transforms an otherwise detached, and essentially individual, acousmatic experience into a collective multi-mediatic one.
3.2. The signing of a new audio visual contract
A (real) anecdote:
9.46 pm, August 2nd. In the filled-to-capacity St Paul's Hall, at Huddersfield University (UK), Julien Legault-Salvail's work Chute Libre (Reference Legault-Salvail2009) for prepared piano, ensemble, live electronics, tape and video projection is being performed in front of the delegates of the 2011 International Computer Music Conference. The first section of this compelling work consists of a long process of dissonant musical accretion, a gradual crescendo in which all instruments and electronic treatments contribute to a steady accumulation of sonic energy, spectrally richer and richer and increasingly impetuous. All musicians are physically involved in the bearing of this irresistible cacophony; their gestural exertion and commitment are fascinating, both on an individual and on a collective level. Yet, in spite of that spectacle of human performative stamina, which by the end of the agglomerative process has reached almost delirious proportions, all the spectators’ eyes, with no exception, are fixed on the large screen suspended just above the frantically moving musicians, their presence ignored. The audience, in its entirety, is transfixed by the images thereon projected. Footage of chromatically saturated drifting skies have by then morphed into a stream of gaseous shapes rapidly shifting across the field of vision and, in turn, into a furious vortex of red and yellow profiles that are being sucked, at incredible speed, into an imaginary gravitational storm. It is as if the audience itself is being dragged into that mesmerising tunnel, unable to unlock their sight from the colourful, mysterious, frightening plunge, staring at its ultimate destination.
As far as this audience was concerned, the ensemble provided nothing more than ‘pit music’ (Chion Reference Chion1994: 80). However powerful as mood setter, it disappeared from the spectators’ visual focus. The images, those particular images at any rate, attracted the viewers’ attention more than the visible, physical, live performing, gesturally engaged musicians. We ought to give careful consideration to Simon Emmerson's judgement on the potentially polarising power of images in relation to music (Emmerson Reference Emmerson1999), a point that is even more relevant when the soundtrack is acousmatic, hence lacking the visual presence an emotive power of a live musical performance.
In visual music the framework within which audiovisual gestures operate is determined by a network of agents continually interacting, converging and opposing one another to form the unique, individual idiom of a particular work. The development of an audiovisual discourse, in the absence of a narrativic leitmotif, is inevitably complex. Electroacoustic sounds themselves are suspended in a multi-dimensional space that offers no time grid and no organisational pitch categories, thus presenting listeners with decoding challenges that, however riveting might they be, are often diverse in different works, and always arduous, especially for the uninitiated (Piché Reference Piché2003). The addition of a deluge of visual shapes, themselves offering vast arrays of design possibilities, adds further variables to an already labyrinthine equation. Yet the unspoken tenet that impels audiovisual composers, including sonic artists and visual artists who commit themselves to collaborations and cross-fertilisations, is that they are not adding further complexity to an already boundless medium; quite the opposite, they are ‘compositing’ the ramifications of two artistic domains to arrive at a harmonious synthesis of the two, a locus where variables explain one another, not solving the equation, but making it, if nothing else, somewhat more transparent.
The audiovisual contract described by Chion postulates the visually driven nature of cinema, a media where sound is ‘that which seeks its place’ (Chion Reference Chion1994: 68) and adds value to the subjects and actions on the screen. However, there is a fundamental difference between the audiovisual contract at the basis of cinematography and the audiovisual contract that governs the relationship between sonic and visual streams in visual music. Such difference has its roots not just in the profound dissimilarities between the two idioms but, perhaps more importantly, in the extremely distant expectations that stakeholders (composers, producers, listeners, analysts, researchers) hold when they ‘sign the contract’ and use it as a framework to negotiate their relationship with the materials. The audiovisual cinematographic contract is signed and used by parties who have more or less implicitly agreed upon some degree of visual centrality, thence on some degree of aural subservience. Conversely, the audiovisual contract at play in visual music is agreed upon by parties who hold the primacy of both ear and eye together as their artistic credo. Therefore, notwithstanding the irresistible power of the image as perceptual attractor, it would be erroneous to assume that such magnet establishes the same potential in the forcefield of electroacoustics as it does in cinema.
4. The language of visual music
When I saw the videotracks by [filmmaker and video artist] Nick Cope I thought ‘this is electroacoustic music made with images instead of sounds!’
Tim Howle, composerFootnote 3
4.1. Audio versus visual gestural analogies
The issue of audiovisual mapping has preoccupied, and still does, practitioners, composers, theorists and engineers within and without the visual music community. The audio-versus-video correspondences, or their commutative video-versus-audio, which we will consider as a synonym of the former, are undoubtedly an important element of our art's semiotics at the poietic (the domain of the composer), aesthesic (the domain of the listener) and neutral (the domain of the analyst) levels (Nattiez Reference Nattiez1990: 17). Nevertheless, we will relate to the question of correspondences in a holistic fashion, indulging ourselves in compositional practices and listening strategies typified by fluid, multi-dimensional inter-relationships: visual music is more than a mapping exercise; the absence of a relationship is itself a relationship.
The strategies and the degrees of audio-versus-visual congruence were discussed in a previous article (Garro Reference Garro2005a), where it was posited that the level of integration between gestural profiles in the two media can be placed within a taxonomic continuum, illustrated in figure 1. On one extreme we rely on purely interpretative inklings, calling upon intuitive renditions and appraisals of the connections between sonic and visual gestures.
Continuum of gestural audio vs. visual association strategies (Garro Reference Garro2005a)
At the opposite end of the continuum, behavioural associations are established on the basis of the mutual profiles of one or more phenomenological attributes in the sounds and in the visuals, whereby the isomorphism created by the designer/composer can clearly be identified as such by the viewer/listener. In intermediate situations the integration between sounds and visuals operates through both interpretative, to some extent, and mapping, to some extent, schema.
Synchronisation is evidently the sine qua non for the establishment of true parametric relationships between audio and video gestural profiles. However, it is equally clear that synchronisation between them is, on its own merit, insufficient to establish parametric links: two events occurring at the same time may be spectromorphologically unrelated. The fact that synchronised gestures are, nevertheless, often misconstrued as being parametrically connected is due to the phenomenon of culturally acquired synchresis (Chion Reference Chion1994: 63); in other words, the tendency to ascribe special correlations to sounds and visuals purely on the basis of their simultaneity. Synchresis is rooted in the mechanisms of perception and cognition but is highly reinforced by the frequent exposure, through film and television, to the results of audiovideo montage: early on in our lives we are tricked into believing that sounds and images are intimately, often causally, related, simply because they have been assembled together, either by way of Foley, or electronic editing or digital mix.
A revised version of the gestural association continuum will therefore include synchresis as an important signpost within the axis (see figure 2). We will keep rendition as additional mark within the continuum and we will rename it ‘intuitive complementarity’ to underline the mutual support materials can confer to each other in the creation of loosely multi-mediatic gestures, even when they do not occur simultaneously. We will also extend the continuum on the left-hand side to include cases in which there is no detectable relationship between materials in the two domains which appear to proceed on separate, unrelated phenomenological trajectories.
Continuum of gestural audio vs. visual association strategies – revised version
The continuum presumes, from left to right, a progressively growing strength of the associative links between sounds and visuals. In most circumstances this will correspond to the experience of viewers/listeners: spectromorphological ties in parametrically mapped passages will provide a more convincing gestural integration than synchresis. In turn, the temporal bonds in synchretic gestures will in most cases yield stronger coalescence than a loose, and often subjective, complementarity between the constituents. It should be pointed out that synchresis or parametric mapping, despite their phenomenological cohesion, might fail to accomplish gestural associations that are aesthetically plausible within the context of the work at hand.
4.2. Beyond gesture: discoursive associations
A few times during our discussion on gestural associative strategies we pointed out the inherently contextual essence of the concepts and categories introduced therein. The lexicon of audiovisual gestures, discussed in the previous section, is only the starting point for the articulation of syntactical relationships between the materials, of whatever nature and in whatever medium, and the genesis of a coherent discourse distinctive to the work in question. As previously pointed out, synchresis, along with contextual clues, naturally represents powerful agents in determining the strength of the audiovisual associations, opening a vast syntactical playground for composers as well as uncertain territories for audiences to venture into. The importance of at least some degree of gestaltism has been highlighted by many theorists and practitioners seeking to integrate sounds, music and images into a cohesive synergetic construct: ‘The principle of synergy, i.e. that the whole is greater than the sum of its parts, is fundamental to the nature of complex interactive dynamic sound and light systems and their resulting forms’ (Pellegrino Reference Pellegrino1983: 208).
Attention has also been given to the fact that fixations with literal mappings of visible and audible gestural profiles can be not only laborious to establish in any meaningful way, but also, compositionally speaking, redundant in most cases (Garro Reference Garro2005a). Synchresis itself, albeit an associative technique less strong than parametrical mapping, is applied at various degrees in different works, often resulting in antonymic audiovisual idioms: from the highly synchretic constructs of Cope/Howle's In Girum (2007) in which audio and video montages progress, inextricably tied on to one another, to the highly syncopated interplay between the sonic and the visual textures in Jean Piché's Australes (2011). In Piché's ‘vidéo-musique’ synchresis is purposefully avoided, as it is regarded as an obstacle to visual music expressive potential and, in the worst cases, as an artistic tautology.
The associative strategies utilised to combine sounds and visuals to form syntactical precepts, at a hierarchical level higher than lexical gesture, are themselves significant facets in determining a cohesive, global idiomatic identity of a piece of visual music. This is particularly germane to collaborative works where a sound designer and a visual artist generally work on their crafts at different times and in different locations. With few opportunities to sculpt ‘four hands’ truly integrated audiovisual objects, they will rather resort to discoursive associative schemes through which the sonic and visual structures meet ‘somehow’, ‘somewhere’ to form general traits of connotation for the whole. An indicative taxonomy (see figure 3) of these discoursive audio-versus-visual associations will inevitably include synchresis, which can be extended from a local lexical device (see the previous section) into a global paradigm when synchronisation between audio and video streams lies at the basis of most passages.
Discoursive audiovisual association strategies
An example of ‘synchretic’ paradigms in visual music discourse can be found in Nick Fox-Gieg's work Traffic Flow II (Reference Fox-Gieg2009) in which the soundtrack features a substratum of broadband, urban modulated noises, in constant flux, from the surface of which a rich assortment of sonic ripples, granular streams and distinctive sound objects emerge, each one of them invariably synchronised with a visual gesture taken from a multifarious collection of sweeping, quickly morphing colourful swirls and surges. There is no observable spectromorphological bond between the audio and the video streams; on the contrary the two occupy opposite idiomatic polarities, the former rooted in a soundscape of city anecdotes, the latter suspended in a phantasmagoria of abstract patterns. However, it is the enduring synchronisation between cohesive sets of sonic and visual emergences that, in combination with their buoyant quality, establishes the particular lingo of this work.
‘Contextual’ strategies (figure 3) utilise the weakest form of audiovisual integration, in which the two parallel streams contribute to the global discourse predominantly on the virtue of coexisting within the context of the same artefact. Synergy, rather than resulting from compositional intent, is left mostly to the viewer's reading of the concurrent stimuli. In the aforementioned work Australes by Jean Piché (Reference Piché2011) the use of contextual associative strategies follows a conscious artistic stance based on a deliberate ‘distantiation’ between sounds and visuals, as opposed to their convergence. The scarce and minimally changing sonorous and visual textures in the works by Simon Longo (Reference Longo2011) extend to the audiovisual domain the principles of ambient music and provide an insight on the ultimate consequences of contextual associations: the images can be regarded purely as one additional stream in the orchestrated whole which may, or may not, intersect contrapuntally any of the other (audible) streams.
‘Allegorical’ strategies work on premises stronger than simple coexistence. Sounds and visuals are similarly manipulated, but not necessarily in synch, so they may be seen as metaphors of one another on the basis of general descriptive categories, even when occurring at different times within the work. For instance, the slowly undulating visual layers in Maura McDonnell and Linda Buckley's Silk Chroma (Reference McDonnell and Buckley2010) can be seen as an allegorical depiction of the subtly rippled sonic textures, with the additional analogy one can draw between the three modal frameworks in the music and the three colour palettes utilised in the sections of the work.
‘Coalescing’ strategies are those which seek to establish audiovisual discourses principally through morphological convergence of the sonic and visual streams in most parts of a visual music work. Thus, such connection is more than merely temporal, and as a matter of fact it does not need to be, for it seeks to incorporate the two components into a recognisable pattern of phenomenological behaviours. For example, in Joseph Hyde's End Transmission (Reference Hyde2009) both visual and sound gestures are sculpted with highly degraded matter: sine waves and simple tones with very narrow spectral focus in the audio track; monochromatic, saturated shapes in the video track. Furthermore, the duration and amplitude envelope of most audio gestures reflect the brightness temporal profile of the white, shifting glows, both akin to closed attack–decay morphotypes with steep exponential onsets and terminations. Interestingly, what could have become an exercise in literal mapping acquires unexpected layers of complexity due to the fact that the synchretic paradigm is repeatedly challenged throughout the work: coalescence does not necessarily imply synchresis.
The diagram in figure 3 reflects the presumption that allegory, synchresis and coalescence all operate on the basis of a common audiovisual context to begin with. Furthermore, the diagram inevitably represents a schematic simplification: usually audiovisual discourse will show traits of all three higher categories, albeit with a penchant for one rather than the other two. It should also be pointed out that the degree of morphological integration, stronger for coalescing strategies but progressively weaker for purely synchretic and allegorical ones, does not necessarily imply or indicate a judgement on the aesthetic success of a visual music work: allegorical, or even merely contextual, discourses can result in beautifully cohesive, accomplished compositions, such as those quoted above as illustrative cases in point. Vice versa, synchresis and coalescence do not automatically guarantee artistic attainment and not even syntactical coherence, for example when they are implemented by means of a contextually redundant palette of disconnected materials.
4.3. Montage in visual music
The relationship between sound montage in electroacoustic composition and cinematographic montage is rather uncomfortable. For a videographer, montage is as important as the image itself for it establishes narratives and clear stylistic signage. Montage is for videography what syntax is for prose writers. For sonic artists, however, sound montage comprises a cyclic series of complex crafts such as acts of sculpture, sonic orchestration and acoustic landscape design.
Furthermore, there exists another difference between sound montage and video montage: transparency and compositing techniques aside, video montage operates only horizontally, juxtaposing scenes, also called ‘clips’ or ‘cuts’ depending on the context, into a sequence. Conversely, a sound artefact is crafted both by means of a horizontal juxtaposition of materials (‘takes’, ‘phrases’) and also through vertical mix of separate sonic streams (‘tracks’, ‘parts’).
The uneasy relationship between video montage and the electroacoustic idiom is often resolved in visual music works utilising video morphing in lieu of cuts or transitions (the traditional arsenal of montage). Scene cuts introduce a powerful shift in the viewer's relationship with the visual construct: a cut, or a transition to a different scene, will switch the attention from the photographic, pictorial and morphological attributes of the images to the story being told through them; from the intrinsic to the extrinsic (Barreiro Reference Barreiro2010). For instance, a simple change of camera angle signals a change in the spatial relationship between viewer and the story being told. Even more dramatically, a cut to a different scene or location or character will be the visual equivalent of a new chapter heading in a book. Cuts and transitions speak of cinematography, which in turn speaks of ‘pit music’ (Chion Reference Chion1994: 80): even the spectromorphologically richest electroacoustic soundtrack may, as a result of video montage, be repositioned to a role of, however exquisite, sonic commentary of the action being shown, or hinted to, in the video.
5. VISUAL MUSIC TECHNIQUES IN PATAHFootnote 4
Patah (Garro Reference Garro2010) is a visual music work produced between 2009 and 2010 in the Music Technology Laboratories at Keele University (UK). It was premiered at the Sonic Arts Oxford series, Oxford Brookes (UK) in February 2010 and it has since been screened in many national and international events, exhibitions and conferences including the Cinesonika First International Film and Video Festival of Sound Design, Simon Fraser University, Vancouver (Canada) in November 2010. A still frame from Patah (see figure 4) was also exhibited as pictorial artefact in two international art festivals: Contemporanea, Udine (Italy) in October 2010 and Minumental Exhibition, Cincinnati (USA), in February 2011.
Black and white reproduction of Patah (still frame)
Patah is an investigation into abstract spectromorphologies articulated in both the audio and the visual domain. Instead of revolving solely around the evolution of audible spectra and the construction of a sonic discourse, such as in my previous acousmatic works, this compositional endeavour has been extended to the integrated audio/video media through the utilisation of computer-generated abstract animations. These are purposefully constructed on a black background to emphasise their ‘object’ quality. On one hand the viewer is forced to contemplate their shifting morphologies and behaviours. On the other hand, the focused characterisation of the abstract shapes on a dark backdrop encourages synchretic and coalescing discoursive strategies as well as synchretic gestural design.
The materials utilised in this work are chosen from a palette of abstract animations and abstract sounds. The whispers, vocal utterances and more or less comprehensible phrases bring this fantasy world much closer to home than it would otherwise be, while adding a textual layer of ambiguity through the shifts between intelligible speech/whispers, and the granular textures/streams obtained from them through computer processing. Similarly, the video track features fragments of disguised reality, in the shape of a cityscape filmed with handheld camera, that emerge from within the animated abstract silhouettes, as glimpses of an alternative existence hidden just behind the dominant layer (see for example the passages 3′10″–3′13″, 10′30″–11′00″ and, more explicitly, the gestural section 11′45″–12′15″).
Most abstract imagery in Patah has a filamentous quality, which creates ‘fractures’, ‘patah’ in Indonesian, within the texture of the shapes themselves. There is nothing to see through these rifts; there is no background. These fibrous entities do not have an environment within which to exist and operate. Thus, the inceptive compositional stratagem consisted in designing the sounds as a synaesthesic manifestation of a visual background: the long, dissonant resonances are the environmental background that you ‘see’ through, behind, around the streaked visible entities.
5.1. Metamorphic tendencies in Patah
More than an iconological trait, the use of morphing represents a strong idiomatic element of my creative work. O Que A Menina Ouve, acousmatic composition (Garro Reference Garro2005b), is constructed utilising entire passages and sections developed through morphing processes which, in terms of the acoustic imagery, are akin to playful transmogrifications. Artistically speaking, the fascination with transformative processes represents a homage to many of the pioneers of sonic art – in particular Trevor Wishart, who utilised extensively such techniques, and indeed researched their technical and idiomatic implications.
Both in my acousmatic and audiovisual works metamorphosis occurs slowly, during time spans of minutes, or faster, in less than 20–30 seconds, or very rapidly, sometimes within the space of a few frames, or fractions of a second. They feature mutations between materials showing various degree of surrogacy (Smalley Reference Smalley1986: 82–3) and mimic, in terms of their duration, and model the gestural trajectories, the textural motions, the pace and inertial responses of phenomena occurring in cosmology, the natural environment or biology, or even those typical of everyday domestic, labour and leisure activities (see Table 1).
Gestural trajectories in metamorphosis processes.
The use of morphing processes in Patah serves different purposes and is motivated by the urgency to highlight the spectromorphological approach to the digital synthesis of visual material, and the coherence of such methods with the stylistic credo that informed my acousmatic compositional projects thus far. The viewer is constantly presented with shifting shapes, unstable boundaries, objects that morph into one another, at various tempos and portamentos. Facets previously seen or heard in a certain passage of the piece are not merely re-presented; they are ‘arrived at’ in the end, or in the middle, of long processes of audio/video mutation prompting a contextual re-view of the past, present and future materials and articulations. An example can be seen/heard in the section 8′48″–11′47″, where parallel morphing processes, operating concurrently in the visual and audio domains, elaborate and transform both visual shapes and sonic streams presented earlier in the piece, posthumously revealing their provenance, exploring further dimensions of their phenomenological being as well as fresh avenues for their audio-versus-visual dimension.
The idiomatic importance of metamorphosis processes in Patah is bolstered by its role as a proxy for montage (see section 4.3). Montage, with its cinematographic interplay of scenes, cuts and camera angles, would introduce layers of visual narrative that would impinge on the syntax of the work. Such syntax is essentially abstracted from the materials themselves (Emmerson Reference Emmerson1986: 23).
5.2. Gestural pairing in Patah
The relationship between visible gestural motion and its sonic counterpart represents the territory in which Patah resolves the hierarchical conflict between two competing messages, hopefully without raising any to the a leading role to the detriment of the other. The end of the first section (3′20″–3′28″), for instance, features a red morphing shape undergoing a process of dilation–rarefaction–dissolution, the progress of which is emulated by the spectral trajectory of the accompanying sound. Both the context and the phenomena at play conjure up a mapping process which, in reality, is much more synchretic than parametric (figure 2). Rather than following a topological isomorphism, the sound notionally augments the scope of the visual event; it inflates it, pushing its energy through the void that is progressively left behind by the dissolving visible streaks. The sound thus buttresses the visual process of dissipation with a sonorous representation of a change in state from liquid to vapour, from dense to rarefied, by means of the ebullient sonic textural motion that characterise the climatic edge of the sound in question.
The second section (3′45″–4′48″) features giganticFootnote 5 shapes undergoing rapid processes of morphing, both chromatic and morphological, sometimes sweeping across the entire visuals field. The accompanying sound gestures were designed not only to reflect the fast motions of the visuals but also to match their energetic profiles and sensorial impact within the context of the piece. In this section the paradigm of the objet audiovisuel is established and then challenged soon after; for example between 4′14″ and 4′21″ a tight motif of sonic gestures is only partially mirrored by the behaviour of the visual gestures, which in fact appear only towards the end of the phrase, in synchronisation with a crepitating granular sound: on one hand the audiovisual integration is interrupted; on the other hand it is, at least conceptually, elevated to a higher status whereby an invisible sweeping filter, for instance, not only produces a change in the sonic qualities but also seems to engender the appearance/disappearance of some visible wavelengths. The visible and the audible are no longer only parallel fields in which material of different constitution manifest themselves; they are instead, and as well, contiguous phenomenological playgrounds for time-varying stimuli. The articulation of spectromorphological properties in Patah does not obey principles of audiovisual mapping and it goes beyond, or beside, the montage of congruous streams. Rather, it arises from a dramaturgical approach in which materials in different media are all actors delivering their lines sometimes in unison, sometimes in dialogue, sometimes in monologues, all of them evoking ‘layers of meaning and interpretation’ (Rudi Reference Rudi2005), all compelled to obey only the internal precepts of the composition.
5.3. Textural motions pairing in Patah
The section between 4′49″ and 8′46″ resolves the tension accumulated in the previous, highly active, parts presenting the viewer with a long process of transformation in which the granularity of the elements, both audio and video, provides the distinctive trait for this movement (the equivalent of a ‘Moderato’ in terms of musical tempo). The abstract entities in this part show second-order surrogacy (Smalley Reference Smalley1986: 82); causally we cannot recognise anything real, or figurative, yet the effervescence of the granular surface-textures, both sonic and visual, reminds us of something somewhat organic, ebullient, mutating, as alive as these matters are allowed to become. The morphological pairing, in this segment of the piece, shows some degree of tight correspondence, at least in the level of granularity within the attack–effluvium continuum (Smalley Reference Smalley1986: 72) exhibited by some of the audible and some of the concurrent visible streams.
6. Conclusions (quasi a manifesto)
For decades the electroacoustic community has been at the forefront of sonic experimentation, pioneering aesthetic solutions and audio techniques later assimilated in film, television and videogames as well as in the recorded music industry. The contemporary art of noises inherently strives to manifest itself through the articulation of sound materials of incomparable beauty: recordings of outstanding fidelity, innovative sound synthesis and processing methodologies, sonic novelty and sophistication, spectral richness, emotive impact and acoustic detail. As for the moving image, it is the film and television industries, with their immense financial resources, which fostered similar technical developments in the fields of animation, high-resolution imagery and 3D cinematography. Advertisements, popular music videos and videogames nurtured generations of viewers, listeners and players increasingly accustomed to a deluge of highly advanced visual effects and to a language based on the limitless plastic manipulation of virtually any element that occupies the vision field, the work by award-winning animator Niko Tziopanos providing just one example (Tziopanos Reference Tziopanos2010).
Visual music ought to follow the example and legacy of these driving forces and cultivate a fresh, experimental language of sounds and images which utilises, and possibly innovates and surpasses, the technical wizardry and idiomatic vigour we have so far witnessed. Pre-eminent visual music practitioners are already showing the way towards motion graphics of great beauty and technical sophistication, strong enough to balance the lush and evocative electroacoustic soundtracks featured in their work. Dennis H. Miller (Reference Miller2008) and Bret Battey (Reference Battey2005) create animations of outstanding painterly quality in which motions, colours, shapes, lines, particles and clusters continuously aggregate, collide and dissipate, creating animated canvases of stunning graphic detail that come to an even more meaningful existence in synergy with the music.
The potential of 3D imagery also ought to be germane to visual music, especially in consideration of the importance of spatial articulation in the electroacoustic idiom, both in terms of virtual acoustic space design and in terms of multi-loudspeaker performative projections of soundtracks in concert spaces, installations and projection halls. Echoing the concern with articulation of space typical of sonic arts, visual music will eventually be disenthralled from the two-dimensional confines of the projection screen and will harness the entirety of three-dimensional field.Footnote 6
The virtual three-dimensional space projected in 3D cinema halls spans all the space in front of the viewer, from left to right, from above to below, from within his or her grasp to the faraway background, yet it cannot avoid the peculiar interference of the screen plane, a black matte box getting in the way of a full, unobstructed view. Holography could overcome this shortcoming, providing a fully open three-dimensional domain for visual shapes to inhabit. Holography, a technique not necessarily suitable for cinematography, is potentially the perfect partner for electroacoustics’ spatio-temporality.
These are possible avenues for the development of a visual music that keeps sound and the experience of the electroacoustic community as vital guiding principles. Visual music is very closely related to the sonic arts, in a variety of ways. The advent of the visual into electroacoustic circles follows logically the artistic leap that emancipated noise and gave it equal status within the musical arts; it enfranchises the visible as an additional (additional to the audible) set of experiential attributes we can use as means for artistic expression. Audiences, of whatever extraction, will not (and should not) engage in a process of intellectual validation of the medium but will, instead, try to enjoy, decode and make (artistic) sense of whatever message is being transmitted and received. The worthwhile line of enquiry ahead of us is not the one which seeks to deliberate upon visual music's legitimacy as post-electroacoustic trend. It is, instead, one that puts the visual music idiom under scrutiny to evaluate whether or not it tells us, sonic arts composers and listeners, anything meaningful and whether or not it does so while firmly and genuinely holding sound at its heart, the place where we want it to be.
