1. INTRODUCTION
This article addresses the creation (and to some degree the analysis) of media-based artworks by considering ways of classifying and linking multimedia morphologies, informed by cognitive linguistics, catastrophe theory and narratological studies.
The semantic features necessary in order to identify transmedial narratives will be delineated, and a modelling approach to narrative will be proposed, based on qualitative mathematical interpretation of morphological components and their syntactical structure. The relation between morphologies and emergent symbolic structures will be investigated, and their relationship as expressed in the acousmatic music field will be interrogated. Finally, a practical application of these concepts for the analysis of a visual music composition will be presented, using Diego Garro’s Patah as an example.
2. TRANSMEDIAL NARRATIVES
From cognitive linguistics, the concepts of morphodynamical classification and conceptual metaphor are particularly crucial to transmedial narratives, since both of these concepts are closely associated with meaning formation mechanisms.
Morphodynamical classification deals with the study of forms and their dynamics; it is a mathematical approach to modelling natural morphologies and their evolution (Petitot and Doursat Reference Petitot and Doursat2011). Morphodynamics are based on René Thom’s topological studies of singularities and morphogenesis, which together formulate the field of catastrophe theory (Thom Reference Thom1975, Reference Thom1989).
Metaphor in cognitive linguistics is defined as the ‘the cognitive mechanism whereby one experiential domain is partially “mapped”, i.e. projected, onto a different experiential domain, so that the second domain is partially understood in terms of the first one’ (Barcelona and Valenzuela Reference Barcelona and Valenzuela2011: 27). In this article, conceptual metaphors will be considered in terms of their function in temporal-based multimedia art forms, focusing on the ways in which metaphorical thinking is involved in the mechanisms of forming meaningful discourse or narrative within acknowledged media constraints.Footnote 1 More specifically, this article questions how this attributed or emergent meaning is derived from morphologies, gestures, structures and their temporal evolution.
On a narratological level, two aspects of this study are significant: the interdisciplinary and transmedial nature of narrative. To interrogate these aspects requires investigation of how narrative projects onto different conceptual domains.Footnote 2 Each domain can encompass a different conceptualisation,Footnote 3 be it of different sign systems, different media types and distinct media identities (materialities), various modalities (language or non-linguistic based, representational or abstractFootnote 4 ), or pragmatics. These different conceptual domains, and their cognitively derived, blended constructs, are connected, mapping the elements of one domain onto those of another. Further analysis of this mapping aggregate can address both the questions of how narrative, as a concept, extends to disciplines other than narratology, and how it could be meaningfully expressed through different media.
In their introduction to Narratology in the Age of Cross-Disciplinary Narrative Research, Heinen and Sommer (Reference Heinen and Sommer2009: 3) propose that ‘narrative is everywhere’ and ‘seems to be a kind of vertex around which other discourses orbit in ever closer proximity’. In Narrative Across Media, Marie-Laure Ryan claims that ‘[n]arratology, the formal study of narratives, has been conceived from its earliest days as a project that transcends disciplines and media’ (Ryan Reference Ryan2004: 1). Claude Bremond points out that this transcendental trait of narratives comes from the very object of narrative studies, the story, which
is independent of the techniques that bear it along. It may be transposed from one to another medium without losing its essential properties. (Bremond Reference Bremond1973: 12 quoted in Ryan Reference Ryan2004: 1, Reference Ryan2006: 3)
The precise nature of the transmediality of narratives remains an open subject of investigation in narratological studies, and its precise definition is therefore beyond the scope of this article. Nevertheless a description of the use of the term in the context of this study is given. A prerequisite for the description to be valid is the assumption of a cognitive narratological stance, identifying narratives as mental constructs – mental images evoked in the recipient given the right stimuli (text or otherwise). In these terms transmedial narratives are indentified as those media-based constructs that evoke mental imagery in the recipient in such a way that some sort of causal and goal-driven development/evolution can be derived. ‘Transmedial’, in the context of this article, does not presuppose the use of multiple media, but is rather intended to indicate elements of narrative that remain invariant during the actualisation of narrative in different media.
This article focuses on narrative elements that can be identified in abstract (i.e. non-representational) forms of art – particularly those elements that enable us to detect coherence and derive meaning from works mainly using non-referential material. Abstract art forms are considered as a type of transmedial narrative, a claim supported by Ryan, who includes them in the category of metaphorical narrative mode (Ryan Reference Ryan2006: 15). Traditional rhetoric considers metaphor as a stylistic feature: a difficult Footnote 5 figurative element that alters ‘[t]he meaning of a word and so requires cognitive processing’ (Fulton et al. Reference Fulton, Huisman, Murphet and Dunn2005: 23). Considering Ryan’s description of transmedial narratives as mental constructs (Ryan Reference Ryan2004, Reference Ryan2006: 7) and of narrative text as a broadly defined story artefact on media, not limited to language-based text (ibid.), metaphorical troping seems to include conceptualisation as one of the involved cognising routines. In that sense metaphor is suggested as a model for organising thoughts and concepts, rather than a rhetorical trope (Spitzer Reference Spitzer2004: 15): ‘[w]e not only speak in metaphorical terms, but also think in metaphorical terms’ (Evans and Green Reference Evans and Green2006: 295); even ‘[l]inguistic expressions that are metaphorical in nature are simply reflections of an underlying conceptual association’ (ibid.).
2.1. Semiotic complexity of transmedial narratives
Examining narrative in terms of the media in which it is conveyed, requires, among other things, examining the semiotic value these media carry for narrative (Ryan Reference Ryan2006: 17–30). Analysis of media as semiotic phenomena requires consideration of the codes and sensory channels that support them, leading to three broad media groupings: verbal, visual and aural (Ryan Reference Ryan2006: 18). Narratological investigation, then, is concerned with the affordances and constraints these different types of groupings (and signs they carry) impose on narrative.
Some signs can carry stronger narrative qualities and significations than others. Ryan identifies language as the natural medium of expression for narratives and states that some media ‘[r]equire a far more extensive gap-filling activity than verbal texts to be interpreted narratively, though they can usually be appreciated aesthetically without paying attention to their narrative message’ (Ryan Reference Ryan2006: 20). These media can nevertheless make original narrative contributions and that, when they are used collectively ‘in multichannel media, each of them builds a different facet of the total imaginative experience’ (ibid.).
For the composer of multimedia narratives, however, considering these affordances in their totality can prove to be a daunting task, especially taking into account semantics, and their involvement in meaning formation.
2.2. Defining narrative
My own multimedia narrative studies focus on structure and its dynamics, primarily in terms of the growth and evolution of distinct narrative states – suggesting interrogation of the causes and events that drive that evolution – and the construal that emerges from associative, metaphorical cognising routines evoked in a receiver.
Ryan proposes eight deductions for evaluating narrativity in a text (Reference Ryan2006: 6), distributed in ‘[t]hree semantic and one formal and pragmatic dimensions’ (Ryan Reference Ryan2006: 8), resulting in a flexible ‘pick and choose’ semantic, minimally formal and pragmatic framework that seems quite attractive in defining narrativity in abstract multimedia art. Such a framework allows the composer to emphasise certain features and to determine the degree of narrativity present in an artwork.
Out of Ryan’s eight deductions we are here particularly interested in those addressing world construction, temporality, causation and reception. These conditions portray narrative as the different possibilities for meaningfully communicating content, the different principles employed to organise and order events in any medium, and consequently the development-progression of spatio-temporal representations of these events as thematic elements in a larger structure. This investigation focuses on the fashion in which these events cause and undergo transformation, not merely in order to carry information, but also to demonstrate causality and relations such that meaning can be derived.
Structurally, the minimum conditions to satisfy the above description are:
-
∙ At least two events or clauses need to be present.
-
∙ There needs to be transition or transformation between events.
-
∙ This transition needs to be causally transparent in order to potentially be considered meaningful.
Interestingly, this is no different from A. C. Danto’s description of the ‘basic structure of all narrative texts’, which defines narrative as: ‘x is f at t1 ○ q happens to x at t2 ○ x is h at t3’ (Stierle 1972: 178 quoted in Altman Reference Altman2008: 5). This formula describes a state (or agent) existing at a particular moment in a time continuum, to which an event happens at another moment, bringing forth a change rendering that state into another identifiable state.
To summarise, this section describes abstract art forms as a metaphorical mode of transmedial narratives, and presents the prerequisites necessary to identify multimedia artwork as transmedial narrative. This is premised on the perspective of cognitive narratology, identifying narrative as a mental construct that involves metaphorical thinking in the process of meaning formation. In an effort to make an informal description of what constructs are considered narratives in the context of this article, some of the semiotic idiosyncrasies and semantic implications of transmedial narratives were identified, as well as the minimum structural conditions considered fundamental for narrative to occur.
3. CONCEPTUALISING A GENERIC DOMAIN
However, even abstracting narrative to a simple formula such as Danto’s does not simplify the compositional and analytic narrative act – not unless one relates the above narrative conditions (states, transitions, causal transparency) to a generic domain,Footnote 6 abstracted enough to be independent, uniformly analogous or even isomorphous in content, yet intricate enough to address specificities.
3.1. Catastrophes and morphogenesis
Intuitively, one can be inclined to conceive of this domain in mathematical terms – mathematical not only in the sense of formal descriptions and quantitative estimations, but more importantly in the sense of abstracted qualitative appreciations.
A two-part question arises concerning the above hypothesis of a generic domain. The first part inquires as to the type of entities represented in this domain, and the second, the level of interaction and relation between these representations and their projections in the phenomenological realm.
One answer to the question of entities is: forms and their dynamics. Forms are here described independent of content, as dynamic, modulated, enveloping processes of material substrates, that is, morphologies. Catastrophe theory deals with this fundamental question of morphologies, described by Thom (Reference Thom1975: 1) as ‘[o]ne of the central problems studied by mankind … the problem of the succession of form’. Thom supports the claim of the independent nature of form: ‘we can construct an abstract, purely geometrical theory of morphogenesis, independent of the substrate of forms and the nature of forces that create them’ (Thom Reference Thom1975: 8).
Catastrophe theory investigates the set of singularities where qualitative discontinuities are observed during the process of a form’s unfolding around an organising centre. These singularities are called catastrophes. If the unfolding of the form is describing a system’s behaviour, in the neighbourhood of the castastrophe set that behaviour would significantly change, and the system would thus collapse into new states, changing form. This change of states, which can be observed geometrically as a form collapsing into another, is also called morphogenesis.Footnote 7
Given the narrative description adopted in section 2.3, it is natural to interrogate the succession of narrative states – and the causes forcing this succession – as a study of morphogenesis.
3.1.1. Salience and pregnance
In identifying what these narrative states (and events affecting them) are, the concepts of salience and pregnance are found to be essential. Thom uses ‘salience of forms’ (Thom Reference Thom1989: 3) to describe the process of forms becoming perceivable – that is, distinct from a continuous background – as qualitative discontinuities arise. A basic distinct auditive discontinuity, for instance, would be the eruption of a sound in a continuum of silence. Another example is found in the introductory section of Bayle’s Lignes et Points (Reference Bayle1966), in which a quiet melodic ‘back and forth’ of a whole tone interval (continuum) is interrupted twice (at 0′13″ and 0′20″) by the sudden eruption of a high pitched ‘cry’-like sound (see ‘cri-silence’ sound typology, in Bayle Reference Bayle2013: 72).
Forms interact and affect each other, and have dynamic propensity, which Thom calls ‘pregnance’ (Thom Reference Thom1989: 6). Pregnancies are contained in salient forms, and can be transported from one to another (think of kinetic energy been transmitted when one billiard ball hits another, causing it to move). One could describe these dynamics as an inherent quality of forms. Saliencies are ontological, while pregnancies are strongly associated with causality and explicate how one form can influence another. Salient forms and their pregnancies are considered structurally stable, that is, they tend to retain their qualitative nature, kinematic and dynamic,Footnote 8 for relatively small disturbances.
One deduction could be that narrative states are also stable structures, distinguishable in a narrative, that change under the influence of dramatic, equally distinguishable events affecting them.
3.1.2. Seven elementary catastrophes and 16 archetypal morphologies
The important contribution of Thom was the realisation that, for systems controlled by up to four parameters (as many as we need to describe phenomena in the four-dimensional space-time continuum), there are just seven elementary catastrophes (Figure 1) describing these changes. Of course the complexity of forms as found in our world arises not only from these elementary catastrophes, but also from emergent properties of continuous interactions (e.g. superposition and dynamic inference) amongst them, producing infinite possibilities. Thom proceeded to identify 16 morphological archetypes (Figure 2) found in these seven models of form evolution, detecting these morphologies in various domains such as optics, biology, semantics, linguistic structures, etc.
Elementary catastrophes.
Archetypal morphologies.
The following section focuses on catastrophic theoretical models applied in semantics, mainly an elaboration of Thom’s archetypes by Wolfgang Wildgen (Reference Wildgen1982), in an initial effort to highlight the relationship between morphological structures and symbolic structures. This investigation is based on Thom’s observation that salient forms are charged with semantic significance (Thom Reference Thom1989, Reference Thom1975), meaning we can recognise the morphologies of forms we understand. This is the start of an investigation in understanding and modelling different ways narratives are constructed, specifically how meaning-formation mechanisms function in narratives and relate to narrative structure (and its dynamics). To clarify why meaning formation mechanisms and semantics are important to narratives, earlier points made in this article should be considered, where narratives were described as mental constructs – symbolic structures formed by the recipient. It is these symbolic structures that make it possible to recognise and associate with narratives. An extra variable here is that understanding these symbolic structures is closely linked to understanding the morphological component of which they are comprised, as that component too is semantically charged.
3.2. Syntax and archetypal morphologies: the semantic archetypes
Analysing the geometry of the elementary catastrophes can provide a convincing and flexible approach to modelling the correlation between form and meaning, especially given that anything meaningful to human beings is mostly manifested in the phenomenological realm of the world as we perceive it – a topological continuum, where catastrophe theory is applicable.
A first step, then, would be to analyse some of the archetypal morphologies and interrogate their semantic significance. Wildgen, among others, elaborates Thom’s application of catastrophe theory in linguistics and semantics, pointing out that ‘[t]he catastrophe theoretic model is a synthesis which reintroduces real time in the stucturalist framework, thereby creating a dynamic model of language’ (Wildgen Reference Wildgen1982: 30).
Wildgen outlines Thom’s indirect correlation between form and meaning, allowing a first-degree semantic interpretation (basic attribution) of the geometrical analysis of ‘[s]imple paths in the bifurcation set of an elementary universal unfolding’ (Wildgen Reference Wildgen1982: 25). These paths are called semantic archetypes. The attributional process is based on the following:
-
1. Stable attractors of an elementary unfolding are interpreted as:
-
∙ satellites around the dynamic centre of a sentence (pronouns, noun phrases, names), concerning linguistic structures;
-
∙ stable entities or classes of entities (individuals, objects, substances, natural kinds, qualities, states and phases in a dynamic frame, etc.), concerning cognitive structures.
-
-
2. Catastrophes in the unfolded geometries are interpreted as semantic archetypes, and are attributed to:
-
∙ verbs, relational terms, expressions of change, events, action and interactions, linking stable states with agents in linguistic structures;
-
∙ changes of state, position, qualities and actions in cognitive structures.
-
Wildgen (Reference Wildgen1982) makes a distinction between elementary, semi-elementary and higher archetypes. This distinction depends on the complexity of the path chosen, on the unfolded geometry of an elementary catastrophe. The difference between elementary and semi-elementary or higher archetypes is that the former can usually be described by one state or simple transitions between states, while the others usually describe a continuum of possible paths between two states, and hence need to be analysed in terms of phase transitions.
Figure 3 describes some of the archetypes as described by Wildgen and some of their semantic correlates.
Archetypes and their semantic correlates.
To summarise, this section investigated fundamental morphological components observed in any perceivable form (geometrical, semantic, aural, etc.). The following section investigates how these morphologies are perceived by the recipient, and how the understanding of morphological components relates to higher symbolic understanding of concepts manifested as phenomena.
3.3. From form to content: what natural morphologies tell us about the world as we ‘know’ it
In essence, the main question here concerns the connection between perceiving and recognising structure in abstract morphologies, and higher symbolic attributions that are necessary for meaningful content to arise. The answer to such a question is of particular importance for understanding underlying processes taking place in transmedial narratives. Section 3.4 will extend this in an investigation concerning the field of acousmatic music – a field which, in a number of ways, can be considered quite well equipped for conveying rich narrative forms:
[W]e will look at the universe of images and try to make explicit rough information concerning their morphological structure … this is possible if we use specific mathematical theories for categorizing the structures into non-symbolic syntactic scripts or frames, which can then be translated into symbolic syntactic structures. (Petitot and Doursat Reference Petitot and Doursat2011: 4)
Petitot (Reference Petitot1989), in an article complementary to Bayle’s article on sound image (i-son) (Bayle Reference Bayle1989), admits the existence of a specifically morphological component of perceptual organisation on which higher order semiosis can be built (Petitot Reference Petitot1989: 171). That component is where ‘[m]ost form-bearing elements can be founded’ (ibid.: 172).Footnote 9 The morphological level – very difficult to theorise without using ‘[p]hysico-mathematical theory of forms and moprhogenesis’ (ibid.) – is the basis for the ‘[h]igher – symbolic – cognitive level of perceptual languages’ (ibid.). Speaking about this morphological level of phenomena – an autonomous level, inhabited by catastrophic infrastructures – Petitot claims that it is manifested through perceptual routines on a morphological level, and displays relative autonomy and its own structural profile (ibid.: 175).
Petitot’s proposition of a principle of double emergence and double organisation deals with the dual enigma of object (the morphological dimension of the natural sensory world) and subject (how meaning attaches to symbol). Petitot identifies two aspects of the natural world: a spontaneous, autonomous, emergent, self-organised morphological aspect, and a projected aspect – a cognitive construction. The morphodynamical classifications of the morphological aspect of the natural sensory world do not constitute a direct input to the symbolic level, but rather to a sub-symbolic, morphological one, which is governed by the same type of qualitative, mathematical classifications (morphodynamics). This morphodynamical correlation makes it easier to identify associations between the objective morphological aspect of the natural world and the sub-symbolic cognitive structures that are the product of conceptual processes in the mind of the recipient. Once sub-symbolic references of morphological information are established, emergent, symbolic, higher conceptual structures arise. The projection of symbolic structures onto the phenomenological level formulates the world as a perceived phenomenon, and in the process these symbolic structures acquire meaning.
Petitot describes the above as double emergence, double organisation: first, the emergence of an interpretation based on morphodynamical classification of critical (catastrophic) auto-organisational phenomena, and second, the emergence of assimilated macro-content in the attractor neighbourhood of a fundamental micro-dynamic.Footnote 10
The phenomenological manifestation of the natural world is identified as a result of three independent processes (Petitot Reference Petitot1989: 178):
-
∙ The emergence of a symbolic conceptual structure from a subsymbolic dynamic level.
-
∙ The projection of this conceptual structure onto the phenomenological world (during which symbolic structures acquire meaning).
-
∙ The emergence of a (spontaneous) structured morphological level from a nearly amorphous substrate, based on observation and mathematical classification (morphodynamics) of catastrophic phenomena.
The significance of double emergence, double organisation (as with the case of semantic archetypes) lies in the fact that an essential continuity between the physical and phenomenological modes of existence is preserved, and phenomenological differences are attributed to a morphological level of spontaneous (mathematical) classifications of natural phenomena.
3.4. The sound image in acousmatic composition
This section concentrates on how the idea of double emergence and double organisation has been applied in the acousmatic music field. Music is a useful example here, since it is arguably one of the most abstracted ‘languages’. We will take acousmatic music as a useful example of the category of ‘multichannel’ media art forms; the term ‘multichannel’ is used here to denote multiple semiotic channels (as opposed to its common usage in electroacoustic music to denote the number of audio channels). In this sense, acousmatic music is significantly more ‘multichannel’ than for example tonal instrumental music – as Smalley states, ‘although acousmatic music can be received via a single sensory mode, this does not mean that other senses lie dormant’ (Smalley Reference Smalley2007: 39) – and is further useful in that it can occupy the full spectrum between reference and abstraction. Finally, musical forms (of primarily abstract content) can be considered the most challenging to grasp due to their immateriality, an aspect that supports the concept of transmediality of narrative.
The work of François Bayle (Reference Bayle1989, Reference Bayle2007 etc.) on the notion of sound image (i-sound) is of great significance for this investigation between morphological and symbolic structures in the field of acousmatic music. Bayle has made observations that parallel those of Petitot (Reference Petitot1989; Petitot and Doursat Reference Petitot and Doursat2011) concerning linguistics and phonology. An advocate of catastrophe theory and archetypical morphologies (Bayle Reference Bayle1989, Reference Bayle2008, Reference Bayle2013) in the field of acousmatic music, Bayle, according to Petitot (Reference Petitot1989: 171), exhibits a wealth of morphological and morphodynamical lexicon in his descriptions of sound images, sound structures and sound organisations – more than a mere vague, poetic approximation, but rather a demonstration of how the previously discussed morphological component can be the basis for higher-order semiosis in music.
Bayle (Reference Bayle1989) establishes an isomorphic relationship between content and the perceptual/cognitive routines involved in interpreting it – between object and subject. The correlation of the object (i-sound) with the audible is reflected as a trichotomy found imprinted in the nature of both the object and the phenomenon: i-sound represented as a triadic im-sound, di-sound and me-sound composite (Bayle Reference Bayle1989: 168, see below), and the audible as a complex process involving appearance, attention and correspondence. This isomorphic relationship is the basis of the process of making musical structures understandable (ibid.: 166) through constant exchanges, shifts and interactions.
As products of the double emergence described by Petitot (Reference Petitot1989), im-sounds are referential, products of isomorphic attributions to a source – an event, an object, a salient form. They dwell in the emergent world of structured morphologies (Bayle Reference Bayle1989; Petitot Reference Petitot1989). Di-sounds are emergent diagrammatic conceptualisations – abstracted, symbolic, schematic structures. They arise as artefacts of the quest for origins, evolutions, consistencies, directions, intentions, dynamics, causalities, paths and flows (Bayle Reference Bayle1989: 167). Projected onto phenomenological reality, these become metaphors (me-sounds): objects for open translation, associations and interpretations, emerging from projective relationships of i-sounds with target figures in the semantic field (Bayle Reference Bayle1989: 168). Thus the cognitive function in music is described both from ‘above’ – symbolic functions rearranging the real according to a projected world – and ‘below’, through the qualitative appreciation of morphological organisations serving as input to our perceptual apparatus (Bayle Reference Bayle1989: 167).
Concerned with structures and their evolution in sonic space, Trevor Wishart (Reference Wishart1996: 93) proposes catastrophe theory as a valid candidate for categorising and modelling sound objects. He recognises objects with dynamic morphology as those in which most, or all, of their properties can be in a state of change. Though he avoids a formal approach to categorisation, he defines structures of certain morphologies in terms of gesture and perceived natural phenomena (ibid.: 102). Robert Hatten describes gestures as ‘significant energetic shapings through time’ (Hatten Reference Hatten2004: 95), and Wishart does not fall far from this description; as multivariable dependent phenomena varying in time, he recognises gestures as dynamic morphologies. If we can generalise from Wishart’s attempt at classification of morphological archetypes of vibrato (Wishart Reference Wishart1996: 120), we can identify three characteristics for the classification of gestures: their magnitude, their first-order morphology and their second-order morphology – in other words, how intense they are, of what type, and how they vary.
In terms of natural perceived phenomena, Wishart uses the term ‘landscape composition’ to signify the process of recognising an imagined source of the sound image in musical works (Wishart Reference Wishart1996: 131–4). He describes the landscape of a composition in terms of three components: the nature of the perceived acoustic space, the disposition of the sound objects in space and the recognition of individual sound objects (ibid.: 140–59). Wishart draws a parallel between composition with sounds and composition with sound-images, mostly referring to representational sound objects. Given Petitot’s descriptions, however (see previous section), the referential nature of image construal is inherent to our conceptual functions, based in part on the natural organisation of morphologies and our schematic representation of them, and therefore does not need to apply to representational material alone (Young Reference Young1996; Andean Reference Andean2010). When remote surrogacy (Smalley Reference Smalley1997: 112) is dominant, there is a ‘[u]seful way of interpreting the “abstract” phenomenon’: ‘matching certain aspects of a sound’s morphology to known source-cause patterns, though these may be fleeting, hybridised or in a state of constant flux. This is […] context-related and always subject to the prejudices of source-cause conditioning’ (Young Reference Young1996: 76). Metaphorical associations such as those described by Wishart (Reference Wishart1996: 165) take place even in the case of abstract material. Establishing projective associations between the material (abstract or representational) and semantic significance – through transformations and contextualisation using aural cues (Young Reference Young1996: 78) – is an important process for the metaphors to be considered coherent.
John Young has addressed the importance of transformation for the sound image. He describes how extended transformations over time ‘[e]stablish the potential of meaning extending beyond the basic real world references – as metaphor translates sign into symbol’ (Young Reference Young1996: 83). Moreover he explicates how mediation (shifts in the surrogacy level) and juxtaposition (sequential or simultaneous combination of polarised orders of surrogacy) express those shifts in the reality-abstraction continuum (Young Reference Young1996: 84).
Young further elaborates (Reference Young2007) on how we might ‘[s]eek strategies to incorporate referential meaning at more than one level’ (Young Reference Young2007: 26), and continues by stating that these levels ‘[h]ave consequences for structure at a purely morphological level’ (ibid.: 27). In the same article he addresses the issue of potential uses of sound imagery, in the context of spectral transformations, in mediating a coherent articulation between the spectromorphological and the referential (see Young Reference Young2007: 28–32). This is quite concisely described by Andean (Reference Andean2010), who proposes that, even in the case of a composer working from a musical rather than a narrative standpoint, the effects of the sound image will ‘[a]lmost inevitably construct a string of mental imagery’ (ibid.: 110). Imagined attributions of source and causal deductions will result in an emergent narrative layer (ibid.: 110).
The above suggests that the acousmatic space-form Footnote 11 (see Smalley Reference Smalley2007: 40), and its artefact the sound image, indicate a large capacity for facilitating metaphorical thought and association (from musical to real-world references), and in that sense can be considered as a well-equipped medium for conveying complex narrative forms:
‘In this landscape, the listening subject works – ‘symbolically replays’ – the music. One ‘pilots’ one’s listening like a vehicle, an apparatus simulating spaces and forms … A tense relationship with appearances is established, full of risk, desire, antipathy, hope. This reproduces the same sensation of strangeness as when a tale is recounted. (Bayle Reference Bayle1989: 170)
4. ANALYSIS OF PATAH
The visual music composition Patah (2010) by Diego Garro (Garro 2014) will now be considered, in an attempt to demonstrate how morphodynamical classifications and catastrophic theoretical models as proposed in this article can be used in analysing transmedial narratives. The aim in this analysis is to achieve an apprehension of the narrative form of the composition as a gestalt. Focus is directed towards particular material (visual or audio) with forms that are salient in the narrative domain. On the one hand, the piece, to a significant extent, has a synchretic disposition (Chion Reference Chion1990: 63); audio and visual materials convene in a harmonised manner, and can therefore be considered assemblages – aggregated wholes. On the other hand, there are sections where the material relates in a rather abstruse fashion; these are the sections where the use of stronger associative grounding, through metaphorical reasoning, is called for. The use of archetypes derived from cuspoids (spiked curves – see Figures 2 and 3) is employed for the sections that display a strong degree of synchresis, as the interaction of the attractors in cuspoid forms is more dramatic and hence the shifts in states more sudden. The sections that are defined by more gradual transitions are associated with umbilics (smooth curves – see Figures 2 and 3); the interaction of the attractors is smoother and the transition of states more gradual.
The schematic representation of the morphodynamical classifications describing narrative evolution in Patah is shown in Figure 4, in which the composition has been divided into eleven parts – marked as Parts A through K – at the moments at which the audio and visual materials exhibit significant shiftsFootnote 12 in behaviour. The timeline is matched to the video online,Footnote 13 and therefore begins at 1′26″, after the video’s opening titles. The figure traces out morphological archetypes at three levels – audio-only (A), visual-only (V) and combined audiovisual (AV) – and are named with reference to the morphologies and catastrophe types listed in Figures 1, 2 and 3.
Morphodynamical classifications describing narrative evolution in Patah.
The work’s opening section, Part A, already displays quite rich and complex morphological character. The material comprises two stable abstract visual forms, and audio streams of long inharmonic, spectral waves with no apparent source references – abstract, non-source-bonded, non-referential material, usually consisting of spectral partials, with slow attack, a long evolution and long decay. Both audio and visual undergo analogous, slow-dynamic transformations; upward movement of the visual shapes is met by gradual enrichment and amplification of higher audio partials, and vice versa. Thom’s ‘almost’ archetype (Figure 2) is employed here, signifying an ‘almost’ emission of a new attractor – a new state. The slowly transformed material seems to oscillate between its established form (its morphological character, in both audio and video), and a new form not yet clearly defined. It is hinted, however, that this new state is going to come through successive ‘morphing’ of the current material. This behaviour is observed three times in this section, and can be modelled as phase transitions of a higher archetype. Wildgen’s archetype (Reference Wildgen1982: 61) of ‘bipolar change through a metastable state’ can be used to describe this failure in emitting a new attractor. These morphologies (as well as others of similar qualities, for example Thom’s ‘failing’ morphology) can be observed in the geometry of Thom’s ‘swallowtail’ (Figure 2). In the resulting geometry, we can find paths that reflect the archetypes of ‘birth-death’ (as in fold) and of ‘bimodal change’ (see Figure 3). In the piece, this is reflected by the behaviour of the ‘almost’ archetype in Figure 4: attractor areas appear and disappear (Jumps 1 and 2) reflecting ‘birth-death’, until there is a final bimodal change of the state altogether – a permanent jump (Jump 3), reflecting ‘bimodal change’. So, returning to their description as ‘phase transitions of a higher archetype’, above: the first phase is Jump 1 (Figure 4), followed by a second jump with a longer duration, until a final jump at 3′15″ (an emission that started at 3′09″) is observed, at which point a separate attractor becomes salient, ‘freed’ from its previous state, and change has been established. At 3′27″ this audiovisual attractor is ‘captured’ (as per Thom’s ‘capture’ archetype) by what will be referred to in this analysis as an ‘acousmatic moment’ (AM), to indicate the presence of solely aural material, with nothing happening in the visuals. This is the first of three purely acousmatic stable states. The materials used in those states (marked in Figure 4 as AM 1, AM 2 and AM 3) are granulated sounds, sounds of soft whispering, utterances and words. These moments tend to be quite semantically charged, given their diversely enriched appearance and centrality in building tension-release gestures in the composition. This semantic prominence is demonstrated further in the analysis by how these acousmatic attractors unfold in complexity throughout the piece and their significance for the narrative.
Part B – 3′40″ to 4′04″ – is defined by a swift, almost violent succession of audiovisual attractors, a fast progression of changing dominance, while each new attractor forces the previous one to its ‘death’. The composition’s character here is quite synchretic. The visual material consists of abstract, but compact (with clear boundaries) forms, reflecting the gestural character of the sound objects – sweeping sounds with fast attacks and fast decays, pulses with a short period, etc. – and exhibiting remote surrogacy. Coupled, they appear as distinct, clearly outlined audiovisual objects, rapidly replacing each other. This attractor behaviour results from cuspoid geometries, with their spiked curves, and in this case we can observe prominent characteristics of the cusp and double cusp geometries described in Figure 3. At 3′40″ in Figure 4, we find two further archetypes: ‘transient existence’ (Figure 3) and ‘failing’ (Figure 2); other archetypes that could be used in this latter case, bearing similar qualities, are those of ‘crossing’ and ‘rejecting’ as well as the ‘changing’ archetype. These morphologies are relevant to the description of the quick shifts and the antagonism between attractors. This erratic succession of attractors ends at 4′04″ with a ‘capture’ morphology.
Part C indicates a shift in the system’s behaviour, collapsing into what appears to be a state of smoother transitions and attractor interaction (even though the material is not much different from the previous section). This behavioural pattern suggests the use of archetypes derived from umbilic catastrophes. At the start of Part C, two attractors can be observed; eventually one of them becomes dominant. In Figure 4, a separate audio attractor outlines the more complex interaction between the aural and visual streams. A subtle but complex exchange between the audio and visual domains, with audio glissandi appearing to drive the visual forms, leads the visual counterpart to its goal (4′38″) in a gradual manner, driven by slight shifts in qualitative aspects of the materials. This is successfully portrayed in the ‘giving’ archetype, which is more apparent at 4′32″, when the aforementioned audio glissandi seem to initiate instability in the visual counterpart – as if there is energy transfer from audio to visual – thereby making the visual forms unstable, and in need of dispersing that energy. This interaction can be represented by the ‘cutting’ morphology, and we can perceive the emitted ‘energy’ as a separate entity after its emission and eventual assimilation by the recipient visual attractor. The emitted energy thereby acquires a symbolic function. This process finishes at 4′38″ with an ‘acousmatic moment’ (AM 2) schematised by the ‘capture’ archetype, in which utterances are clearly present, while by the end of AM 2 extreme vocal transformations dislocate the recipient from the familiarity and intimacy of these utterances and destabilise the attractor. We are nevertheless left with the impression that someone is trying to say something. This dislocation from the stable vocal sounds prepares us for the next part, in which the material is again non-referential in nature.
Part D has qualities similar to the previous part as far as its dynamics are concerned. The material, however, is slightly different, since one of the two visual attractors is referential (photos of buildings, urban landscape, etc.), while the abstract visual attractor appears as a textural stream with continuous, internal, slow-dynamic motion, instead of as a compact abstract visual form. The audio comprises long, abstract spectral streams, which, like their visual counterpart, are textural in nature, implying internal motion and momentum. The archetype used to demonstrate the complex mediation between the aural and the visual is that of the ‘messenger’. In this higher archetype, one attractor seems to lead, guiding an emitted attractor to the field of a further attractor in a gradual manner. The dynamic landscape in this morphology constantly shifts between non-dominant attractors. This process is interrupted at 6′37″ by the presence of a separate attractor, which we once more perceive as an audiovisual assemblage (compact visual form/gestural sound material). This attractor acquires dominance and pushes the previous attractors to their death, but quickly fades out at 6′51″, emitting a new attractor and signifying a new state.
This new state has very similar qualities with the previous part (as described before 6′37″); however, the process of mediated interaction described by the messenger archetype forces the system to cross an equilibrium at 8′38″ after the gradually intensified use of audio glissandi and colour saturation. The result is a quite violent change and the birth of yet another new state.
Part F is again a domain of complex morphologies and interactions between attractors (remote surrogacy sounds – long spectral waves/glissandi; pulse/beating sound; compact abstract visual forms), and is again smooth and gradual in nature. For a significant length of time the visual domain is dominated by one nearly stable attractor. At 9′03″ a significant but subtle audio attractor makes an appearance, marked by a subtle but noticeable visual flash. The audio attractor manifests as a quite stable but not dominant form of beating, similar to that of a heart; this attractor can therefore be seen as a ‘beating’ archetype, usually used to describe a physical movement or an instrumental action, giving the impression here of an effect of temporality and sentience. Around 10′17″ metastable states and jumps appear, along with ‘giving’ archetypes, and the interaction between audio glissandi and the visuals introduces mobility to the system and leads to a permanent jump at 10′51″ back to Part F’s initial state.
The system quickly settles around a new audiovisual attractor (compact visual forms with slow internal dynamics, with long subtle spectral waves) that captures the previous attractor (concluding at 11′34″), while a separate stable audio form is manifested in the presence of transformed but recognisable vocal material reciting the words ‘something’ and ‘something out’. The interaction between the materials used in the two media streams (i.e. between the audio and the visuals) becomes more complex. ‘Metastable states’ emerge; from 11′34″ to 11′46″ the dynamics become faster in both the audio and the visual materials, with synchronicity between fast sweeping sounds and constant transformation of the visual forms, until 11′46″ when the third and most prominent ‘acousmatic moment’ occurs (AM 3). With no visuals present at this point, the recipient can clearly hear the words ‘something out there’, but this time the observed archetype does not seem to be an elementary one (such as ‘capture’ as in AM 1 and 2). The elaborations and techniques used by the composer to reach this particular moment in the composition, the gradual build-up of utterances leading to the appearance of the phrase ‘something out there’, contains such symbolic value that the symbolism here becomes a separate entity, as a result of which the ‘cutting’ archetype can be proposed.
Furthermore, this charged symbolic gesture is carried through to the next part (with the use of footage from urban landscape as a possible reference for ‘out there’). Hence the morphology that follows is the ‘fastening’ (sometimes termed ‘composition’) archetype. The dynamics have changed however; the abstract material quickly dominates, and morphologies similar to those described in Part B are observed.
Between 12′43″ and 13′19″, the system returns to morphologies similar to those found in sections D and E.
Gradual mediations in Part I lead to a new state that once more carries a strong symbolic identity, which can be connected to the previous semantically charged ‘acousmatic moment’ at AM 3. This time the visuals morph into what seems to be a clear representation of a room, as if the composer wants to create referential association with a kind of visual ‘in there’ that connects with the ‘something out there’ at AM 3; this time, however, it is the visuals that carry these strong referential associations. Another important observation is that for the first time we perceive visual depth, and can relate to prepositions such as behind, out of, on, over, etc., a reference that was difficult to deduce in previous sections. This state ends with the sudden death of the attractor, a frontier of existence signified by a downward intervallic harmonic motion from what seems to be C to F.
The last Part, which enters suddenly, again demonstrates multiple ‘metastable states’, appearing as subtle interactions between audio transformations of long abstract spectral waves and abstract slow-dynamics, non-compact textural visual streams, until a final ‘capture’ at 14′58″ leaves solely the audio stream to continue until it fades.
The above analysis is a middle-level analysis,Footnote 14 based on observation of discrete events, their temporal evolution, and their structural formation into higher-level groupings. The detection of these events is a product of qualitative estimations of their morphological character (e.g. recognising stable states, salient forms and pregnancies in the material). The grouping of these events into syntactical structures reveals more about their semantic significance, while awareness of the processes bringing those structures to the fore as structural components further uncovers their semantics. Semantic interpretation of the structured morphological formations, patterns and their evolution allows for a narrativisation of the flow and transitions from one identifiable state to the next. In the process, certain events stand out for their functional role (see for instance the ‘acousmatic moments’ above), and symbolic associations are formulated (see, for example, the ‘symbolic’, or ‘room’, scene in Part J). This narrativisation process does not result in a ‘narrative’ in the classical sense of the term (as found in language-based media for instance), but does satisfy the narrative definition given in section 2 of this article. The evolution of the material and succession of states acquires meaningfulness for the composition – even in the case of abstract, non-referential material – from metaphorical associations (birth/death of agents, emissions, spatial semantics like inside/outside, on/under, etc.). These narrativised descriptions, products of double emergence and double organisation, disclose important information concerning the flow and evolution of events, their morphological profiles, and the metaphorical associations they evoke.
5. CONCLUSION
This article has been an attempt to interrogate the link between morphological components and higher-order symbolic structures, and to explicate the importance of this connection for transmedial narratives. We considered the nature of transmedial narratives, their semiotic complexity and semantic dimensions, and outlined the need for a generic conceptualisation domain that could be used to represent this complexity. Catastrophic theoretical models were proposed, particularly the idea of semantic archetypes, in order to identify narrative structure and its evolution, independent of the media involved and semiotic channels used in those narratives. Starting from the work of Petitot, and using the field of acousmatic music and the work of Bayle as examples, a relation between morphological and symbolic structures was identified.
Most importantly, this article has attempted to demonstrate that there is a morphological structure to meaning, and that morphologies, as perceived, carry meaning in their structure. This is what makes structures and forms noticeable (Thom Reference Thom1975, Reference Thom1989): the fact that they are semantically charged. The more distinguishable the morphology and stable the form, the more meaningful they seem to be, an idea that was beautifully expressed by Bayle with the notion of metaforms, which he describes as constructs that are formed through a bijective (even isomorphic) relation between forms and their dynamics (source forms), and their target forms in the semantic field (Bayle Reference Bayle1989: 168). Working with metaforms brings both the symbolic and morphological aspects to the table, and so offer interesting potential for further work on narratives as mental constructs and their use in abstract art forms and multimedia art works.
Acknowledgements
I would like to express my gratitude towards the reviewers and editors of this issue, who through their reviews and feedback helped significantly to communicate the ideas presented in this article.
