Multimodality and the Diagram in Goodwin’s Work and in Semiotic Approaches

Goodwin’s work is relevant for semioticians for at least two reasons: on the one hand, his work is crucial for his conception of the heterogeneity of sign systems (verbal, visual, gestural), which plays a central role in scientific discourse and practices in hard sciences such as geology as well as in his own work as an analyst of interactions. On the other hand, his work is semiotically crucial for the way he approaches the relationship between the production of images and other visual devices in scientific experimentation. In fact, Goodwin’s theory focuses on the problematic relationship between reasoning, perception, and graphic inscriptions in science, a relationship that, in Peircean theory, is covered by the notion of diagram. I believe these two aspects of his work (multimodality and diagrammatic reasoning) are capable of enlightening and renewing semiotic reflection—and vice versa.

Before addressing the matter of the diagram as an abstract/concrete means of reasoning, I will briefly look at the problem of multimodality, which in Goodwin’s work is foremost an issue of notation of different languages such as verbal, visual, and gestural ones. This matter of the relation between heterogeneous sign systems is also a crucial concern in contemporary French School semiotics in the lineage of Greimas, whose perspective is very close to Goodwin’s viewpoint.

In the history of semiology, the attempts of neither Emile Benveniste nor of Roland Barthes have been able to conceive of an articulation and organization of visual, musical, and/or gestural signs that would be as complex as for verbal language. Benveniste thus refuses to recognize other sign systems as having the same status as natural language: these other sign systems would need the mediation of verbal discourse in order to stand in a semiological and meaningful relationship. According to him, systems that do not have units and rules to govern their relations would be dependent on natural language for any description or for any “interpretance.” As Benveniste stated in “Semiology of Language” (Reference Benveniste1981), verbal language is the system for interpreting other signs—and society more generally. For Benveniste, the handicap of nonverbal signs is indeed the lack of distinct constitutive units and rules that manage the linguistic system, such as the rules of selectivity (on the paradigmatic axis) and recurrence (on the syntagmatic axis). The problem, for theorists of the relations between the verbal and the visual such as Benveniste, is the presumed liberty of nonverbal signs. Gestures, sounds, and images would all lack a repertoire, a system of distinct signs, and syntactic rules to govern their syntagmatic dimension—there would be no grammatical rules that would guarantee the intelligibility of gestural or pictorial statements. Nonverbal signs would be unable to ensure the predictability of their occurrences or their transmissibility through a universally accepted system of notation. This problem also concerns plastic arts: “Therefore, the meaning of art may never be reduced to a convention accepted by two partners. New terms must always be found, since they are unlimited in number and unpredictable in nature; thus they must be redevised for each work and, in short, prove unsuitable as an institution. On the other hand, the meaning of language is meaning itself, establishing the possibility of all exchange and of all communication, and thus of all culture” (Benveniste Reference Benveniste1981, 16).Footnote ⁴ Greimas himself, in “Figurative Semiotics and the Semiotics of the Plastic Arts” (Reference Greimas1989, 637), went a step further than Benveniste who did not acknowledge the existence of a system of visual language: “Such a system, which is said to exist but which is unknown to us, can have a chance of being grasped and made explicit only through an examination of the semiotic processes—that is, of the ‘visual texts’—by which it realizes itself. Knowledge of particular planar objects can lead to knowledge of the system which underlies them. This means that if the processes are grasped in their realized form, they presuppose the system as a virtual one, and thus as one that can be represented only through an ad hoc, constructed language.” It is clear that Greimas does not renounce conceiving of a visual system, though he is aware that such a system is not universal and that it does not depend on a “general” perception. If there is indeed a system, this would be under the condition of it being reconstructed a posteriori through the analysis of relations of kinship between the various images.

The challenge of contemporary semiotics, and in particular of visual semiotics, which aims to go beyond the semiological conception of language, is to demonstrate that the image possesses a level of articulation that does not depend on minimal elements.Footnote ⁵ Visual language is not a set of isolable parts, and every image is a composition that emerges from the tension between the conflicting centrifugal and centripetal tendencies present in the markings—which the French mathematician René Thom (Reference Thom1983) calls “forces”—and from the relations between local and global divisions, between the contour and the “centers of attention,” the latter being the zones within the image where the markings are intensively concentrated or where a visual isotopy is broken.Footnote ⁶

Goodwin’s work can contribute to this demonstration, based on his analyses of linguistic practices observed in natural situations of interaction. These practices function, for semiotics, as forms of experiments that provide a foundation for our hypotheses on visual language. More fundamentally, Goodwin’s approach makes it possible to raise anew the question of distinct units, which he reformulates within the very framework of the notation of verbal language. In Goodwin’s approach, the discontinuity of the units that make up natural language (phonemes, words) is transposed into a “continuous” fact by notation in order to account for the continuum of oral exchange (through the notation of prosody, for example), as can be seen in the transcription excerpt between Chuck and Chil, the latter being an aphasic speaker whose vocabulary is limited to a few words (fig. 1).

Figure 1.

Excerpt from transcription (Goodwin Reference Goodwin2013, 10)

While many theorists have tried to reduce the language of images, which is made up of forms and forces (Thom Reference Thom1983; Deleuze Reference Deleuze2003), to a system of distinctive units, Goodwin does the opposite: he tries to account for the forms and forces in verbal interactions without making their significance dependent on distinctive units—which verbal language possesses and which, according to Benveniste, guarantee a verifiable semantic analysis.

If, generally speaking, any kind of notation tries to make discontinuous and to regulate what is continuous and apparently free of all rules—think of the notation of dance and of other more or less formalized gestures, such as musical conducting—Goodwin, however, analyzes the modulation of the continuous dynamics in interaction. In a way, the notation of oral discourse approaches verbal language as if it were of the same complexity as a musical performance or a ballet. In fact, it is not only the meaning of words that guides Goodwin in the succession of turns of speech in the transcription but the composite whole that words form with gestures and their rhythms, the very rhythms that are arranged according to the practical relevance of the situation and not through a priori codifications of verbal language and its system. Goodwin’s segmentation of conversation and of collective practices follows a procedure that we have called, in semiotics and in the case of the segmentation of visual language, “plastic” as opposed to “figurative” (Greimas Reference Greimas1989). Figurative segmentation follows a preordered and lexical meaning that can be read in isolated units, whereas plastic segmentation is suprasegmental. Note that, traditionally, linguistics has opposed the segmental level (that of the phonemes, the distinctive units of the spoken chain) to the suprasegmental level, which designates components of prosody, among other things (melodic pitch, volume, quality of the voice, length associated with the duration of the sounds, etc.). The segmentation chosen by Goodwin is therefore based on the rhythms of the perceptual flow of attention and on visual saliencies, thereby respecting the continuum of visual (and gestural) language and the fact that any opposition between signs is always tensive and gradual, as stated by tensive semiotics (Zilberberg Reference Zilberberg2012).

Goodwin’s work is therefore important from a semiotic point of view, not only because it approaches visual and gestural signs as independent of verbal language while cooccurring with it but also because it questions any approach that analyzes verbal language as a fixed system of units isolated from signs that accompany language use, showing the usefulness of analyzing the manner in which continuous gesture modulates the significance of language use in a multimodal practical framework. In a sense, we could dare say that the notational system in Goodwin’s reflection works as a diagrammatic tool that overtakes the discontinuity of words spoken to reorganize a suprasegmental meaning based on multimodality.

The Diagram in Goodwin’s Work

The diagram is, in my view, at the core of Goodwin’s thinking. In fact, he has used this notion in its Peircean sense on multiple occasions. A diagram is a form of schematization that Goodwin already described in the 1990s as follows: “In order to generate a data set, collections of observations that can be compared with each other, scientists use coding schemes to circumscribe and delineate the world they examine. When disparate events are viewed through a single coding scheme, equivalent observations become possible” (Reference Goodwin1994, 608; emphasis added). The notion of diagram, referred to using other terms, was present in earlier articles on scientific practice, including “Professional Vision” (Goodwin Reference Goodwin1994). But it is in his recent work in particular that the power of the diagrammatic tool plays a major role in understanding the process of scientific exploration (Goodwin Reference Goodwin2018, 436). To give an initial definition, a diagram is, according to Goodwin, a graphical representation of the forms emerging from the relation between heterogeneous discourse practices.Footnote ⁷ This representation ensures the possibility for a translation between such discourse practices forming part of a new, experimental whole allowing one to understand something beyond current stabilized knowledge.

This idea of heterogeneity as a basis of diagrammatic thinking is supported by an eminent theorist of the diagram, philosopher Nelson Goodman, who states that the diagram results from abstraction and selection procedures in relation to a complex perceptual field (see Goodman Reference Goodman1976). In this sense, the diagram is a device that selects, from different experiences, common possibilities for translating them within a whole.Footnote ⁸ This conception of a homogeneous totality achieved by a translation and transposition of heterogeneous experimental practices has been described by Peirce (Reference Peirce and Buchler PWP ) as a manipulable experimental form of relations,Footnote ⁹ that is, as something having the property of being perceptually and intellectually evident once a final stabilized and inscribed form has been achieved at the term of the process of experimentation/manipulation.Footnote ¹⁰

We can thus understand the notion of diagram as a device capable of translating heterogeneous sign systems, which allows us to conceive of commensurability between different types of systematicity, such as those of verbal, gestural, perceptive, and visual signs. The diagram is therefore a device that allows for translations in order to identify commensurabilities that allow it to signify beyond the expressive capacities of verbal language. As Goodwin (Reference Goodwin1994, 611) writes: “A theory of discourse that ignored graphic representations would be missing both a key element of the discourse that professionals engage in and a central locus for the analysis of professional practice. Instead of mirroring spoken language these external representations complement it, using the distinctive characteristics of the material world to organize phenomena in ways that spoken language can’t, for example by collecting records of a range of disparate events onto a single visible surface.” When Goodwin addresses the phenomenon of translation between multiple inscriptions, in the case, for example, of the activities of scientists, archaeologists, or geologists, he states that the search for commensurability aims at the construction of a whole. This whole can be taken to be the result of a demonstrative process that, by the mere fact of bringing together and translating heterogeneous signs through a single perspective (in a sense, an abstract form), produces a surplus of knowledge—as also stated in the Kantian and Peircean traditions. In his paper “Professional Vision,” Goodwin (Reference Goodwin1994, 620) makes a very structuralist statement that could also be compatible with the Peircean theoretical tradition, as far as the conception of diagram is concerned: “As talk and image mutually enhance each other, a demonstration that is greater than the sum of its parts emerges.”

This notion of diagram has been developed not only in Goodwin’s book published in 2018 but also in an unpublished work by Smith, Mogk, and Goodwin on the inscriptional practices of geologists during their fieldwork (Smith et al. Reference Smith, Mogk and Goodwin2015), where the term “diagram” appears to describe a process of scientific experimentation and not just a schematic visualization—as we’ll see in section 3. In section 3, I will return to the relationship between diagram and discovery, but for the time being, I would emphasize that in the next section, I consider Goodwin’s conception of transcription—which concerns gestures, gazes, verbal exchanges, and perceptive movements within a shared action—to already be an example of something more than a diagram, that is, of the diagrammatization of experience. Through transcription, one seeks to represent heterogeneous syntaxes and to constitute them into a whole in which the heterogeneities are still perceptible but that at the same time makes them comparable and translatable.

The Diagram in the Process of Scientific Investigation: The Case of the “First Inscriptions” in Geology

The diagram is addressed twice throughout this article: once as a tool for understanding and transcribing that which is multimodal (as, for instance, in the case of the notation of conversations) and once as a research instrument for furthering thought and enhancing the work conducted within various scientific fields. In both cases, the diagram has to be intended as a tool leading to discovery within the practices of scientific investigation and experimentation.

In this section, I will examine a video produced by Goodwin and two of his collaborators, Michael Smith and David Mogk, as part of a presentation titled “Creating the First Inscription,” which consists of a talk given at the “Writing Fieldwork: A Symposium on the Place of Writing in the Field Sciences” conference that took place at Princeton University on April 24–24, 2015, in which a set of data collected from the field by geologists is analyzed (Smith et al. Reference Smith, Mogk and Goodwin2015).Footnote ¹¹ The video of the presentation thus shows not only the geologists’ practices of inscription by means of drawing and annotation as they are recorded in the field but also the annotation and visualization practices of Goodwin and his collaborators who analyze and comment on the practices of the geologists.

In a first image extracted from this presentation (fig. 2), we find what we have already described above regarding the notation of conversations, that is, a graphical representation translating between multiple types of systematicity: that of professional visual experience and that of tracing a drawing in a notebook (on this geological experience, see also Smith [Reference Smith2018]). This figure shows that what is diagrammatic is not the representation of what is seen (“maps and drawings”) but rather the relationship between the scientist’s gaze directed toward the rock and the tracing of the drawing. Through this example, it is possible to formulate the difference between the diagram and diagrammatic thinking, the latter being inspired by the former during an ongoing scientific activity or more generally during a cognitive act of experimentation and discovery. The diagram can be drawn on a substrate as a realized figure, but its destiny, contrary to the destiny of a fixed theoretical model, is to change through further manipulations and experimentations.

Figure 2.

“The first inscription in the geological domain” (Smith et al. Reference Smith, Mogk and Goodwin2015)

Smith et al. (Reference Smith, Mogk and Goodwin2015) specify in the figure that these are not representations that belong to iconicity understood as a similarity between two entities but rather diagrams that are “visual propositions that can be tested and which guide subsequent action.” In this sense, Goodwin is in complete agreement with Peirce’s theory where the diagram is an experimental instrument that must be tested and retested before being stabilized into what Peirce calls a “symbol” (a stabilized form). Peirce (Reference Peirce and Buchler PWP ) states that diagrams make it possible to see shapes emerging from the experimental tests of various materials (or syntaxes). At the end of a series of manipulations and of mutual translation and transpositions, diagrammatic functioning creates a whole, something evident as a form.

In the schema in figure 3, the whole chain of diagrammatic reasoning is shown: the scientist’s gaze toward a rock in an exchange with a colleague discussing the hypotheses, the tracing of the drawing, and the drawing as a tool for future stabilization/institutionalization of this partial knowledge to be tested again. This figure makes visible the connection between several actions: the verbal exchange, the orientation of the gaze toward the rock that the video records, and the drawing being traced.

Figure 3.

“Diagrammatic process in the geological domain” (Smith et al. Reference Smith, Mogk and Goodwin2015)

Let’s examine this process in more detail. The diagrammatic reasoning that we see taking place in figure 3 will be consolidated through the discussion of these proposals among the group of researchers. Once these negotiations have been stabilized and accepted, and once these proposals have been transformed into competences incorporated by the scholarly community, they will be delegated to objects that will themselves materialize the competences of the community from which they come. In this regard, consider the Munsell color chart described by Goodwin (Reference Goodwin2000), which incorporates within a single object the competences of the scholarly archaeological community. Tools such as the Munsell color chart thus become “black boxes,” as Latour (Reference Latour2013) would call them, because they bring together and stabilize or institutionalize the skills acquired by the actors, by storing them within an object. These skills can be put to the test during new investigations that will lead to the emergence of other forms of classification or categorization: the objects will then be actualized again to be transformed. We’ll come back to this process in section 4, according to the perspective of cultural dynamics expressed in French School semiotics.

Let’s return to the notion of diagram. We have defined diagrammatic reasoning as a process of translation between several representational systems: a diagram allows us to see how observation, drawing, and scientific reasoning (through the formulation and modulation of hypotheses) are mutually elaborated. The diagram therefore does not function as a representation but rather as a device for investigating and constructing relations between different experiences that need to be made commensurable. Moreover, Goodwin and his collaborators have shown that the scholar understands what he or she draws by drawing it and that his or her perception is enriched by the practice of inscription.Footnote ¹²

It should also be emphasized that the diagram is the site of commensurabilities that are still available for review and correction. Contrary to “images” in the Peircean sense,Footnote ¹³ the relations constructed by the diagrammatic device preserve “gaps” (for example, in relation to what is perceived, the drawing selects the relevant features to be retained).Footnote ¹⁴ These gaps are a major characteristic of the diagram because they allow its transposition to different future experiments.Footnote ¹⁵

In their commentary on figure 3, the three researchers involved state that their aim is to explain the path from perceptual appearance to categorization in order to produce a scientific description. In other words: “How is sensory experience in the actual setting where the work of a community is accomplished transformed into the abstract types that organize its discourse?” (Goodwin Reference Goodwin2018, 16). Each scholar’s sensorium must relate to knowledge that has already been stabilized and that is solicited by the new challenge that will lead their sensorium and, potentially, their professional tools to be transformed. In a way, we could say that the trajectory from the object sought through questioning to the scientific object as an entity that has been acknowledged by the scientific community—at least by part of it—is mediated by a diagrammatic stage. During this stage multiple tests seek to negotiate and stabilize the representational and intellectual forms of the scientific object. What characterizes this type of diagrammatic device is therefore that it allows a form of visibility stemming from the reciprocal transposition of seeing, representing, and reasoning.

Semiotizing Cultural and Scientific Practices: Sedimentation and Novelty

The diagrammatic stage allows the passage from multiple experiments to the stabilization of scientific knowledge, that is, the path from attempts and tests to the recognition of something new that could afterward be integrated with the stabilized norms of a discipline (or of a culture). In the French School of semiotics, these stabilized norms are called “virtualized practices,” because they have integrated and normalized something that has been a novelty in the past and that is currently available for new usages. To understand this path from novelty to virtualization (sedimentation) and from sedimentation to novelty, it is necessary to introduce the notion of enunciation and, specifically, the concept of “enunciative praxis,” which is useful for reformulating the problem of the transformation of the linguistic system by the process, that is, of the transformation of langue (stabilized knowledge) by parole (innovative performance) and vice versa. Langue and parole, in fact, have not to be addressed solely in relation to verbal language but, for at least forty years, have been addressed in accordance with the Greimasian distinction between competence and performance, also in relation to experience and, notably, to the accumulation of experiences in ordinary life and in the sciences.

In contemporary semiotics, the notion of enunciation covers everything that relates to the practice of discourse intended as an intersubjective action.Footnote ¹⁶ The theory of “enunciative praxis” has been developed at the end of the 1990s (Fontanille Reference Fontanille2006). This theory is in my view in line with Goodwin’s concerns about the relationship between a substrate of linguistic habits and the production of novelty. In both Fontanille’s and Goodwin’s work, it is a matter of conceiving of a dynamic between the sedimentation of existing structures of signification and the creativity inherent to any ongoing process.

But let’s take a step back. According to a very general definition, the theory of enunciation allows us to conceive of a mediation between Saussurean langue, that is, a system of virtualities, and parole, that is, the actualizations of langue in discourse—which are supposed to broaden the spectrum of the virtualities of language. Since Benveniste, in fact, the theory of enunciation has greatly evolved and has enabled to take into account the fact that langue is a system of historically attested virtualities, built on the practices of the speakers, and not on pure grammatical virtualities. Goodwin’s substrate is a concept that is very close to Saussurean langue: it is a stabilized pool of linguistic habits to be solicited during exchanges. Goodwin (Reference Goodwin2018, 5) describes the substrate as a basis for building new skills and knowledge as follows: “New structures for the accomplishment of consequential action are progressively created by performing systematic transformative operations on what already exists.”Footnote ¹⁷

The substrate is therefore identifiable as a reservoir of usages that have been sedimented throughout history and that play a part in the structuring organization of local action. This sedimentation also takes the form of various tools, such as the Munsell color chart used in geology and archeology fields, studied by Goodwin in several articles (Goodwin Reference Goodwin2000) and in his book (Reference Goodwin2018), and that function as instruments—namely, objects materializing the skills acquired within scientific disciplines and during the training of professional experts. This stabilization of practices in the form of tools such as the Munsell color chart makes it possible to plan future action and thus prepares the production of novelty within the process of an action in progress. The production of novelty in an ongoing action is not only of local value but has also a structuring character. In a certain way, not only is the substrate the result of a stabilization and of a schematization of usages but any local action is part of a more general schematizing operation and contributes to new sedimentations of practices and even to the transformation of the substrate.

In semiotics, this relationship between novelty in an ongoing action and the sedimentation of practices was formalized by Fontanille through the notion of enunciative praxis. The relationship between what is possible, relevant, and feasibly programmable and what is produced by the act of adjustment between actors and objects during a specific interaction no longer relates solely to the relationship between virtuality and realization, as in the first structuralist theory, but concerns multiple degrees of discursive presence. They can be thought of in a complex manner in terms of various modes of existenceFootnote ¹⁸—as schematized in figure 4.

Figure 4.

Schema of modes of existence in enunciative praxis (Fontanille Reference Fontanille2006, 199)

In my opinion, this schematization of the movements inherent to the production of meaning in discourse practices has the merit of making visible the complexity of our linguistic operations caught between creativity and sedimentation, because it multiplies the steps and the nuances in this process. The position of the virtual, of Saussurean langue, is beyond the scope of the schema because it is abstracted from the dynamics of enunciative praxis. The virtual comprises everything that is a priori possible in a system, in contrast to the virtualized, which covers every discourse that has been concretely produced and sedimented and that is solicitable—every discourse that has undergone a process of virtualization. Actualization concerns the process, in linguistic production, of passing from the reservoir to action (acquiring the competence), while realization is the action of putting something into discourse (mise en discours). Finally, potentialization is the reverse process that follows realization, that is, a process of putting significations on standby so that they may subsequently be virtualized.

The enunciative praxis is not the sum of all discourses performed but the locus of a discursive schematization that makes it possible to account for the thickness of our linguistic performances, caught between projection (protention) and backgrounds in memory (retention). This schema highlights the fact that each discourse has a discursive depth, based on what Goodwin calls the substrate, that is, a reservoir that is partly removed from the field of practice in course of realization but that can be partly resolicited by a movement of appropriation and actualization. This movement of actualization is to be understood as a selection in relation to everything that has previously been virtualized and that is available and reutilizable. The operation of selection implied by the movement of actualization is made relevant by Goodwin (Reference Goodwin2013, 11), who states that “the current substrate organizes coherence in gathering together in a limited, but uniquely appropriate, collection of resources implicated in the organization of the specific action now in progress.”

If we use the terms of the enunciative praxis scheme regarding figure 3, this schema shows the way in which the competence of the experts (actualization) is mutually and dynamically formed and transformed. The drawing (realization) will be then transmitted to the professional community once it is completed (potentialization). Virtualization will follow potentialization once the hypotheses of the experts form stabilized knowledge through scholarly manuals and other educational instruments.

Fontanille’s proposal enriches Saussure’s reflection on the binary relationship between langue and parole and Goodwin’s relationship between substrate and ongoing action, for it complexifies the relationships, including the temporal one, between the underlying substrate and the novelty emerging in the action through intermediate stages that mark the transition from the virtualized to the realized: the stages of actualization and of potentialization.

In his last book, Goodwin (Reference Goodwin2018, 436), in my view, takes a step further in describing the dynamics of exchange, and he clearly establishes the relation between these dynamics and diagram theory: “I have been repeatedly struck by the way in which the operations performed by subsequent speakers upon the patterned organization of elements visible (or hearable) in earlier utterance seem to have some of the characteristics of diagrammatic reasoning. The way in which both new action and structured utterances are built through systematic transformation of the resources and patterning visible in an earlier utterance seems to constitute what Peirce called ‘experiments upon [a] diagram.’” To explain this idea of patterned organization present in speaking turns, he offers the illustration in figure 5 to the readers.

Figure 5.

“Building new action by performing accumulative transformations on materials created by earlier actors” (extracted from Goodwin Reference Goodwin2018, 431).