1 Introduction
The study of visual reasoning requires an analysis of visual content. While one might suppose that the visual arts are the most obvious place to begin any investigation into visual reasoning, the complications of artistic style and expression threaten to obscure the basic features of visual form and information to which any general analysis must be expected to attend. In contrast, images in scienceFootnote 1 are objects that aim to convey information while avoiding subjective, expressive or otherwise inappropriately persuasive effects and as such stand as ideal visual examples from which to develop a generic account of visual content and reasoning. It is, after all, uncontroversial that a central function of scientific images is to facilitate reasoning.
Many theorists writing about images in science emphasize their ubiquity in scientific communication, education and practice (see, for instance, Elkins, Reference Elkins1995; Ruse, Reference Ruse1996: 269; Sargent, Reference Sargent1996; Kemp, Reference Kemp2000; Kaiser, Reference Kaiser2005). Less common is the development of a robust analytic method for discerning the content of these images, seamlessly integrated into a general scientific epistemology that explains how scientists reason with images. This lacuna is paradoxical, especially among those trained in the Anglo-American tradition who are typically impressed by the power of good analysis. There is prima facie reason to expect that just as an analytical method that goes beyond surface grammar to logical structure is an invaluable tool for clarifying linguistic content and reasoning with sentences, so an analytical method that goes beyond the surface image, laying bare the machinery of pictorial representation, would be a powerful tool for clarifying pictorial content and reasoning with images. Arguably, an analytical method for parsing pictorial content is crucial for building a general epistemology of images in science that has robust norms for good image construction, which could ultimately provide a basis from which to develop general accounts of visual reasoning.
Perhaps the dearth of analytic approaches to visual content rests on a tacit agreement within Anglo-American aesthetics that the problem of devising such a theory has been for some time effectively solved. Nelson Goodman's Languages of Art (Reference Goodman1968) offers an account of representation and an analysis of visual content that has widely been taken to either fulfil the job of the analysis of visual content, or show that the entire programme is sufficiently fraught to justify its rejection. Goodman's theory is explicitly epistemic, so there is reason to expect his account to provide an analysis of visual content that applies neatly to scientific images and elucidates visual reasoning generally. My primary goal in what follows is to suggest that this is not the case; a general Goodmanian approach to scientific image analysis is unlikely to succeed. My method is to focus on what appears to be one of the most careful and thorough attempts to date to put Goodman's analytic system to work to this end, Laura Perini's ‘The truth in pictures’ (Reference Perini2005). Here Perini explicates and applies Goodman's analytical tools for the purpose of carefully analyzing and assessing the content of images as they appear on the page (hereafter read to include other rendering surfaces—e.g., blackboards, posters, computer screens, etc.). Starting from the claim that the epistemic efficacy of scientific images depends on their having the capacity to be true, Perini utilizes Goodman's (Reference Goodman1968) account of the interpretive conventions for symbol systems as an analytical method for demarcating types of visual representation and parsing their content. Perini's analysis is not only interesting because of her dependence on Goodman's theory of representation but also for her attempt to engage the subject at this general and foundational level. Arguably, Perini makes the best case possible for the rigorous application of Goodmanian conventionalism to this analytical task. Thus, insofar as her project fails to provide an adequate method of analysis, it speaks, if not decisively then persuasively, against the promise of a Goodmanian conventionalism as an approach to understanding the content of visual representations.
Perini's central argument revolves around analyses of three images that each exemplify certain kinds of representations common in science. A graph (Figure 1) and a chemical diagram (Figure 2) are taken as exemplars of the type of Goodmanian symbol system that can be parsed and assessed. An electron micrograph (Figure 3), in contrast, is treated as a worrying but marginal case; though it requires special accommodation that effectively speaks against the Perini–Goodman theory, this is not taken as sufficiently significant to warrant the theory's rejection. I argue that, far from marginal, the electron micrograph is a type of representation that both is paradigmatically pictorial and plays a crucial epistemic role in science. The manner of the micrograph's mechanical production allows it to be used as evidence and as a means for discovery and also makes it unlikely that a conventionalist system of analysis will be able to usefully parse it. Ultimately, I suggest that John Willats (Reference Willats1997) provides analytical tools that can both identify pictures as a type of visual representation and characterize their form and content, elucidating pictures like the electron micrograph in a way that is more psychologically plausible and retains the important and useful features of Goodman's conventionalism. As such, Willats’ account promises a more viable groundwork for parsing the content of an important subset of scientific images and explaining their roles in reasoning processes. While it might be argued that the Perini–Goodman system successfully offers a method for analyzing the content of types of images that I will call ‘visual languages’, pictorial content is better elucidated by a Willats-style analysisFootnote 2. Perini has, however, offered a new benchmark in terms of analytic rigour and depth, redefining what should be expected from a general analysis of and epistemology for scientific images. Moreover, she has directed the discussion of the role of images in scientific reasoning to the question of content as a kind of first issue that must be decided before a complete account can be devised.
A graph. Figure 4 from Reference PenefskyHarvey S. Penefsky, 1977, Perini's figure 3
Reprinted by permission from Macmillan Publishers Ltd: Mechanism diagram (from Jan Pieter Abrahams et al., Reference Abrahams, Leslie, Lutter and Walker1994) (see also Perini (Reference Perini2005: 275, figure 4)
Electron micrograph (from Humberto Fernandez-Moran, Reference Fernandez-Moran1962). Perini's figure 2 (p. 272)
2 Preliminary considerations: epistemic function, truth and language
There can be little doubt that Goodman's aesthetics has had a central role in the development of aesthetics in Anglo-American philosophy (see Elgin, Reference Elgin1997; Giovannelli, Reference Giovannelli2009). One of the attractions of Goodman's theory for those concerned with rigorous analysis is that it reduces pictorial representations to denoting, quasi-linguistic symbol systems and thus retains something of the familiar structure of propositional logic and truth functionality. For Goodman, images, like sentences in natural or formal languages, are symbols that belong to a symbol system and schema with rules of reference and that denote (or fail to denote) states of affairs. This raises the question of whether truth-bearing capacity is actually required for pictures to be epistemically efficacious. Certainly, in traditional propositional attitude epistemologyFootnote 3 the importance of the truth-bearing capacity is simply assumed. A proposition that could not possibly be true is a contradiction or nonsense. It seems that Perini adopts this approach and assumes that something like what philosophers call the syntactic approach to theories accurately describes scientific reasoning. This approach reconstructs scientific explanations as arguments in which theories (in the guise of universal generalizations), descriptions of background conditions and bridge principles (defining theoretical concepts in terms of observables) are the premises. Perini notes that scientists’ talks and published articles ‘amount to arguments’ and that ‘[s]cientists treat figures as integral parts of their arguments whose strength and soundness depend on visual representations as much as they do on linguistic representations’ (p. 262). She explains, ‘Validity, strength, and soundness are understood in terms of the truth conditions of premises and conclusions, so representations that contribute to arguments must have the capacity to bear truth’ (p. 263, emphasis mine).
There is, however, no need to adopt this account of scientific justification and, indeed, one might think that the semantic approach, which treats theories either as models or as being in some sense instantiated in or interpreted through models, is currently a preferred candidate for an epistemology of scientific images. After all, some scientific images appear to be models, at least on some accounts (consider a chemical diagram of a benzene ring), and others accompany models; thus they may be expected to be epistemically related to models. Whether as mediators between theory and world (Cartwright, Reference Cartwright1999; Morrison & Morgan, Reference Morrison and Morgan1999) or as constitutive of theories (Giere, Reference Giere1988: 82–86), models are not the kind of things that are thought to be true, understood on a traditional conception, where truth is a property of propositions. Other notions of epistemic success—such as fitting out (Cartwright, Reference Cartwright1999), similarity in relevant respects and to a sufficient degree (Giere, Reference Giere1988: 81), conformation (Longino, Reference Longino2002: 115) or even ‘true enough’ (which is defined to include felicitous falsehoods; Elgin, Reference Elgin2004)—are prevalent in semantic approaches to theory. Thus finding a role for images in reasoning need not rest on parsing their content so as to fit inferential systems designed for linguistic representations. Once we recognize that truth-bearing capacity may not be required for epistemic significance, a Goodmanian quasi-linguistic analytic method may look less appealingFootnote 4.
Despite clearly stating that assigning meaning to a particular visual representation should not depend on its being mediated by some other representation, that is, being translated into a linguistic representation (pp. 274–275), Perini's primary approach to such assessments depends on translation into language, albeit a metalanguage (following Tarski, Reference Tarski1956). She explains, ‘To define truth for a symbol system you need a systematic way to (1) name each representation in the system and then (2) state the fact the symbol represents’ (p. 276). She continues:
A statement of the definition of truth for a visual system requires a way to assign a linguistic name to each symbol based on its structural form, and also a linguistic [or metalinguistic] expression of the content of each representation. A definition of truth for a visual symbol system consists of a statement of this form for every figure f in a system. Name (f) is true IFF statement (f). (p. 276, emphasis hers)
Thus, for Perini, analysis rests on the capacity to name the image, identify its characters and translate it into something linguistic.
However, Perini explains that this approach is just ‘a method used to demonstrate that a system exhibits an appropriately systematic relationship between the form of its symbols and the states of affairs they refer to because such a systematic relationship determines truth conditions for symbols of a system’ (p. 280). Translatability into language is only one route to truth-bearing capacity; there is also this back-up method, where truth bearing rests on ‘an appropriately systematic relationship between the form of [a symbol system's] symbols and the states of affairs they refer to’. Indeed, she suggests that translation is a special case of this more generic method despite the fact that this back-up method only appears when translation fails. What Perini means by ‘appropriately systematic’ is not entirely clear. Some clarification happens when Perini makes use of this backdoor in her analysis of the problematic case of the electron micrograph, but at the expense of effectively abandoning the Goodmanian approach. I will, however, bracket this back-up method until we address her treatment of this example below.
3 Perini's account of Goodman's analytical method
With these preliminary considerations aside, we can now consider Perini's method for parsing pictures. First, she distinguishes linguistic representations from visual representations, which are further categorized as either linguistic or pictorial using Goodman's methods for analyzing syntactic and semantic features of symbol systems. Perini departs from Goodman, however, in giving resemblance relations a role in representation. Although this amendment to Goodmanian conventionalism appears at first glance to add intuitive plausibility to her account, in fact it fails to address serious arguments against both conventionalism and resemblance theories. Ironically, what Perini takes to be a simple amendment to Goodman's account seriously weakens her position and effectively motivates its rejection in favour of Willats’ alternative approach.
3.1 The linguistic and the visual
Perini locates the crucial difference between linguistic representations and visual representations in the former being fundamentally sequential whereas the latter are two dimensional (2D). The difference is that spatial positioning of serial representations is arbitrary with respect to meaning, whereas the ‘the referential role of spatial relations is the fundamental feature of visual representations’ (p. 264). She explains, ‘The spatial features of a figure can refer to spatial relations (as in a diagram of a molecule), temporal relations (timelines), relations between properties (graphs) etc…. Because of this fundamental feature, the visible forms of visual representations are related to their referents’ (p. 264).
It is not entirely clear that what are in fact two demarcations—(i) the sequential versus the 2D and (ii) having spatial relations that are fundamentally referential (or semantically significant) versus having arbitrary spatial relations—line up the way Perini suggests. In order to determine this, an account of what counts as ‘significant’ and what counts as ‘arbitrary’ form is needed. After all, the form of a Venn diagram, one of Perini's own examples of a 2D representation capable of bearing truth (p. 264), is arbitrary with respect to its referents in a number of important ways. The same Venn diagram can represent ‘There is at least one butter knife in Wyoming’ and ‘Some cats are black’. Labels are often used to specify the empirical content of diagrams and, indeed, many representations incorporate both 2D and sequential (linguistic) elements and thus are mixtures of both types of representation. When linguistic components are removed some graphs are interchangeable; without labels, one graph of a linear function that intersects the origin and has a slope of 1 is much like another. Intuitively, it seems that Perini must be right—something about significant form must distinguish the visual from the linguistic. However, the long list of possible referents of visual significant form—‘spatial relations,… temporal relations,… relations between properties…, etc.’—by their very vagueness, give little sense of how an analysis of significant form should proceed.
3.2 Parsing visual representations with Goodman's tools
This is where Perini introduces Goodman's interpretive conventions and his analysis of syntactic and semantic features of symbol systems (p. 267), in order to classify various types of visual representationFootnote 5. This classification scheme determines which visual representations have precisely specifiable content (i.e., are translatable into natural language) and thus can bear truth. Following Goodman, Perini understands visual representations as symbol systems, consisting of characters (constituted by marks on the page), rules for combining characters to form other characters, and rules for character interpretation. Interpretation has two aspects—(i) syntactic, identifying symbols, and (ii) semantic, associating the correct referent with the symbols (p. 270)Footnote 6.
Goodman's criteria for symbol system classification address both parts of the interpretative process but as the semantic criteria play little role in parsing the visual representations themselves, I focus on the syntactic. As I am particularly interested in Perini's application of Goodman's system, I quote her explanation of it:
In syntactically disjoint systems each mark is assigned to at most one character. English is syntactically disjoint because any of the marks that appear in a particular word are instances of exactly one letter of the alphabet. There might be an infinite number of characters (e.g., the standard symbols for fractions) but each mark is an instance of only one. Syntactically nondisjoint systems contain some marks that are instantiations of more than one character…. In such a system, the character assignment of at least some marks will be either undecidable or context dependent.
Another way to categorize symbol systems is by distinguishing between those whose characters are all differentiable from one another and those with characters that are undistinguishable in principle. A system is syntactically articulate when any mark that does not belong to two characters can be determined not to be an instance of either one or the other. Written English is a syntactically articulate system; markings for each letter of the alphabet can be determined not to be instances of other letters. But many visual representations are not syntactically articulate. For example [and this is a warning of things to come], the identity of the electron micrograph is determined by the exact array of black and white, including gradation in tone… Limitations on measurement make it impossible to determine the form of this character with complete precision. As a result, the exact identity of the figure as a unique character cannot be determined.
Systems with infinitely many characters are syntactically dense if the characters are ordered such that for any two characters there is another ordered between them….
Goodman calls systems that are syntactically disjoint and articulate linguistic systems….
…. [These] can be contrasted with systems which are syntactically inarticulate and dense. I will refer to such systems as pictorial systems, since this category includes the kinds of images we think of as pictures (photographs, perspective drawings, courtroom sketches, etc.). The difference between pictorial representation and representation with articulate and discrete syntax is exemplified in the contrast between the sweeping second hand of an analog clock and the discrete integers of a digital clock. (pp. 270–273, emphasis hers)
Using Goodman's criteria and her own distinction between visual and sequential representations, Perini identifies ‘three basic types of symbol used in science: serial representations with linguistic syntax (natural language sentences, mathematical formulas), visual representations with linguistic syntax (diagrams), and visual representations with pictorial syntax and semantics (some graphs, micrographs, etc.)’ (p. 273). I will refer to these types of representation as languages, visual languages and visual pictures, respectively.
While Perini's account is fairly straightforward, a few features are particularly noteworthy. First, the conventionalism of this approach is evident at the most basic level of image analysis; marks represent by virtue of being characters in a symbol system. Second, the very features that determine that a representation is pictorial—syntactic inarticulateness and density—are precisely those that we can expect to cause problems for translation into language as by definition they indicate that characters in the symbol system cannot be exactly determined. Finally, this approach produces a counter-intuitive classification of types of visual representation. Even if the class of visual pictures includes many of ‘the kinds of images we think of as pictures’, it excludes a good many others and, moreover, includes some images that are not intuitively pictorial (p. 273). Strangely, some graphs count as pictorial representations because their axes are dense (e.g., Figure 1), whereas other graphs (for instance, bar charts scoping over relatively small numbers of discrete, countable objects), by virtue of being syntactically articulate, are expressions in a visual language. Chemical diagrams are also visual languages, despite intuitively seeming pictorial, at least in comparison to most graphsFootnote 7. Of course, intuitions can be misleading and, arguably, the point of an analytic method is to provide something more robust than intuitions to guide one's understanding and assessment of represented content. Nonetheless, if an equally rigorous or elucidating analytic method is available, which better conforms to our intuitions, the counter-intuitive aspects of the Goodman–Perini analysis speak against it.
3.3 Amendment to Goodman: convention and resemblance
It is perhaps a concern about intuitive plausibility that inspires Perini to make a role for resemblance relations in visual representation. This is a major departure from Goodman's strict conventionalism, which denies any role for resemblance in realistic or naturalistic representation: ‘Realistic representation’, writes Goodman, ‘depends not upon imitation or illusion or information but upon inculcation’ (Goodman, Reference Goodman1968: 38). Although Perini is by no means in a minority in rejecting this view (see Arrell, Reference Arrell1987: 41), her amendments appear ad hoc; and through attempting to avoid the prima facie implausibility of Goodman's conventionalism, she not merely remains vulnerable to some of the serious challenges to conventionalism, but also to some of the serious challenges from Goodman's conventionalism. While intuitively it seems resemblance must have some role in at least some visual representations, Perini fails to appreciate how the psychology of perception and Goodman's canny criticisms constrain what that role might be.
Perini follows Craig Files’ (Reference Files1996) suggestion that although Goodman is right that convention, not resemblance, entirely determines which objects are representations, he is wrong when it comes to his claim that convention alone determines content. Perini explains:
Goodman thinks that there is nothing intrinsic to the form of image [sic], or its causal history, that determines its meaning independent of conventions;… while this conventionality may initially seem to exclude any role for resemblance in addressing the [question of what determines the content of a representation], it does not.
Content can… be determined by conventions that relate symbol and reference through resemblance relations. For example a botanical print represents a particular species because the print has properties (color, spatial features, etc.) that resemble those of the plant. (pp. 268–269)
Convention still determines which, if any, resemblance relations count for any given representational system (p. 269), so all the content depends on convention even if it sometimes also makes use of resemblance relationsFootnote 8. Thus Perini's amendments still imply a thoroughgoing conventionalism.
It is difficult to see how any conventionalism of this type can survive the challenge from the capacity of many non-human animals to glean pictorial content. This capacity is so widely accepted that countless psychological experiments simply assume it (e.g., Perrett et al., Reference Perrett, Hietanen, Oram, Benson and Rolls1992). To suggest that non-human animals—for instance, macaques or pigeons—are following pictorial conventions and deciphering symbol systems makes nonsense out of the term ‘convention’. Furthermore, to suggest that there is a psychological, non-conventional explanation for non-human animals’ capacities to glean pictorial content and a conventional explanation for these capacities in humans is ad hoc and suspect in light of evolutionary continuity. Certainly, most humans, unlike most other animalsFootnote 9, can also make use of conventions in the creation and perception of visual content (as is clear from the many visual representations that are rich with symbolism or that combine image and text). However, the capacity of non-human animals to see content in pictures suggests, prima facie, that the visual experience of pictures is in some sense akin to the visual experience of the objects they depictFootnote 10.
Nonetheless, naïve resemblance theories are also fraught with problems, as Goodman's criticisms demonstrate. Files’ discussion is directed towards one part of Goodman's argument—the claim that representation cannot be based on resemblance, as resemblance is a symmetrical relation whereas representation is not (Reference Files1996: 398). But Goodman also attacks resemblance itself—the idea that there exists ‘the way a thing is’ or ‘the way a thing looks’ that can be copied and captured in a static visual representation. There are, after all, multiple ways any given thing is. As Goodman explains, ‘… the object before me is a man, a swarm of atoms, a complex of cells, a fiddler, a friend, a fool, and much more. If none of these constitute the object as it is, what else might? If all are ways the object is, then none is the way the object is’ (p. 6). The idea that visual representation concerns copying an aspect of an object or its appearance is little better. Botanical drawings, Perini's example of pictures that resemble their objects, are a case in point. These drawings typically picture the plant as it is never seen—against a plain, monochromatic background, outside its ecological niche and without shade, disease or infestation. Frequently, depending on whether the plant was drawn from a live specimen or a pressed one, the orientation of the leaves and flowers in the image do not resemble those typical of living members of the species in their various natural settings.
Goodman also attacks those representational methods that supposedly produce resemblances because they replicate the exact array of light that would hit the viewer's retina in the state of affairs represented—scientific perspective and photographs (pp. 10–19). Such pictures are often taken to be exemplars of accurate representation; but, as Goodman notes, it is only if the eye is precisely placed, motionless and in perfect lighting conditions that they do actually replicate this array of light. Such conditions are ‘grossly abnormal’ (p. 13). Moreover, recreating the array of light produced by a state of affairs seen from a particular point in space and time is far from recreating the experience of that state of affairs from that position. ‘Even where both the light rays and the momentary external conditions are the same, the preceding train of visual experience together with information gathered from all sources can make a vast difference in what is seen’ (p. 14). Not only previous experience, but also interests, needs and prejudices determine what is seen. No eye is innocent. ‘[The eye] selects, rejects, organizes, discriminates, associates, classifies, analyzes, constructs…. Nothing is seen nakedly or naked’ (pp. 7–8). Finally, as Goodman notes, for all their exact replication of light arrays, photographs frequently fail to look like the objects they represent: ‘… the photograph of a man with his feet thrust forward looks distorted, and Pike's Peak dwindles dismally in a snapshot. As the saying goes, there is nothing like a camera to make a molehill out of a mountain’ (p. 15).
Certainly, Goodman's criticisms of naïve resemblance theories, particularly those that privilege perspective or reporting to picture things as they are or appear, are persuasive; but replacing them with a thoroughgoing conventionalism seems an unpalatable alternative. Not merely is such conventionalism counter-intuitive (even to advocates of Goodman's theory like Perini), but it is also unable to account for the cross-species psychology of pictorial perception. We need something in between—an account that shows how visual form is significant, relating it to visual experience and the interested eye. This account should be psychologically plausible and avoid Goodman's criticisms while also providing a method for parsing pictures—a means of analyzing visual form to determine represented content. Such an account has been developed by John Willats.
4 Willats’ alternative analytical tools
In Art and Representation: New Principles in the Analysis of Pictures Footnote 11, Willats explores the wide variety of ways in which the spatial relations within a state of affairs can be represented on a surface. These various methods have various strengths and weaknesses and it makes little sense to treat one as ‘truly capturing’ the ways things are or appear. Grounded in the history of art, technical drawing and developmental psychology, Willats’ analytical method fundamentally relates pictures to visual experience and objects. Rather than identifying marks as characters and translating pictures into linguistic expressions, determined by the conventions of one of an indefinitely large set of symbol systems, Willats’ taxonomy parses marks in terms of their basic visual properties, showing how they create the image of a 3D scene. At the heart of his analysis is the striking ‘double reality’ of pictures—our capacity to see a house or ship in what is, at the same time, recognizably a set of marks on a page (Willats, Reference Willats1997: 93, referring to Richard Gregory and Richard Wollheim).
Although his complete technical tool kit and its explication are extensive and rather intricate, Willats’ basic distinctions can be fairly easily grasped with examples. He distinguishes what he calls denotation systems—which identify and classify the various visual constituents that visually construct shapes and spatial information from marks—from projection (or drawing) systems, which constrain how the spatial information of the depicted scene is transferred to the page. Willats’ technical vocabulary allows easy comparison of visual representations that utilize different projection and denotation systems, without resorting to appeals to realism or naturalism, which inevitably fall prey to Goodman's criticisms of resemblance theories rehearsed above. Indeed, ultimately Willats’ method reveals why Goodman's criticisms, while having bite, do not entail a rejection of resemblance tout court.
There are three basic types of projection system: orthogonal, oblique and perspective. Although the reader is doubtlessly familiar with perspective, I quote Willats’ brief overview as it introduces useful technical terms (specifically, the ‘picture plane’ and ‘orthogonals’):
In pictures in perspective… objects that are further away in the scene are shown to a smaller scale than objects that are closer to the viewer, and the lines representingFootnote 12 edges that lie perpendicular to the picture plane, or orthogonals as they are called, converge to a vanishing point…. (Willats, Reference Willats1997: 2)
The picture plane is understood as an imaginary plane located between the scene and the viewer, intersected by projection lines or rays coming from the objects pictured in the scene (Figures 4 and 5 illustrate picture planes). ‘The geometry of these intersections forms the geometry of the picture’ (p. 37). Interestingly, perspective effects are not exhausted by geometry. Other ways of creating the appearance of distance—diminishing clarity of form, diminishing tonal contrast, diminishing colour saturation and blurring distant objects (pp. 141–143)—are familiar devices in perspective painting, but entirely distinct from the formal geometrical properties dictated by the projection system and so are kept clearly separated from the projection system level of analysis.
The primary geometry of orthogonal (or orthographic) projection. Reprinted with permission from Fred Dubery and John Willats (Reference Dubery and Willats1983: 15, figure 8) (see also Willats, Reference Willats1997: 43, figure 2.3)
Oblique projection. Reprinted with permission from Fred Dubery and John Willats (Reference Dubery and Willats1983: 30, figure 33) (see also Willats, Reference Willats1997: 53, figure 2.14)
While perspective is more familiar (arguably through inculcation), oblique and orthogonal projectionFootnote 13 are equally legitimate and are far better suited to representing certain spatial features of a scene accurately.
In pictures in oblique projection… objects are shown to the same scale irrespective of their distance from the viewer, and the orthogonals are parallel and run at an oblique angle across the picture surface [see Figure 4]…. In pictures in orthogonal projection… objects are shown to the same scale irrespective of distance, but there are no orthogonals. Instead, edges in the scene that lie perpendicular to the picture plane… are represented by points; and planes that are perpendicular to the picture plane… are represented by single lines [see Figure 5]. (pp. 2–4)
Traditional East Asian and medieval European art provide some of the more familiar examples of oblique projection, but technical drawings also sometimes make use of this system as it gives a suggestion of depth while retaining true relative sizes and true shapes of objects parallel to the picture plane. As oblique projection can also produce ambiguities, typically technical drawing is done using orthogonal projection, which shows to scale all true relative sizes and shapes of features parallel to the picture plane, but (confusingly) has no orthogonals. In technical drawings that use orthogonal projection, additional relevant shape information that is not visible from the view is shown through dashed (hidden) lines and/or through one or more alternate views, where the pictured object is rotated at 90° to the main view. Each of these projection systems map spatial relations in a state of affairs into corresponding relations on the page; within each system visual form is significant but in different ways.
Psychological facts about human perception further complicate matters, bringing to the fore Goodman's insight that none of these projection systems create resemblances in any straightforward way. Perceptual constancies, such as shape constancy, ‘the tendency to see an object as its true shape regardless of the viewing angle’ and size constancy, ‘the tendency to see an object as its true size regardless of the viewing distance’ (Willats, Reference Willats1997: 40) inform not only the perception of objects, but also the perception of pictures. Such constancies explain the diminishing of Pike's Peak and why one's feet may appear unnaturally enlarged in an unflattering snapshot. Furthermore, just as in experience we may have difficulty recognizing objects when we see them in certain unfamiliar positions, so in various different orientations to the picture plane, represented objects will be more or less easily recognized (Willats, Reference Willats1997: 79–84). Though ordinary visual experiences are determined by the interested eye and the world, the construction of the appearances of objects in pictures is often the result of a combined effort of the viewer's interested eye and past experience and the artist's practiced hand.
Of course, projection systems form only one main branch of Willats’ analytical method. Indeed, it is not even the most general level of spatial analysis as many pictures make no use of a formal projection system but still spatially represent spatial relations in the world. Thus, for instance, subway maps represent the spatial order of stations and the intersections of routes, but not true relative lengths and shapes. These basic spatial, or topological, relations are, interestingly, the first to be mastered in children's development, when a child learns to draw a face with the eyes above the mouth and all facial features inside the region representing the head (Willats, Reference Willats1997: 13–15, 70–75).
Because denotation systems concern the visual features of the marks that constitute the image they provide the most basic level of analysis, starting with the mark—the pigment on the pageFootnote 14. Whereas Goodman and Perini suggest that marks are assigned charactersFootnote 15, for Willats marks are used to represent or comprise the most elementary units of shape information on the surface (points, lines and regions). These ‘picture primitives’ in turn represent the most elementary units of shape information in the scene—‘scene primitives’, which can be 3D (lumps, sticks and slabs), 2D (surfaces), 1D (edges) or 0D (corners). These scene primitives then represent objects (pp. 93–108).
To facilitate understanding of the above terminology, I provide an example in Figure 6. It is natural to name and describe what are in fact marks on a page as two cubes; Willats’ analysis reveals how marks create the picture of the cubes. Interestingly, the cubes can be seen as being in two separate scenes or as in one scene. If they are in one scene, the cube on the right appears to be closer to us than the cube on the left because we tacitly infer from their orientations that they are resting on a single surface. However, the difference between the projection and denotation systems used to represent each cube fight against seeing them as together in this way. The projection system for the cube on the left is obliqueFootnote 16 and that on the right is perspectiveFootnote 17. At the most basic level of the denotation system—the domain of marks—the cubes are rendered using the same materials, some kind of ink (or, if you are reading an electronic copy of the paper, pixels). The most obvious difference between the two cubes lies in their picture primitives. The cube on the left is constructed from 2D regions, some of which are primarily extended in one direction and others that are equally extended in both dimensions, while that on the right has only points as picture primitives, which form another type of picture primitive, lines. The scene primitives appear to be the same in both images and it is by virtue of this that they construct the same type of object. The left hand image has edges, represented by regions extended in one dimension, and surfaces, represented by regions equally extended in two dimensions, and the right has edges and surfaces represented by shared lines (comprised of points). There is, however, a possible difference with the representation of the corners. The corner that attaches to the three shown sides of the image on the left has two possible readings: the corner might be seen as coming out towards the viewer, and under such a reading the object is a cube. Alternatively, it might be seen as going back away from the viewer, in which case the object is something like an empty box with two sides and bottom removed. The true sizes retained in oblique projection facilitate this gestalt switch between alternate readings of this corner and thus the object. (Necker cubes depend on the same ambiguity.) The foreshortening in the perspective projection of the cube on the right resists a similar alternative reading. (To see the cube on the right in this way we are forced to imagine not only that the two sides and the bottom have been removed, but also that the two remaining square sides have been cut down into smaller irregular quadrilaterals.)
Two cubes drawn so as to illustrate the different systems of analysis discussed by Willats
Clearly, the identification of characters and translation into language play no role in Willats’ analysis of pictorial content (or at least no more than they do in ordinary visual experience). For Willats, pictures transfer shape and spatial information from states of affairs (including imaginary states of affairs) onto pages. Goodmanian conventions are only relevant in the limited sense that symbols and linguistic expressions are sometimes added to pictures. While the choice of projection and denotation systems in some specific contexts may be determined by convention (as with the use of orthogonal line drawings in engineering), Goodman's analysis offers no insight into the structure or significance of these kinds of conventions. In fact, projection and denotation systems are often determined not by convention but by the physical constraints of the image-making procedure and medium, as, for instance, with the various methods of picture production through film exposure (ranging from X-rays to stop motion photography).
Understood through Willats’ analytical method, different projection systems are seen to display different types of shape and spatial information in a scene well or poorly, and different denotation systems obscure or reveal different visual features. Thus different systems are appropriate for representing different types of states of affairs, depending on what features are of interest. Because the systems of picture construction are precise, there is no need to appeal to some vague notion of resemblance that might make the account vulnerable to Goodman's criticisms. Moreover (and in contrast to Perini's distinction between visual languages and visual pictures, discussed above), Willats’ system provides a way of demarcating the pictorial from the non-pictorial that matches our intuitions. A picture is simply that which is amenable to all levels of his analysis—something that represents the spatial information from a state of affairs on a 2D surface.
5 The three examples
We are now ready to test these two analytical methods against Perini's examples. As we shall see, neither system can completely and satisfactorily parse all three examples; Willats’ approach stalls on the graph (Figure 1), Perini's stalls on the electron micrograph (Figure 3). However, whereas the limits of Willats’ analysis suggest that the graph is not pictorial, the failings of Perini's theory speak against her account, at least insofar as it is supposed to explain the truth in pictures. By her own lights, the electron micrograph is the most clearly pictorial example, but the least amenable to her Goodmanian analysis. Interestingly, her account of visual language survives these problems and might be considered a friendly addition to Willats’ approach, though an in-depth account of this joint method will have to wait until a later time.
5.1 Perini's analysis
Perini's least problematic example, the chemical diagram (Figure 2), is syntactically disjoint and articulate and is thus part of a visual language. The figure can be decomposed into atoms and the meaning of the figure is easily determined by the arrangement of these atoms because they and their rules of combination are stipulatedFootnote 18. In Perini's words, ‘The interpretative convention used to understand the figure determines a particular state of affairs, which can be expressed linguistically…. The concept of truth for such systems can be specified because every representation in the system can be given both a name and a statement of the fact that makes it true, both expressed as a recursive function of the arrangement of atomic characters’ (pp. 277–278).
Things are a bit trickier when we look at the graph (Figure 1). It is pictorial on Perini's account as it lacks a completely articulate syntax because the values in the axes are dense (between any two characters there is another ordered between them) and thus cannot be decomposed into atoms and translated. But Perini claims we can still get a robust concept of truth for the system. We can identify some atomic characters—the axes, terms for values, the circles and the continuous line (p. 278). Horizontal and vertical axes each correspond to a distinct feature of the referent (ligand binding and enzyme concentration) and thus we can identify the characters of the circles and the continuous line. Values represented by any curve can be represented mathematically and ‘[b]ecause both the symbols and their referents can be assigned linguistic representations, a systematic definition of the concept of truth is possible’ (p. 278). It appears that the graph, despite being syntactically dense, has a truth value because we can articulate the truth conditions for particular points on the curve, due to the significant form of the graph being translatable into mathematical expressions.
This explanation draws attention to some troubling aspects of Perini's theory. Pictorial representations are defined as failing to be syntactically articulate, but capacity to bear truth exactly rests on being able to precisely decipher the content of a representation because it is syntactically articulate. It seems that what makes a visual representation pictorial undermines its capacity to bear truth! The graph is only able to straddle this apparent contradiction by being dense as a whole, but allowing articulate content to be drawn from it sentence by sentence, precisely determined by the relevant interpretive conventions. Employing the categories defined above, despite being a visual picture (as it is impossible to assign some marks particular characters), we can treat it as a visual language. At this point it appears that rather than showing ‘The truth in pictures’, Perini has given us an account of the truth in visual languages.
This brings us to the last image, the electron micrograph (Figure 3), which is pictorial in both Perini's technical sense and intuitively and seems to defy Goodman–Perini-style analysis. Consistent with the procedure outlined in section 2 above, Perini suggests naming the figure by specifying it pixel by pixel (p. 279), but neither this norm nor any other approach seems able to unambiguously translate the relevant content of the picture, which, according to Perini, ‘depends on our visual perception of shapes’ (p. 280). There are no atomic characters thus the relation between symbol form and content cannot be precisely specified. At this point Perini opens the backdoor, mentioned above, and reminds us that all we really need is ‘an appropriately systematic relationship between the form of [a symbol system's] symbols and the states of affairs they refer to’ (p. 280). In the case of the electron micrograph, she suggests that this systematic relationship is produced by the process of creating it. Electrons are beamed through a thin sample of biological material surrounded by an electron deflecting stain. The light areas of the micrograph correspond ‘geometrically’ (p. 281) with the unstained areas of the sample. Thus, according to Perini, the systematic relation between the form of the image and the states of affairs represented is a kind of truth, resulting from a mechanical process.
Although the visual form and thus content of the electron micrograph depends on its mechanical production, Perini does not recognize that moving from symbol systems determined entirely by human convention—the diagram and the graph—to a type of image determined in part by autonomous physical causes poses serious problems for her conventionalism. After all, significant form, what parts of the image on the page are meaningful, is stipulated in the first cases, but discovered in the latter. The relative autonomy of visual pictures produced by machines from human interventions allows visual representations generated by radio telescopes, bubble chambers, MRIs, microscopes, etc. to serve as evidence for theories and ground discoveries of new objects. (Scientists may try, initially, to impose interpretive conventions onto images of newly discovered objects but they will later be rejected if they prove empirically inadequate and thus they cannot alone be constitutive of the content and the ‘truth’ of the picture.) While conventions that explicitly stipulate characters and syntax may over time come to play an important role for viewing these same kinds of pictures, unless represented content is at least initially determined by something more than convention it is difficult to see how an evidential role for machine-produced pictures can get off the groundFootnote 19.
But there are more serious problems for Perini's view. She suggests that the mechanical production of the electron micrograph produces a systematic relationship by virtue of which the micrograph can be true. We might suppose that mechanical production guarantees a systematic relationship between the appearance of the state of affairs and the representation, but, as noted above, there is no one way any object appears and, moreover, certain mechanical procedures can be plagued by noise or introduce other systematic problems (such as false attachment or other ambiguities concerning apparent depth). We can see from Willats’ account that there are many different ways of projecting a 3D state of affairs onto to a 2D surface and many different denotation systems that can have markedly different visual forms and still present the same content. Perini's vague remarks about the micrograph being a ‘geometrical projection’ (p. 281) and being imperfectly correlated with the spatial features of the sample (p. 281) are simply insufficient and, indeed, seem to call for something like Willats’ analysis to do the job properly.
Despite her own caveats, Perini argues that her Goodmanian approach is the only way to explain how images like the electron micrograph can have any kind of epistemic significance at all. Because a never before seen micrograph is still comprehensible (as witnessed by the possibility of discovery through pictures), Perini maintains that the relation between symbol and content cannot be arbitrary and thus the micrograph must be part of a symbol system with interpretive conventions that determine meaning. ‘Unless there was a systematic relation between the form of the symbol and the state of affairs it represents, it seems impossible to explain how such a figure could be understood’ (p. 281). This seems to suggest a false dilemma: Either the relation between a visual representation and its content is mediated by some conventional Goodmanian symbol system or it is arbitrary. Willats offers a third alternative, showing that a non-arbitrary relation between visual representation and represented state of affairs can be mediated by a combination of an analogy between the psychology of perceiving ordinary objects and the psychology of perceiving pictures and the skills of viewing (as opposed to reading) that one practices (often tacitly) in seeing a scene in the marks on a pageFootnote 20. Willats’ approach still allows that there are various different roles for convention, but the content of a picture does not reduce to them.
In the end, Perini's failure to get a successful analysis of the electron micrograph, because its pictorial features defy Goodmanian analysis, suggests that Goodman's approach is poorly suited for devising a careful and complete analysis of scientific images, and visual content generally. We can see now that Perini's attempt to correct Goodman's rigid conventionalism by drawing in resemblance (discussed in Section 3.3 above), while reflecting a proper discomfort with Goodman's theory, fails to appreciate both Goodman's insight into the problems with resemblance and the variety of ways in which visualizable states of affairs can be projected onto a page. When applied to the electron micrograph Perini's Goodmanian-conventionalism-plus-resemblances approach does little to help us decipher the content of the visual picture or account for its epistemic success. Moreover, because the features that Perini cites as pictorial are precisely those that Goodman considers ‘symptoms of the aesthetic’ we may doubt that his theory will ever be able to adequately address visual pictures in the sciences. While Perini makes a fascinating attempt to use Goodman's aesthetics to ground a generic analysis of visual representation in science, in the end she shows why a Goodmanian approach is unlikely to provide a complete account of visual content and reasoning in the sciencesFootnote 21.
5.2 Willats’ analysis
Willats’ alternative analytical method is able to engage Perini's examples at the points where her analysis is most paradoxical. Consider the electron micrograph. Willats’ tools reveal how the relation between its form and its content is non-arbitrary and comprehensible, yet poorly described as a symbol system composed of characters. The systematic relation between representation and object is determined by projecting a 3D scene onto a 2D surface—a procedure in this case, achieved mechanically. The ‘geometric projection’ and the relevance of ‘our visual perception of shapes’, to which Perini alludes, can be precisely cashed out in Willats’ approach. The orthogonal projection system determines which features are significant and what information can be gleaned from them, that is, true relative size, true shape and relative position in the two dimensions parallel to the picture plane. This analysis provides a tractable means of assessing whether the projection system of the images produced captures the spatial features of the cell membrane ultrastructure that are of interest.
The denotation system requires another level of analysis. The marks here are dots or areas of ink on a page. The picture primitives are regions and lines. The scene primitives appear, at least to my untrained eye, to include a black background surface and a number of 3D lumps and sticks on top of or perhaps within it (this is difficult to decipher because orthogonal projection is employed). The analysis at the level of objects is the tricky bit here (at least for me, though perhaps not for my reader) because I am looking at a decontextualized representation with an untrained eye. Indeed, it is precisely the identity of the objects and an understanding of their relations that the scientists who first view such images attempt to discover. But the image is clearly a scene as it shows shape representations of physical objects—a cell membrane ultrastructure and its parts—that have more complex attributes than their shapes. Again, Willats’ analytical tools offer a means for assessing the image in terms of whether the features of the marks produce picture primitives that obscure or reveal the important shape information for identifying and characterizing the objects pictured. The mechanical process determines the form of the image and the form determines the content. In addition, the counter-factual dependence of image on object entailed by its mechanical production provides a kind of epistemic access to the object that justifies accepting that the spatial relations that constitute the parts of the object and their relations are as they appear in the image.
As for the chemical diagram the marks are again ink on the page. But in this case the picture primitives are all lines, some dashed, many inscribing regions. The scene primitives are trickier as it is not clear whether we are meant to see these shapes as 2D or 3D. However, this is irrelevant to identifying the objects represented. In this case, object identification appears to be straightforward as the significance of shapes of the objects is stipulated. Here the notion of a visual language is useful as each part of the representation denotes a particular object or relation and in order to understand its precise content each part must be translated and read. There are, however, reasons to think that it is also a visual picture, though a complete account would require additional conceptual tools. Although there is no projection system, very rudimentary topological relations are preserved; the extended regions can be understood as molecules interacting and changing over time. This may reasonably be understood as a kind of scene (or, perhaps, a set of related scenes).
While, given Perini's analysis, we might expect that the graph will also have this dual role, on Willats’ account it is does not. The marks are as before; the picture primitives are lines, but there are no scene primitives, there are no elementary units of shape, comprising objects in a scene. There are no similarities or analogies with the visual experience of what is represented; the spatial features of the graph are arbitrary with respect to the physical objects themselves, except indirectly through abstracted quantities. The graph is, indeed, completely dependent on linguistic representations identifying x and y axes in order to have any empirical content at all. The interchangeability of the graph—its capacity to represent any two sets of objects or properties that can be quantified and related using the same mathematical function—is indicative of its being only a visual language. Thus, in this case, Willats’ analytical method cannot help us to parse and assess content as there is nothing pictorial about the graphFootnote 22. This shows how Willats’ method is able to demarcate pictorial from non-pictorial representations. Only if the marks on the page comprise picture primitives, comprising scene primitives, comprising objects in a scene is an image a picture in Willats’ sense.
That this demarcation scheme appears to be consistent with our intuitions is only one of a number of strengths of Willats’ system. As we have seen, because his approach fundamentally relates pictures to visual features of states of affairs it avoids a number of the problems arising from Goodman–Perini conventionalism. No characters need be identified; nothing need be translated because pictorial representation is entirely autonomous from language, grounded in the psychology of perception. Moreover, the analytical methods for parsing pictorial content themselves suggest norms for assessment. In contrast, the many problems that plague Perini's approach suggest that a Goodmanian conventionalism may be a dead end for a general epistemology of scientific images. Even some of Perini's own remarks relating the electron micrograph to ‘our visual perception of shapes’ and observing ‘geometrical’ correspondence seem more appropriate for Willats’ system than her own, thus lending it tacit support. It would be a mistake, however, to move too fast as the Goodman–Perini approach still shows promise for the type of images that I have called visual languages, which suggest that there may still be a role for Goodmanian conventionalism in an epistemology of scientific images after all, albeit a limited one.
6 Drawing conclusions
Certainly, Perini is engaged in a very important project—devising a general method for analyzing the content of scientific images so as to elucidate their role in scientific reasoning—and she makes an interesting case for a Goodmanian conventionalism. However, the problems with her account suggest that even if it can tell us something about visual languages, it cannot be an appropriate basis for an epistemology of pictures. Fortunately, Willats offers a very powerful analytical method that can serve as a basis for constructing a better theory. The strength of Willats’ method is grounded on his sensitivity to the practices of image production and the psychology of perception. He focuses attention not simply on the objects depicted, but instead on the marks on the page and how they construct content. Thus he reminds us of the many different ways in which content can be pictorially represented and offers tools for us to develop analyses of the strengths and weaknesses of various different rendering methods and styles.
Willats’ analytical method is wholly different from that designed for language or from a quasi-linguistic approach. His mode of analysis suggests that the logic of pictures might be quite different from that of words, and asking whether a pictorial representation is true may be much like asking whether one's current perceptual experience is true—a kind of category mistake. This pushes us to consider the different epistemic functions that visual representations may serve and encourages us to look to the literature on models and the semantic approach to theories for guidance. The fact is that in order to understand the content of images and elucidate visual reasoning we may need to stop trying to apply analytic methods designed for language to these distinctive content bearers. What we cannot abandon is a careful method for analyzing the content of scientific images.
Acknowledgements
Important aspects of this work were developed during my dissertation research. While Dr Kathleen Okruhlik was integral to the development of that project, the central ideas that are discussed here owe most to Dr Patrick Maynard, who introduced me to the works of Kendall Walton and John Willats and showed me how beautifully they complement each other. Thank you also to the Philosophy Department at Dalhousie University for feedback on an earlier draft and particularly to Dr Mélanie Frappier whose criticism and support have been invaluable.
