2. SUPPORT FOR GESTURES IN COMPUTER-SUPPORTED COLLABORATIVE DESIGN

Gestures can be considered in relation to the design of computer systems that support cooperative (design) work in a number of ways. For one, we might concentrate on how existing gestural interactions are made around systems or consider how system elements might provide additional resources that people can gesture about. There is also the possibility of allowing people to gesture to systems by having their gestures interpreted as system commands. Each of these areas has potential implications for system design, but here our interest is in another area: how gestures might be relayed through a system to remote parties engaged in a collaborative design activity. This raises several difficult challenges, which researchers in computer-mediated communication have struggled to address. One core issue is how to provide a remote embodiment (or representation) for the gestures of users.

Telepointers are one relatively simple and widely used technique for transmitting (what might be considered as) pointing gestures of remote participants. Telepointers are simply screen-based cursors that are replicated over a network so that each user of a groupware application can see the cursors of all the other users. From a gestural perspective, such systems are very limited. As Buxton (Reference Buxton2007) remarks, they restrict the user “. . . to the gestural vocabulary of a fruit fly!” (p. 253). Nevertheless, users are still able to make powerful use of telepointers, such as to provide a limited embodiment, awareness of the actions of others, and for gestural communication between users (Dyck et al., Reference Dyck, Gutwin, Subramanian and Fedak2004). The representation of the remote participant may be conveyed as a conventional mouse cursor, or may be combined with additional information such as a trace of the user's path of movement over time (Gutwin & Penner, Reference Gutwin and Penner2002), a visual indication of the accuracy of the position (Dyck et al., Reference Dyck, Gutwin, Subramanian and Fedak2004), or information such as the user's name or picture (Sánchez et al., Reference Sánchez, Strazzulla and Paredes2008). Researchers have also investigated the possibilities of how multiple people might interact, for instance, by aggregating or delegating control of a telepointer among interactants (Osawa, Reference Osawa2006, Reference Osawa2007).

The idea of a remote pointing device has also been explored in remote physical space, beyond the confines of the groupware screen. In a system designed to support remote instruction of operators undertaking physical tasks, Yamazaki et al. (Reference Yamazaki, Yamazaki, Kuzuoka, Oyama, Kato, Suzuki and Miki1999) describe a system consisting of a remotely controlled laser pointer mounted on a mobile, radio-controlled carriage, and coupled with a camera and audio link. The (albeit limited) mobility of this “GestureLaserCar” allowed an instructor and operator to mutually coordinate their positioning in relation to a task. The instructor could control the movements of the laser pointer with a mouse to point at tools, objects, bodies, and so forth, in physical space. The laser light could also be made more intense by pressing the mouse button or blinked by repeated clicking. A subsequent system called GestureMan substituted a mobile remotely controlled robot platform for the car to provide greater freedom of movement (Kuzuoka et al., Reference Kuzuoka, Oyama, Yamazaki, Suzuki and Mitsuishi2000). With this system, it was observed that operators noticed changes in the direction of the instructor's gaze and could use this to orient to the area that the instructor was about to talk about.

Video has also been widely explored as a means of mediating gestural interactions in computer-mediated communication. Although video might at first seem well suited to representing gestures, in practice, it has proven problematic as a medium for conveying remote gestural interactions. The main problem is that video introduces distortions into interpersonal communication, which interfere with their interaction, including aspects such as awareness of gaze, relative size of gestures, and differential access to a mutual environment (Heath & Luff, Reference Heath and Luff1991). However, it has also been found that over extended periods, users are able to develop interaction mechanisms, which can overcome some of these problems (Dourish et al., Reference Dourish, Adler, Bellotti and Henderson1996).

Many of the more promising attempts at the use of video as a means of mediating or linking gestures between remote parties have been those that employ video in relation to some shared resource around which the interaction can be organized and made intelligible, rather than through a stand-alone video link. This would seem especially relevant for developers of computer-supported collaborative systems, because of the important role that shared resources and objects often play in design activities (Brereton & McGarry, Reference Brereton and McGarry2000).

The VideoWhiteBoard was designed for remote shared collaborative drawing (Tang & Minneman, Reference Tang and Minneman1991). A video feed of the remote drawing partner and their drawing was projected onto the rear of a semitransparent drawing surface with an accompanying audio link. A local partner could draw on the front side with a regular dry-erase marker and the remote partner would appear as a shadowy figure standing just on the other side of the wall. Collaborators could therefore see and hear the gestures and talk of their remote partner in relation to the drawing that they made and get a sense of how far their partner was standing from the wall by the blurriness of the shadow. Through observations of the system in use, participants were seen to make a variety of gestures, such as pointing gestures referencing parts of the drawing, elicitations such as cocking of the hand to the ear to request the partner to speak up and full body gestures such as shrugs. Another system, ClearBoard, used the same basic metaphor of a remote participant standing behind the whiteboard, but transmitted full-color video rather than a shadow only (Ishii & Kobayashi, Reference Ishii and Kobayashi1992). This allowed participants to maintain shared awareness of eye gaze and thereby supported the transition from shared drawing to face-to-face discussions.

Another metaphor that has been explored with video is to convey video of the physical workspace of a participant to their remote partner, in order to provide a shared reference for gesturing. In the DOVE system, a remote instructor could view and make pen gestures on a video stream of a participant's workspace in order to guide them through a simple assembly task. The annotated video was then displayed back to the participant on a separate monitor in the workspace along with audio from the instructor (Fussel et al., Reference Fussell, Setlock, Yang, Ou, Mauer and Kramer2004). This has been taken further by projecting video of the hands of a remote “helper” down onto a desktop surface at which a local participant was engaged in an assembly task, thus producing a “mixed reality ecology” (Kirk et al., Reference Kirk, Crabtree and Rodden2005).

Collaborative virtual environments are distributed virtual realities designed to support collaborative activities (Churchill & Snowdon, Reference Churchill and Snowdon1998). Users are typically represented in such systems as avatars. Although the potential value of providing avatars with the ability for gestural expression has long been recognized (Benford et al., Reference Benford, Bowers, Fahlén, Greenhalgh and Snowdon1995), gestural interactions are still not well supported and often rely on the typing of text commands to trigger predefined scripted gestures (Moore et al., Reference Moore, Ducheneaut and Nickell2007). Although some systems allow for pointing gestures to be made, such gestures are markedly more difficult to make and interpret in a collaborative virtual environment than they are in the real world (Wong & Gutwin, Reference Wong and Gutwin2010). The utility of adding support for pointing gestures to collaborative virtual environments has been investigated in the context of a simple design task of rearranging furniture within a room (Hindmarsh et al., Reference Hindmarsh, Fraser, Heath, Benford and Greenhalgh1998). Although there was a benefit from the use of gestures, several interaction problems were apparent due to each participant only having a fragmentary view onto the world, needing to refer to objects explicitly and verbally even when visible to both partners, and due to the disruption of interactional resources normally used to make sense of another's activity (Hindmarsh et al., Reference Hindmarsh, Fraser, Heath, Benford and Greenhalgh1998).

As our review illustrates, a number of systems have been developed to support gesture in remote computer-supported cooperative work, ranging from devices dedicated to support pointing specifically (such as tele- and laser pointers), to devices involving video that in various ways are meant to convey gesture (and other information) more generally in a manner as close as possible to face-to-face interaction. Each of these devices clearly has their own justification, but also their own problems. In particular, with respect to pointing, most devices appear to be designed to support pointing primarily as an indexical practice only, that is, as a practice for coordinating the attention of multiple participants toward a particular object, for referencing objects in the environment or for drawing relations between objects. As we will illustrate in this paper, however, the role of pointing in collaborative design as well as presumably in any other interactional context is much more multifaceted than merely being an indexical practice. In particular, we will show how pointing is used to (re-)establish intersubjectivity, that is, understanding or agreement, between participants and that this is accomplished through a variety of forms of pointing. With this rather complex picture in mind, we will consequently propose a bricolage approach to the support of pointing in mediated cooperative design, where the focus is not so much on developing devices that can mimic pointing per se, but rather on providing resources that can be used by participants in interaction to achieve the actions and activities that they might otherwise achieve through pointing. Before launching our analysis and our concluding proposal for how to address pointing in mediated cooperative design, we briefly introduce the material on which we base our observations, our method of analysis, and devices of description.

3. BACKGROUND TO THE DATA

The data that we consider for analysis comes from a design workshop conducted at Sønderborg Participatory Innovation Research Center (SPIRE) with the participants from a newly formed collaborative project focusing on designing a new type of sustainable energy generator that can replace the noisy, polluting, and fault-prone diesel engines that are currently used to power independent camps and shelters for landmine clearing operations in Angola. The project collaborators and participants in the workshop include partners from industry (manufacturers of devices such as solar panels and fuel cells), a nongovernmental organization involved in projects in developing countries, and researchers and developers from two Danish universities. Figure 1 shows the team members at one of the two group tables at the workshop. Although we have since analyzed this data for the participants’ uses of gestures, this was not the purpose of the workshop, nor was it the original reason video was taken of the session. Both the workshop and its documentation on video were organized as project events, and they would have taken place as planned with or without this particular study in mind. In this regard, our analysis is of a natural design activity at the early stages of a newly formed project.

Fig. 1.

The participants in focus at the workshop. Clockwise from left foreground: Anna, an engineer from a fuel cell manufacturer; Brian, an engineer from a manufacturer of production equipment; Claes, a university professor specializing in power grid management; Daphne, an engineer from a solar panel manufacturer; and Ewan, a manager of a small manufacturer of solar hot water systems. The final participant, Finn, who is an engineer and university researcher in the field of participatory innovation, is not visible in this picture.

The entire collaborative project, as well as this particular workshop, was organized as a participatory innovation effort, in the sense that care was given to involving the voices of the potential users of the final design, as well as people with a stake in the new product, for example, manufacturers. As the potential users in this case were primarily located in Africa, whereas the stakeholders from industry were located in Denmark, SPIRE sought to bring in the users’ perspectives through playing short videos on various themes from the users perspective (e.g., user activities, dimensions, maintenance, transport of comparable equipment) at various points during the workshop, in order to help the participants design the generator (see Yliriksu & Buur, Reference Ylirisku and Buur2007). The videos were played on a screen that is not visible in Figure 1, but it is located on the wall to the left of the camera view.

The particular design workshop we are focusing on took place on the second occasion the stakeholders had met each other. After the partners introduced their technologies to each other, they spent 1 h sketching their ideas out, with the assistance of an experienced sketcher. The finished sketches were pinned to the board visible in Figure 1 right behind two of the participants, Claes and Daphne. In the second part of the workshop, the participants used various objects such as cardboard and foam pieces as well as toy trucks, motorcycles, and figurines to design and build coarse small-scale models of their final product, a generator that could be transported either on a truck or a motorcycle. A preliminary model can be found on the table between the participants in Figure 1. The data we consider for our study is taken exclusively from the second part of the design activity, that is, where the idea boards have already been prepared and the model is being built. This second part of the workshop lasted for just over 2 h.

5. ANALYSIS: POINTING AS A PRACTICE FOR ESTABLISHING INTERSUBJECTIVTY IN A DESIGN WORKSHOP

A crucial aspect of cooperative design activities, and of almost every activity in which human beings are involved, is the establishment, negotiation, and maintenance of intersubjectivity. In other words, in order for a cooperative activity to be even partially successful the coparticipants need to be able to understand each other, to share a certain level of “common ground” (Clark et al., Reference Clark, Schreuder and Buttrick1983), to agree on what they are doing, what they have done, and where they are going. In the following, we will demonstrate how pointing is one of the devices with which participants in cooperative design can manage intersubjectivity. At its most simple, pointing can be understood as an indexical practice with which one speaker calls something to the attention of others (Tomasello, Reference Tomasello, Enfield and Levison2006), that is, to designate an entity in the environment and locate it in space so that others are able to identify what is currently being talked about (Liszkowski, Reference Liszkowski, Enfield and Levison2006; Visser, Reference Visser, McDonnell and Lloyd2009). However, we will demonstrate that “designating an entity” is in itself a practice, a practice for establishing intersubjectivity between participants because it provides a possibility to inquire about, check, affirm, or correct own or other participants’ current interpretation of the ongoing interaction and design task. In order to show this, we have chosen four cases in which one or more points are produced. Although the cases we show may not be exhaustive in terms of the numerous variations with which pointing can be produced in its various contexts, they are meant to be representative cases that together illustrate that pointing can be employed as a practice for establishing intersubjectivity. We also wish to underscore a finding by Visser (Reference Visser, Kopp and Wachsmuth2010), that there is no one-to-one match between the function of pointing and the form of the point. We extend this finding here by showing how the particular function of pointing is determined by its specific place in a sequence of interaction, rather than its gestural form.

5.1. Pointing to check understanding

We begin by illustrating how pointing can be employed as a means for first checking understanding and then correcting understanding. In the following example, the participants in our design workshop thus produce various points to make sure everyone understand where they currently are in the design process. Although Anna and Claes are adding further units to their joint design and talking and laughing about that (lines 01–03), Daphne, who was not present at the beginning of the workshop, inquires whether the item in Claes's hand is a fuel cell (line 04, Fig. 3). She thus reveals herself to be lacking understanding at this point, although her naming of the item as a fuel cell suggests that she has reason to believe that she does understand part of the design.

Fig. 3.

Daphne points to the thing in Claes's hand (around line 05).

5.1.1. Example 1

Claes is building a structure. Anna leans over the table and picks up some material. Both Anna and Claes's actions are fully visible to Daphne.

• 01 Anna:    This one is it's a fuel cell. Then there's a tube here.

• 02 Claes:    heh heh heh [hah hah heh

• 03 Anna:                       [heh heh heh heh h[eh

• 04 Daphne:                                                 [What's§ that one is that a fuel cell,

• 05 Daphne:                                                             §lifts hand with extended

• 06               finger and points to structure held by Claes

• 07               §(0.3)

• 08 Daphne: §retracts hand

The point produced by Daphne in lines 05–06 (Fig. 3) is placed at the same time as she produces the reference “that one.” With this, Daphne thus establishes a link between the real world and her talk, making it clear to the others what she is referring to and drawing their attention to that object. In addition, the point along with Daphne's verbal contribution performs a practical task of checking whether her current understanding of their joint design project is correct. In other words, Daphne points to the object not only to get the other's attention to that object but also to make sure that intersubjectivity in the form of joint understanding has been reached at this point. As we will see in the continuation of this extract, this is not the case, as Daphne's identification of the object as a fuel cell is wrong.

• 09 Claes:   No this one th[at is that§ that's §this one

• 10 Anna:                          [No.

• 11 Claes:                                      §Turns head and upper body in direction of board

• 12 Claes:                                                  §Points with hand holding structure

• 13                                                                      toward board

• 14                (0.3)

• 15 Daphne: §Torques head and upper body toward board

• 16                (1.0)

• 17 Claes:   §Scoots on chair toward board

• 18 Claes:   You're just getting (.) a fast (.) (guided tour)

• 19               §Look it's that one.

• 20               §Points to specific part on board

• 21               (0.8)

• 22 Claes:   That's like (.) e:h (.) the core unit.

Here, Claes corrects or repairs (Schegloff et al., Reference Schegloff, Jefferson and Sacks1977) Daphne's understanding that the object in his hand is meant to represent a fuel cell, and does so, among other things, by pointing to various things in their environment. First, he uses a point to indicate that the item in his hand is identical to an item on the board, thus implying that it cannot be a fuel cell (lines 12–13, Fig. 4). He then moves to the board and locates this item specifically by pointing to a drawing of the core unit (line 20, Fig. 5). This point is accompanied by a “look-prefaced” utterance (Sidnell, Reference Sidnell2007), which in itself serves to call attention to the link between the item in his hand and the core unit. Finally, he states explicitly that the thing he holds in his hand and the item he is pointing to on the board is the core unit. Both Claes's verbal and gestural contributions are thus addressed to establish intersubjectivity between the participants, by making sure that Daphne who has revealed herself to have misunderstood the design now understands correctly. Although Claes's pointing thus serves to locate and identify various objects for Daphne, this designation of entities is in itself done in the service of remedying Daphne's faulty understanding of the design.

Fig. 4.

Claes points toward the board (around line 12).

Example 1 thus illustrates how pointing may serve a particular interactional function and is used in a particular interactional context, namely, one in which a potential problem needs to be solved. The point produced by Daphne is thus part of her activity of checking whether she has understood what was going on correctly (has identified a part of the design in the intended manner). Likewise, Claes uses his points as a resource for establishing mutual understanding, although he does so by correcting Daphne's misunderstanding. In other words, all of the points in this example are not merely indexing a particular referent or item but are doing so in the service of solving a misunderstanding.

Visser (Reference Visser, McDonnell and Lloyd2009) argues that there is no one-to-one correspondence between the form and function of a gesture and as Figures 3–5 illustrate, this is also true of pointing that is employed in the service of solving or correcting a misunderstanding. Thus, the points produced by Daphne in lines 05–06 (Fig. 3) and by Claes in line 20 (Fig. 5) very precisely locate a particular item (in Claes's case, naming it), whereas the first point produced by Claes in lines 12–13 (Fig. 4) merely suggests a direction and a place toward which the others should look.

Fig. 5.

Claes points to the board (around line 20).

As Figures 3 to 5 together illustrate, pointing is not a single, definable movement, even when employed for one particular function, that of solving a problem of understanding. Rather, pointing can be performed in a variety of ways to index where coparticipants in the current state of talk should direct their attention (see also Goodwin, Reference Goodwin and Kita2003; Haviland, Reference Haviland and Kita2003; Kendon & Versante, Reference Kendon, Versante and Kita2003). Moreover, pointing is in itself part of a larger set of physical practices with which something can be brought to attention. In Figure 4, for instance, it is not the point alone that gives the direction in which to turn the attention, but also the direction of Claes's gaze and the posture of his body. It is thus not pointing, as a simple iconic gesture that is of interest or relevance here, but rather the action of calling attention to an object (whether this be done by pointing the index finger in the direction of an object, turning toward an object, gazing at an object, referring to an object) and it is this action that the participants in Extract 1 employ to solve a problem of understanding.

5.2. Pointing to correct or repair understanding

In the following example we see how pointing (thus calling attention to an object) is enlisted to solve a problem of misunderstanding, that is, to correct a misunderstanding. Here, the group is in the process of finishing their model, when Finn (in line 01) states that there need to be wires connecting the different parts (from the devices producing electricity to the devices using electricity). The wires have already been put in place by Claes at an earlier stage, where he even notified the others verbally of what he was doing. Nevertheless, Finn has apparently failed to understand or has forgotten this.

5.2.1. Example 2

• 01 Finn:    And then there should be some wires connected somehow.

• 02             (1.5)

• 03 Claes:   Yes b§ut the wires are there.

• 04 Claes:            §points to model

• 05              (0.3)

• 06 Finn:     Oh that's right it's there

In line with the more specific points employed by Daphne and Claes in example 1, Claes here (in line 04) employs a point that exactly locates where the wires are positioned (see Fig. 6).

Fig. 6.

Claes locates where the wires are positioned (around line 04).

By doing so, Claes provides visual evidence for his concurrent statement that the wires are in place (line 03). With this, he implies that Finn has somehow missed something, leading him to the faulty conclusion that the wires are still missing. By directing Finn's attention to the wires, Claes thus manages to correct Finn's misunderstanding, as evidenced also by Finn's subsequent response in line 05, where he produces the realization token “oh that's right” with which he treats the information given by Claes as something he has only now understood (or remembered; Emmertsen & Heinemann, Reference Emmertsen and Heinemann2010).

Our next case similarly illustrates how pointing is employed to enlist an object to correct misunderstanding between the participants, but in this case the quality of the pointing is very different to that in example 2. The object referred or pointed to is an abstract, distant location, and the pointing is thus correspondingly conceptually complex (Haviland, Reference Haviland and Kita2003), done as a backward, over-the-shoulder point in the direction of the screen where the video about Africa has been shown earlier. This point is illustrated in Figure 7. As the trace in the picture shows, the pointing gesture in this example is not the stereotypical gesture of an arm and index finger extended out from the body, but a much subtler gesture made with a flick of the fingers back over the shoulder to the video without moving the hand away from the area of the face.

Fig. 7.

A backward, over-the-shoulder point in the direction of the screen (around line 07).

5.2.2. Example 3

• 01 Claes:    Nyeah but what could we could get four

• 02                hundred watts from one.

• 03                (0.6)

• 04 Daphne: eh Yeah (in a couple of years)

• 05                (0.6)

• 06 Claes:     Y[es = No but §down there. down there.

• 07 Claes                              §points backwards to  videoscreen

• 08 Daphne: At the moment it's two hundred eighty.

                                                              = Yes no[ now

• 09 Claes:                                                            [Yes

• 10 Daphne: sorry yes

Here, the participants of the workshop are trying to figure out how many different units they need in their design in order to provide sufficient resources (electricity, heating, and water) at the site they are designing for. In lines 01–02, Claes formulates what Labov and Fanshel (Reference Labov and Fanshel1977) term a “B-event” statement, a statement that concerns something that someone else has better access, and hence, greater rights to know about. In our case, Claes makes a stipulation about the efficiency of solar panels in a context where Daphne, who works for the solar panel company, is present. Claes displays his awareness of this by directing his gaze at Daphne, thus selecting her as the recipient of his statement (Goodwin, Reference Goodwin and Psathas1979). At the same time, however, Claes reformulates his utterance from being a question (“what could we”) to being a statement (“We could”), in effect answering his own (unstated) question. Whereas the question form, for instance, “what could we get from a panel,” would have allowed Daphne to provide her expert knowledge as information to Claes, the statement form merely invites her to confirm (or disconfirm) the information given. As can be seen from line 04, Daphne produces what initially may appear to be a confirmation, a “yes.” However, the structural aspects of her response are designed to project a negative, “dispreferred” (Pomerantz, Reference Pomerantz, Atkinson and Heritage1984) answer, in that it is delayed by 0.6 s and furthermore preceded by a hesitation marker “eh.” Finally, Daphne adds the qualifying “in a couple of years” to her “yes,” thus canceling out her confirmation as being valid only at some point in the future, but not now.

At this point then, Daphne has basically told Claes that his assumption about how many watts a solar panel can produce is wrong. Corrections or disconfirmations such as these can be delicate social matters; they are indications of a breakdown in intersubjectivity and social solidarity which interaction in general is designed to achieve and maintain (Sacks, Reference Sacks1995; Clayman, Reference Clayman2002). As illustrated in examples 1 and 2, pointing can be employed exactly in such positions of potential breakdown. In previous examples, pointing was thus employed as part of a larger practice, to solve problems of misunderstanding. In Example 3, the potential breakdown in intersubjectivity is more severe, as in this case Daphne has specifically disconfirmed (and thus corrected) something that Claes has stated in a manner that suggested he was an expert. Again, pointing is here deployed to enlist an object as being part of the participants’ shared experience, and again this is done in order to solve interactional trouble. Thus, Claes, by pointing in the direction of the video screen (line 07, Fig. 7), enlists the setting in which their design will eventually be placed (Africa) and uses this shared experience of having seen the conditions in Africa as the grounds for his statement about how much energy a solar panel can produce, suggesting it was based on the local contingencies of the use context, not on the specifications of the solar panels themselves. In turn, Daphne accepts these contingencies, confirms that Claes's stipulation about the efficiency of the solar panels was correct and even apologizes for having made a faulty correction (see the “sorry, yes” in line 10), and the interactional trouble is thus resolved.

5.3. Pointing to preempt trouble

Examples 1, 2, and 3 together illustrated that pointing can be employed not only to direct coparticipants’ attention toward an object but also to do so to solve a problem of intersubjectivity in a context where some interactional trouble has occurred, that is, where one participant has failed to understand, has misunderstood, or even been corrected. Pointing in this context of trouble thus serves (along with other practices) as a resource for solving a problem of intersubjectivity that the participants are facing and need to deal with before moving on to the next part of their design task.

In our next example pointing is also employed to enlist an object. However, in this case, the point and its accompanying verbal actions appear to be used not just to solve trouble but to preempt it.

5.3.1. Example 4

• 01 Brian:      But that could be the box or what,

• 02                 (0.6)

• 03 Brian:     so you wouldn't have to carry[   [

• 04 Daphne:                                                  [You [could

• 05 Ewan?:                                                              [Yes

• 06 Daphne (just break) with to ([find out)

• 07 Brian:                                      [§you know this one

• 08 Brian:                                       §Points to board

• 09               (1.4)

• 10               point is retracted to scratch nose

• 11 Ewan:  But then it's still easier to have a thin

• 12               or a telescope you can pull up.

In line 01, Brian proposes that a pole needed for getting the solar panels in a high enough position on the ground could be used at the same time as part of the box within which to carry the solar panels to the site. As the 0.6-s silence in line 02 shows, Brian does not get any immediate uptake of his proposal. Proposals are contingent upon a coparticipant's acceptance and a lack of immediate uptake such as in this example is interpreted and treated as a problem by the proposer as well as by coparticipants (Heinemann et al., Reference Heinemann, Mitchell, Buur, Darras and Belkhamsa2009). However, studies show that participants in interaction (Svennevig, Reference Svennevig2008) have a preference for interpreting indications of trouble in the least problematic way possible, for instance, by treating a lack of uptake as a problem of hearing or understanding, rather than as a problem of agreement. There could be many different explanations for why Brian's proposal is not ratified straight away, in particular, in this case because (a) it is not obvious to whom (if anyone in particular) he is addressing his proposal and (b) two of the group (Claes and Daphne) have temporarily left the table to glue some pieces together and thus may not have heard his proposal at all. In line 03, Brian displays his interpretation of the lack of uptake as being a problem of understanding by adding another part to his utterance, further detailing what his proposal would entail in practical terms (“then you wouldn't have to carry”). By initiating this part with a conjunction (“then”), Brian constructs this utterance as being part of his earlier turn, thus effectively erasing the (0.6 s) silence and giving his coparticipants another chance at affirming his proposal. At this point, Daphne, who is at a different location in the room and, together with Claes, engaged in trying to glue a part of the design together responds with something that could be a ratification of Brian's proposal (the “you could” in line 04). When Brian turns to look at Daphne, however, it becomes obvious that her utterance was directed at Claes and had nothing to do with Brian's proposal. Turning back, still without any uptake of his proposal, Brian makes a further specification of what he meant with “that could be the box,” this time by pointing (in line 08) to and thus locating for the others what he meant to refer to with “that.” Here, the point is done with an extended index finger toward the board that the participants have earlier constructed together, and targeting a particular item there (Fig. 8). As in our previous cases, Brian's point thus enlists (part of) an item that is part of their shared experience and directs the other participants’ attention toward it. As Figure 8 illustrates, this particular pointing movement is very similar to the mid-specific point in Figure 4 and example 1 and thus lies somewhere between the very specific point delivered by Daphne and Claes in example 1 (Figs. 3 and 5), in contrast, and the very broad, unfocused point in example 3 and Figure 7.

Fig. 8.

Brian points to the board (around line 08).

With the point and its accompanying verbal contribution, Brian once again treats the lack of uptake as a problem of understanding, rather than it indicating that the others do not like his proposal. By pointing to the board, he thus attempts to identify what he was talking about, in order to make it easier for the others to understand and hence to agree with his proposal. As it happens, he does not succeed in soliciting agreement from the others, as is evident from Ewan's turns at talk in lines 11 to 12, where the proposal is effectively discarded by comparing it unfavorably to another solution. Brian's lack of success, however, should not distract from his integrating a pointing gesture that locates a specific item he has referred to as part of a larger practice for pursuing agreement with a proposal he has problems getting ratified by the others. In this way, he is seeking to avoid or preempt a potentially upcoming disagreeing response, that is, a response that would reject his proposal.

6. SUMMARY

Through these examples, we have tried to illustrate that although pointing certainly is a way of fixing linguistic references to objects in the world and thus identifies and locates these objects for others, this is not the only, or even primary, thing that is accomplished through pointing. Our analysis shows that “designating an entity,” for instance by pointing, is a practice that can be employed by the participants in this design workshop in order to solve or prevent problems of intersubjectivity (e.g., misunderstandings or disagreements). We have furthermore sought to demonstrate that this employment of pointing is not dependent on a particular form, shape, or size of the point, but that both specific and abstract pointing, with or without an extended index finger can be used for this purpose, the crucial thing being that some item in the physical (or imagined) environment is enlisted to “solve” the interactional trouble. Naturally, pointing is not the only way in which the environment can be so enlisted (other practices that may figure prominently in this regard could be gaze, other gestures, as well as the actual handling or placing of objects (e.g., see Clark, Reference Clark and Kita2003). Likewise, gestural reference to the physical environment is not the only way participants can manage problems of intersubjectivity, although this may serve a more prominent role in the context of performing a practical task such as designing technical and material configurations. However, it is noteworthy that pointing seems to be a tool readily employed to do this kind of “trouble” management. We did not, for instance, see designers pointing at particular objects (e.g., solar panels) just because they happen to be talking about those elements of the system at the moment. Pointing is accountably more than just fixing a point of others’ attention, or specifying and disambiguating a linguistic reference for the sake of clarity.

With this in mind, we also observe that in our data, “shared” objects are not simply shared by virtue of communal physical or visual access. The wires, the fuel cells, and so forth, became shared by being coopted into the work of achieving intersubjectivity. That is, sharing physical spaces and objects does not of itself enable the participants to share objects in a social or intersubjective sense. For research concerned with understanding design practice, this is an important point.

As we look to possible applications of analyses like this one, it is shared experiences, and the ability to refer to shared experiences, that are the more relevant practices to try to support in computer-mediated communication, rather than providing shared access to objects, spaces, or visual fields. We feel entitled to say this on account of the ways in which physical objects, spaces, and visual fields were enlisted in order to index shared experiences and to head off intersubjective “troubles,” rather than being important for simply being physically present, being objects of particular types or qualities, or because they were relevant semiotic markers of conversational objects.

We readily agree with previous research that gestures can take various forms and that they can serve a range of functions in design. However, of interest, with the analytic lens that we have employed here, we have shown how the particular function of pointing (as a tool to achieve intersubjectivity) is not determined by its form or the manner of its production, but draws its function from its sequential placement in interaction: that is, what the current social circumstances are, what has just happened, and what it is that hangs on whatever happens next. It is precisely this local sequential context that is omitted from classification schemes that abstract the various functions of gesture, yet it is this context that furnishes each particular gesture with its meaning and usefulness. Our recommendation for future research on gesture in design would be to first build up a substantial corpus of naturalistic examples of gesture that preserve the interactional details of their production, prior to glossing over these details in service of generic classifications schemes, particularly as gesture research in design is still searching for semantic and/or context-related regularities in form–function relationships (Visser, Reference Visser, Kopp and Wachsmuth2010).

7. IMPLICATIONS FOR COMPUTER-MEDIATED COMMUNICATION

In this paper, we have presented an analysis of the use of pointing gestures within a face-to-face design meeting. Although such studies are useful in themselves for the kinds of things they tell us about gestures in design (which is an area still deserving much research), our aim here is also to explore the implications of our findings for the development of computer-mediated communication systems, in order that they might be improved to support gestural interactions between remote designers.

In relation to our focus on intersubjectivity, we have shown in the examples how pointing served to support people to check understandings with each other, to correct and preempt misunderstandings, and to pursue agreement. The analysis also shows that pointing gestures do not conform to strict preexisting standards of form but encompass a diversity of movements ranging from quick flicks of the fingers to more stereotypical pointing gestures involving an extended arm and index finger. Our analysis also highlights that there is an array of contextual details through which pointing becomes intelligible. Prominent among these are the relation of pointing gestures to ongoing talk, how pointing gestures are linked to gaze, the positioning of pointing gestures with objects in the environment, and how people position themselves in relation to each other and the space when pointing.

We are cognisant that findings of the sort presented above can be difficult to translate into ready implications for systems design. The approach we have taken in our analysis, which is to attend to the interactional detail of particular examples, militates against a broad abstracting approach in the analysis itself and in its application to design. Nevertheless, from a system development perspective, one implication for design that can be taken from these findings is that system designers should at least be aware that pointing gestures might encompass a wider range of forms and purposes than is usually assumed, and may relate to a wider range of contextual details. Providing designers with this wider understanding would be beneficial both in order that they make sure that potentially important aspects of interaction around pointing are not interfered with by new systems, as well as for the possibility that such systems might actually respond to or mediate these aspects of interaction.

Our focus in this paper has been on the analysis of several instances of pointing gestures in a face-to-face design activity. The task of designing and building a system that would actually respond to or mediate these aspects of interaction is beyond the scope of what we can present here, so we must remain somewhat vague in how our findings might be translated into specific features of a system. Instead, we offer a higher level argument for a particular approach to the design of such systems, that our findings would support.

One problem for systems designers in making use of our findings is that in laying out the richness and subtlety of face-to-face interactions, it is likely that the interactions offered by computer-mediated systems will be found lacking by comparison. A plausible response would be to focus on bringing the interactions provided by systems closer to those of face-to-face interactions, such as by providing extra multimodal information (video display of gestures, relation to objects, and verbal activity) (Tang & Leifer, Reference Tang and Leifer1988), making computer representations responsive to a wider range of pointing gesture forms (Tory et al., Reference Tory, Staub-French, Po and Wu2008), or questioning whether alternative pointing modalities provide an adequate substitute for face-to-face pointing gestures (Visser, Reference Visser, McDonnell and Lloyd2009).

Although it certainly seems a worthwhile undertaking to continue to improve the fidelity of computer-mediated interactions for design and keep a critical eye on the adequacy of current interfaces, we also wary of an implicit assumption that naturalism is the key to improving systems’ abilities to act as mediators of gestural interactions. As a goal for design, naturalism is problematic for several reasons. It leads to the idea of a more or less real representation, which implies that there will be a most natural system, a singular solution to the problem of mediating gestures (cf. Matthews, Reference Matthews2006). It is also problematic as a goal because it is difficult to know exactly what aspects of a gesture make a representation natural. As observed by many gesture researchers (and as borne out by our analysis), it is not only the physical form of a gesture that is important for its meaning but also details such as how it relates to speech, the space within which it is performed, the material artefacts, its sequential position, and the particulars of its performance. It seems unlikely that a boundary could be placed on what a system would need to include in its representation in order to make it natural “enough.”

Many of the more interesting computer-mediated communication systems rely on unnatural devices as part of the interaction. For instance, the projection of a remote helper's hands into a local workspace in the same orientation as the hands of their local partner as demonstrated in Kirk et al.'s (2005) mixed ecologies approach would be a highly unnatural position if it were enacted face to face because it would entail the helper speaking aloud their instructions while reaching their arms around either side of the partner from behind.

An alternative approach to naturalism, which we believe could be more productive for system design, is to aim to support the creation of bricolage solutions: to favor the design of small, partial solutions that users (as bricoleurs) could combine and bring into use in order to convey their gestural actions. However, what might such a bricolage approach look like in concrete terms?

Designers of computer-mediated communication systems may not have to look very far for good examples. Consider the VideoWhiteboard system introduced earlier. Constructed as it was from off the shelf video cameras, projectors, and speaker phones that were artfully arranged in relation to two semitransparent but otherwise unspectacular drawing surfaces, it actually provides a wonderful example of the spirit of bricolage as used in design (Buxton, Reference Buxton2007). From this perspective, a strategy for supporting end users as bricoleurs might be to try to identify the recurring elements of systems (e.g., projector–camera pairings, display surfaces, drawing implements) and consider how these might be incorporated into new kinds of systems that users could bring together in particular ways to suit their needs.

We see it as important to consider how this bringing together might be done both in the moment-to-moment interactions within a design meeting as well as in the more deliberate ahead of time processes of configuration—such as the day before setting up of a room. When engaging in activities of bricolage as part of the set up for a design meeting on the day before, we expect that users would be able to take their time to configure a system for the particular needs that they anticipate their meeting will have (whether they need a VideoWhiteboard, or a VideoPond). In the case of the moment-to-moment interactions within a design meeting, we expect a more ad hoc approach, for instance, improvizing a shared drawing implement as a pointer, or repositioning a camera–projector over a newly created model in order to make a presentation to remote participants. The key here is to think of computer-supported cooperative design not in terms of systems that are provided for use by system designers, but as configurations that are brought into use by user-bricoleurs.

Intersubjectivity and its moment-to-moment management are not, we think, productively conceptualized as “problems” for systems developers to remedy once and for all through the innovative design of new computer-mediated communication systems. Rather, they are participants’ matters, for them to negotiate in situ. In this respect, the issue for system design is not about replicating the natural affordances of face-to-face environments, but about providing the users various resources that can be pointed to and gestured with as resources for design participants to achieve and maintain their intersubjective understanding of an unfolding design situation.