3.1. scope

Cognitive science researchers are interested in a subject that is not directly accessible to observation: processes in the mind and brain, thoughts and thought processes. Although language is an everyday medium used to express thought, there are nevertheless limits to the scope of research that can be addressed through language data analysis. As a first step, therefore, it needs to be clarified to what extent language is a suitable medium to convey insights relevant to the research question at hand.

As a starting point, a simple heuristics when considering the scope for CODA is this. Anything that can be meaningfully verbalized by speakers, can be meaningfully analyzed using systematic linguistic methods. As already indicated, the phenomena that a researcher can identify by a close look at linguistic choices may go beyond whatever the speakers themselves verbalize explicitly or would be aware of. Systematic linguistic analysis can thus run deeper than conscious awareness, but it cannot exceed the data resource itself – it cannot address cognitive processes that do not have any reflection in language at all. Consequently, the range of research questions that can be meaningfully addressed via language is mostly limited to non-automatic cognitive processes, excluding memory retrieval, activation and recognition, automated procedures, sudden insights or realizations, and the like. Some of these can be reported after the fact to the extent that they leave a trace in short-term memory (Ericsson & Simon, 1993). Moreover, think-aloud data may to some extent reflect cognitively crucial moments not by explicit formulation but by subtle features such as hesitation markers, pauses, changes in intonation, and the like. Nevertheless, many research issues of interest to cognitive science researchers concern unconscious levels of cognitive processing that will not find any reflection in language and will therefore need to be addressed in different ways.

While the general scope of potential applications is more extensive, related research traditions fall into two main areas, both of which are combined and extended by CODA. The first tradition concerns mental representations, namely the linguistic representation of conceptualized information, such as perceptually available or memorized scenes. Typically, this involves description tasks related to an experimentally controlled scenario presupposing no particular cognitive effort (except memory, if the scene to be described is no longer perceptually available). Here a close analysis of linguistic detail is fairly common in order to address the mental representation of perceived information, leading to a broad variety of significant insights (e.g., Berman & Slobin, 1994; Nuyts & Pederson, Reference Nuyts and Pederson1997; Taylor & Tversky, 1996).

The second area concerns the analysis of complex cognitive processes as identified by verbal protocols produced along with cognitively challenging tasks (Ericsson & Simon, 1993), such as problem solving (Newell & Simon, 1972) or decision-making (Ranyard, Crozier, & Svenson, 1997). The method of having people think aloud during such tasks, or provide a retrospective report of what they were thinking, is based on early work by Newell and Simon (Reference Newell and Simon1972), and has over the past decades been widely used across many different domains, including spatial cognition (Gugerty & Rodes, 2007), medical areas (Kuipers, Moskowitz, & Kassirer, 1988), design studies (Purcell & Gero, 1998), reading research (Afflerbach & Johnston, 1984), usability (Krahmer & Ummelen, 2004), and many more. Linguistic data of this kind can be seen as an external representation of some aspects of what is going on in the mind. In particular, think-aloud protocols and retrospective reports provide procedural information that complements other data, such as decision outcomes and behavioral performance results.

Conventionally, the focus of verbal protocol analysis lies on the content of verbal data, addressing those aspects (e.g., particular thought processes or strategies) that the speakers are themselves aware of (or ‘heed’; Ericsson & Simon, 1993). The content-based inspection of verbal reports, particularly if carried out by experts in the problem domain and set against a substantial theoretical background (Krippendorff, 2004), often leads to well-founded specific hypotheses about the cognitive processes involved (see, for instance, a detailed script analysis in Kuipers et al., 1988).

However, the implications of particular choices of linguistic structure have rarely been taken into account in this line of research. Here, CODA provides a substantial step forward by pointing to the cognitive significance of specific linguistic features, operationalizing and validating content categories, and extending the scope of research to cognitive aspects that are not necessarily consciously available to the speakers and therefore do not get to be verbalized explicitly. This opens up avenues for addressing research issues that have not yet been investigated using verbal data at all, ideally through triangulation with other kinds of data.

To introduce our running example illustrating CODA procedures, Tenbrink and Seifert (Reference Tenbrink and Seifert2011) examined how speakers devise route plans for a holiday round trip on Crete. Previous research had highlighted a range of conceptual strategies available for planning complex routes. Hayes-Roth and Hayes-Roth (1979), for example, examined abstract processes reflected in verbal protocols and proposed a cognitive model of planning on this basis. Further research examining scenarios related to the well-known Traveling Salesman Problem (TSP) task was mainly based on behavioral results along with computational modeling, allowing for conclusions about underlying strategies such as clustering mechanisms (e.g., Graham, Joshi, & Pizlo, 2000) or nearest neighbor heuristics (e.g., Best & Simon, 2000). Now, Tenbrink and Seifert (Reference Tenbrink and Seifert2011) investigated the extent to which results from this research extended to holiday tour planning by examining trajectories, cognitive focus, and conscious strategies for designing a route. Relevant cognitive processes were deemed likely to be verbalizable, related to previous research (e.g., Hayes-Roth & Hayes-Roth, 1979; Tenbrink & Wiener, 2009). Furthermore, Tenbrink and Seifert (Reference Tenbrink and Seifert2011) were interested in the challenge of conceptually shifting between a two-dimensional map in small-scale space (used for planning) and the three-dimensional real-world situation in large-scale space that travellers are confronted with (see Pick, Heinrichs, Montello, Smith, & Sullivan, 1995, for a verbal protocol study of this challenge). This domain discrepancy is an integral part of any kind of in-advance travel planning. The time and location where planning takes place does not correspond to the time and place of traveling; to ensure a pleasant journey, it is necessary to imagine what kinds of consequence each decision during planning might have for the travel in the real world. While it was unlikely that participants would directly comment on this challenge, the required conceptual shifts should be represented in the language used to describe the problem-solving procedure, in ways yet to be explored. In a nutshell, the research questions for this study were:

• • What kinds of problem-solving strategies are used when planning a holiday route with multiple goals, and how do they relate to TSP-related research?

• • To what extent do humans refer to a real-world environment when the only available information is a map, and how do they switch between concepts of map-related planning and concepts of traveling in the real world?

• • How do these two aspects relate to each other?

3.2. data collection techniques

Having specified a research question that fits to the scope of CODA as just outlined, the next consideration concerns how to collect data and prepare them for analysis. Since CODA addresses linguistic (and, relatedly, conceptual) patterns by examining how some content is expressed or structured (beyond what is said), speakers (i.e., the participants of empirical studies) need to be allowed to make their own linguistic choices rather than choosing from a restricted set of options. The specific elicitation methods used in a study need to be chosen carefully based on the research question at hand, keeping in mind that each particular text type implies particular patterns of linguistic choices (e.g., Biber, 1989) that may not be related to the given task as such.

In the area of mental representations, the elicited verbal description should optimally reflect the speakers’ conceptualization of a perceived scene or event. For example, participants could be shown a picture and asked a question about it that triggers a description (e.g., Carlson & Logan, 2001; Henderson & Ferreira, Reference Henderson and Ferreira2004; Holsanova, 2008). Notably, the precise formulation of the question and other discourse factors systematically affect the participants’ description. As Vorwerg and Tenbrink (Reference Vorwerg, Tenbrink, Barkowsky, Knauff, Ligozat and Montello2007) showed, a question asking about the location of an element in a picture (“Where is the object?”) triggers far more detailed descriptions than a question about the identity of an element (“Which one is the object?”), although both questions can generally be answered in similar ways, as exemplified by “it is (the object) to the left of the square”. Responses to ‘Where’ questions contained more projective terms and modifiers, as in “slightly to the top and left of the square”, whereas responses to ‘Which’ questions tended to be short and simple, as in “the one to the left” or simply “the circle”.

In the area of problem-solving processes, Ericsson and Simon’s (Reference Ericsson and Simon1993) framework provides a good basis for identifying the cognitive significance of certain text types. Most prominently, information verbalized during the task (think-aloud protocols) and retrospective reports are supposed to reflect cognitive processes within short-term memory fairly directly, and can therefore be recommended as preferred elicitation methods. Ericsson and Simon (Reference Ericsson and Simon1993) provide elaborate procedural advice towards optimal elicitation for both methods. The main idea is to encourage participants to speak out loud what they are (or – in the case of retrospective reports – were) thinking, rather than guiding them towards particular trains of thought. The ‘Appendix’ provides an example instruction that can be used to train participants to think aloud, which is necessary if only to clarify what is expected from them.

It has been observed that, under certain circumstances, the requirement to verbalize may promote a better understanding of the task itself (Krahmer & Ummelen, 2004) – or it may lead to an impairment (Schooler, Ohlsson, & Brooks, 1993). Based on a range of studies showing effects in either direction (or none), it now seems clear that it cannot be generally predicted whether the requirement to think aloud will affect task performance in a particular research setting. Therefore, studies using think-aloud protocols typically involve another (control) group of participants who are asked to perform the same task without thinking aloud, allowing for a comparison of behavioral results.

Additionally, relying on think-aloud data alone may often not be sufficient since verbalizations during the task may be incomplete in various respects (Ericsson & Simon, 1993). Other types of verbalizations have different effects. Therefore, it may be useful to combine several methods of data collection, both with respect to other types of verbal data and with respect to triangulation (addressed separately below). In the following, I will briefly address some other widely used types of language data elicitation, which may be suitable for different purposes.

3.3. data preparation techniques

Following data collection, the next procedural step is to prepare the verbal data for analysis by transferring them into manageable units. Handwritten language needs to be transferred to an electronic format; spoken language needs to be transcribed. If transcription is involved, the required level of detail and types of feature to be represented in the protocol need to be considered carefully. Since CODA typically addresses semantic or conceptual levels of linguistic structure (see Section 3.4), an orthographically correct transcription of what is meant may be more useful than an exact representation of pronunciation patterns. However, it is typically useful to transcribe markers of hesitation (such as hm, uh, uh-uh) and the like in systematic ways. Such sounds are not produced at random; they usually carry a meaning relevant for communicative and/or current cognitive processes. Analyzing them systematically can be enlightening for research areas involving uncertainty (Lindsey, Greene, Parker, & Sassi, 1995; Tenbrink, Bergmann, & Konieczny, 2011) or confusion, cognitive effort, dialogic negotiation, and other issues (Brennan & Williams, 1995). Regardless of whether or not a systematic analysis of hesitation markers is carried out, including this information in the transcript supports the interpretation of utterance meaning.

To represent discourse functions of intonation in a feasible and well-established (though coarse) manner, punctuation markers can be used in the conventional way, i.e., question marks for (semantically identifiable) questions, exclamation marks for the (unusual) case of an exclamation, commas for a continuing intonation contour even if a sentence is grammatically complete, and a full stop to signal ostensive completion (e.g., as signaled by a falling intonation contour). Non-linguistic events can be noted in brackets, such as (laughter) or (noise). The convenient transcription software f4^{Footnote 3} uses time stamps; this allows for capturing the temporal development as well as extended pauses. Short and mid-utterance pauses can be represented in brackets using numbers for seconds. More intricate non-verbal contributions such as gestures and facial expressions should be considered with respect to their relevance to the research goals; transcribing these can be extremely time-consuming, especially if no established conventions for the particular distinctions needed are known (see Brösamle, 2013, for an extended specialized gesture transcription project). Furthermore, actions may be crucial for the interpretation of language. Generally, all relevant information needs to be included on a suitable level of detail. Here, the focus will remain on language.

Once the language data are available in electronic format, they will typically need to be segmented into smaller units serving as the basis for analysis. The length and definition of a unit depend on the research goals (see Krippendorff, 2004, for elaborate discussion of unitization). Any kind of quantitative analysis must build on carefully defined (operationalized) units. However, this goal may not always be easy to achieve for smaller units of analysis. In some cases it may be sufficient to establish smaller units simply for practical purposes, e.g., as a basis for line-by-line annotation (see Section 3.4). Researchers may decide to relate quantitative analysis results to the overall number of words or other clearly delineated larger units, while retaining smaller units simply for purposes of managing the annotation process.

A notion that may be a useful start for line-by-line analysis is that of a ‘possible sentence’ (Selting, 2000). Selting specifies this notion in terms of turn-constructional units (TCUs) as follows:

While Selting developed her notion of TCUs for spoken language, a similar idea can also be used for written language in order to obtain manageable units of a similar size; these may be shorter than the sentences suggested by the writer’s use of punctuation. Moreover, in dialogue, turn changes between speakers are clear cases of new units, and syntacticallycompletesentences are also units. Semantic/pragmatic completion is reached, for instance, when a speaker turns from a description of a specific item or spatial array to the next. For some purposes the notion of conversational game established in the dialogue modeling literature may be useful (Carletta, Isard, Isard, Kowtko, Doherty-Sneddon, & Anderson, 1997). Further useful ideas about segmentation and the definition of discourse units can be found, for instance, in Allen (Reference Allen2000), Degand and Simon (Reference Degand and Simon2009), Denis (Reference Denis1997), and Krippendorff (Reference Krippendorff2004). However, for many purposes it may not be necessary to identify a specific operationalization of unit definitions, especially if no quantitative analysis directly relies on unit counts. Then, segments can be intuitively defined as convenient for the analyst.

Tenbrink and Seifert (Reference Tenbrink and Seifert2011) first rendered the collected handwritten reports into digital format. Unitization was done on the basis of informational chunks, similar to the notion of TCU as described above (Selting, 2000, i.e., “the smallest interactionally relevant complete linguistic unit” in the given context), as exemplified here:

1. 1. Zentralen bzw. großen Ort mit Flughafen gesucht (Sitia)

   Searched for a central or large town with an airport (Sitia)

2. 2. Dann habe ich überlegt wie weit man in 2 Wochen so kommen könnte.

   Then I considered how far it is possible to travel in two weeks

3. 3. Ziel war es möglichst viel zu sehen

   The aim was to see as much as possible

4. 4. und weit in den Westen zu kommen.

   and to get far into the west.

5. 5. Große Städte als Übernachtungsplätze:

   Large cities for overnight stays:

6. 6. viel Unternehmungensmöglichkeiten [sic!] für Touristen

   many activity options for tourists

This unitization made it possible to analyze chunks of text in a straightforward line-by-line analysis.

3.4. content analysis

Following data preparation, the first step of any analysis of natural language data is to gain a clear grasp of the content of the data, i.e., the speakers’ meaning in producing the verbalizations. An intuitive understanding of the range of content produced by the speakers should precede any closer analysis as described next. Based on the content and guided by the research question that motivates the study at hand, the analyst will need to make decisions about which aspects to pursue further and capture systematically.

Content analysis, as described by Krippendorff (Reference Krippendorff2004) and Ericsson and Simon (Reference Ericsson and Simon1993), can involve extremely complex analysis procedures (see Crampton, 1992, for an insightful example). In particular, Krippendorff (Reference Krippendorff2004) describes content analysis as a research technique suitable for making reliable inferences from texts; this represents a notion that considerably exceeds simple, superficial, intuitive text comprehension. In problem-solving studies, it is often possible to identify a range of conceptual strategies, representations, and processes that are directly described by participants (cf. Ericsson & Simon, 1993; for CODA-related studies see, e.g., Gralla et al., 2012; Hölscher et al., 2011; Tenbrink & Wiener, 2009). Moreover, content analysis provides a first basis for categorizing the data with respect to each segment’s relation to the discourse task (related to the research question at hand). For instance, in Tenbrink, Coventry, and Andonova (2011), utterances were categorized as to whether they described an object’s location or orientation, both, or neither. These categories were identified because they emerged as prominent types of content produced by the speakers, with clear effects on the targets of the research design.

In Tenbrink and Seifert (Reference Tenbrink and Seifert2011), content analysis served to inspire the more fine-grained coding procedures (as described in Section 3.5). While examining how participants described the procedures of planning a holiday tour and how they shifted between the conceptual domains of planning and traveling, it became clear that the participants sometimes explicitly commented on the former issue, while the latter remained implicit in language. To capture explicit comments, the authors extracted any mention of conscious strategies describing the holiday tour design procedure. Examples are: “After that, I also looked for a ‘peaceful’ place with a range of attractions in the vicinity”, which exemplifies the importance of spatial vicinity for travel planning; and “It was also important to avoid traveling the same route twice, but rather, traveling some sort of circle (ellipse)”, which exemplifies the concept of a suitable overall trajectory as well as the avoidance of repetition. As a next step, linguistic indicators for each of the identified strategies (e.g., region, area, side, and mainland indicate a conceptual segmentation of the environment into regions) were determined in order to operationalize the strategy allocation process (see Tenbrink & Seifert, 2011, for details).

Concerning domain shifts, the general impression emerged that participants were mentally at two places at once rather than showing awareness of a conceptual shifting process. A closer inspection of the language the participants used led to the identification of linguistic markers indicating each of the domains involved (the current conceptual domain of planning alongside that of traveling on a remote island) as well as the (implicit) shifts between them. The details of this part of the analysis will be described in Section 3.5.

3.5. analysis of linguistic features

Building on the identified content categories, the next step in the analysis is to identify features of the linguistic representations that can be interpreted as reflecting characteristic conceptual phenomena. As illustrated above, some aspects of language reflect cognitive aspects that go beyond conscious reflection by individual speakers, and that are not necessarily directly observable in linguistic content. Speakers are typically unaware of the cognitive structures that are reflected in particular ways of framing a representation linguistically. Furthermore, they are not consciously aware of the network of options (or ‘social semiotic system’; Halliday & Matthiessen, 1999) that allows for a range of linguistic choices beside their own. For instance, a sentence like The car is next to the tree will be intuitively produced without considering alternative options like The tree is next to the car, The oak partially covers the Bentley, or other linguistic representations of the same scene. Nevertheless these choices are meaningful. From a cognitive point of view (Talmy, 2000), The car is next to the tree is more standard than The tree is next to the car, since the movable object (car) is referred to as a locatum in relation to the fixed object (tree) as relatum. Besides cognitive principles, discourse-related factors (such as the current topic of the conversation) may lead speakers to intuitively choose other options. Depending on the context of its production, a choice like The oak partially covers the Bentley may reflect the speaker’s way of perceiving a pictorial configuration as well as their attention to details (oak rather than tree, Bentley rather than car). Along these lines, different ways of referring to the same situation reveal the speaker’s conceptual perspective (Schober, 1998), without the speaker necessarily being aware of this effect.

To capture these issues, it is useful to first examine the data qualitatively so as to identify linguistic features that are relevant to the research issues at hand. This means a close look at the ways in which central aspects relating to cognitive processes are expressed in language. For instance, it may be interesting to examine whether decision points in route descriptions are linguistically represented as given, and backgrounded, or rather highlighted as new elements. An examination of the language used to refer to these locations will then lead to the identification of the relevant linguistic repertory, such as definite and indefinite articles, modifiers, syntactic position, and the like.

The next step is then to annotate the data on a line-by-line basis so as to capture crucial qualitative insights systematically. Apart from gaining quantitative insights by counting numbers of occurrences of particular phenomena, systematic patterns can then be identified based on the features’ distribution throughout the data (see Section 3.6).

This approach to linguistic data analysis is fundamentally discourse-analytic. Discourse analysis generally means analyzing texts with respect to their linguistic (and contextual) features, adopting a specific analysis perspective that is relevant with respect to a particular motivation – bearing in mind that linguistic analysis can almost never be regarded as exhaustive. Crucially, in contrast to psycholinguistic experimentation, which typically relies on precise predictions and controlled settings, the identification and detailed (qualitative) description of relevant linguistic phenomena and their interpretation relative to the research question is seen as primary. Quantitative data then serve to highlight the relative role of the detected phenomena within the text, and further support can be gained through inferential statistics (e.g., comparing results for different conditions; see Section 3.6).

In other areas of discourse analysis, research motivations include identifying distinctive features of text types (de Beaugrande, 1980), e.g., for purposes of data mining (Leidner & Schilder, 2010), specifying dialogue structure (e.g., to inform automated dialogue systems, as in Shi, Jian, & Rachuy, 2011), and uncovering gender bias or a hidden political agenda (as in Critical Discourse Analysis, e.g., van Dijk, 1993). In CODA, the perspective adopted is to identify linguistic features that are potentially indicative of cognitive processes and representations.

It is crucial at this central procedural step to provide clear and crisp definitions and operationalizations for annotation criteria, typically aiming for mutually exclusive variables within a category (see Carletta et al., 1997, for a particularly useful discussion of these issues for the area of dialogue annotation). In the following, I outline some examples for selective discourse analysis perspectives relevant for research in cognitive science, motivating systematic annotations of specific linguistic features in the collected data.

First, the way in which texts (of any type) are linearly structured, and the way in which the information is presented, can be expected to relate systematically to the way the underlying cognitive processes are structured. This concerns both the text as a whole, revealing, for instance, temporal and causal relationships developing gradually, and smaller portions of the text, for example information packaging within single clauses (Halliday, 1994). For instance, with respect to whole texts, Tenbrink and Wiener (Reference Tenbrink and Wiener2009), as well as Gralla et al. (Reference Gralla, Tenbrink, Siebers, Schmid, Miyake, Peebles and Cooper2012), exploited the temporal structure of participants’ problem-solving reports for the purpose of proposing a generalized procedure for the type of problem at hand. These accounts served to illustrate how different cognitive strategies, which in earlier literature were treated as contrasting problem-solving approaches, were integrated over time by the problem-solvers in the studies. With respect to smaller units of text, Tenbrink and Ragni (Reference Tenbrink, Ragni, Stachniss, Schill and Uttal2012) highlighted regular principles within participants’ ways of describing abstract configurations, both on the level of whole configuration descriptions and with respect to individual analysis units that described the relation of two objects to each other. This analysis revealed systematic visual attention patterns with respect to small-scale space. Rather than conceptualizing a configuration as a whole and describing its details in random order, people mostly followed conventional patterns of reading (left to right), and furthermore tended to base each individual object location description on the previous one (e.g., A is at the top left, B is next to A, and C below B).

Second, for many purposes, a close look at the speakers’ lexical choices in references to objects and notions can be revealing (see also Krippendorff, 2004). Since there may be a range of different options, the particular reference type chosen by a speaker highlights the role of a particular semantic or conceptual field within the current verbalization task. In many cases, specific lexical items that have been identified by cognitive linguists (e.g., Talmy, 2000) as reflecting cognitive structure may be indicative of underlying conceptual patterns. For example, explorations of the chosen level of detail (Daniel & Denis, 2004; Vorwerg & Tenbrink, 2007), hierarchical description levels (Plumert, Carswell, de Vet, & Ihrig, 1995), and the underlying conceptual perspective (Tenbrink, Coventry, & Andonova, 2011; Tenbrink, Ross, Thomas, Dethlefs, & Andonova, 2010; Tversky, 1999) reveal the flexibility of these concepts relative to changes in the task scenario. For example, Plumert et al. (Reference Plumert, Carswell, de Vet and Ihrig1995) showed how the discourse task as well as the spatial configuration of landmarks affected the order and hierarchical structuring of spatial descriptions. Tenbrink et al. (Reference Tenbrink, Bergmann, Konieczny, Carlson, Hölscher and Shipley2010) found that speakers interacting with other humans frequently shift between perspectives in route dialogue, whereas speakers communicating with an automatic dialogue system refrain from doing so and mostly stick to the (arguably) simpler perspective choice.

Moreover, it can be useful to trace the development of reference types over time, particularly for complex cognitive processes. If reference types change during a problem-solving task, this “can be interpreted as the trace of changes in the functional organization of the subject’s representation” (Caron, 1996, pp. 24f.). A detailed analysis of the semantics of the lexical choices in the relevant action context may highlight the significance of this kind of conceptual change. For example, in a comparison of think-aloud protocols and instructional discourse in a dollhouse assembly setting, Gralla (Reference Gralla2014) shows that reference choice is influenced by prior knowledge about the function of referent objects. Speakers who had been shown a picture of the fully assembled dollhouse often used pronouns in initial reference to a part in focus, reflecting its integration into an existing mental representation. In contrast, participants without prior knowledge initially tended to use definite noun phrases containing domain-unspecific nouns. Moreover, reference choice was also influenced by gradual change in the comprehension of the situation. Subsequent references reflected mental re-conceptualizations via the assignment of specific functions to objects. These principles, which systematically affected the distribution of pronoun use as well as lexical choices, were mediated by the communicative purposes in an instruction context.

Third, it is often worthwhile to examine the use of explicit discourse markers, which may serve multiple purposes, and have been intensely researched both for English and for German (Fischer, 2006; Grosz & Sidner, 1986; Schiffrin, 1987). According to Caron (Reference Caron, Caron-Pargue and Gillis1996), the use of discourse markers in think-aloud protocols reveals how the participant construes the concepts and relations involved, without serving any specific communicative purpose that might influence this construal. For example, connectives (such as before, because, while) explicitly structure the represented contents. Furthermore, certain markers that are particularly prominent in spoken language may reflect hierarchical thought processes (see also Bégoin-Augereau & Caron-Pargue, 2003); for instance, occurrences of Okay, now … may signal the completion of a subprocess together with the start of a new one. Caron (Reference Caron, Caron-Pargue and Gillis1996) specifically proposes that, if modal expressions such as can, must, have to occur together with interjections such as oh, well, this

Other verbal cues, such as pauses, lapses, and self-repairs, may be indicative of other types of cognitive process, depending on the task situation (Lindsey et al., 1995; Tenbrink, Bergmann, & Konieczny, 2011).

Generally speaking, a detailed analysis of linguistic features is the most central aspect of CODA. There is no theoretical limit to the types of linguistic feature that can be or should be attended to in this part of the analysis. After all, language relates to cognition in many different ways, as shown by linguistic theory and previous research. Crucially, linguistic features need to be identified that are relevant for the research question at hand. The analyst will need to take account of practical limitations; while many aspects may be interesting to examine and discuss in depth, it will typically only be feasible to pick a few of them and focus on these. Identifying the most relevant features in a linguistic dataset may take time and effort, but will prove worthwhile if the analysis is then carried out systematically.

In terms of our running example, Tenbrink and Seifert (Reference Tenbrink and Seifert2011) identified switches between different conceptual domains by classifying lexical choices in the collected retrospective reports. In particular, they examined the linguistic choices in each analysis unit with respect to the following features:

• • identification of an underlying (explicit or implicit) agent of a described action that could be identified unambiguously as a planner or traveler (e.g., due to the nature of the instruction in this task, I was typically the planner, whereas they referred to traveling);

• • nouns, verbs, and adjectives/adverbs that could be identified as indicators of the planning or the traveling activity (where map, decide, and important were typical indicators of planning, while relaxation, experience, and beautiful referred to traveling);

• • temporal markers that clearly belonged to the planning or the traveling domain (identifiable by the temporal scope involved, as illustrated by references to days and hours).

In terms of a line-by-line analysis of the units cited above, this looks as shown in Figure 1. Importantly, each annotation category was carefully defined in an annotation documentation file so as to avoid confusions and inconsistencies in unclear cases. For instance, places and activities that were directly represented in the map (such as the ‘large town’ Sitia) had a corresponding element both in the planning and in the traveling domain, and therefore could not be regarded as indicating either of the domains, explaining the zeros in line 1 (Figure 1). Similarly, activities (Unternehmungsmöglichkeiten) were indicated as symbols in the map. Consequently, the only nouns that were counted as indicators of a specific conceptual domain were Übernachtungsplätze ‘overnight places’ and Touristen ‘tourists’.

Fig. 1.

Snapshot of the line-by-line annotation of conceptual domains in Tenbrink and Seifert (Reference Tenbrink and Seifert2011).

As in this example, the linguistic analysis in Tenbrink and Seifert (Reference Tenbrink and Seifert2011) started from a qualitative examination of indicators of a particular phenomenon (a cognitive domain or a conceptual strategy), followed by the identification (annotation) of their occurrences in the dataset using number counts in the data table. While this is a frequent procedure in CODA, data tables can also represent and support more complex analysis procedures. Consider the following example, taken from Vorwerg and Tenbrink (Reference Vorwerg, Tenbrink, Barkowsky, Knauff, Ligozat and Montello2007), where level of detail in the description of an element in a picture was addressed by a close examination of linguistic features, as represented in Figure 2. Here, the first two columns provide the number code for the condition and the picture that is described by the utterance represented in the third column. The annotation starts in the fourth column, where mention of direction terms is categorized as follows: ‘1’ for one direction term (e.g., top for picture 1), ‘2’ for more than one direction term (e.g., northwest and lower for picture 2), and ‘0’ for descriptions containing no direction term at all (as for picture 7). The fifth column asks whether the direction term (if any) is modified by a precisifier such as most (for pictures 8 and 10); other examples would be slightly (left) and directly (above). If the direction term remains unmodified, the code is ‘1’; if it is modified, the code is ‘2’; and if there is no direction term the code is ‘0’. Note that this annotation creates nominal rather than ordinal or cardinal categories. Alternatively, the number of direction terms or precisifiers could have been counted. The last column, however, differs in this respect since it asks for a type of relatum (rather than existence or frequency). Here no relatum is coded as ‘0’ and the box as relatum as ‘1’; alternatives found in the data were the speaker (e.g., in front of me) or the picture itself (e.g., leftmost in the picture) as relata, and these were assigned different number codes. Instead of numbers, it is equally possible to use lexical categories (e.g., box, speaker, etc.) in the annotation columns.

Fig. 2.

Snapshot of the line-by-line annotation of level of detail in Vorwerg and Tenbrink (Reference Vorwerg, Tenbrink, Barkowsky, Knauff, Ligozat and Montello2007).

3.6. reliability

Following line-by-line annotation, the annotations need to be checked for reliability, as outlined, for example, by Krippendorff (Reference Krippendorff2004). This is an established way of assessing to what extent the definitions used for analysis were operationalized sufficiently for use by different annotators. The aim is to rely not only on experts to use these definitions, who typically draw on additional background knowledge that is hard to determine, but also other coders who should not need extensive additional training. Where applicable and feasible, inter-coder reliability should be tested statistically, for example using Krippendorff’s alpha (using the tool provided by Hayes & Krippendorff, 2007, for the SPSS/PASW statistics software package). As shown by Krippendorff (Reference Krippendorff2004), the widely used measure Cohen’s kappa (Cohen, 1960) has the flaw of allowing for systematic disagreement, which may then go unnoticed.

Tenbrink and Seifert (Reference Tenbrink and Seifert2011) let independent coders annotate a representative subset of the data in parallel, and obtained favorable Krippendorff’s alpha values for each annotation category.

3.7. identification of patterns

Having accomplished a detailed and reliable annotation of the linguistic features of individual participant data, the results need to be systematically related to the features of the setting in which the language was produced. This is most conveniently accomplished by transferring counts of annotation results per participant into a spreadsheet. Patterns to be identified may concern the features of a verbally represented scene, or the problem-solving process, different conditions, scenes, or situations, subgroups of participants or individuals, or different text types. For instance, Daniel and Denis (Reference Daniel and Denis2004) identified the features of route descriptions that were relevant for conciseness in all collected data, and then compared the results between conditions (which differed in the extent to which conciseness was explicitly asked for). Where other kinds of data, such as behavioral results, are available, the analysis results furthermore need to be related to these (see Section 3.8). Establishing such inter-relationships between types of evidence is not only useful in terms of validation (Krippendorff, 2004) but also in terms of accounting for the significance and impact of any kind of change in the situation, its conceptualization, and its representation in language.

In Tenbrink and Seifert (Reference Tenbrink and Seifert2011), the line-by-line annotation of conceptual domains based on linguistic indicators was used to identify units that referred only to the traveling or planning domain, both, or neither. A close examination of these led to the identification of linguistic markers of conceptual shifts (such as let in I wanted to let the couple travel once round the western part of the island). Number counts for each category per participant were transferred to the spreadsheet, and then related to various categories relevant for the study, namely different modes of travel for different participants as well as different conceptual planning strategies. Together, these analyses revealed how participants used spatial strategies to design a travel plan relevant for a conceptual domain that is distinct from the currently perceived scene, taking into account scenario features such as the mode of travel involved. There were striking parallels between this naturalistic and conceptually complex holiday planning task and previous results for simpler and more abstract spatial planning problems. Furthermore, the conceptual domain shifts highlighted by the linguistic analysis resonates with theories on conceptual flexibility, such as that reflected by the adoption of various perspectives.

Depending on the sample size, existence of different conditions, distribution of results, and the like, the analysis may now be supported by statistical procedures. Since the type of data and feature distribution will vary from case to case, no recommendations for specific statistical tests can be made here. Previous statistical analyses related to CODA range from simple t-tests, chi-squares, and ANOVAs to more complex mixed-effects logistic regression models.

Obviously, statistical validation of observed patterns is highly desirable and provides substantial support for the linguistic analysis results whenever it can be achieved. It should be recognized, however, that this is notoriously hard to obtain based on unconstrained language data, and statistical significance is not the only valid evidence of cognitive phenomena. In various areas of cognitive science (e.g., artificial intelligence, human−computer interaction, cognitive modeling, and others), it is actually common to examine individual case studies, to obtain proof of concept, and the like. Showing that phenomena exist (maybe systematically under distinct circumstances) can be a decisive step forward in the understanding of the human mind. Case studies and the identification of qualitative patterns can therefore be regarded as inspiring explorative insights, leading towards more controlled study designs that can shed further light on the observed phenomena.

In terms of our running example, Tenbrink and Seifert (Reference Tenbrink and Seifert2011) focused on descriptive statistics (relative frequencies). Much of the analysis was qualitative, showing the repertory of conceptual strategies as well as domain shifts as represented in language across various scenario types. The quantification provided an assessment of the relative role of these processes within the overall procedures and verbalizations.

3.8. triangulation and extensions

As observed by many researchers following the tradition of verbal protocol analysis (based on Ericsson & Simon, 1993), language may in many ways be insufficient for gaining access to cognitive processes and representations to the extent desirable for a research purpose. A systematic linguistic analysis (as just described) partially remedies this by a closer examination of linguistic choices than available through content analysis, building on established insights concerning their significance. Nevertheless, it is highly beneficial to collect other types of evidence that can complement the insights gained from language, as well as to relate insights gained from verbal protocol analysis to established (or newly developed) theories and models.

To link the results of linguistic analysis with other findings, cognitive science provides a wide range of methodologies and opportunities for triangulation. For example, measures such as memory or behavioral performance data, decision outcomes, reaction times, and eye-tracking data can provide (further) insights on cognitive activities, some of which remain below the threshold of participants’ awareness. Eye-tracking behavior, for instance, primarily reveals implicit patterns of attention of which participants are rarely aware (Findlay, 2004; Thomas & Lleras, 2007). Neuroimaging (e.g., fMRI; see Cabeza & Klingstone, 2001, for an overview) provides insights on the activation of particular parts of the brain, for example during the conceptualization of objects, which triggers activation of certain lexemes (Boutonnet, Athanasopoulos, & Thierry, 2012).

Synthetic approaches such as cognitive modeling (e.g., ACT-R; Anderson, 2007; Anderson, Bothell, Byrne, Douglass, Lebiere, & Qin, 2004; Anderson & Lebiere, 1998) suggest mechanisms and procedures across all cognitive levels and modules, including those that can only be hypothesized at the current state of research. In particular, cognitive models represent theories to explain the observable data consistently by concrete assumptions about the underlying non-observable processes, covering the full range of cognitive processes from conscious human decisions and strategies through to neural activities. The result of a computationally implemented cognitive model can then be compared with observable insights from various sources, including fMRI (Fincham, Carter, van Veen, Stenger, & Anderson, 2002; Ragni, Fangmeier, & Brüssow, 2010) as well as language data, feeding back into improved models. For example, Gugerty and Rodes (Reference Gugerty and Rodes2007) provide an ACT-R model of the strategies and cognitive processes involved in cardinal direction judgments, based on participants’ verbalized strategies and enhanced by further findings about human direction conceptualizations and other processes that were not directly reported by participants.

Another extension is to feed the results into practical applications. The overarching goal motivating the study reported in Tenbrink and Seifert (Reference Tenbrink and Seifert2011), for instance, was the development of spatial assistance software for planning holiday routes (Seifert, 2008). Other studies have focused on the development of intuitive human−robot or human−computer interaction (e.g., Moratz & Tenbrink, 2006; Tenbrink et al., 2010).

3.9. summary

Table 2 provides an overview of the steps involved in CODA along with some key questions to be considered by the researcher, and key aspects involved in a step. Each step can be treated more or less elaborately in a specific research process.

table 2.

Overview of steps in the CODA procedure