2.1 Audio view tools

The first category of tools displays the waveform of the audio file and allows users to make annotations time-aligned to the audio signal. This category includes Emu (Cassidy and Harrington Reference Cassidy and Harrington2001), Speech Viewer (Sutton, Cole, deVilliers et al. Reference Sutton, Cole, de Villiers, Schalkwyk, Vermeulen, Macon, Yan, Kaiser, Rundle, Shobaki, Hosom, Kain, Wouters, Massaro and Cohen1998), Praat (Boersma and Weenik Reference Boersma and Weenink2005), and WaveSurfer (Sjölander and Beskow Reference Sjölander and Beskow2000). Figure 1 shows the interface for WaveSurfer. These tools often allow multiple annotation tiers, which can be used for different annotation tasks. For instance, the ToBI annotation scheme (Pitrelli, Beckman and Hirschberg Reference Pitrelli, Beckman and Hirschberg1994) uses one tier for the word transcription, a second for intonation tags, a third for break indices, and a fourth for miscellaneous information, including disfluencies. These tools often have powerful signal analysis packages for displaying such phenomena as the spectrogram, F0 contour, and voicing. Some tools display a video as well, such as Anvil (Kipp Reference Kipp2001).

Fig. 1.

The interface of WaveSurfer.

These tools are intended for tasks like word and phoneme transcription, acoustic event annotation (such as intonational phrasing and word accents), and utterance segmentation. However, these tools are not good for coding higher level information, such as speech acts or hierarchical discourse structure, because these tasks require that users understand a much larger context. For these tasks, often there is no need to see the exact timing of each word or to see the abandoned speech (which does not contribute to the dialogue); instead, users want to see how utterances are segmented, what they are intended to convey, and how they are related to each other. Thus users need a view that suppresses information that is not directly relevant, such as the exact timing of words, and emphasizes what is important, such as utterance segmentation.

2.2 Utterance-by-utterance view tools

The second category of tools displays a dialogue as a sequence of utterances and formats the utterances line-by-line, like a movie script. These tools are used for annotating the higher-level structure of dialogue. DAT allows users to annotate utterances or groups of utterances with the DAMSL scheme (Allen and Core Reference Allen and Core1997; Core and Allen Reference Core and Allen1997). MMAX (Müller and Strube Reference Müller and Strube2003) allows for multiple annotation tasks, such as speech acts and co-reference. N.b. (Flammia Reference Flammia1998) allows users to annotate discourse structure as a hierarchical grouping of utterances. All three tools take as input a dialogue as a sequence of speakers' utterances. This simplification can make it difficult for users to truly understand what is happening in the dialogue, especially when there are disfluencies, abandoned utterances, and overlapping speech.

2.3 Tools with multiple views

Transcriber (Barras, Geoffrois, Wu and Liberman 2000) gives users two views of a dialogue. The interface of Transcriber is shown in Figure 2. It takes as input an audio file of a dialogue. The user can mark utterance boundaries and transcribe what was said in each utterance. A second view, which organizes the dialogue as a sequence of speakers' utterances, allows users to annotate them with speech acts and to group them into topics.

Fig. 2.

The interface of Transcriber.

Transcriber lacks support for annotating some common dialogue phenomena. First, Transcriber is weak in handling overlapping speech. If there is overlapping speech, users must treat the overlap as a special utterance, spoken by both speakers. For some typical cases of overlap (see Section 4.1), this creates a convoluted interpretation of the dialogue. Second, Transcriber does not provide direct facilities for annotating disfluencies. Third, although Transcriber offers flat (topic) segmentation, hierarchical segmentation is not supported.

Transcriber has some basic abstraction mechanisms. The utterance-by-utterance view abstracts away the detailed timing of words for coding speech acts and discourse blocks. However, more powerful abstractions, such as abstracting away disfluencies, abandoned speech, and overlapping utterances, are not provided.

2.4 Toolkits

In addition to existing tools, there are several toolkits that allow researchers to build tools for a wide range of annotation tasks. AGTK (Bird, Maeda, Ma and Lee Reference Bird, Maeda, Ma and Lee2001) is a toolkit based on a knowledge representation scheme called Annotation Graphs (Bird, Day, Carofolo et al. Reference Bird, Day, Carofolo, Henderson, Laprun and Liberman2000). The toolkit provides functions for reading and writing annotations, displaying a waveform, transcribing a segment of speech, and annotating segments with multiple sets of labels. These modules can be used with a programming language (C++, Tcl/Tk, or Python) to build an annotation tool.

A second toolkit is the MATE Workbench (McKelvie, Isard, Mengel et al. Reference McKelvie, Isard, Mengel, Moeller, Grosse and Klein2001). Users specify their annotation task using the MATE style sheet language, which is an XML specification of (i) the data that users will annotate, (ii) how to annotate the data, and (iii) how to display the annotated data. This toolkit can be used for creating tools that let users perform multiple annotation tasks, including part-of-speech tags, co-reference, and utterance tags.

A third toolkit is the NITE XML Toolkit, which is intended for annotating multi-modal data (Carletta, Isard, Isard et al. Reference Carletta, Evert, Heid, Kilgour, Robertson and Voormann2003). This toolkit provides two ways of building an annotation tool. First, similar to AGTK, the toolkit provides modules that can be used with the Java programming language to build an annotation tool. Second, similar to MATE, the toolkit can take as input a style sheet to create an annotation tool.

These toolkits suffer the following problems. First, they do not provide built-in support for a number of annotation tasks, such as annotating disfluencies, overlapping speech, and hierarchical discourse structure. Second, the toolkits do not advocate nor provide built-in abstraction mechanisms useful for understanding and annotating dialogue. No support is provided for abstracting out disfluencies, abandoned speech, and overlapping utterances in an utterance-by-utterance view; nor is support provided for displaying annotations in a time-aligned audio view. Although the toolkits are flexible enough that users can program almost any functionality that is needed, it is unclear how much effort it would take to implement the functionalities listed above to build a tool as powerful as DialogueView.

3.1 WordView

The first view in DialogueView is WordView, which takes as input two audio filesFootnote ¹ (one for each speaker), the words said by each speaker together with their start and stop times (in XML format), and shows the words time-aligned with the audio signal.Footnote ²Figure 3 shows WordView with an excerpt of a dialogue from the Trains corpus (Heeman and Allen Reference Heeman and Allen1995b). This view is ideal for seeing the exact timing of speech, especially overlapping speech.

Fig. 3.

Utterance segmentation in WordView.

WordView gives users the ability to select a region of the dialogue and to play it. Users can play each speaker channel individually (Play System and Play User) or both combined (Play Both). This capability is especially useful for overlapping speech, where users might want to listen to what each speaker said individually, as well as to hear the overlap between the speakers. The Play System Cleanly and Play User Cleanly buttons provide functionality to play the speech with disfluencies removed, which will be addressed in Section 4.3.

3.2 Segmenting word sequences into utterances

One function of WordView is to allow users to break up the speech of each speaker into segments, which we refer to as utterances. Figure 3 shows a dialogue excerpt segmented into utterances. A vertical bar indicates the start time of the utterance. The end time is not explicitly shown; it is simply the end time of the last word in the utterance. In the example, the top speaker's utterances are “and then take the remaining boxcar down to”, “Corning”, and “yeah”, and the bottom speaker's utterances are “okay”, “right”, “to du-”, and “Corning”. Users can easily move, insert or delete utterance boundaries. WordView ensures that boundaries never fall in the middle of a word.

As can be seen in the example, utterances of one speaker are allowed to overlap with the utterances of another speaker. This allows us to easily annotate backchannels, which often overlap with the other speaker's utterances, as illustrated by the bottom speaker's “okay” at 76.3 seconds. This example would be cumbersome for the tool Transcriber (Barras et al. Reference Barras, Geoffrois, Wu and Liberman2001) to handle.

In our own annotation scheme, we have defined an utterance as a semantically meaningful contribution that is not affected by subsequent speech from the other speaker (Heeman and Allen Reference Heeman and Allen1995a). We felt that the backchannel of the bottom speaker during the top speaker's utterance of “and then take the remaining boxcar down to” did not change the top speaker's actions. However, we felt that the bottom speaker's attempted completion of the top speaker's utterance did have an effect on the top speaker, which is why the top speaker's speech was broken into two utterances.

3.3 UtteranceView

The annotations in WordView are utilized in building the next view, UtteranceView. This view shows the utterances of two speakers as if it were a script for a movie. To derive a single ordering of the utterances of the two speakers, we use the start time of each utterance as annotated in WordView. We refer to this process as linearizing the dialogue (Heeman and Allen Reference Heeman and Allen1995a). The order of the utterances should show how the speakers are sequentially adding to the dialogue, and is our motivation for defining utterances as being small enough so that they are not affected by subsequent speech of the other speaker. Figure 4 shows UtteranceView with the dialogue excerpt from Figure 3.

Fig. 4.

Utterance segmentation in UtteranceView.

UtteranceView abstracts away from the detailed timing information of the words that were said. Instead, it focuses on the sequence of utterances. This view is ideal for annotations that require a larger context of what is happening in the dialogue, such as speech act annotation. Of course, if users want to see the exact timing of the words in the utterances, they can examine WordView, as it is displayed alongside UtteranceView. Although both views have scroll bars that allow the user to independently position them anywhere in the dialogue, both have a navigation button (Sync) that repositions the other view to the same place. Furthermore, changes made in WordView are immediately propagated into UtteranceView, and hence users will immediately see the impact of their annotations. For example, if a user were to delete the utterance boundary between “to du-” and “Corning” in WordView in Figure 3, then “to du- Corning” would be shown as a single utterance in UtteranceView.

UtteranceView also includes playback buttons. To playback, the user first highlights a sequence of utterances with the mouse. The user can then press Play Both to hear the audio starting from the start time of the first highlighted utterance to the end time of the last highlighted utterance. This is similar to Play Both in WordView. The second playback, Play Order, is slightly different. It takes the linearization into account and dynamically builds an audio file in which each utterance in turn is concatenated together, and a 0.5 second pause is inserted between each utterance. This gives the user an idealized rendition of the utterances, with overlapping speech separated. In the example from Figure 4, utterances u38 through to u40 would be each played sequentially, rather than with overlapping between u38 and u39, and u39 and u40. By comparing the Play Both and Play Order playbacks, users can aurally check if their linearization of the dialogue is correct.

UtteranceView also has a print facility. This produces a postscript file with the script-like rendition of the dialogue as it is displayed in UtteranceView.

3.4 Annotating utterances

In either UtteranceView or WordView, users can annotate utterances with various tags. Although the notion of segmenting word sequences into utterances is built into the tool, what the utterances can be annotated with is not. The annotation scheme is specified in a configuration file. This makes the tool useful for users who want to apply different annotation schemes, and makes it easy to experiment while developing a scheme. Utterances can be annotated in either WordView or UtteranceView. WordView is more suitable for tags that depend on the exact timing of the words, or a very local context, such as whether an utterance is abandoned or incomplete (see Section 4.2). UtteranceView is more suitable for tags that relate the utterance to other utterances in the dialogue, such as whether an utterance is an answer, a statement, a question, or an acknowledgment. Here, the exact timing of the words is not important, but the larger context is, which UtteranceView shows. Figure 5 shows UtteranceView with the utterance tags displayed underneath each utterance. Whether an annotation tag can be used in WordView or UtteranceView (or both) is specified in the configuration file. Which view a tag is used in does not affect how it is stored in the annotation files.

Fig. 5.

UtteranceView with utterance tags displayed.

To illustrate how the configuration file works, we present a simplified version of the DAMSL annotation scheme (Allen and Core Reference Allen and Core1997; Core and Allen Reference Core and Allen1997). Utterances can be annotated in UtteranceView with tags of forward function, backward function, and a comment. Forward functions include statement, information request, and suggestion. Backward functions include agreement, answer, and understanding, which can be either acknowledgement, repetition, or completion. The following shows the specification of these tags in the configuration file, which is simply a text file that can be edited in any text editor.

The first line specifies what utterance tags should be selected in UtteranceView, namely Forward, Backward and Comment. The keyword anyof specifies that any number of the items on the righthand side can be annotated, while the keyword oneof (as in line 6) specifies that only one member of the righthand side can be selected. Tokens on the righthand side can be further decomposed, as we see for Forward in line 2. Tokens are assumed to be of type boolean unless otherwise specified; see, for example, UttrTags.Forward.Other in line 3. DialogueView will use the configuration file to create the annotation panel shown in Figure 6, which will pop up if the user double-clicks on an utterance in UtteranceView.

Fig. 6.

Sample utterance annotation panel.

An added feature of DialogueView is that it allows users to quickly locate particular words or utterance tags in a dialogue, just as one can search for words in a text file. Both WordView and UtteranceView support this search feature. This feature can be used to find interesting utterances or patterns. For example, one might be interested in analyzing discourse markers, such as “like” or “well”; users could search for the word they are interested in and quickly view every instance of it in the dialogue, along with the context surrounding the word. Likewise, if they are interested in all utterances tagged as completions, they can easily find these as well.

3.5 Grouping utterances into blocks

In UtteranceView, users can annotate hierarchical groupings of utterances. We call each grouping a block, and blocks can have other blocks embedded inside of them. This blocking function is similar to what N.b. provides (Flammia Reference Flammia1998), but rather than showing the structure with indentation and color, we use boxes to show the extent of each block. Figure 7 shows a dialogue excerpt with discourse blocks annotated. To create a segment, users highlight a sequence of utterances and then press the Make Segment button. Users can change the boundary of a block by simply dragging either the top or bottom edge of the box. The tool ensures that block edges are never moved in such a way as to cause two blocks to be interleaved, where one is not entirely contained in another.

Fig. 7.

Grouping utterances into blocks in UtteranceView.

DialogueView supports hierarchical blocks because many linguistic theories of discourse argue that conversation is organized as a hierarchical tree-like structure (e.g. Riechman-Adar 1984; Grosz and Sidner Reference Grosz and Sidner1986; Polanyi Reference Polanyi1988; Mann and Thompson 1988; Carberry and Lambert Reference Carberry and Lambert1999). These theories propose that speakers can continue talking about a topic, introduce a new subtopic, or finish with the current one. A few researchers, however, have proposed that speakers can interleave topics, as when comparing two different options (e.g. Ramshaw Reference Ramshaw1991; Rose, Di Eugenio, Levin and Van Ess-Dykema Reference Rose, Di, Levin and Van Ess-Dykema1995; and Hagen Reference Hagen1999); however, we currently do not support such structures.

3.6 Annotating blocks

Just as utterances can be tagged, so can discourse blocks. Many researchers of discourse theory have proposed that there are different types of discourse blocks. For instance, Carletta et al. (Reference Carletta, Isard, Isard, Kowtko, Doherty-Sneddon and Anderson1997) distinguish between dialogue games and transaction blocks; Traum and Hinkelman (Reference Traum and Hinkelman1992) classify blocks by their purpose, such as construct plan, summarize, elaborate, and clarify. In Grosz and Sidner's (1986) theory, discourse blocks also have a purpose associated with them. In DialogueView, users can define the block tags as appropriate for their own annotation scheme in the configuration file. In fact, in our annotation scheme, we annotate all of the aspects discussed above. A simplified version of our scheme is shown in the following.

Note that we use color to distinguish different block types; for example, red for transaction blocks and blue for dialogue games.

4.1 Overlapping speech

There are cases where both speakers talk at the same time. Capturing this overlap properly is critical for understanding what is happening in the dialogue. Hence, it is imperative that users can properly annotate overlap and that the tool properly displays the overlap. We classify overlap into three types, and provide a mechanism in DialogueView for effectively dealing with each type.

The first type of overlap is where the beginning of one utterance overlaps with the ending of the previous utterance. In Figure 3, the second instance of “Corning” (by the bottom speaker) overlaps slightly with the first instance of “Corning” (by the top speaker). Such overlaps are very common in dialogue, as speakers are very good at anticipating the end of the other speaker's turn (Beach Reference Beach1991). These overlaps are not problematic, as UtteranceView will display them in the proper order.

The second type of overlap is where the overlapping utterance is responding to what the other speaker is currently saying, but only the first part of it. For instance, in Figure 3, the bottom speaker said “okay” during the middle of an utterance, perhaps to tell the top speaker that everything was understood so far. However, in the UtteranceView of Figure 7, it may appear that the “okay” of u36 is acknowledging the entire utterance, rather than just a part of it. Hence, users could be misled as to what is happening. In order to remove this source of confusion, we have an utterance tag for this type of overlap, which we annotate in WordView. The following shows the specification of the overlap tag in the configuration file.

In order that users can easily see what has been tagged as overlap, we do the following: in WordView, we color the utterance segmentation marker in blue; in UtteranceView, we indent the utterance and put ‘+’ on either side (similar to the Childes scheme, cf. MacWhinney Reference MacWhinney2000).

The third type of overlap occurs where one speaker is responding to something that the other speaker said, but responds after the other has resumed speaking. This is illustrated in Figure 8, where the top speaker finished utterance u43 and then paused for over a second. The bottom speaker then acknowledged the utterance with the “okay” of u45, but this happened a fraction of a second after the top speaker started speaking again with u44. As shown in Figure 7, the linearization of the dialogue has the “okay” of u45 following u44 as if it were acknowledging that utterance. One could tag u45 with the overlap tag. However, the fact that u45 overlaps with u44 is not critical in understanding what is occurring in the dialogue, as long as u45 is linearized before u44. This can be done in WordView by moving the start marker of u45 so that it is before the start of u44, as shown in Figure 9.Footnote ⁴ After the start marker has been moved, UtteranceView is automatically reformatted as shown in Figure 12.

Fig. 8.

Misleading utterance segmentation.

Fig. 9.

Altered start time for “okay” utterance.

In summary, overlapping speech can be handled in three ways. The overlap can be ignored if it is not critical for understanding what is going on in the dialogue; the utterance can be explicitly tagged as overlap; or the start time of the utterance can be changed so that the overlap does not need to be tagged.

For the audio playback of Play Order, if there is overlap that has not been explicitly tagged as such, the utterances are simply concatenated together according to the start time, as explained in Section 3.3. If the overlap is explicitly marked, we keep the overlap from the audio file. Users can thus aurally check if their treatment of the overlap is correct.

4.2 Incomplete utterances, abandoned utterances, and completions

In the previous section, we discussed how overlapping utterances can be annotated so that UtteranceView succinctly captures what is happening. In this section, we discuss three more utterance tags that can be used to better format UtteranceView, and thus help abstract away from the details in WordView. These tags are for abandoned utterances, incomplete utterances, and completions.

Abandoned: The speaker abandoned what they were saying and the speech had no impact on the rest of the dialogue. This mainly happens when one speaker loses the bid to speak next. Figure 7 shows a number of abandoned utterances, namely u42, u46, u48, u50, u58, and u62. Figure 10 shows two of these abandoned utterances: u46 and u48. In u46, the bottom speaker tried to help the top speaker finish the utterance, but abandoned the attempt, which can be heard by a decreasing volume in the speech. In u48, the bottom speaker started to speak, but then the top speaker cut in, and the bottom speaker gave up the turn.

Fig. 10.

Abandoned utterances.

Incomplete: The speaker did not make a complete utterance, either intonationally, syntactically, or semantically. Unlike the previous category, incomplete utterances do impact the dialogue behavior. Figure 3 has two incomplete utterances: u35 and u38. In both instances, the other speaker completes the utterance, which is a strong indicator that the utterance should not be tagged as abandoned.

Completions: The speaker attempted to complete the utterance of the other speaker. This is similar to the co-operative completions of Linell Linell (Reference Linell1998). Completions are often viewed as evidence that an utterance was understood (Schober and Clark Reference Schober and Clark1989), which is why we included them as a subtype of the Understanding tag (see Section 3.4). Figure 3 gives two examples of completions: u38 and u39.

Abandoned and incomplete utterances are tagged in WordView and completions in UtteranceView. However, regardless of where they are tagged, we want to use these tags to better format UtteranceView. Incomplete utterances are displayed with a trailing ellipsis marker “. . .” and completions with a preceding one. Figure 12 shows UtteranceView with these markings, which makes it much easier to see what is going on in the dialogue. As for abandoned utterances, these are not displayed in UtteranceView, as they do not impact the subsequent development of the dialogue. Hence, users do not need to see them in annotating utterance tags in UtteranceView and segmenting the discourse structure. Since abandoned utterances are not displayed in UtteranceView, they are also skipped in the audio playback of Play Order. The specification for these tags is given in the following, and makes use of the Completion tag as specified in Section 3.4.

4.3 Speech repairs

DialogueView has a special facility for annotating speech repairs. A speech repair is the disfluency where a speaker goes back and repeats or changes something that was just said (Heeman and Allen 1999). The following gives an example of a repair.

The reparandum is the speech that is being replaced, and the interruption point (ip) is the end of the reparandum. This can be followed by an optional editing term (et), such as “um”, “uh”, or “let's see”, which helps mark the repair. The piece of speech that replaces the reparandum is called the alteration.

As the annotation of speech repairs requires careful listening to the speech, and the repairs have strong interactions with utterance boundaries, their annotation is best done in WordView. To annotate a repair, the user highlights a sequence of words and then tags the sequence as a reparandum or an editing term. The user also chooses the type of repair. Figure 11 shows how speech repairs are displayed in WordView. The words in the reparandum are underlined and displayed in a different color. The words in the editing term are displayed in a similar manner.

Fig. 11.

Speech repair example.

The speech repair scheme is specified in the configuration file. The following specifies the scheme used by Heeman and Allen (Reference Heeman and Allen1999).

The first line says that there are three types of repairs: cancel repair (can), modify repair (mod), and abridge repair (abr). The next three lines specify the constituents of each type of repair, in terms of Reparandum and EditingTerm, which are reserved words.

After a speech repair is annotated in WordView, there is little need to show the repaired words in UtteranceView. Hence, we show only the cleaned up speech in which we have removed the reparandum and editing terms of speech repairs. This is what accounts for the difference between Figures 7 and 12 in utterances u33, u43, u44, u52, u54, u63, u65, and u69. As can be seen, the cleaned up utterances are easier to understand than the original ones. Note that after a user annotates a speech repair in WordView, UtteranceView is immediately updated, which lets the user visually verify their annotations.

Fig. 12.

Effect of overlap, abandoned, incomplete, and completion tags.

DialogueView also allows users to aurally verify their speech repair annotations. WordView has two playback buttons, Play System Cleanly and Play User Cleanly, that play the selected region of speech but with the annotated reparanda and editing terms skipped over. We have found this useful in deciding whether a speech repair is correctly annotated. If one has annotated the repair correctly, the edited speech will sound fairly natural. We conducted a human-subject experiment to investigate the usefulness of this playback (Yang, Strayer and Heeman 2003). Subjects were divided into two groups: the experimental group had access to the Play Cleanly buttons and the control group did not. After a short training period, both groups annotated repairs using WordView for six dialogue excerpts. The results showed that the experimental group did better in identifying the scope of repairs, giving a relative improvement of 29% in accuracy, although the performance in detecting repairs was not improved. In addition to aurally separating each utterance, the Play Order button in UtteranceView also skips over reparanda and editing terms.

There are a few cases in which one might want to see the speech repairs. First, speech repairs sometimes indicate lack of certainty on the part of the speaker, and one might want to see this conveyed in UtteranceView. Second, the alteration of a speech repair might have an anaphoric reference to an object introduced in the reparandum. For instance, in utterance u69 in Figure 7, the system said,“so engine E one wasn't maybe it wasn't the best thing.” When this is cleaned up in Figure 12, it becomes “maybe it wasn't the best thing.” The referent of “it” is no longer shown, which could mislead the user. This, however, happens very rarely. In any event, a user can still see exactly what happened by looking at WordView.

4.4 Closing blocks

Even with removing abandoned utterances and cleaning up speech repairs, users can still be overwhelmed by the amount of detail in UtteranceView. This makes it difficult to understand the higher-level structure of the dialogue, which is needed for annotating discourse blocks. Consider utterances u35 to u41 in Figure 12 where the two speakers are collaborating in building an utterance. In our annotation scheme, we call this a grounding block (Clark and Shaeffer 1989; Traum and Nakatani Reference Traum and Nakatani1999). After the user has annotated it as a grounding block, it would be convenient if the block was replaced by a single utterance that captures the essence of what was said in the block: “and then take the remaining boxcar down to Corning,” as if spoken by the system. Utterances u65 to u68 also comprise a grounding block and can be replaced by “so that's one hour more than we have,” as if spoken by the user. Now consider utterances u44 and u47, which form a dialogue game. Here, the system asks, “oh and then we have to unload, right?” and the user responds with, “right.” Here again, it would be convenient to replace this with a single utterance: “oh and then we have to unload,” as if spoken by the system. Figure 13 shows the dialogue excerpt from Figure 12 but with the three example blocks closed (shown as blocks b12, b13, and b17).

Fig. 13.

Dialogue excerpt with three of the blocks closed.

As can be seen, closing the three annotated blocks greatly reduces the amount of detail that users see, which should help them find higher-level discourse blocks. Thus, for doing discourse segmentation, users can start by finding smaller discourse blocks and then work their way up to higher-level ones. Figure 14 shows the dialogue excerpt with blocks b11, b15 and b17 closed, which reduces the excerpt to only 6 lines. Note that for block b11, we took some liberties in composing the summary. Fewer lines and less text makes it easier for the user to keep track of what is going on in the dialogue.

Fig. 14.

Dialogue excerpt with all of the blocks closed.

To support the capability of closing blocks, each block is associated with a summary, which the user needs to fill in. When a block is closed, it is replaced by its summary, which is displayed as if it were said by the speaker who initiated the block. This is consistent with our previous work on initiative and control in dialogue (Strayer, Heeman and Yang 2003).Footnote ⁵ To re-open the block, the user can simply double-click on it.

4.5 BlockView

In addition to WordView and UtteranceView, we are experimenting with a third view, which we call BlockView. Figure 15 gives an example of this view. The bottom half of the figure corresponds to the dialogue excerpt that we have been using in our examples. As can be seen, this view shows the hierarchical structure of the discourse by displaying the summary for each block, indented appropriately. BlockView gives a very concise view of the dialogue. It is also convenient for navigating in the dialogue. By highlighting a line and then pressing Sync, the user can see the corresponding part of the dialogue in UtteranceView and WordView.

Fig. 15.

BlockView.

6.1 Comparing speech repairs

ACT has several different modes. In one mode, it only highlights differences in the speech repair annotations. Utterances with different speech repairs will be highlighted in orange in each UtteranceView. This is shown in Figure 18 for utterance u33. To see whether the difference is in the type of the repair or in the extent of the repair, the users will need to look at their exact annotations in WordView.

6.2 Comparing utterance segmentations

Users can also have ACT highlight differences in utterance segmentation. Differences are shown in red in each UtteranceView. Rather than focusing on individual utterances, our strategy is to highlight regions of utterances that may have been affected by a single disagreement in the annotations. To illustrate, consider the two utterance segmentations in Figure 18, which correspond to the dialogue excerpt in Figure 3. For simplicity, we refer to utterances in the first user's annotation with a superscript ‘A’, and the second user's with a ‘B’. Clearly, utterance u38^A is different than u38^B because it contains different words. Utterance u40^A has no corresponding utterance in the second user's annotation, so it is clearly different. We also view utterances u39^A and u39^B as being different. This is because the preceding context of the utterances are different. The preceding context of u39^A does not include “Corning”, while the preceding context of u39^B does. ACT marks utterances as different if they either do not have the same content or their preceding context is different. Hence, the regions of utterances u38^A to u40^A and u38^B to u39^B are marked as different. Our definition of what constitutes different utterances results in fewer regions being marked than when utterances are marked individually.

6.3 Comparing utterance tags

ACT can also highlight differences in utterance tags. Utterance u34 is highlighted in yellow as a tag difference in Figure 18; the first user annotated it as an agreement and the second as an acknowledgement. Note that differences in whether utterances are annotated as overlap, incomplete, or abandoned also count as utterance tag differences.

When comparing utterance tags, users might want to focus on major distinctions and ignore minor ones. Users specify which distinctions should be ignored in the configuration file. This is especially useful when developing an annotation scheme, as it allows users to focus on the glaring differences and temporarily ignore the more subtly differing aspects. For example, the difference between acknowledgement and agreement can be ignored by specifying the following rule:

More formally, suppose that an utterance has tags A from one user and B from another user. The tags are treated as being identical if the following holds:

As a second example, users might want to only highlight differences where they disagreed on whether an utterance had a forward or backward function. This can be specified by the following two rules, where ‘*’ can match any string:

6.4 Comparing blocks

Even though UtteranceView for each user can display discourse blocks, we have found it useful to include a special view for comparing discourse segmentation, which we call BlockDiffView and show in Figure 19. Before comparing block segmentations, ACT requires that there be no differences in the utterance segmentation (and linearization), but differences in speech repairs and utterance tags are allowed. BlockDiffView shows the utterance numbers along the top, with the discourse blocks of each user displayed underneath.

Fig. 19.

Block comparison in the BlockDiffView.

To help users compare their block annotations, we categorize each block B as one of three types:

1. Same: There is a corresponding block in each of the other users' annotations that has the same start and end utterances. Examples in Figure 19 are b4 and b7 in the top user's annotation, which are the same as b4 and b9 in the bottom user's annotation respectively. These blocks are displayed in black.

2. Inconsistent: There is a block B′ in another user's annotation that overlaps with B but B′ does not embed nor is embedded in B (Strayer et al. Reference Strayer, Heeman, Yang, Kuppevelt and Smith2003). Examples in Figure 19 are b2, b3, and b6 in the top user's annotation, and b3, b7, and b10 in the bottom user's annotation. This is similar to the crossing brackets metric for parser evaluations (Harrison et al. Reference Harrison, Abney, Fleckenger, Gdaniec, Grishman, Hindle, Ingria, Marcus, Santorini and Strzalkowski1991). An inconsistent block is displayed in red.

3. Consistent: All other blocks. Examples in Figure 19 are b5 in the top user's annotation, and b2, b5, and b8 in the bottom user's annotation. A consistent block is displayed in yellow.

Ideally, blocks from different users' annotations would be the same. However, when a block is not the same, but it is consistent, this could just mean that the users are annotating at different levels of granularity. On the other hand, when a block is inconsistent, it could mean that the users have very different interpretations of what is going on in the dialogue, which is a much more serious concern. Distinguishing inconsistent blocks from consistent ones helps users focus on more important differences.

As one of the uses of ACT is for consensus annotation, we allow users to change their speech repairs, utterance segmentation, and utterance tags in ACT, similarly as can be done in DialogueView. Currently ACT does not support block modifications.