2.1 The original Communicator corpora

The Communicator corpora (2000 and 2001) consist of spoken human–machine dialogues in the domain of telephone flight reservations. The users always try to book a flight but they may also try to select a hotel or rent a car. The dialogues are primarily ‘slot-filling’ dialogues, with information being presented to the user at the end of the conversation.

The Communicator corpora have recently been released by the Linguistic Data Consortium (LDC). A particular problem is that although the Communicator corpus is the largest publicly available corpus of speech-act-annotated dialogues, it does not meet our requirements on corpus annotation for dialogue strategy learning, user simulation, and representation of dialogue context. For example, the user dialogue inputs were not annotated with speech act classifications, and no representation of dialogue context was included. Moreover, there was no information about the status of the slots, which is critical for learning dialogue strategies and user simulations.

The original Communicator corpora have previously been annotated (but only for the system's side of the dialogue) using the DATE (Dialogue Act Tagging for Evaluation) scheme (Walker and Passonneau Reference Walker and Passonneau2001) described in Section 2.2. Figure 1 shows an extract from the 2001 collection. For user utterances both the speech recognition output and the human transcription of the user's input are provided but there is no speech act tagging. Also, for each dialogue there is information about the actual and perceived task completion and user satisfaction scores, based on the PARADISE evaluation framework (Walker, Kamm and Litman Reference Walker, Kamm and Litman2000). For the user satisfaction scores, users had to answer questions in a Likert scale (1–5) about the ease of the tasks they had to accomplish, whether it was easy or not to understand the system, their expertise, whether the system behaved as expected, and if they would use the system again in the future or not.

Fig. 1.

Example user turn from the original Communicator corpus, simplified from the original XML format; asr is the output of the speech recogniser, and ne_asr is the output of the speech recogniser tagged with named entity information.

The 2000 collection contains 648 dialogues recording the interactions of humans with nine systems, and the 2001 collection contains 1,683 dialogues with eight systems. Table 1 shows some statistics of the two collections. In the 2000 collection each turn contains only one utterance but in the 2001 corpus a turn may contain more than one utterance. More details about the Communicator corpora can be found in Walker et al. (Reference Walker, Aberdeen, Boland, Bratt, Garofolo, Hirschman, Le, Lee, Narayanan, Papineni, Pellom, Polifroni, Potamianos, Prabhu, Rudnicky, Sanders, Seneff, Stallard and Whittaker2001a, Reference Walker, Passonneau and Boland2001b, Reference Walker, Rudnicky, Aberdeen, Bratt, Garofolo, Hastie, Le, Pellom, Potamianos, Passonneau, Prasad, Roukos, Sanders, Seneff, Stallard and Whittaker2002).

Table 1.

Statistics of the 2000 and 2001 Communicator data

We now present the DATE scheme used in the original Communicator corpus and then our extension of it, including the dialogue information state annotations. This annotation scheme has become known as the ‘TALK context annotation framework’ and it has an associated XML document type definition (DTD).Footnote ³

2.2 The DATE annotation scheme

The system utterances in the original Communicator corpus are annotated using the DATE scheme (Walker and Passonneau Reference Walker and Passonneau2001). The DATE scheme was developed for providing quantitative metrics for comparing and evaluating the nine different DARPA Communicator spoken dialogue systems. The scheme employs the following three orthogonal dimensions of utterance classification:

• •

• •

• •

The conversational domain dimension categorises each utterance as belonging to a particular ‘arena of conversational action.’ Here ‘about_task’ refers to the domain task (in Communicator this is air travel, hotel, and car-rental booking), and ‘about_communication’ refers to conversational actions managing the communication channel (e.g. ‘are you still there?’). ‘situation_frame’ utterances manage the ‘culturally relevant framing expectations’ in the dialogue (e.g. that the conversation will be in English, or that the system cannot issue airline tickets).

The task–subtask dimension relates to a model of the domain tasks that the dialogue system is designed to support. In Communicator there were two main tasks: booking a flight (top_level_trip), and ‘ground’ which was to determine whether the user also wanted to book a car rental and/or a hotel. The subtasks were elements such as finding the dates and times of the flights.

The speech act dimension relates to the utterance's communicative goal. The speech acts used are relatively standard, and are described in detail in Walker and Passonneau (Reference Walker and Passonneau2001). Note that in the Communicator data only the system's side of the dialogue is already annotated using the DATE scheme.

2.3 Extending DATE: The new utterance annotation scheme

Given these system utterance annotations, the first part of our task of producing the new annotations is to interpret the user's input and find its effect on the dialogue context. In other words, we need to associate the user utterances with the correct speech acts and tasks. This process is complicated by the fact that each utterance may involve multiple tasks and that each task could include multiple speech acts.

To accommodate the annotation of user utterances, we needed to extend the DATE scheme in several ways. One problem was that the original annotation of the Communicator data does not distinguish between the different origin and destination cities for different legs of a multiple-leg trip. Thus the tag ‘dest_city’ could be used for any type of destination, regardless of whether the trip is single or multiple-leg. However, we believe that it is important to annotate these distinctions so that there is no overwriting of the values in filled slots such as ‘dest_city,’ ‘depart_date,’ etc. For this reason we extended the set of subtasks to include the distinction between different journey legs. Another problem is that the original annotation of the Communicator data does not distinguish between departure and arrival dates or times, and sometimes it contains times which have been labelled as dates. Thus we also made this distinction, and fixed these mislabellings.

We use the following extended tasks (the equivalent of subtasks in the DATE scheme) and speech acts for annotating user utterances. These are in addition to the DATE scheme (Walker and Passonneau Reference Walker and Passonneau2001) used for the system prompts annotation

• •

• •

• •

Assigning 〈speech act, task〉 pairs to user utterances is basically a dialogue act recognition taskFootnote ⁴ where each dialogue act tag is a 〈speech act, task〉 pair. Our case is very similar to dialogue act tagging with multidimensional tag sets, which assigns a combination of tags to each utterance segment (Lesch, Kleinbauer and Alexandersson Reference Lesch, Kleinbauer and Alexandersson2005). Although only one tag can be assigned per utterance segment (‘from Boston’ is unambiguously a ‘provide_info(orig_city)’ and ‘to Orlando’ a ‘provide_info(dest_city)’), each user utterance may consist of more than one segment, the boundaries of which are not defined, thus possibly leading to disagreements in the number of tags assigned to that utterance by different annotation systems or human annotators. This makes the task of our automatic annotation system even more challenging.

Several dialogue act recognition techniques have been described in the literature so far, e.g. n-grams (Reithinger and Maier Reference Reithinger and Maier1995; Reithinger and Klesen Reference Reithinger and Klesen1997; Webb, Hepple and Wilks Reference Webb, Hepple and Wilks2005), hidden Markov models (Ries Reference Ries1999; Stolcke et al. Reference Stolcke, Coccaro, Bates, Taylor, Ess-Dykema, Ries, Shriberg, Jurafsky, Martin and Meteer2000), Bayesian networks (Keizer and op den Akker Reference Keizer and op den Akker2007), neural networks (Kipp Reference Kipp1998; Ries Reference Ries1999), transformation-based learning (Samuel, Carberry and Vijay-Shanker Reference Samuel, Carberry and Vijay-Shanker1998), etc. All these techniques assume that some part of the corpus is already annotated with dialogue acts and can be used for training statistical models. None of these statistical methods can be applied in our case because the original Communicator corpus does not include any annotations for the user's side of the dialogue; in other words, there is no complete corpus that could be split for training and testing purposes. Therefore we are restricted to use traditional parsing techniques for each user utterance taking into account information from the previous system utterances. In particular, the speech acts and tasks are computed using a phrase spotting semantic parser which we describe in Section 3.1.

As already mentioned, because a given user utterance may contain multiple speech acts and tasks, each user utterance is potentially annotated with lists of these features. These lists are kept synchronised so that the lengths and ordering of the features are kept equal. For example, if the user gives the departure date, reprovides information about the origin city and also provides new information about the destination city then ‘SpeechAct’ will be ‘[provide_info, reprovide_info, provide_info]’ and ‘Task’ will be ‘[depart_date, orig_city, dest_city].’ This representation allows the pairing of speech acts and tasks to be extracted from the annotation.

2.4 The TALK context annotation scheme

Using these annotations of the system and user utterances, our system then computes context annotations for the states which follow each utterance. Figure 2 shows an example information state as it has been annotated by this automated annotation system. It corresponds to the state following the user utterance shown in Figure 1. Dialogue annotations are saved in XML format but here the information state is presented in plain text for reasons of readability. Several of these features simply specify the utterance annotations for the previous utterance. Others specify various book-keeping features such as turn number, or low-level features (discussed in Section 5.2). The most interesting features from the point of view of context annotations are those specifying which slots have been filled or confirmed, and those which accumulate information about the whole dialogue history.

Fig. 2.

Example dialogue context/Information State. User-provided information appears between [ ] brackets.

Computing context annotations for the state which follows each utterance means computing the internal state of the system at that point in the dialogue. Thus our automatic annotation system can be thought of as a dialogue system running in a reverse mode (reading backwards from the system prompts and recognised user utterances) and therefore there is no guarantee that the state produced by the automatic annotation system will be an accurate simulation of the real state that the dialogue system was in. Nevertheless, as will be shown in Section 6, the states produced by the automatic annotation system are good approximations of the real states. We must also keep in mind that the purpose of the automatic annotations is to generate corpora to be used in machine learning experiments for learning dialogue strategies, user simulations, and models for context-dependent interpretation and speech recognition. Machine learning methods are good at compensating for errors in the training data.

The most difficult problem to solve in annotating dialogue contexts is determining (via processing discussed in Section 3) what slots have been filled, confirmed, or even emptied, by a user utterance. We define a piece of information as ‘confirmed’ (according to the system's perspective) only if it has been positively confirmed. Thus confirmation processing can only take place after system utterances labelled as explicit or implicit confirmation. There is no need to have a separate field for the value of the confirmed slot because the value which is confirmed must be the same as the value with which the slot has been filled.

One concern about labelling confirmation using only the information available in the original Communicator corpus (and not the real state of the dialogue system itself) is that it assumes that the Communicator systems had some notion of confirmation in their algorithms. For this reason, we only specify slot confirmation when the systems attempt confirmation via an explicit or implicit confirmation action. Moreover, only the speech recognition output is used for processing and deciding on the slots that will be filled or confirmed. The human transcription of the user's input is only considered for computing error rates, as explained in Section 5.2. This also ensures that we do not base our annotation on information that the systems did not have at runtime.

Note that the history-level annotations/logs of states should be computable from the other levels over the history of utterances. Therefore the features of the dialogue, task, and low levels can be expanded after the annotation/logging is completed. With some dialogue managers, e.g. those by Larsson and Traum (Reference Larsson and Traum2000) and Bos et al. (Reference Bos, Klein, Lemon and Oka2003), this can be achieved by logging whole Information States.

Note also in Figure 2 the difference between the groups of Information State fields {FilledSlotsHist, FilledSlotsValuesHist, ConfirmedSlotsHist} and {FilledSlotsStatus, FilledSlotsValuesStatus, ConfirmedSlotsStatus}. The former fields give us information about the exact order in which slots have been filled or confirmed and may contain several instances of the same slot, e.g. slot ‘orig_city’ could be confirmed twice. The latter fields (‘FilledSlotsStatus,’ etc.) inform us about the current status of the slots and thus may only contain one instance per slot. This is very important because if a confirmed slot is refilled with a new value it will remain in the ‘ConfirmedSlotsHist’ field even though its new value has not been confirmed yet. The history of speech acts and tasks is also included in our annotations.

Regarding the reward level, task completion (actual and perceived) metrics and user satisfaction scores, e.g. based on the PARADISE evaluation framework (Walker et al. Reference Walker, Kamm and Litman2000), may be used. As mentioned in Section 2.1 the original Communicator corpora included such metrics, and these features are used in Henderson et al. (Reference Henderson, Lemon and Georgila2005, Reference Henderson, Lemon and Georgila2008). Other reward features that could be computed are the dialogue duration, the number of turns, whether the user hung up, the number of slots that were filled and confirmed in a slot-filling dialogue, etc.

The TALK project context annotation framework was initially developed specifically for annotating the Communicator data. It has also been used for a corpus of in-car MP3 player dialogues (Rieser et al. Reference Rieser, Kruijff-Korbayová and Lemon2005a, Reference Rieser, Kruijff-Korbayová and Lemon2005b; Rieser and Lemon Reference Rieser and Lemon2006, Reference Rieser and Lemon2008) and has been adopted for the DiSCoH project (Andreani et al. Reference Andreani, Di Fabbrizio, Gilbert, Gillick, Hakkani-Tür and Lemon2006), but in general there are several other features one might want to include in dialogue context annotations. For the dialogue level, such features include changes to issues/questions under discussion (Ginzburg Reference Ginzburg, Seligman and Westerstahl1996), changes to common ground (Clark and Brennan Reference Clark, Brennan, Resnick, Levine and Teasely1991; Traum Reference Traum1994), obligations (Traum and Allen Reference Traum and Allen1994), system and user intentions (Grosz and Sidner Reference Grosz and Sidner1986), syntactic and semantic parses, and salient NPs. For the task level, such features include the number of database query results (for slot-filling/information-seeking dialogues), user goals, and confidence in each slot (e.g. low/medium/high). For the low level, such features include the amplitude of the speech signal, and word-based confidence scores. For multimodal systems, one might also want to log information about the input modalities, available output modalities, and XY co-ordinates of mouse or touch-screen gestures.

A notable constraint on the information to be useful for machine learning of dialogue strategies and context-dependent interpretation and speech recognition is that all captured features should in principle be available to a dialogue system at runtime – so that, for example, a dialogue system using a learnt policy can compute its next action in any state. This excludes, for example, word error rate from the state information usable for Reinforcement Learning, since it can only be computed after transcription of user speech. In this case, for example, automatic speech recognition (ASR) confidence scores should be used instead. It also means that we need to automatically annotate the ASR hypotheses of the systems, rather than the transcribed user utterances.

3.1 Parsing utterances in context

Note that in the original Communicator corpus only the system's side of the dialogue was already annotated using the DATE scheme. The first job of the automatic annotation system is thus to compute speech acts and content for the user's side of the dialogue. In order to do this multiple stages of parsing are required – to determine the content of the user's utterance and then to determine the speech act that it performs in context.

These computations are performed using Prolog clauses in DIPPER. In the original Communicator corpus, the user utterances (both speech recognition output and human transcription) are enriched with tags indicating the semantics of some words, e.g. cities, dates, airlines, etc. (see Figure 1). This, in addition to the fact that user utterances tend not to be very complex in this type of dialogue, facilitates parsing and thus a keyword-based parser seems to be adequate for this first stage of parsing in our task. Thus if the user says ‘from Los Angeles’ it is straightforward for the parser to fill the slot ‘orig_city’ with the value ‘Los Angeles.’ If however, the user said ‘Los Angeles’ the parser would need information from the previous context. If the previous system utterance was tagged as ‘request_info(orig_city)’ then the parser would understand that ‘Los Angeles’ is related to the ‘orig_city’ slot. As we will see in the confirmation strategies (Section 3.2) and the example that follows (Section 4), the inferences required here are not always so simple. Since some system dialogue moves in the Communicator data were not always reliably annotated, the automatic annotation system does not always rely only on the original speech act and task tags of the system utterances but also parses the system utterances.

3.2 Confirmation strategies

When computing confirmation, it is important to take into account the different ways in which dialogue systems use various types of confirmation. In general, the systems in the Communicator corpus typically follow one of three general confirmation strategies.

In the first strategy the system asks the user to fill a slot, then asks for confirmation (explicit or implicit), and moves to the next slot if the user confirms, or may keep asking for confirmation if the user does not cooperate. In the second strategy the system asks the user to fill several slots and then attempts to confirm them in one single turn. That means that the system's turn could consist of several utterances labelled as ‘explicit_confirm’ or ‘implicit_confirm.’ A third strategy, which is a variation of the second strategy is when the system tries to confirm several slots in a single utterance, e.g. ‘explicit_confirm(trip),’ ‘implicit_confirm(orig_dest_city).’ Before confirmation the slots could be filled either in a single turn or in multiple turns.

For the first and third confirmation strategies it proves adequate to look only one or two steps backwards in the history of system utterances, whereas for the second strategy looking further back is required. To cover all these cases, when the annotation system parses the user input, it looks backwards to all the system utterances between the last user utterance and the current one (not including the last user utterance). From these system utterances, we take into account only utterances with the following speech acts: request_info, explicit_confirm, implicit_confirm, and offer. Other utterances (e.g. instructions) are not taken into account because they do not affect whether a slot will be filled or confirmed.

Note that the annotation system first extracts the speech acts and possible tasks related to the current user utterance and then attempts to detect confirmation based on this information. Any kind of disambiguation required, e.g. to decide whether the speech act should be tagged as ‘provide_info’ or ‘reprovide_info,’ is done before confirmation. However, if speech act or task ambiguity remains unresolved it will be dealt simultaneously with confirmation, e.g. if the user uttered a city name we cannot be sure whether it refers to an origin or destination city until we consider the context. The reason for this sequential procedure is that it could be the case in a different corpus that the user's speech acts and tasks were already annotated. In that case the annotation system would only have to compute the confirmed slots. Thus the separation of parsing and confirmation computations serve the purpose of modularisation.

For the first two confirmation strategies, the annotation system should check whether the tasks extracted by parsing the user's utterance are included in the task labels of the previous explicit or implicit confirmation system prompts. Then according to the type of speech act (yes_answer, no_answer, reprovide_info, provide_info, correct_info, etc.) the system confirms one or more previously filled slots or fills one or more new ones.

The third confirmation strategy involves system utterances such as ‘You are travelling from Boston to Chicago on the 5th of May. Is that correct?’ If the user says ‘yes’ then this is a confirmation of three different slots, so the value of the ‘Task’ feature is ‘[orig_city, dest_city, depart_date].’ To infer this the annotation feature needs to be aligned with the ‘Task’ feature, its value is ‘[yes_answer, yes_answer, yes_answer].’

In the Appendix, the processes for assigning speech acts and tasks to each utterance, and calculating confirmations and dialogue context, are presented in the form of pseudocode.

5.1 Specific parsing rules for the Communicator data

As discussed in Section 2.3 when the annotation system parses the user input it has to decide whether the information provided refers to a single or a multiple-leg trip. For a continuation trip, user input parsed as dest_city, depart_date, depart_time, arrive_date and arrive_time will be tagged as continue_dest_city, continue_depart_date, continue_depart_time, continue_arrive_date, and continue_arrive_time. In the same way, for a return trip, depart_date, depart_time, arrive_date, and arrive_time will be tagged as return_depart_date, return_depart_ time, return_arrive_date, and return_arrive_time. For a continuation trip the origin city of one leg is the destination city of the previous leg. For a return trip, the origin city of the inward leg is the destination city of the outward leg and the destination city of the inward leg is the origin city of the outward leg. Thus we do not need tags such as continue_orig_city, return_orig_city, return_dest_city.

5.2 Confidence scoring for the Communicator data

Ideally, in future dialogue corpora, we would have dialogue data that contains ASR confidence scores. Unfortunately the Communicator data does not have this information. However, the Communicator data contains both the output of the speech recognition engine for a user utterance and a manual transcription of the same utterance carried out by a human annotator. We consider the word error rate (WER) to be strongly related to confidence scores and thus each time a user utterance is read from the XML file a third agent is called to estimate error rates (the ComputeErrorRates agent). Four different error rates are estimated: classic WER, WER-noins, sentence error rate (SER), and keyword error rate (KER).

The classic WER, is defined by the following formula:

where N is the number of words in the transcribed utterance, and Nins, Ndel, Nsub are the number of insertions, deletions, and substitutions, respectively, in the speech recognition output. WER-noins is WER without taking into account insertions. The distinction between WER and WER-noins is made because WER shows the overall recognition accuracy whereas WER-noins shows the percentage of words correctly recognised. The sentence error rate (SER) is computed on the whole sentence, based on the principle that the speech recognition output is considered to be correct only if it is exactly the same as the manually transcribed utterance. All the above error estimations have been performed using the HResults tool of HTK (Young et al. Reference Young, Evermann, Gales, Hain, Kershaw, Moore, Odell, Ollason, Povey, Valtchev and Woodland2005), which is called by the ComputeErrorRates agent. Finally the keyword error rate (KER) is also computed by ComputeErrorRates (after the utterance has been parsed by DIPPER) and shows the percentage of the correctly recognised keywords (cities, dates, times, etc.). This is also a very important metric regarding the efficiency of the dialogues. Similarly to WER, KER cannot be computed at runtime but we assume that it is strongly correlated with the confidence score of the parser.

It should be noted that speech phenomena such as pauses, fillers, noises, etc. that are transcribed by human annotators are not taken into account when error rates are estimated because most speech recognisers do not include them in their outputs, even though they are considered by their acoustic models. Therefore if such phenomena were included while estimating errors there would not be a fair comparison between the speech recognition output and the human transcription of the utterance.

6.1 Automatic evaluation

We evaluated our automatic annotation system by automatically comparing its output with the actual (ATC) and perceived (PTC) task completion metrics as recorded in the original Communicator corpus. Our evaluation is restricted in the 2001 corpus because no such metrics are available for the 2000 data collection. Systems in the 2001 collection were generally more complex and there was a large performance improvement in every core metric from 2000 to 2001 (Walker et al. Reference Walker, Rudnicky, Aberdeen, Bratt, Garofolo, Hastie, Le, Pellom, Potamianos, Passonneau, Prasad, Roukos, Sanders, Seneff, Stallard and Whittaker2002). Therefore if the 2001 annotations are evaluated as successful, it is expected that the same will be true for the 2000 annotations. If the final state of a dialogue – that is, the information about the filled and confirmed slots – agrees with the ATC and PTC for the same dialogue, this indicates that the annotation is consistent with the task completion metrics. We consider only dialogues where the tasks have been completed successfully – in these dialogues we know that all slots have been correctly filled and confirmedFootnote ¹⁰ and thus the evaluation process is simple to automate.Footnote ¹¹ For example, if a dialogue that is marked as successful consists of a single leg and a return leg then the expected number of slots to be filled and confirmed is six (‘orig_city,’ ‘dest_city,’ ‘depart_date,’ ‘depart_time,’ ‘return_depart_date,’ and ‘return_depart_time’). If the automatic annotation system filled three and confirmed two slots then the accuracy for this particular dialogue for filled slots and confirmed slots would be 50 per cent and 33.3 per cent, respectively. For dialogues that are marked as unsuccessful or not marked at all (as in the 2000 corpus) there is no straightforward way to calculate the number of slots that should have been filled or confirmed. This automatic evaluation method cannot give us exact results – it only indicates whether the dialogue is annotated more or less correctly.

We have applied our automatic evaluation method on the flight-booking portions of the automatically annotated Communicator corpus. The results are that, for dialogues where ATC or PTC is marked as ‘1’ or ‘2’ (i.e. where the flight-booking portion of the dialogue was successful or was perceived by the user to be successful),Footnote ¹² the current automatic annotations for the whole corpus showed 93.9 per cent of the required slots to be filled (filled slots accuracy) and 75.2 per cent of the slots to be confirmed (confirmed slots accuracy). For the baseline annotations the accuracy for the filled and confirmed slots was 65.4 per cent and 50.9 per cent, respectively. Therefore when we use the advanced automatic annotations the absolute increase in accuracy of the filled and confirmed slots over the baseline annotations is 28.5 per cent and 24.3 per cent, respectively. Detailed results are depicted in Table 2.

Table 2.

Automatic ISU annotation accuracy for the Communicator 2001 data (automatic evaluation, AUTO: automatic annotation system, BASE: baseline system)

The IBM system did not confirm and therefore we could not obtain results for the ‘confirmed slots accuracy.’ In cases where the system attempts to confirm more than one slot in a single turn (second and third confirmation strategies), if the user gives a simple ‘no_answer’ there is no way for the annotation system to detect the slot that the ‘no_answer’ refers to. The system assumes that the ‘no_answer’ refers to all the slots under confirmation. This can lead to fewer slots being confirmed. One of the rules that the annotation system used for confirmation calculation in the version described in Georgila, Lemon and Henderson (Reference Georgila, Lemon and Henderson2005b) was that only filled slots could be confirmed, mostly to ensure that the system policies trained with the Communicator annotated corpus (e.g. using Reinforcement Learning) would be reasonable. This rule caused problems in cases where for example the system knew the user's residence and therefore did not ask for the ‘orig_city’ but in the sequel tried to confirm it, or when the user gave a negative confirmation to a filled slot value (thus the filled slot was emptied) but the system performed a second confirmation request with an alternative slot value. In that case even if the user gave a ‘yes_answer’ the slot would not be confirmed because it was not filled anymore. The above observations explain the low scores of the COL and ATT systems (and to a lesser extent CMU) for ‘confirmed slots accuracy’ in the earlier work of Georgila et al. (Reference Georgila, Lemon and Henderson2005b). In the current version of the automatic annotation system the rule that caused problems (of not being able to confirm a slot unless it had been filled first) has been removed and the COL and CMU scores for confirmed slots accuracy have risen significantly (especially COL). The results presented in Table 2 show that the baseline annotations are generally poor and that the user does not always behave as expected. Therefore user utterances should be properly parsed in their context instead of just being tagged with the 〈speech act, task〉 pair that best matches the previous system request. In particular, IBM, MIT, and also COL had very general opening questions in their dialogues, e.g. ‘What are your travel plans?’ or ‘Please tell me about the first leg of the trip.’ that prompted the user to provide much information at the same time, which was beyond the capabilities of the baseline system.

6.2 Evaluation with respect to hand-annotated data

The results of the automatic evaluation tell us whether the system succeeded in filling and confirming the right slots but not whether their values were correct. By inspection of the annotated Communicator dialogues it appears that most of the time slots are filled and confirmed with the correct values or they are not filled/confirmed at all. However, in order to have more accurate results, evaluation with respect to hand-annotated data was also performed. We randomly selected six dialogues from each system of the 2001 collection, making forty-eight dialogues in all for hand annotation. There were no constraints regarding whether the dialogues were actually completed or perceived by the user as completed. The only constraint was not to have more than one dialogue from the same user so that more variety was captured.

We extracted the final information state of the flight-booking portion of the automatically annotated dialogues and examined the ‘FilledSlotsStatus,’ ‘FilledSlotsValuesStatus,’ and ‘ConfirmedSlotsStatus’ fields. As mentioned in Section 2.4, these fields give us information about the current status of the slots, i.e. whether they are filled or confirmed, and their values. For example, if ‘orig_city’ is filled with the value ‘Boston’ and then with the value ‘Orlando,’ only the value ‘Orlando’ will be saved in the ‘FilledSlotsValuesStatus’ field. Then we manually inspected the flight-booking portion of the dialogue from beginning to end and we noted down the slots that should be filled or confirmed and their values, according to what happened in the dialogue and with respect to the ASR hypotheses, rather than the utterance transcriptions. By comparing the status of the slots as produced by our automatic annotation system with the status that the slots should have according to the manual annotation, we can measure the precision, the recall, and the F-score of our automatic annotations.

Precision and recall are defined by the following formulae:

where C is the number of correctly tagged slots, I1 is the number of incorrectly tagged slots that should have been tagged with a different value, I2 is the number of incorrectly tagged slots that should not have been tagged at all, and MC is the number of slots that were missed by the automatic annotation system or the baseline system (i.e. not tagged at all). Using the above formulae, precision, recall, and consequently F-score can be computed for filled slots and confirmed slots.

In our manual annotation a slot is considered as correctly filled if it is filled with the correct value. Moreover, a slot is considered as confirmed only if the system attempted to confirm it. It will not be considered as confirmed just because the user accepted the flight offer. This is in order to have a fair comparison with our automatic annotations. Both the automatic annotation system and the baseline system tag a slot as confirmed only after the dialogue system has explicitly or implicitly tried to confirm that particular slot. In addition, a slot is confirmed if its value in the ‘FilledSlotValue’ field has been confirmed.

Detailed results of precision, recall, and F-score for filled and confirmed slots are given in Tables 3 and 4, respectively.

Table 3.

Precision, recall, and F-score of filled slots (including their values) for the Communicator 2001 data (manual evaluation, AUTO: automatic annotation system, BASE: baseline system)

Table 4.

Precision, recall, and F-score of confirmed slots (including their values) for the Communicator 2001 data (manual evaluation, AUTO: automatic annotation system, BASE: baseline system)

The results for precision, recall, and F-score for the automatic annotation system are relatively high for all systems apart from LUC. For ATT, CMU, and COL the scores computed for confirmed slots from the evaluation with regard to the hand-annotated data are much higher than the automatically derived scores of Section 6.1, which proves that the assumption that the successful dialogues had all slots confirmed is too strong. Precision of confirmed slots is much higher than recall, which means that the automatic annotation system decides to tag a slot as confirmed only if there is strong evidence for doing so.

As with the automatic evaluations reported earlier, the recall and the F-score of the baseline system for the confirmed slots are low for ATT, BBN, COL, IBM, MIT, and to a lesser extent CMU. The performance of the baseline for LUC is poor for the same reasons that the automatic annotation fails to produce good precision and recall values (details are given in Section 6.4). The precision of the baseline system is very high, some times even higher than the precision of the automatic annotation system, due to the fact that the baseline is very conservative in its predictions. However, the recall is always considerably lower than the recall of the automatic annotation system for both filled and confirmed slots.

By using the automatic annotations instead of the baseline ones the absolute gain in the F-score for filled and confirmed slots is approximately 10 per cent and 15 per cent, respectively, with or without including the poor results of LUC.

For these forty-eight dialogues we also manually tagged each user utterance with 〈speech act, task〉 pairs and used this as our gold standard with which we compared both the automatic and the baseline annotations. The results are depicted in Tables 5 and 6.

Table 5.

Precision and recall of 〈speech act, task〉 pairs for the Communicator 2001 data (manual evaluation, AUTO: automatic annotation system, BASE: baseline system)

Table 6.

Precision, recall, and F-score per slot type for the Communicator 2001 data (manual evaluation, AUTO: automatic annotation system, BASE: baseline system)

Table 5 shows that the precision and recall of the automatic annotation system for all 〈speech act, task〉 pairs are 70.5 per cent and 76.4 per cent, respectively. The precision is a little lower than the recall because the automatic annotation system had the tendency to overgenerate. For example if the ASR output was ‘yes I'd no I'm travelling to London’ the automatic annotation system would tag it as ‘[yes_answer(null), no_answer(null), provide_info(dest_city)]’ instead of ‘[provide_info(dest_city)].’ This is due to the fact that the keyword-based parser cannot handle speech repairs. Each row in Table 5 gives the gain we can get in precision and recall by ignoring some unimportant sources of error. Thus, if we ignore cases that are tagged with empty values for both the speech act and the task by the system but the human annotators tag them as ‘provide_info(null)’ or the opposite, precision rises to 72.3 per cent and recall to 78.4 per cent. Also, both the automatic annotation system and the baseline system tag user utterances which contain the words ‘yes’ or ‘no’ as ‘yes_answer’ or ‘no_answer,’ respectively even though the utterance is not necessarily an answer. That of course does not cause problems with the status of the filled and confirmed slots, and if it is ignored the precision of the automatic annotation system rises from 70.5 per cent to 72.2 per cent and the recall from 76.4 per cent to 78.3 per cent. If the distinction between ‘provide_info’ and ‘reprovide_info’ is ignored the gains for precision and recall are 2.4 per cent and 2.6 per cent, respectively. As we can see the results for the baseline are consistently lower. By ignoring all of the three above-mentioned sources of error the best results that can be obtained are 76.4 per cent and 82.9 per cent for the automatic annotation system (precision and recall) and 68.0 per cent and 68.7 per cent for the baseline system. The initial F-scores for the automatic annotation system and the baseline system are 73.3 per cent and 64.3 per cent, respectively and the best F-scores we can obtain by ignoring the three sources of error are 79.5 per cent and 68.3 per cent. All the values in Table 5 have been calculated without taking into account cases where both the human annotators and the systems tagged the user utterance with empty lists for both the speech act and the task, i.e. when they agreed that the utterance did not contain relevant information to the dialogue task. If we want to give some credit to the automatic annotation system and the baseline system for successfully detecting such cases then the results for precision, recall, and F-score become for the automatic annotation system 71.0 per cent, 76.3 per cent, and 73.5 per cent and for the baseline 65.7 per cent, 66.2 per cent, and 66.0 per cent. The improvement becomes higher if we also ignore the problems of distinguishing between ‘provide_info’ and ‘reprovide_info,’ etc.

Table 6 shows precision, recall, and F-score per slot type. Again the automatic annotation system outperforms the baseline one. The recall for ‘arrive_time’ (both in the automatic and baseline annotations) and for ‘airline’ (only in the baseline annotations) is very low because there are not many instances of these slots and thus the effect of each error on the total score is inflated. Furthermore, the ‘airline’ slot is optional and the dialogue systems rarely prompt the user to provide this slot, which explains the low recall of the baseline.

Manual annotation has also shown that the hotel and car rental bookings have been well annotated.

6.3 Indirect evaluation via user simulations

As a form of indirect evaluation we build user simulation models. For learning dialogue strategies it is rarely (if ever) the case that enough training data from real dialogues with human users is available to sufficiently explore the vast space of possible dialogue states and strategies. Thus, simulated users are critical for training stochastic dialogue systems (often of the order of 100,000 training dialogues are required), and for evaluating candidate dialogue policies. Simulated users simulate real-user behaviour, i.e. the way users would interact with the system in order to accomplish their goals (e.g. book a flight, get tourist or product information, etc.). The basic idea is to use small corpora to train stochastic models for simulating real-user behaviour. Once such a simulated user is available, any number of dialogues can be generated through interaction between the simulated user and the dialogue policy, and these simulated dialogues can be used with statistical optimisation methods such as RL (Georgila et al. Reference Georgila, Henderson and Lemon2005a, Reference Georgila, Henderson and Lemon2006, Reference Georgila, Wolters and Moore2008b; Schatzmann et al. Reference Schatzmann, Georgila and Young2005a). Also user simulations make it feasible to test the performance of different dialogue policies against simulated users in an efficient and inexpensive way.

We train n-grams of 〈speech act, task〉 pairs from both the automatic system annotations and the baseline annotations. Our reasonable assumption is that the higher the quality of the user simulation model the better the training data it was trained on. To assess our user models we need to compare their output with real responses given by users in the same context. For this purpose we run our user simulation models through the 〈speech act, task〉 pairs of the hand-annotated data.Footnote ¹³ As explained previously, user simulations are to be used for automatic dialogue strategy learning. For this reason we desire user simulations which do not always act in the same way in identical states – we need to explore the policy space, so some reasonable amount of variation is required. Our n-gram models produce actions based on a probability distribution learnt from the training data and not always the action with the highest probability. Therefore our evaluation metrics should take into account this distribution. We use expected precision (EP) and expected recall (ER) to quantify how closely the synthetic turn resembles the real-user turn. Unlike precision and recall that are calculated always choosing the action with the highest probability (Schatzmann et al. Reference Schatzmann, Georgila and Young2005a), EP and ER are calculated taking into account the probability distribution of actions that the user model can produce at a given state. Thus we measure the expected value of precision and recall if we ran the user model for a large number of times through the same states found in the data, each time generating an action according to the distribution produced by the model. We also use perplexity (PP) as a measure of the number of possible actions that the user model has to choose between at a given state. For a full discussion on the above metrics see Georgila et al. (Reference Georgila, Henderson and Lemon2005a, Reference Georgila, Henderson and Lemon2006) and Schatzmann et al. (Reference Schatzmann, Georgila and Young2005a, Reference Schatzmann, Weilhammer, Stuttle and Young2006). Results are depicted in Table 7. The user model based on the automatic annotations performs better than the one trained on the baseline annotations, which once more proves that the automatic annotation system is superior to the baseline one. The automatic annotations are especially better with regard to expected recall and for higher values of n. For low values of n, the baseline user model suffers less from data sparsity issues since the baseline annotations do not produce the variety of the automatic ones. With regard to perplexity again the user model trained on the automatic annotations performs better.

Table 7.

Expected precision, expected recall, and perplexity obtained when n-gram user simulation models trained on the automatic and baseline annotations run through the hand-annotated data (AUTO: user simulations trained on the automatic annotations, BASE: user simulations trained on the baseline annotations)

Another indirect indication of the good quality of the annotations and proof of their usefulness is that they have been used to train successful dialogue strategies (Henderson et al. Reference Henderson, Lemon and Georgila2005, Reference Henderson, Lemon and Georgila2008; Frampton and Lemon Reference Frampton and Lemon2006) and user simulations in Georgila et al. (Reference Georgila, Henderson and Lemon2005a, Reference Georgila, Henderson and Lemon2006) and Schatzmann et al. (Reference Schatzmann, Georgila and Young2005a, Reference Schatzmann, Stuttle, Weilhammer and Young2005b).

Furthermore, in Lemon et al. (Reference Lemon, Georgila, Henderson and Stuttle2006a) we have demonstrated that a policy learnt from our Communicator 2001 automatically annotated dialogues performs better than a state-of-the-art hand-coded dialogue system in experiments with real users. The experiments were done using the ‘TownInfo’ multimodal dialogue system of Lemon et al. (Reference Lemon, Georgila, Henderson and Stuttle2006b). The learnt policy (trained on the Communicator data) was ported to the city information domain, and then evaluated with human subjects. The learnt policy achieved an average gain in perceived task completion of 14.2 per cent (from 67.6 per cent to 81.8 per cent at p < .03) compared to a state-of-the-art hand-coded system.

6.4 Additional error analysis

We now explore some additional patterns of error in the automatic annotations, which are specific to the flight reservations domain. In all systems there are system prompts asking the user whether he/she would like to have a continuation or return trip. Unfortunately the DATE tag ‘continue_trip’ or ‘return_trip,’ respectively, is the same regardless of whether the user chooses to have a continuation/return trip or not. The automatic annotation system has to parse the system prompt and the user reply and decide on the type of trip. For all systems except LUC, these system prompts always have a consistent form. For example, for continuation trips ATT always uses the following type of question ‘Is Detroit your final destination?’ Thus if the user replies ‘yes’ there will be no continuation trip. On the other hand, for CMU the structure of the question is different: ‘Would you like to go to another city after Los Angeles?’ Here ‘yes’ means the opposite, that is, there will be a continuation trip. LUC switches between the two types of questions, which means that if the annotation system fails to parse the system output there is no default solution to follow as in the other systems. In addition, in the LUC dialogues sometimes there is no question about continuation or return trips and the annotation system has to deduce that from the context, which makes the annotation task even harder. As a result, sometimes the new values from continuation or return legs overwrite the initial values of the outward trip. This mistake was not captured by the automatic evaluation because the annotation system did not recognise the existence of a continuation/return trip, and thus the number of slots that it computed as expected to be filled or confirmed did not include continuation and return slots.