2 Related work

Early question-answering (QA) systems, such as Baseball (Green et al. Reference Green, Wolf, Chomsky and Laughery1961) and Lunar (Woods Reference Woods1973) were carefully hand-crafted to answer questions in a limited domain, similar to the QA components of ELIZA (Weizenbaum Reference Weizenbaum1966) and SHRDLU (Winograd Reference Winograd1972). However, there has been a resurgence of QA systems following the TREC conferences with an emphasis on answering factoid questions. This work on text-based question-answering which is comprehensively summarized in Maybury (Reference Maybury2004), range widely in terms of linguistic sophistication. At one end of the spectrum, there are linguistically motivated systems (Katz Reference Katz1997; Waldinger et al. Reference Waldinger, Appelt, Dungan, Fry, Hobbs, Israel, Jarvis, Martin, Riehemann, Stickel and Tyson2004) that analyze the user's question and attempt to synthesize a coherent answer by aggregating the relevant facts. At the other end of the spectrum, there are data intensive systems (Dumais et al. Reference Dumais, Banko, Brill, Lin and Ng2002) that attempt to use the redundancy of the web to arrive at an answer for factoid style questions. There are also variants of such QA techniques that involve an interaction and use context to resolve ambiguity (Yang, Feng and DiFabbrizio Reference Yang, Feng and DiFabbrizio2006). In contrast to these approaches, our method matches the user's query against the questions in a large corpus of question-answer pairs and retrieves the associated answer.

In the information retrieval community, QA systems attempt to retrieve precise segments of a document instead of the entire document. In Tomuro and Lytinen (Reference Tomuro and Lytinen2004), the authors match the user's query against a frequently asked questions (FAQ) database and select the answer whose question matches most closely to the user's question. An extension of this idea is explored in Xue, Jeon and Croft (Reference Xue, Jeon and Croft2008) and in Jeon, Croft and Lee (Reference Jeon, Croft and Lee2005), where the authors match the user's query to a community collected QA archive such as Yahoo! (2009) and MSN-QnA (2009). Our approach is similar to both these lines of work in spirit, although the user's query for our system originates as a spoken query, in contrast to the text queries in previous work. We also address the issue of noisy speech recognition and assess the value of tight integration of speech recognition and search in terms of improving the overall performance of the system. A novelty in this paper is our method for addressing dynamic questionsFootnote ¹ as a seamless extension to answering static questions.

Also related is the literature on voice-search applications (Google 2009; Microsoft 2009; YellowPages 2009; vlingo.com 2009) that provide a spoken language interface to business directories and return phone numbers, addresses and web sites of businesses. User input is typically not a free flowing natural language query and is limited to expressions with a business name and a location. In our system, users can avail of the full range of natural language expressions to express their information need.

And finally, our method of retrieving answers to dynamic questions has relevance to the database and meta search community. There is growing interest in this community to mine the ‘hidden’ web – information repositories that are behind web forms – and provide a unified meta-interface to such information sources, for example, web sites related travel, or car dealerships. Dynamic questions can be seen as providing a natural language interface (NLI) to such web forms, similar to early work on NLI to databases (Androutsopoulos Reference Androutsopoulos1995).

3 Speech-driven question retrieval system

We describe the speech-driven query retrieval application (Qme!) in this section. The user of this application provides a spoken language query to a mobile device intending to find an answer to her question. User input could be either a short business listing query (e.g. pizza hut near urbana illinois) or a complete question (e.g. how do I fix a leaky dishwasher, what movies are playing in los angeles california).

As shown in Figure 1, we classify questions to Qme! into three categories and apply different retrieval methods.

1. (1) Business listing questions: These questions typically contains a business name, optionally followed by a city and state to indicate the location of the business, such as pizza hut near urbana illinois. User input with a business category (laundromats in madison) and without location information (hospitals) are some variants supported by this application. The result from speech recognition is parsed to identify the business name (aka SearchTerm) and the city/state segments that the user is interested in. The parsed information is used to search a business listing database of over 10 million entries to retrieve the entries pertinent to the user query.

2. (2) Static questions: These are general user queries that are not constrained to be of any specific question type such as what, where, when, how questions. The answers to these questions remain mostly the same irrespective of when and where the questions are asked. Some examples of such queries include what is the fastest animal in water?, how do you make a bloody mary mix?, when is George Washington's birthday?, why is the sky blue?. The result of the speech recognizer is used to search a large corpus of human generated question-answer pairs to retrieve the answers pertinent to the user's static questions. This ensures high accuracy for the answers retrieved (if found in the archive) and also allows us to retrieve related questions on the user's topic of interest.

3. (3) Dynamic questions: The answers to these questions depend on when and where they are asked. For such questions, the above method results in less than satisfactory and sometimes inaccurate answers. Examples of such questions are what is the stock price of General Motors?, who won the baseball game last night?, what is playing at the theaters near me? For the dynamic questions, the answers are retrieved by querying a web form from the appropriate web site (for example, www.fandango.com for movie information).

Fig. 1.

The architecture of the speech-driven question--answering system.

For all user queries, the result from the speech recognizer can be a single-best string or a weighted word lattice. The retrieved results are ranked using different metrics discussed in the next section. In Table 1, we show the results that the system returns for business listings query audi automobiles. In Figure 2, we illustrate the answers that the system returns for the static question what are the symptoms of a heart attack and the dynamic question what movies are playing on sunday. Note that the system returns all the movie listings and timing information for shows on Sunday for the dynamic question.

Table 1.

Listings retrieved for query audi automobiles

Fig. 2.

Retrieval results for static and dynamic questions using Qme!.

4 Tightly coupling speech recognition and search

Most of the speech-driven search systems (Acero et al. Reference Acero, Bernstein, Chambers, Ju, Li, Odell, Nguyen and Zweig2008; Bacchiani et al. Reference Bacchiani, Beaufays, Schalkwyk, Schuster and Strope2008; vlingo.com 2009) use the 1-best output from the speech recognizer (ASR) as the query for the search component. Given that ASR 1-best output is likely to be erroneous, this serialization of the ASR and search components might result in suboptimal search accuracy. As will be shown in our experiments, the oracle word/phrase accuracy using n-best hypotheses is far greater than the 1-best output. However, using each of the n-best hypothesis as a query to the search component is computationally sub-optimal since the strings in the n-best hypotheses usually share large subsequences with each other. A lattice representation of the ASR output, in particular, a word-confusion network (WCN) transformation of the lattice, compactly encodes the n-best hypothesis with the flexibility of pruning alternatives at each word position. Two examples of WCNs are shown in Figures 3 and 4. The weights on the arcs are to be interpreted as costs and the best path in the WCN is the lowest cost path from the start state to the final state. Note that the 1-best path in Figure 4 is how old is mama, while the input speech was how old is obama which also is in the WCN, but at a higher cost.

Fig. 3.

A sample word confusion network for a business listing query.

Fig. 4.

A sample word confusion network with arc costs as negative logarithm of the posterior probabilities.

In order to obtain a measure of the ambiguity per word position in the WCN, we define the (average) arc density of a WCN as the ratio of the total number of arcs to the number of states in the WCN. As can be seen, with very small increase in arc density, the number of paths that are encoded in the WCN can be increased exponentially. In Figure 5, we show the improvement in oracle-path word and phrase accuracies as a function of the arc density for our data set. Oracle-path is a path in the WCN that has the least edit-distance (Levenshtein Reference Levenshtein1966) to the reference string. It is interesting to note that the oracle accuracies can be improved by almost 10% absolute over the 1-best accuracy with small increase in the arc density.

Fig. 5.

Oracle accuracy graph for the WCNs at different arc densities.

4.1 Representing search index as an FST

In order to exploit WCNs for Search, we have implemented our own search engine instead of using an off-the-shelf search engine such as Lucene (Hatcher and Gospodnetic Reference Hatcher and Gospodnetic2004). We index each business listing (d) in our data that we intend to search using the words (w_d) in that listing. The pair (w_d,d) is assigned a weight (c _(wd,d)) using different metrics, including the standard tf * idf, as explained below. This index is represented as a weighted finite-state transducer (SearchFST) as shown in Figure 6 where w_d is the input symbol, d is the output symbol and c _(wd,d) is the weight of that arc.

Fig. 6.

An example of an FST representing the search index related to the business listings repository.

The FST search index for the question answer repository is built as follows. We index each question-answer (QA) pair from our repository ((q_i,a_i), qa_i for short) using the words (w_qi) in question q_i. This index is represented as a weighted finite-state transducer (SearchFST) as shown in Figure 7. Here a word w_qi (e.g old) is the input symbol for a set of arcs whose output symbol is the index of the QA pairs where old appears in the question. The weight of the arc c _(wqi,qi) is one of the similarity based weights discussed in Section 4.1. As can be seen from Figure 7, the words how, old, is and obama contribute a score to the question-answer pair qa25; while other pairs, qa150, qa12, qa450 are scored by only one of these words.

Fig. 7.

An example of an FST representing the search index related to the QA repository.

4.3 Search process using FSTs

Given a user's speech query, the process of obtaining search results is defined by (3) to (8). In these equations, º indicates composition of two finite state transducers (or one transducer and one acceptor); and π₂ indicates that the output of the given transducer is projected (or preserved) to obtain a new automaton.

The user's speech query, after speech recognition (either 1-best or WCN), is represented as an FSA Q. The n-grams of Q and their counts are constructed using the function CountNgrams and the result is represented as another FSA (N_Q) as shown in (3). For this construction, we follow the algorithm presented in Allauzen, Mohri and Roark (Reference Allauzen, Mohri and Roark2003). The result is that, if Q is an unweighted FSA with one path (e.g. a 1-best ASR output), N_Q contains the set of n-grams of Q as paths and the frequencies of those n-grams as the path weights. If Q is a weighted FSA (e.g. a WCN), the paths in the N_Q represent the set of n-grams in the WCN and the path weights correspond to the weighted frequencies of the n-grams of Q.

The weighted FSA N_Q is then composed with the SearchFST (4). For numerical stability, the weights are converted as negative logarithm scores, and the composition operation adds these scores. We retrieve the 1-best path (R ₁) from the result of the composition D ₁ (5). R ₁ represents that particular question with the best score (a combination of acoustic, language model and search score) in the repository that also contains the n-grams in N_Q. We use R ₁ as the rescored output from the ASR, compute its n-grams N _{R 1} (6) and compose N _{R 1} with the SearchFST (7) to obtain all the arcs (w_q,d _wq,c _{(wq,d wq)}) where w_q is a query term, d _wq is a either a QA index or a business listing index containing the query term and, c _{(wq,d wq)} is the weight associated with that pair. Using this information, we aggregate the weight for a document (which is either a QA pair or a business listing) (d _q) across all query words (using fsmdeterminize) and rank the retrieved QAs in the descending order of this aggregated weight.Footnote ³ We select the top N QA pairs (8) from this ranked list.

(3)

(4)

(5)

(6)

(7)

(8)

While the computational complexity of the n-gram counting is O(|q|), where |q| is the number of arcs in q, the composition operation on two FSTs is O(m * n) where m is the number of states of the first FST and n is the number of states of the second FST. Determinization of FSTs in general is O(2^p) where p is the number of states, however, in our approach, the FST D2 has n+1 states for an n-gram based model. We have built this model on over 30 million documents in the SearchFST, without any scalability problems. Extensions to this model using lazy evaluation techniques (Mohri, Pereira and Riley Reference Mohri, Pereira and Riley1998) can be used in cases where scalability becomes an issue.

In Figure 8, we illustrate the result of reranking the WCN shown in Figure 3 using the search relevance weights of each word in the WCN. It must be noted that the least cost pathFootnote ⁴ for the WCN in Figure 3 is ballys automobiles while the reranked 1-best output in Figure 8 is audi automobiles. Given that the user's query was audi automobiles, the listings retrieved from the 1-best output after reranking are much more relevant than those retrieved before reranking, as shown in Table 2.

Table 2.

Listings retrieved for query audi automobiles before and after ASR WCNs were rescored using search relevance weights

Fig. 8.

A WCN rescored using word-level search relevance weights.

The process of retrieving documents using the Levenshtein-based string similarity metric can also be encoded as a composition of FSTs as shown in (9) and 10. The SearchFST encodes the corpus of question-answer pairs as shown in Figure 10. The EditFST (Figure 9) allows for substitution, insertion and deletion of terms and the composition operation (º) allows the matching of the user's query string to the questions in the corpus through edit operations.

(9)

(10)

Fig. 9.

An EditFST encoding the edit operations with substitution, deletion and insertion costs.

Fig. 10.

An example of an FST representing the search index for static queries.

5 Experiments and results

In this section, we present the results of our experiments on tight coupling the ASR and Search components. We show that we not only improve the relevance of search results, but we also improve the speech recognition accuracy. We illustrate this for both the business listing queries as well as the general static queries.

5.1 On business listings queries

We used 852 speech queries collected from users using a mobile device based speech search application. We ran the speech recognizer (Goffin et al. Reference Goffin, Allauzen, Bocchieri, Hakkani-Tur, Ljolje, Parthasarathy, Rahim, Riccardi and Saraclar2005) on these queries using a four-gram language model that is built from the corpus of user queries and created word-confusion networks such as those illustrated in Figures 3 and 4. These 852 utterances were divided into 300 utterances for the development set and 552 for the test set.

5.1.1 ASR experiments

The speech recognizer used to recognize these utterances incorporates an acoustic model adapted to speech collected from mobile devices and a four-gram language model that is built from the corpus of questions. The baseline ASR word and sentence (complete string) accuracies on the development set are 63.1% and 57.0% while those on the test set are 65.1% and 55.3%, respectively.

In Table 3, we summarize the improvements obtained by rescoring the ASR WCNs based on the different metrics used for computing the word scores according to the search criteria. The largest improvement in word and sentence accuracies is obtained by using the rescoring metric: . The word-level accuracy improved from the baseline accuracy of 63.1--63.9% after rescoring while the sentence-level accuracy improved from 57.0% to 58.3%. Thus, this rescoring metric was used to rerank the 552 WCNs in the test set. After rescoring, on the test set, the word-level accuracy improved from 65.1% to 65.9% and sentence-level accuracy improved from 55.3% to 56.2%.

Table 3.

Performance of the metrics used for rescoring the WCNs output by ASR

5.1.2 Search experiments

To analyze the Search accuracy of the baseline ASR output in comparison to the ASR output, reranked using the reranking metric, we used each of the two sets of ASR outputs (i.e. baseline and reranked) as queries to our search engine, SearchFST (described in Section 3). For the search results produced by each set of queries, we computed the precision, recall, and F-score values of the listings retrieved against the listings retrieved by the set of human transcribed queries (Reference). The precision, recall, and F-scores for the baseline ASR output and the reranked ASR output, averaged across each set, is presented in Table 4. For the purposes of this experiment, we assume that the set returned by our SearchFST for the human transcribed set of queries is the reference search set. This is however an approximation for a human annotated search set and it does introduce a bias in the results. In on-going work, we are working on creating a query set annotated with human annotated relevancy judgments.

Table 4.

The recall, precision and F-scores obtained from the baseline ASR output compared to those obtained by the reranked ASR output on the search task

In Table 4, by comparing the search accuracy scores corresponding to the baseline ASR output to those corresponding to the reranked ASR output, we see that reranking the ASR output using the information repository improves both the precision and recall of the search.

5.2 On general static queries

We have a fairly large data set consisting of over a million question-answer pairs collected by harvesting the web. In order to evaluate the retrieval methods discussed earlier, we use two test sets of QA pairs: a Seen set of 450 QA pairs and an Unseen set of 645 QA pairs. The queries in the Seen set have an exact match with some question in the database, while the queries in the Unseen set may not match any question in the database exactly.Footnote ⁵ The questions in the Unseen set, however, like those in the Seen set, also have a human generated answer that is used in our evaluations.

For each query, we retrieve the 20 most relevant QA pairs, ranked in descending order of the value of the particular metric under consideration. However, depending on whether the user query is a seen or an unseen query, the evaluation of the relevance of the retrieved question-answer pairs is different as discussed below.Footnote ⁶

5.2.1 Evaluation metrics

For the set of Seen queries, we evaluate the relevance of the retrieved top-20 question-answer pairs in two ways:

1. (1) Retrieval Accuracy of Top-N results: We evaluate whether the question that matches the user query exactly is located in the top-1, top-5, top-10, top-20 or not in top-20 of the retrieved questions.

2. (2) Coherence metric: We compute the coherence of the retrieved set as the mean of the BLEU-score between the input query and the set of top-5 retrieved questions. The intuition is that we do not want the top-5 retrieved QA pairs to distract the user by not being relevant to the user's query.Footnote ⁷

For the set of Unseen queries, since there are no questions in the database that exactly match the input query, we evaluate the relevance of the top-20 retrieved question-answer pairs in the following way. For each of the 645 Unseen queries, we know the human-generated answer. We manually annotated each unseen query with the Best-Matched QA pair whose answer was the closest semantic match to the human-generated answer for that unseen query. We evaluate the position of the Best-Matched QA in the list of top twenty retrieved QA pairs for each retrieval method.

5.2.2 Results

On the Seen set of queries, as expected the retrieval accuracy scores for the various retrieval techniques performed exceedingly well. The unigram based tf.idf method retrieved 93% of the user's query in the first position, 97% in one of top-5 positions and 100% in one of top-10 positions. All the other retrieval methods retrieved the user's query in the first position for all the Seen queries (100% accuracy).

In Table 5, we tabulate the results of the Coherence scores for the top-5 questions retrieved using the different retrieval techniques for the Seen set of queries. Here, the higher the n-gram the more coherent is the set of the results to the user's query. It is interesting to note that the BLEU-score and Levenshtein similarity driven retrieval methods do not differ significantly in their scores from the n-gram tf.idf based metrics.

Table 5.

Coherence metric results for top-5 queries retrieved using different retrieval techniques for the seen set

In Table 6, we present the retrieval results using different methods on the Unseen queries. For 240 of the 645 unseen queries, the human expert found that there was no answer in the data repository that could be considered semantically equivalent to the human-generated response to that query. So, these 240 queries cannot be answered using the current database. For the remaining 405 unseen queries, over 60% have their Best-Matched question-answer pair retrieved in the top-1 position. We expect the coverage to improve considerably by increasing the size of the QA archive.

Table 6.

Retrieval results for the unseen queries

5.2.3 Speech-driven query retrieval

For this experiment, we use the speech utterances corresponding to the Unseen set as the test set. We use a different set of 250 speech queries as the development set. In Table 7, we show the Word and Sentence Accuracy measures for the best path in the WCN before and after the composition of SearchFST with the WCN on the development and test sets. We note that by integrating the constraints from the search index, the ASR accuracies can be improved by about 1% absolute.

Table 7.

ASR accuracies of the best path before and after (in parentheses) the composition of SearchFST

In order to investigate the impact of WCNs on search accuracy, we computed the search results by integrating the ASR WCNs with the SearchFST on the speech utterances of the Unseen set. As shown in line 5 of Table 6, the integration of the ASR WCNs with the SearchFST produces higher search accuracy (70.12/78.52) compared to ASR 1-best (69.13/75.81). This indicates that the WCNs have information that can be utilized effectively to improve Search retrieval accuracy.

5.2.4 Role of stop words

An important aspect of query-to-question search is the role of stop words. In traditional search, stop words in the query and the document are often filtered out prior to search. However, since our queries are short, directed questions about ten words long, filtering out the stop words leaves very little information content in the query. For example, two questions varying only in tense such as Who is the president of the US? versus Who was the president of the US? are quite different questions, however, filtering out the stop words leaves behind only the words president and US in each of the two queries; hence, both queries may return the same question-answer pairs without any regard to the time-dependency factor inherent in the question. Prompted by such pitfalls of stripping out stop words, we experimented with the relative weighting of stop words and content words. The relative weighting mechanism is only applicable to the unigram based models.

In this experiment, prior to search, we weighted each of the Seen query words using a weighting parameter β as shown below using a predefined list to identify stop words.

(11)

(12)

The parameter β stayed constant for each of the 450 queries in the set of Seen queries.

With this differential weighting, we retrieved the top-20 question-answer pairs using the unigram tf.idf based search. We then computed the rank of the user's query in the list of top twenty retrieved questions. The top-1 retrieval accuracy is shown in the Figure 11 below as we vary β from 0 to 1.

Fig. 11.

The effect of varying the β value on top-1 retrieval accuracy on the Seen data set.

To find the β value that produced the best results (i.e. positioned the location of the exact match as close to the top of the list of retrieved questions), we systematically varied the parameter β from 0 to 1.

6 Finite-state transducer-based parser

In this section, we present a method for parsing the segments of a user's utterance to identify the parameters to be passed to query the information repository. These methods work not only on 1-best ASR output but can transparently scale to take word lattices from ASR as input. We encode the problem of parsing as a weighted finite-state transducer (FST). This encoding allows us to apply the parser on ASR 1-best as well as ASR WCNs using the composition operation of FSTs.

We formulate the parsing problem as associating with each token of the input a label indicating whether that token belongs to one of a search term (st), location term (lt) or neither (null). Thus, given a word sequence (W = w ₁, . . ., w_n) output from ASR, we search of the most likely label sequence (T = t ₁, . . ., t_n), as shown in (13). We use the joint probability P(W,T) and approximate it using an k-gram model as shown in (14) and (15). The sequence w _i−1^i−k+1 represents the k − 1 consecutive words of the history and t _i−1^i−k+1 represents the k − 1 consecutive tags of this history.

(13)

(14)

(15)

A k-gram model can be encoded as a weighted finite-state acceptor (FSA; Allauzen et al. Reference Allauzen, Mohri, Riley and Roark2004). The states of the FSA correspond to the k-gram histories, the transition labels to the pair (w_i,t_i) and the weights on the arcs are −log(P(w_i,t_i | w _i−1^i−k+1,t _i−1^i−k+1)). The FSA also encodes back-off arcs for purposes of smoothing with lower order k-grams. An annotated corpus of words and labels is used to estimate the weights of the FSA. A sample corpus is shown in Table 8.

Table 8.

A Sample set of annotated sentences

The FSA on the joint alphabet is converted into an FST. The paired symbols (w_i,t_i) are reinterpreted as consisting of an input symbol w_i and output symbol t_i. The resulting FST (M) is used to parse the 1-best ASR (represented as FSTs (I)), using composition of FSTs and a search for the lowest weight path, as shown in (16). The output symbol sequence (π₂) from the lowest weight path is T*.

(16)

Equation (16) shows a method for parsing the 1-best ASR output using the FST. However, a similar method can be applied for parsing WCNs. The WCN arcs are associated with a posterior weight that needs to be scaled suitably to be comparable to the weights encoded in M. We represent the result of scaling the weights in WCN by a factor of λ as WCN ^λ. The value of the scaling factor is determined empirically. Thus the process of parsing a WCN is represented by (17).

(17)

7 Experiments

We have access to text query logs consisting of 18 million queries annotated with SearchTerm and LocationTerm labels. In addition to these logs, we have access to 11 million unique business listing names and their addresses. We use the combined data to train the parameters of the two parsing models as discussed in the previous sections. We tested our approaches on three data sets, which in total include 2,686 speech queries. These queries were collected from users using mobile devices from different time periods. Labelers transcribed and annotated the test data using SearchTerm and LocationTerm tags.

We use an ASR with a trigram-based language model trained on the query logs. Table 9 shows the ASR word accuracies on the three data sets. The accuracy is the lowest on Test1, in which many users were non-native English speakers and a large percentage of queries are not intended for business search.

Table 9.

ASR performance on three data sets

We measure the parsing performance in terms of extraction accuracy on the two non-filler slots: SearchTerm and LocationTerm. Extraction accuracy computes the percentage of the test set where the string identified by the parser for a slot is exactly the same as the annotated string for that slot.

In Tables 10 and 11, we report the parsing performance for the FST-based approach. We note that the FST-based parser on a WCN improves the SearchTerm and LocationTerm extraction accuracy over ASR 1-best, an improvement of about 1.5%. The performance under ‘Oracle path’ shows the upper bound for the parser using the oracle pathFootnote ⁸ from the pruned WCN. We pruned WCN by only retaining arcs that are within cthresh (=4 in our experiments) of the lowest cost arc between two states.

Table 10.

SearchTerm extraction accuracy using the FST approach

Table 11.

LocationTerm extraction accuracy using the FST approach

We evaluated the impact of parsing performance on search accuracy. In order to measure search accuracy, we first collected a reference set of search results for our test utterances. For this purpose, we submitted the human annotated two-field data to the search engine (http://www.yellowpages.com/) and extracted the top 5 results from the returned pages. The returned search results are either business categories such as ‘Chinese Restaurant’ or business listings including business names and addresses. We considered these results as the reference search results for our test utterances.

In order to evaluate our voice search system, we submitted the two fields resulting from the query parser on the ASR output (1-best/WCN) to the search engine. We extracted the top 5 results from the returned pages and we computed the Precision, Recall and F1 scores between this set of results and the reference search set. In Table 12, we report the search performance. The overall improvement in search performance is not as large as the improvement in the slot accuracies between using ASR 1-best and WCNs.

Table 12.

Search performances using the FST approach