1.1 Historical development

Early QA systems were developed to enable users to ask increasingly complex queries about well-structured data sets, such as baseball statistics (Green et al., Reference Green, Wolf, Chomsky and Laughery1961) and analyses of rock and soil samples returned from the Apollo lunar missions (Woods, Kaplan and Nash-Webber, Reference Woods, Kaplan and Nash-Webber1972). Given the complexity of the queries and the fact that users were often unaware of the underlying database, early QA development focussed on such issues as handling questions that could not be parsed or that could not be interpreted as a valid database query; correctly resolving syntactic and referential ambiguity in user questions; handling differences in how user and system conceptualized the domain (e.g., user queries about the age of lunar rocks versus data on potassium/rubidium ratios, as well as differences between what user and system believed to be true in the domain (Kaplan, Reference Kaplan, Brady and Berwick1982; Mays, Joshi and Webber, Reference Mays, Joshi and Webber1982; Pollack, Reference Pollack1986; Webber, Reference Webber, Brodie and Mylopoulos1986) and identifying, in the case of distributed databases, what information needed to be imported from where, in order to answer the user's question (Hendrix et al., Reference Hendrix, Sacerdoti, Sagalowicz and Slocum1978).

While devising interactions to handle mismatches between user and system beliefs about the domain was a step toward IQA systems (Section 1.3), the enterprise of database QA was essentially abandoned in the late 80's for lack of audience take-up: Ordinary people lacked access to large data sets, and managers whose companies maintained them lacked sufficient interest. Companies such as Symantic ended up abandoning their software for then state-of-the-art database QA (Hendrix, Reference Hendrix1986).

With the advent of the web, came ever increasing amounts of information that people wanted to access. Search methods pioneered in document retrieval (a long-stagnant field newly revitalized by the web) became commonplace. And QA too found new life, with the idea that users would find not just relevant web pages or documents, but the particular information they were seeking.

This new vision of QA is very different from QA over databases, where questions map onto a computation carried out over a database – for example, computing whether some property held of some set (e.g., ‘Are all employees enrolled in a health plan?’), computing property values (e.g., ‘What is the average concentration of iron in ilmenite?’), computing which entities had some possibly complex set of properties (e.g., ‘Samples that have greater than 13 percent olivine’), etc. In contrast, within the current QA paradigm, the answer to a question is meant to be found rather than computed, and QA became viewed as natural extension to information retrieval (IR), accelerated by its adoption as a track within the Text Retrieval Conference (TREC).

Initially, TREC QA focused on factoid questions – those questions of a specific type, such as who, what and where, that can be answered by a short word or phrase. From 1999 to 2007, TREC QA advanced to address increasingly large document collections, increasingly complex questions and ever more complex evaluation strategies. While the basic approach to factoid QA is now well understood, challenges remain, including the fact that performance in answering list questions (identifying all entities that have some property), is notoriously poor compared to single-entity factoid questions. A second challenge is the fact that the same answer may be conveyed in so many different ways, even within a single sentence, that even using lexical resources to extend search based on pattern matching may not help find answers (Kaisser and Webber, Reference Kaisser and Webber2007). Shallow inference based on a long-tail of infrequent patterns may help in recognizing answers, but it is not yet clear how this could be exploited in finding answer candidates. A third challenge is the fact that current techniques cannot identify answers whose evidence is distributed over multiple sentences (either over more than one adjacent sentence or over an arbitrary set of sentences from across the corpus, comparable to joins in database QA. Bouma et al., Reference Bouma, Fahmi, Mur, van Noord, van der Plas and Tiedemann2005 have called these ‘which’ questions, as in ‘Which ferry sank southeast of the island Utö?’, which requires combining evidence that some entity sank southeast of the island Utö with evidence that the entity is a ferry. Even factoid questions still present a challenge, such as ones involving superlatives (e.g., the most, the worst, the largest, etc.) or averages that can only be answered through a calculation that has not yet been performed or whose value depends on when it is performed, such as relative dates. As noted before, current methods are limited to cases where the answer has already been computed and indexed by the terms used in the question.

Moreover, development of QA inside an IR program has meant that preconceptions from IR have carried over into the present QA system development, including the idea of the perfect query (with respect to the user's desired information) – the query that the user would have asked if only they were familiar with the underlying data. This idea underlies recent research in interactive IR (Dumais and Belkin, Reference Dumais, Belkin, Voorhees and Harman2005), which focuses on the relevance of feedback techniques, to allow the user to modify the query, over some iterations. While this is a form of interaction, it does not reflect the potential for users to change what they want to know as they learn more. One important class of users for whom this is true and whose needs are being addressed through work on complex QA and the ARDA AQUAINT program, a US research program that is driving the development of analytical QA, are professional information analysts employed by government and industry. Their questions are not factoid in nature and cannot be satisfied by phrases or one word answers. Complex questions (such as what are the long term implications of China's one child policy?) require sophisticated approaches for the identification of relevant information, which may involve hypotheses and their consequences, analogies and comparisons, none of which plays any part in factoid or even definition QA. These needs of these users can rarely be satisfied through one or more independent questions.

1.2 Dialogue systems

Whilst the idea of continuous Natural Language interaction with users is fairly new to QA, the idea has had a significant history of its own, going back almost as long, under the rubric dialogue systems. In one of the earliest systems, SHRDLU (Winograd, Reference Winograd, Schank and Colby1973), users engage in dialogue with an animated or robotic agent that is capable of various actions and ‘aware’ of its own behavior, including its interaction with the user. As such, dialogues involve the user asking or telling the system to perform some action or achieve some goal; the system accepting the request/command or rejecting it (along with an explanation); the system asking for clarification of some aspect of the request/command; the user responding with clarifying description; the user asking the system questions about the state of its world or its previous actions or its plans and the system responding with an appropriate description.

Interactions with SHRDLU and with Vere and Bickmore's Basic Agent (Vere and Bickmore, Reference Vere and Bickmore1990) were through typed text, where later systems supported limited spoken interaction (Allen et al., Reference Allen, Miller, Ringger and Sikorski1996; Lemon et al., Reference Lemon, Bracy, Gruenstein and Peters2001; Eliasson, Reference Eliasson2007). Because these systems do not have any goals of their own, apart from those adopted in response to user requests/commands, and because no way was provided for users to collaborate with them, dialogues capabilities lack comparable questions, requests, or any initiative from the system.

More recent dialogue systems have played the tutor role in Intelligent Tutoring Systems. Here, it is the system that has goals – for example, assessing the student's knowledge, correcting the student's errors and imparting information that the student is missing. Dialogues can thus involve the system introducing and describing a topic for tutoring; asking the student about some aspect of the problem; explaining why the student's response is correct or incorrect and/or reminding the student of something already said previously during the interaction, with (in all cases), the student replying with an answer or saying s/he does not know. Again, the earliest tutoring systems, like Sophie (Brown and Burton, Reference Brown, Burton, Bobrow and Collins1975), interacted through typed text, while later systems such as ITSPOKE (Litman and Forbes-Riley, Reference Litman and Forbes-Riley2006) allow for spoken interaction.

The most industrially relevant role played by dialogue systems is in information provision and user assistance, such as in helping users to book flights (Hemphill, Godfrey and Doddington, Reference Hemphill, Godfrey and Doddington1990; Walker, Passonneau and Boland, Reference Walker, Passonneau and Boland2001; Seneff, 2004; Demberg and Moore, Reference Demberg and Moore2006) or find a place to have dinner (Walker, Whittaker and Stent, Reference Walker, Whittaker and Stent2004) routing user telephone calls to the appropriate staff person (Gorin, Riccardi and Wright, Reference Gorin, Riccardi and Wright1997; Chu-Carroll and Carpenter, Reference Chu-Carroll and Carpenter1999) or handling directory enquiries (De Roeck et al., Reference De Roeck, Kruschwitz, Scott, Steel, Turner and Webb2000; Lehtinen et al., Reference Lehtinen, Safra, Gauger, Kaspar, Pardo and Louloudis2000). All such tasks can be viewed in terms of getting the user to fully or partially instantiate some frame, which the system then evaluates, presenting the results to the user as a basis for further interaction. In such cases, the user's goal may be anywhere from completely formed to vague. It may or may not be possible to achieve exactly as specified, and as such, may need to be reformulated on the basis of additional knowlege and relaxed constraints. Dialogues can thus involve the system asking the user questions related to values of frame elements; the user specifying such values (either precisely or vaguely); the system listing and/or describing the results (when too numerous to list); the user choosing some item from among the results or modifying or replacing some already specified values and the system requesting confirmation of its understanding.

Recent research on dialogue systems has involved widening their dialogue capabilities (Demberg and Moore, Reference Demberg and Moore2006), using data-driven models of dialogue (Hardy et al., Reference Hardy, Biermann, Inouye, Mckenzie, Strzalkowski, Ursu, Webb and Wu2004), learning optimal dialogue strategies for a given task (Henderson, Lemon and Georgila, Reference Henderson, Lemon and Georgila2008) and exploiting formal, semantic approaches to information update (Traum and Larsson, Reference Traum, Larsson, vanKuppevelt and Smith2003). A key-emerging element of dialogue approaches is their inherent generality – the potential for subdialogue structures independent of task or application (such as for error correction or clarification) that are vital for allowing users to explore open collections of data.

1.3 IQA

The emerging paradigm of IQA can be placed at the intersection of these two research directions in QA and dialogue systems. We define IQA as a process where the user is a continual part of the information loop – as originator of the query, as arbitrator over information relevance and as consumer of the final product. This mode of operation is useful for both factoid and analytical or complex QA, but perhaps provides greatest benefit in those cases where information need is as yet vague, or is comprised of multifaceted complex concepts, or contains user misconceptions or subjective opinions – all cases where the expected information content returned is complex and contains a greater degree of variability or ambiguity. IQA systems borrow from dialogue systems, their interaction, emphasis on completion of user task, their handling of incomplete or underspecified (as well as overspecified) user input and the constraint and relaxation phases of the query process, whilst remaining focussed on large or open domains, such as the Internet.

In June 2006 we organised a workshop on IQA, in conjunction with the Human Language Technology (HLT) 2006 conference, in Brooklyn, NY, USA. The six papers in this special issue are expanded, reviewed and significantly revised versions of papers from among those presented there.