Evaluating conversation and dialogue is difficult. Researchers have tried to develop evaluation metrics that capture the utility of dialogue, such as task completion, dialogue length, and number of corrections, as well as more qualitative measures such as the appropriateness of individual utterances in the flow of the interaction (cf. Danieli and Gerbino 1995; Walker et al. 1997; Traum, Robinson and Stephan 2004; Webb et al. 2010), but ultimately it often comes down to the kind of analysis used by Justice Potter Stewart of the United States Supreme Court to define obscenity, in the 1964 test case of Jacobilis vs. Ohio, that ‘I know it when I see it’. Often it is only when we observe or participate in good dialogue do we know that it is indeed good.
In that vein then we see this book from Warwick and Shah, a good portion of which is comprised of transcripts of interactions between humans and artificial conversational agents, or chat-bots, where those agents are attempting to fool human judges into believing that the agents are themselves human as part of the so-called Turing Test. This premise has great potential to show people the complexities of dialogue, why it’s hard, and what tricks artificial agents employ to go about attempting to convince humans that they are not, in fact, artificial.
No doubt cashing in on the successful 2014 movie of the same name, Turing’s Imitation Game is a book that purports to have two goals; to introduce the reader to Alan Turing, and his most widely known contribution to artificial intelligence, the Turing Test. Second, to present evidence from a number of organized events where human judges and chat-bots interact, and to demonstrate that the test as described by Turing himself has been passed.
The book starts with a brief overview of Turing, the man, discussing his time at Bletchley, and membership of the Ratio Club, a dining club from 1949 to 1954. It begins with approximately a page of text that summarizes his work on the Enigma encoder at Bletchley Park, and then the following six pages deal with his membership in the Ratio Club, consisting mostly of a series of peripherally related interviews with fellow members of the club. This then is not an in-depth biography, more a gentle introduction to both Turing, and the context of the time he was operating in.
Part one proper of the book then covers Turing’s central ideas in his seminal paper on machine thinking and intelligence (Turing 1950), a brief introduction to Artificial Intelligence (AI), the controversy around the imitation game as proposed by Turing, a historical account of conversational systems, and matters arising from early Turing Tests. That all this information is presented in 70 pages means that the material is not as fleshed out in as much detail as it perhaps deserves.
To cover this material, to present a story for the layperson about the key ideas Turing was looking to address with his test, and to examine the work that has been inspired by it, is both possible and extremely desirable. There is renewed interest in AI in general, and a rise in the use of language interaction with the emergence of Siri, the Google Assistant and Amazon Alexa. However, this book is too often caught between the layperson’s market and the language of the academic. While some of the explanation is clear, often the language used is too academic, too opaque to be much use to a layperson. Perhaps a better target audience is an undergraduate or graduate student in computer science or some related field, who will already have familiarity with concepts introduced in this book, such Godel’s theorem, and prior knowledge of the views of others such as Marvin Minsky and Mark Halpern on the Turing test itself.
For example, the overview section on AI, in Chapter 3, is both very short, and presents only a subset of topics comprising AI. This is probably not sufficient to introduce a layperson to the topic, but can serve as a launch point for deeper investigation should the reader be so motivated. The section on natural language processing for instance, a key technology in chat-bot creation, is just over one-page long, and largely consists of a single sentence each on co-reference resolution, discourse analysis, information retrieval, named entity recognition, part-of-speech tagging, parsing, question-answering, relationships and word sense disambiguation (in that order). Using the transcripts of the chat-bots that appear later in the text, the authors could have illustrated the nature of dialogue, in terms of what’s good, what’s bad, and how those are achieved using computational approaches, but unfortunately that opportunity is missed.
In his original paper, Turing presented nine ‘arguments’ against the validity of his proposed test. In this text, only the argument from disability is highlighted, but gives us some insight into philosophical objections to the test. For a comprehensive overview of those objections, and the rather more recent potential objections to the test as presented, see the excellent overview of AI by Copeland (Reference Copeland1993). In any case, the goal of part one of this new book seems clear. The authors want us to believe that the test as proposed by Turing is a valid and important test of intelligence.
Having established to their satisfaction the importance of the Turing test, part two of the book is an account of three live trials between judges and chat-bots, or three Turing Tests as the authors prefer. In fact, what we see are three instances of variations of the Loebner prize. The Loebner prize is mentioned, but never fully explained, which is unfortunate. Missing too is any critique of the Loebner competition. Instead, we have a very thorough, test by test, transcript by transcript walk through some of the misclassifications made by judges of particular artificial agents.
This had the potential to be an extremely valuable contribution, and it is interesting to see the interactions between agents and judges. Linking these observations back to previous sections on challenges to the test, or in developing conversational agents, would have significantly elevated the text. Instead, the denouement of the book centers on the predictions of one of the authors that the initial predictions of Turing, that an artificial agent would fool 30% of its interrogators, occurred at a final test at Bletchley Park in 2014, with multiple systems interacting with multiple judges in a round-robin test of conversational ability.
Spoiler alert. According to the authors, this 30% threshold is achieved, and a system passes the Turing Test as defined by Turing himself. As the authors themselves say: ‘It is clear that the standard reached by machine communication is now up to the level indicated by Alan Turing in his 1950 paper.’ (p. 191).
Certainly as presented in this book, it is clear that some number of human judges misclassified one particular system a number of times. These claims have been made before; on several occasions, judges have been fooled by these kinds of entities, including the well-known Cleverbot in 2011, and at least apocryphally the most famous system, Eliza (Weizenbaum Reference Weizenbaum1966). The question of the significance of this event is not really addressed. Society and expectation have changed to the point where it seems probable that Turing himself would be underwhelmed by the conversational capability of the modern chat-bot. Performing under the strict limitations imposed by a 1950s thought exercise, today’s artificial agents are able, just, to fool some number of humans throughout a short exercise. But, when it is equally true that the same artificial conversational agents would fail to perform in a 5-min chat in the pub about what was on TV the previous evening, without attracting derision and ridicule, then is the ability to pass a test such as this impactful?
This book is a welcome attempt to explain a complex philosophical and computational problem, and address the question of how we might judge intelligence in today’s highly connected, interactive world. By focusing on the exact substance of the test, and not the intention, and by using language that situates it too far from the layperson’s perspective, the book ultimately falls short of its potential.
