Smart Bots?

The A-IQ project evaluates conversational artificial intelligence (AI) performance through a behavioural perspective, measuring observable external behaviours rather than internal processes considering algorithms developed pre-generative AI. This approach is valuable for applied research targeting end-user perception and where it is not necessary to consider internal processes. The Interdisciplinary Artificial Intelligence Model was developed, consisting of seven domains to solve problems, from which the Interdisciplinary Artificial Intelligence Quotient Scale (iAIQs Scale) was created. The iAIQs Scale contains 62 questions that are evaluated by human testers through a multi-level system of response categories. The A-IQ tests were conducted on Google Now (2018 release) and Google Assistant (2021 release), Siri (Apple), Cortana (Microsoft) and Alexa (Amazon). The results revealed that Siri had the best overall performance due to high scores in the working memory domain, while Cortana scored highest in explicit knowledge. However, no conversational AI scored in the critical or creative thinking domains. Testing, conducted in 2021 (pre-ChatGPT release), showed improvements in Google Assistant’s performance, followed by Alexa, while Siri showed minimal improvements. A prototype was developed to automate the testing process and facilitate continuous monitoring. Limitations were identified regarding reproducibility and objectivity, but the A-IQ project contributes to the evolving field of human–machine interaction, focusing on communication. This chapter focuses specifically on conversational agents developed prior to the release of ChatGPT, with generative AI systems examined in Chapter 15.

Keywords

artificial intelligence behavioural perspective interdisciplinary AI domains conversational AI performance monitoring algorithmic IQ test

Information

Type: Chapter
Information: The Cambridge Handbook of Behavioural Data Science , pp. 234 - 263
DOI: https://doi.org/10.1017/9781108939010.019 [Opens in a new window]
Publisher: Cambridge University Press
Print publication year: 2026

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

AI in the UK (2018). AI in the UK: Ready, willing and able? Report of Session 2017–19. Select Committee on Artificial Intelligence, HL Paper 100, The Authority of the House of Lords.Google Scholar

The bAbl project (latest update 2015). https://github.com/facebookarchive/bAbI-tasks (last accessed 3 October 2025).Google Scholar

Ball, O. E. & Torrance, E. P. 1984. Torrance tests of Creative Thinking streamlined scoring workbook, figural A. Scholastic Testing Service.Google Scholar

BBC News (2015). Intelligent Machines: AI had IQ of four-year-old child. Available at: www.bbc.com/news/technology-34464879 (last accessed 3 October 2025).Google Scholar

Dunn, J. (2017). It looks like Apple has some work to do if it wants Siri to be as smart as Google Assistant. Available at: http://uk.businessinsider.com/siri-vs-google-assistant-cortana-alexa-knowledge-study-chart-2017-6?r=US&IR=T (last accessed 4 October 2017).Google Scholar

Feng, L. & Shi, Y. (2014). The Search Engine IQ Test based on the Internet IQ Evaluation Algorithm. Procedia Computer Science, 31, 1066–1073.Google Scholar

Hendler, J. (2020). Conversational AI can propel social stereotypes. Available at: www.wired.com/story/opinion-conversational-ai-can-propel-social-stereotypes/ (last accessed 3 October 2025).Google Scholar

Hernández-Orallo, J. (2016). Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artificial Intelligence Review, 48, 1–51. doi:10.1007/s10462-016-9505-7.Google Scholar

Jiang, J., Awadallah, A. H., Jones, R., Ozertem, U., Zitouni, I., Kulkarni, R. G., et al. (2015). Automatic online evaluation of intelligent assistants, in Proceedings of the 24th International Conference on World Wide Web (WWW ‘15), Florence, Italy, 18–22 May, 506–516.10.1145/2736277.2741669CrossRef Google Scholar

Kaufman, J. & Sternberg, R. (2010). The Cambridge handbook of creativity. Cambridge University Press.10.1017/CBO9780511763205CrossRef Google Scholar

Knuth, D. (1972). The art of computer programming, Volume 3: Sorting and searching. Addison-Wesley.Google Scholar

Kostopoulos, L. (2020) Report: Decoupling human characteristics from algorithmic capabilities. Available at: https://standards.ieee.org/initiatives/artificial-intelligence-systems/decoupling-human-characteristics.html (last accessed 3 October 2025).Google Scholar

Legg, S. & Hutter, M. (2007). Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4), 391–444.10.1007/s11023-007-9079-xCrossRef Google Scholar

Liu, F. & Shi, Y. (2014). The Search Engine IQ Test based on the Internet IQ Evaluation Algorithm, Proceedings of the Secondi. International Conference on Information Technology and Quantitative Management, Procedia Computer Science, 31, 1066–1073.10.1016/j.procs.2014.05.361CrossRef Google Scholar

Liu, F., Shi, Y. & Liu, Y. (2017). Intelligence quotient and intelligence grade of artificial intelligence. Annals of Data Science, 4(2), 179–191.10.1007/s40745-017-0109-0CrossRef Google Scholar

Monett, D. & Lewis, C. W. P. (2018). Getting clarity by defining artificial intelligence: A survey, in Philosophy and Theory of Artificial Intelligence 2017, edited by Müller, V. C., SAPERE 44. Springer, 212–214.10.1007/978-3-319-96448-5_21CrossRef Google Scholar

Monett, D. & Lewis, C. W. P. (2020). Definitional foundations for intelligent systems, part I: Quality criteria for definitions of intelligence. In Proceedings of the 10th Anniversary Conference of the Academic Conference Association, AC 2020, edited by Vopava, J., Douda, V., Kratochvil, R., & Konecki, M.. MAC Prague Consulting Ltd, 73–80.Google Scholar

Monett, D., Lewis, C. W. P. & Thórisson, K. R. (eds.) (2020). Special Issue ‘On Defining Artificial Intelligence’ commentaries and author’s response. Journal of Artificial General Intelligence, 11(2), 1–100.10.2478/jagi-2020-0003CrossRef Google Scholar

Nilsson, N. J. (2010). The quest for artificial intelligence: A history of ideas and achievements. Cambridge University Press.Google Scholar

PWC (2019). Voice first. Available at: www.pwc.in/assets/pdfs/research-insights/2019/voice-first.pdf (last accessed 3 October 2025).Google Scholar

Russell, S. & Norvig, P. (2012). Künstliche Intelligenz. Ein moderner Ansatz. 3., aktualisierte Auflage. Artificial Intelligence. A modern Approach. Pearson Education.Google Scholar

Sarikaya, R., Crook, P. A., Marin, A., et al. (2016). An overview of end-to-end language understanding and dialog management for personal digital assistants, in 2016 IEEE Spoken Language Technology Workshop (SLT), IEEE, 391–397. https://doi.org/10.1109/SLT.2016.7846294.CrossRef Google Scholar

Statista (2019). Worldwide intelligent/digital assistant market share in 2017 and 2020, by product. Available at: www.statista.com/statistics/789633/worldwide-digital-assistant-market-share/ (last accessed 3 October 2025).Google Scholar

Sternberg, R. J. (1994). Encyclopedia of human intelligence. Macmillan.Google Scholar

Sternberg, R. J. & Detterman, D. K. (1986). What is intelligence? Contemporary viewpoints on its nature and definition. Ablex Publishing Corporation.Google Scholar

Stone, P., Brooks, R., Brynjolfsson, E., et al. (2016). Artificial intelligence and life in 2030. One hundred year study on artificial intelligence: Report of the 2015–2016 study panel. Stanford University. http://ai100.stanford.edu/2016-report.Google Scholar

Tschopp, M. (2018). Psychology of artificial intelligence: Foundations, range and implications from a humanities perspective. Scip labs.Google Scholar

Vallati, M. (2020). Will AI take over? Quantum theory suggests otherwise. Available at: https://theconversation.com/will-ai-take-over-quantum-theory-suggests-otherwise-126567 (last accessed 3 October 2025).Google Scholar

Wang, P. (1995). Non-axiomatic reasoning system: Exploring the essence of intelligence. PhD Dissertation, Indiana University. Available at: https://cis.temple.edu/~pwang/Publication/thesis.pdf (last accessed 3 October 2025).Google Scholar

Wang, P. (2020). On defining artificial intelligence: Author’s response to commentaries. Journal of Artificial General Intelligence, 11(2), 73–86.Google Scholar

Warren, T. (2018). Microsoft no longer sees Cortana as an Alexa or Google Assistant competitor. Available at: www.theverge.com/2019/1/18/18187992/microsoft-cortana-satya-nadella-alexa-google-assistant-competitor (last accessed 3 October 2025).Google Scholar

Waytz, A., Epley, N. & Cacioppo, J. T. (2010). Social cognition unbound: Insights into anthropomorphism and dehumanization. Current Directions in Psychological Science, 19(1), 58–62. https://doi.org/10.1177/0963721409359302.CrossRef Google Scholar PubMed

Weston, J., Bordes, A., Chopra, S. & Mikolov, T. (2015). Towards AI-complete question answering: a set of prerequisite toy tasks. Available at: http://arxiv.org/abs/1502.05698 (last accessed 3 October 2025).Google Scholar

Zimbardo, P., Gerrig, R. & Graf, R. (2008). Psychologie. 16, aktualisierte Auflage. Pearson Education.Google Scholar

Accessibility standard: WCAG 2.2 AAA

Why this information is here

This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.

Accessibility Information

The HTML of this chapter complies with version 2.2 of the Web Content Accessibility Guidelines (WCAG), offering more comprehensive accessibility measures for a broad range of users and attains the highest (AAA) level of WCAG compliance, optimising the user experience by meeting the most extensive accessibility guidelines.

Content Navigation

Table of contents navigation
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.

Index navigation
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.

Reading Order & Textual Equivalents

Single logical reading order
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.

Short alternative textual descriptions
You get concise descriptions (for images, charts, or media clips), ensuring you do not miss crucial information when visual or audio elements are not accessible.

Full alternative textual descriptions
You get more than just short alt text: you have comprehensive text equivalents, transcripts, captions, or audio descriptions for substantial non‐text content, which is especially helpful for complex visuals or multimedia.

Visualised data also available as non-graphical data
You can access graphs or charts in a text or tabular format, so you are not excluded if you cannot process visual displays.

Visual Accessibility

Use of colour is not sole means of conveying information
You will still understand key ideas or prompts without relying solely on colour, which is especially helpful if you have colour vision deficiencies.

Use of high contrast between text and background colour
You benefit from high‐contrast text, which improves legibility if you have low vision or if you are reading in less‐than‐ideal lighting conditions.

Book contents

14 - Smart Bots?

Summary