Skip to main content Accessibility help
×
Hostname: page-component-74656f9884-2dmjx Total loading time: 0 Render date: 2026-11-09T12:08:01.355Z Has data issue: false hasContentIssue false

2 - Can AI Be Explainable?

Published online by Cambridge University Press:  24 July 2026

By
Simone Stumpf
Edited by
Daniele Quercia  and
Marios Constantinides
Daniele Quercia
Affiliation:
NOKIA, Cambridge
Marios Constantinides
Affiliation:
NOKIA, Cambridge
Get access

Summary

This chapter provides an introduction to Explainable AI (XAI), its purposes, key concepts, and terminology, and its importance to Responsible AI. It discusses the aim of explanations to improve understanding and to calibrate trust and reliance, leading to improved system use, and how this contributes to improved transparency in Responsible AI. It describes global and local explanations and how information in explanations can be presented in various ways. The current state of the art in XAI techniques, such as LIME, SHAP, Counterfactual explanations, Grad-CAM, and their limitations, are presented. Multiple perspectives on explanations from computer science, HCI, and social science that question the scope of explanations and how explanations are generated are discussed. As a consequence, opportunities and challenges for the application of XAI in Responsible AI, such as XAI in fairness and bias, the development process, and technological advances, are identified. Reflections are offered on why explanations are not commonly employed in Responsible AI so far, in order to set future research directions and encourage the removal of barriers standing in the way to its successful adoption in practice.

Keywords

explainable artificial intelligenceinterpretable machine learning

Information

Type
Chapter
Information
Operationalizing Responsible AI , pp. 25 - 41
Publisher: Cambridge University Press
Print publication year: 2026

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Aamodt, A., and Plaza, E. 1994. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Communication, 7, 39–59.CrossRefGoogle Scholar
Ahmed, S., Balasubramanian, H., Stumpf, S., Morrison, C., Sellen, A., and Grayson, M. 2020. Investigating the intelligibility of a computer vision system for blind users. Proceedings of the 25th International Conference on Intelligent User Interfaces (New York: Association for Computing Machinery), pp. 419–429. https://doi.org/10.1145/3377325.3377508.CrossRefGoogle Scholar
Alvarez Melis, D., and Jaakkola, T. 2018. Towards robust interpretability with self-explaining neural networks. 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.Google Scholar
Baum, D., Baum, K., Gros, T. P., and Wolf, V. 2023. XAI requirements in smart production processes: A case study. In Longo, L (ed.), Explainable Artificial Intelligence (Cham: Springer Nature Switzerland), pp. 3–24. https://doi.org/10.1007/978-3-031-44064-9_1.CrossRefGoogle Scholar
Bibi, N., Courtney, J., and Curran, K. M. 2024. Enhancing multiple sclerosis diagnosis with eXplainable AI. IET Conference Proceedings, 2024, 218–225. https://doi.org/10.1049/icp.2024.3308.CrossRefGoogle Scholar
Bove, C., Lesot, M.-J., Tijus, C. A., and Detyniecki, M. 2023. Investigating the intelligibility of plural counterfactual examples for non-expert users: An explanation user interface proposition and user study. Proceedings of the 28th International Conference on Intelligent User Interfaces (New York: Association for Computing Machinery), pp. 188–203. https://doi.org/10.1145/3581641.3584082.CrossRefGoogle Scholar
Bunt, A., Lount, M., and Lauzon, C. 2012. Are explanations always important?: A study of deployed, low-cost intelligent interactive systems. Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces (New York: Association for Computing Machinery), pp. 169–178. https://doi.org/10.1145/2166966.2166996.CrossRefGoogle Scholar
Cambria, E., Malandri, L., Mercorio, F., Nobani, N., and Seveso, A. 2024. XAI meets LLMs: A survey of the relation between explainable AI and large language models. arXiv preprint arXiv:2407.15248.Google Scholar
Conati, C., Barral, O., Putnam, V., and Rieger, L. 2021. Toward personalized XAI: A case study in intelligent tutoring systems. Artificial Intelligence, 298, 103503. https://doi.org/10.1016/j.artint.2021.103503.CrossRefGoogle Scholar
Doshi-Velez, F., and Kim, B. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.Google Scholar
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., and Weisz, J. D. 2021. Expanding explainability: Towards social transparency in AI systems. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery), pp. 1–19.Google Scholar
Explainable AI in Fraud Detection. (n.d.). www.milliman.com/en/insight/Explainable-AI-in-fraud-detection (last accessed June 9, 2025).Google Scholar
Gunning, D., and Aha, D. 2019. DARPA’s Explainable Artificial Intelligence (XAI) Program. AI Magazine, 40, 44–58. https://doi.org/10.1609/aimag.v40i2.2850.CrossRefGoogle Scholar
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., and Yang, G.-Z. 2019. XAI: Explainable artificial intelligence. Science Robotics, 4(37). https://doi.org/10.1126/scirobotics.aay7120.CrossRefGoogle ScholarPubMed
Hasling, D. W., Clancey, W. J., and Rennels, G. 1984. Strategic explanations for a diagnostic consultation system. International Journal of Man-Machine Studies, 20, 3–19. https://doi.org/10.1016/S0020-7373(84)80003-6.CrossRefGoogle Scholar
Herlocker, J. L., Konstan, J. A., and Riedl, J. 2000. Explaining collaborative filtering recommendations. Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (Philadelphia: Association for Computing Machinery), pp. 241–250. https://doi.org/10.1145/358916.358995.CrossRefGoogle Scholar
Hoffman, R. R., Mueller, S. T., Klein, G., and Litman, J. 2023. Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Frontiers in Computer Science, 5, 1–15. https://doi.org/10.3389/fcomp.2023.1096257.CrossRefGoogle Scholar
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., et al. 2019. The (un)reliability of saliency methods. In Samek, W, Montavon, G, Vedaldi, A, Hansen, L. K, and Muller, K.-R (eds.), Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (Cham: Springer International Publishing), pp. 267–280. https://doi.org/10.1007/978-3-030-28954-6_14.CrossRefGoogle Scholar
Kulesza, T., Burnett, M., Wong, W.-K., and Stumpf, S. 2015. Principles of explanatory debugging to personalize interactive machine learning. Proceedings of the 20th International Conference on Intelligent User Interfaces (New York: Association for Computing Machinery), pp. 126–137. https://doi.org/10.1145/2678025.2701399.CrossRefGoogle Scholar
Kulesza, T., Stumpf, S., Burnett, M., and Kwan, I. 2012. Tell me more?: The effects of mental model soundness on personalizing an intelligent agent. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems (New York: Association for Computing Machinery), pp. 1–10. https://doi.org/10.1145/2207676.2207678.Google Scholar
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., and Wong, W. 2013. Too much, too little, or just right? Ways explanations impact end users’ mental models. 2013 IEEE Symposium on Visual Languages and Human Centric Computing, pp. 3–10. https://doi.org/10.1109/VLHCC.2013.6645235.CrossRefGoogle Scholar
Lee, J. D., and See, K. A. 2004. Trust in automation: Designing for appropriate reliance. Human Factors, 46, 50–80. https://doi.org/10.1518/hfes.46.1.50-30392.CrossRefGoogle ScholarPubMed
Liao, Q. V., Gruen, D., and Miller, S. 2020. Questioning the AI: Informing design practices for explainable AI user experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery), pp. 1–15. https://doi.org/10.1145/3313831.3376590.Google Scholar
Lim, B. Y., and Dey, A. K. 2010. Toolkit to support intelligibility in context-aware applications. Proceedings of the 12th ACM International Conference on Ubiquitous Computing (New York: Association for Computing Machinery), pp. 13–22. https://doi.org/10.1145/1864349.1864353.CrossRefGoogle Scholar
Lim, B. Y., and Dey, A. K. 2011. Investigating intelligibility for uncertain context-aware applications. Proceedings of the 13th International Conference on Ubiquitous Computing (New York: Association for Computing Machinery), pp. 415–424. https://doi.org/10.1145/2030112.2030168.CrossRefGoogle Scholar
Lim, B. Y., Dey, A. K., and Avrahami, D. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. Proceedings of the 27th International Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery), pp. 2119–2128. https://doi.org/10.1145/1518701.1519023.Google Scholar
Lundberg, S. M. and Lee, S.-I. 2017. A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems (Red Hook, NY: Curran Associates Inc.), pp. 4768–4777.Google Scholar
Maltbie, N., Niu, N., Van Doren, M., and Johnson, R. 2021. XAI tools in the public sector: a case study on predicting combined sewer overflows. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York: Association for Computing Machinery), pp. 1032–1044. https://doi.org/10.1145/3468264.3468547.CrossRefGoogle Scholar
Miller, T. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38. https://doi.org/10.1016/j.artint.2018.07.007.CrossRefGoogle Scholar
Mojsilovic, A. 2021. Introducing AI Explainability 360. IBM Research. https://research.ibm.com/blog/ai-explainability-360 (last accessed June 9, 2025).Google Scholar
Molnar, C. 2020. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (Lean Publishing).Google Scholar
Mothilal, R. K., Sharma, A., and Tan, C. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (New York: Association for Computing Machinery), pp. 607–617. https://doi.org/10.1145/3351095.3372850.CrossRefGoogle Scholar
Nimmo, R., Constantinides, M., Zhou, K., Quercia, D., and Stumpf, S. 2024. User characteristics in explainable AI: The rabbit hole of personalization? Proceedings of the CHI Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery), pp. 1–13. https://doi.org/10.1145/3613904.3642352.Google Scholar
OpenXAI. n.d. OpenXAI. https://openxai.github.io/ (last accessed June 9, 2025).Google Scholar
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., et al. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8779–8788. https://doi.org/10.1109/CVPR.2018.00915.CrossRefGoogle Scholar
Porfirio, R. P., Santos, P. A., and Madeira, R. N. 2024. Enhancing digital agriculture with XAI: Case studies on tabular data and future directions. Companion Proceedings of the 26th International Conference on Multimodal Interaction (New York: Association for Computing Machinery), pp. 211–217. https://doi.org/10.1145/3686215.3689201.CrossRefGoogle Scholar
Ribeiro, M. T., Singh, S., and Guestrin, C. 2016. “Why should I trust you?”: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York: Association for Computing Machinery), pp. 1135–1144. https://doi.org/10.1145/2939672.2939778.CrossRefGoogle Scholar
Saarela, M., and Podgorelec, V. 2024. Recent applications of explainable AI (XAI): A systematic literature review. Applied Sciences, 14, 8884. https://doi.org/10.3390/app14198884.CrossRefGoogle Scholar
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. 2019. Grad-CAM: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision. https://doi.org/10.48550/arXiv.1610.02391.CrossRefGoogle Scholar
Shneiderman, B. 2021. Responsible AI: Bridging from ethics to practice. Communications of the ACM, 64, 32–35.CrossRefGoogle Scholar
Shneiderman, B., and Maes, P. 1997. Direct manipulation vs. interface agents. Interactions, 4, 42–61. https://doi.org/10.1145/267505.267514.CrossRefGoogle Scholar
Shortliffe, E. H., Davis, R., Axline, S. G., Buchanan, B. G., Green, C. C., and Cohen, S. N. 1975. Computer-based consultations in clinical therapeutics: Explanation and rule acquisition capabilities of the MYCIN system. Computers and Biomedical Research, 8, 303–320. https://doi.org/10.1016/0010-4809(75)90009-9.CrossRefGoogle ScholarPubMed
Sotos, J. G. 1990. MYCIN and NEOMYCIN: Two approaches to generating explanations in rule-based expert systems. Aviation, Space, and Environmental Medicine, 61, 950–954.Google ScholarPubMed
Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T., Sullivan, E., Drummond, R., et al. 2007. Toward harnessing user feedback for machine learning. Proceedings of the 12th International Conference on Intelligent User Interfaces (Honolulu, Hawaii: Association for Computing Machinery), pp. 82–91. https://doi.org/10.1145/1216295.1216316.CrossRefGoogle Scholar
Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., et al. 2009. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human–Computer Studies, 67, 639–662.CrossRefGoogle Scholar
Stumpf, S., Sullivan, E., Fitzhenry, E., Oberst, I., Wong, W.-K., and Burnett, M. 2008. Integrating rich user feedback into intelligent user interfaces. Proceedings of the 13th International Conference on Intelligent User Interfaces (New York: Association for Computing Machinery), pp. 50–59. https://doi.org/10.1145/1378773.1378781.CrossRefGoogle Scholar
Swartout, W. R. 1983. XPLAIN: A system for creating and explaining expert consulting programs. Artificial Intelligence, 21, 285–325. https://doi.org/10.1016/S0004-3702(83)80014-9.CrossRefGoogle Scholar
Theissler, A., Spinnato, F., Schlegel, U., and Guidotti, R. 2022. Explainable AI for time series classification: A review, taxonomy and research directions. IEEE Access, 10, 100700–100724. https://doi.org/10.1109/ACCESS.2022.3207765.CrossRefGoogle Scholar
Tintarev, N., and Masthoff, J. 2007. A survey of explanations in recommender systems. Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering Workshop (IEEE Computer Society), pp. 801–810.Google Scholar
Wachter, S., Mittelstadt, B., and Russell, C. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. SSRN. https://doi.org/10.2139/ssrn.3063289.CrossRefGoogle Scholar
Warren, G., Byrne, R. M. J., and Keane, M. T. 2023. Categorical and continuous features in counterfactual explanations of AI systems. Proceedings of the 28th International Conference on Intelligent User Interfaces (New York: Association for Computing Machinery), pp. 171–187. https://doi.org/10.1145/3581641.3584090.CrossRefGoogle Scholar
Wiegand, G., Schmidmaier, M., Weber, T., Liu, Y., and Hussmann, H. 2019. I drive: You trust: Explaining driving behavior of autonomous cars. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (New York: Association for Computing Machinery), pp. LBW0163:1–LBW0163:6. https://doi.org/10.1145/3290607.3312817.Google Scholar

Accessibility standard: WCAG 2.2 AAA

Why this information is here

This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.

Accessibility Information

The HTML of this chapter complies with version 2.2 of the Web Content Accessibility Guidelines (WCAG), offering more comprehensive accessibility measures for a broad range of users and attains the highest (AAA) level of WCAG compliance, optimising the user experience by meeting the most extensive accessibility guidelines.

Content Navigation

Table of contents navigation
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.
Index navigation
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.

Reading Order & Textual Equivalents

Single logical reading order
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.
Short alternative textual descriptions
You get concise descriptions (for images, charts, or media clips), ensuring you do not miss crucial information when visual or audio elements are not accessible.
Full alternative textual descriptions
You get more than just short alt text: you have comprehensive text equivalents, transcripts, captions, or audio descriptions for substantial non‐text content, which is especially helpful for complex visuals or multimedia.
Visualised data also available as non-graphical data
You can access graphs or charts in a text or tabular format, so you are not excluded if you cannot process visual displays.

Visual Accessibility

Use of high contrast between text and background colour
You benefit from high‐contrast text, which improves legibility if you have low vision or if you are reading in less‐than‐ideal lighting conditions.

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×