Book contents
- Human-Centered Machine Learning
- Reviews
- Human-Centered Machine Learning
- Copyright page
- Contents
- Contributors
- 1 Introduction
- Part I Human-Centered Machine Learning in the Arts and Humanities
- Part II Doing Human-Centered Machine Learning
- Part III Humans and Machine Learning
- 8 Human-Centered Explainable AI (XAI)
- 9 Humans Teaching Machine Learning Agents
- 10 Modeling the Social and Normative Context of Human–Agent Interactions
- 11 Interviews: Teams
- Part IV Machine Learning in Its Social Context
- Part V Machine Learning and Humanity
- References
9 - Humans Teaching Machine Learning Agents
from Part III - Humans and Machine Learning
Published online by Cambridge University Press: 11 September 2026
Book contents
- Human-Centered Machine Learning
- Reviews
- Human-Centered Machine Learning
- Copyright page
- Contents
- Contributors
- 1 Introduction
- Part I Human-Centered Machine Learning in the Arts and Humanities
- Part II Doing Human-Centered Machine Learning
- Part III Humans and Machine Learning
- 8 Human-Centered Explainable AI (XAI)
- 9 Humans Teaching Machine Learning Agents
- 10 Modeling the Social and Normative Context of Human–Agent Interactions
- 11 Interviews: Teams
- Part IV Machine Learning in Its Social Context
- Part V Machine Learning and Humanity
- References
Summary
This chapter describes how nonexpert humans can train reinforcement learning (RL) agents. There are numerous ways to learn from human instruction – some have been studied more than others since this is an open field of research. A traditional machine learning approach focuses on the objective performance of the RL agent, optimizing metrics such as training time and reward. Verifying these RL systems is often done with simulated rather than real human input. We believe that to be a limiting approach. In this chapter, we will show: (1) a few methods of humans teaching RL agents; (2) how a great deal of information can be learned from testing these agents with human-subject studies; and (3) how objective performance does not need to be sacrificed in order to create a positive user experience. We can create a much better interactive machine learning system when designing and verifying from a human-centered perspective. We’ll cover different teaching methods, including advice and critique, and how they impact the user experience and objective performance of the agent. We’ll also discus how teaching methods are impacted by many factors, such as rhetorical arguments, immediacy of feedback, and issues of language processing. We’ll also cover a three-step process for the design and verification of interactive machine learning algorithms, and recommendations to improve the human experience.
Keywords
Information
- Type
- Chapter
- Information
- Human-Centered Machine Learning , pp. 201 - 221Publisher: Cambridge University PressPrint publication year: 2026
References
Argall, B. D., Browning, B., and Veloso, M. (2008). Learning robot motion control with demonstration and advice-operators. In Intelligent Robots and Systems, 2008. IROS 2008. IEEE/RSJ International Conference on, pages 399–404. IEEE.Google Scholar
Bartneck, C., Kanda, T., Mubin, O., and Al Mahmud, A. (2009a). Does the design of a robot influence its animacy and perceived intelligence? International Journal of Social Robotics, 1(2):195–204.CrossRefGoogle Scholar
Bartneck, C., Kulić, D., Croft, E., and Zoghbi, S. (2009b). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, 1(1):71–81.CrossRefGoogle Scholar
Cederborg, T., Grover, I., Isbell, C. L., and Thomaz, A. L. (2015). Policy shaping with human teachers. In IJCAI, pages 3366–3372.Google Scholar
Chernova, S. and Thomaz, A. L. (2014). Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning, 8(3):1–121.CrossRefGoogle Scholar
DiSalvo, C. F., Gemperle, F., Forlizzi, J., and Kiesler, S. (2002). All robots are not created equal: The design and perception of humanoid robot heads. In Proceedings of the 4th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, pages 321–326. ACM.CrossRefGoogle Scholar
Duffy, B. R. (2003). Anthropomorphism and the social robot. Robotics and Autonomous Systems, 42(3):177–190.CrossRefGoogle Scholar
Fong, T., Nourbakhsh, I., and Dautenhahn, K. (2003). A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3):143–166.CrossRefGoogle Scholar
Griffith, S., Subramanian, K., Scholz, J., Isbell, C., and Thomaz, A. L. (2013). Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems, pages 2625–2633.Google Scholar
Heinrichs, J. (2017). Thank You for Arguing: What Aristotle, Lincoln, and Homer Simpson Can Teach Us About the Art of Persuasion. Three Rivers Press (CA).Google Scholar
Isbell, C., Shelton, C. R., Kearns, M., Singh, S., and Stone, P. (2001). A social reinforcement learning agent. In Proceedings of the Fifth International Conference on Autonomous Agents, pages 377–384. ACM.CrossRefGoogle Scholar
Isbell, C. L., Kearns, M., Singh, S., Shelton, C. R., Stone, P., and Kormann, D. (2006). Cobot in LambdaMOO: An adaptive social statistics agent. Autonomous Agents and Multi-Agent Systems, 13(3):327–354.CrossRefGoogle Scholar
Joshi, M., Khobragade, R., Sarda, S., Deshpande, U., and Mohan, S. (2012). Object-oriented representation and hierarchical reinforcement learning in Infinite Mario. In Tools with Artificial Intelligence (ICTAI), 2012 IEEE 24th International Conference on, volume 1, pages 1076–1081. IEEE.Google Scholar
Knox, W. B. and Stone, P. (2010). Combining manual feedback with subsequent MDP reward signals for reinforcement learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, volume 1, pages 5–12. International Foundation for Autonomous Agents and Multiagent Systems.CrossRefGoogle Scholar
Krening, S. (2018). Newtonian action advice: Integrating human verbal instruction with reinforcement learning. arXiv:1804.05821.Google Scholar
Krening, S. and Feigh, K. M. (2018a). Characteristics that influence perceived intelligence in AI design. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting.CrossRefGoogle Scholar
Krening, S. and Feigh, K. M. (2018b). Interaction algorithm effect on human experience with reinforcement learning. Transactions on Human-Robot Interaction, 1(1).Google Scholar
Krening, S. and Feigh, K. M. (2019). Effect of interaction design on the human experience with interactive reinforcement learning. In Proceedings of the 2019 Designing Interactive Systems Conference, pages 1089–1100.CrossRefGoogle Scholar
Krening, S., Harrison, B., Feigh, K. M., Isbell, C., and Thomaz, A. (2016). Object-focused advice in reinforcement learning. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pages 1447–1448. International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
Kuhlmann, G., Stone, P., Mooney, R., and Shavlik, J. (2004). Guiding a reinforcement learner with natural language advice: Initial results in RoboCup soccer. In The AAAI-2004 Workshop on Supervisory Control of Learning and Adaptive Systems.Google Scholar
MacGlashan, J., Babes-Vroman, M., DesJardins, M., Littman, M., Muresan, S., Squire, S., Tellex, S., Arumugam, D., and Yang, L. (2015). Grounding English commands to reward functions. In Proceedings of Robotics: Science and Systems, Rome, Italy.Google Scholar
Maclin, R., Shavlik, J., Torrey, L., Walker, T., and Wild, E. (2005). Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In Proceedings of the National Conference on Artificial Intelligence, volume 20, page 819. AAAI Press; MIT Press.Google Scholar
Meriçli, C., Klee, S. D., Paparian, J., and Veloso, M. (2014). An interactive approach for situated task specification through verbal instructions. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, pages 1069–1076. International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
Phillips, T. (2010). Put your money where your mouth is: The effects of southern vs. standard accent on perceptions of speakers. Social Sciences, pages 53–56.Google Scholar
Rakić, T., Steffens, M. C., and Mummendey, A. (2011). Blinded by the accent! The minor role of looks in ethnic categorization. Journal of Personality and Social Psychology, 100(1):16.CrossRefGoogle ScholarPubMed
Sivamurugan, M. S. and Ravindran, B. (2012). Instructing a reinforcement learner. In FLAIRS Conference.Google Scholar
Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A. G., Teller, S. J., and Roy, N. (2011). Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, volume 1, page 2.CrossRefGoogle Scholar
Thomaz, A. L. and Breazeal, C. (2008). Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence, 172(6–7):716–737.CrossRefGoogle Scholar
Thomaz, A. L., Hoffman, G., and Breazeal, C. (2006). Reinforcement learning with human teachers: Understanding how people want to teach robots. In Robot and Human Interactive Communication, 2006. ROMAN 2006. The 15th IEEE International Symposium on, pages 352–357. IEEE.Google Scholar
Accessibility standard: WCAG 2.2 AAA
Why this information is here
This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.Accessibility Information
The HTML of this chapter
complies with version 2.2 of the
Web Content Accessibility Guidelines
(WCAG), offering more comprehensive accessibility measures for a broad range
of users and attains the highest
(AAA)
level of WCAG compliance, optimising the user experience by meeting the most extensive
accessibility guidelines.
Content Navigation
Table of contents navigation
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.
Index navigation
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.
Reading Order & Textual Equivalents
Single logical reading order
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.
Short alternative textual descriptions
You get concise descriptions (for images, charts, or media clips), ensuring you do not miss crucial information when visual or audio elements are not accessible.
You get concise descriptions (for images, charts, or media clips), ensuring you do not miss crucial information when visual or audio elements are not accessible.
Full alternative textual descriptions
You get more than just short alt text: you have comprehensive text equivalents, transcripts, captions, or audio descriptions for substantial non‐text content, which is especially helpful for complex visuals or multimedia.
You get more than just short alt text: you have comprehensive text equivalents, transcripts, captions, or audio descriptions for substantial non‐text content, which is especially helpful for complex visuals or multimedia.
Visualised data also available as non-graphical data
You can access graphs or charts in a text or tabular format, so you are not excluded if you cannot process visual displays.
You can access graphs or charts in a text or tabular format, so you are not excluded if you cannot process visual displays.
Visual Accessibility
Use of high contrast between text and background colour
You benefit from high‐contrast text, which improves legibility if you have low vision or if you are reading in less‐than‐ideal lighting conditions.
You benefit from high‐contrast text, which improves legibility if you have low vision or if you are reading in less‐than‐ideal lighting conditions.