Hostname: page-component-7b7567c8c-fvjsw Total loading time: 0 Render date: 2026-11-17T09:54:59.336Z Has data issue: false hasContentIssue false

When the Body Speaks and the Tone Echoes: Exploring GAI Multimodal Emotion Recognition

Published online by Cambridge University Press:  31 July 2026

H. Dery*
Affiliation:
School of Counseling and Human Development, University of Haifa, Haifa, Israel
D. Piterman
Affiliation:
School of Counseling and Human Development, University of Haifa, Haifa, Israel
Z. Elyoseph
Affiliation:
School of Counseling and Human Development, University of Haifa, Haifa, IsraelDepartment of Brain Science, Imperial College, London, United Kingdom
E. Refoua
Affiliation:
Department of Psychology, Bar-Ilan University, Ramat-Gan, Israel
G. Meinlschmidt
Affiliation:
Department of Psychology, Trier University, Trier, GermanyDepartment of Digital and Blended Psychosomatics and Psychotherapy, University of Basel and University Hospital Basel, Basel, Switzerland
K. Bar
Affiliation:
School of Computer Science, Reichman University, Herzliya
D. Hadar Shoval
Affiliation:
Department of Psychology and Educational Counseling, Max Stern Yezreel Valley College, Emek Yezreel
A. Geller
Affiliation:
Ruppin Academic Center, Emek Hefer, Israel
*
*Corresponding author.

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the 'Save PDF' action button.
Introduction

This study examined the social-cognitive capabilities of three Generative Artificial Intelligence (GAI) models from the Gemini family—Gemini Pro 1.5, Gemini Pro 2, and Gemini Flash 2—focusing on their ability to recognize human emotions from nonverbal cues, specifically vocal tone and bodily gestures. Grounded in mentalization theory—the capacity to attribute mental states to oneself and others, which underpins adaptive social functioning and therapeutic change—the research investigated the extent to which GAI models can approximate human-like emotion recognition.

Objectives

The research investigated the extent to which GAI models can approximate human-like emotion recognition from nonverbal channels and whether the accuracy of this recognition is influenced by emotional valence (positive vs. negative).

Methods

The study employed a comparative experimental design. Stimuli were drawn from the validated EU-Emotion Stimulus Set (O’Reilly et al., 2016) and EU-Emotion Voice Database (Lassalle et al., 2019). The dataset included 36 dynamic video clips representing bodily gestures and 42 audio recordings conveying vocal expressions. The GAI models were evaluated using zero-shot prompting under standardized conditions. Altogether, the study comprised 9,360 independent trials across both modalities. Human accuracy benchmarks were derived from the normative validation data of the EU-Emotion datasets, allowing for direct and controlled comparisons between model and human performance.

Results

The findings revealed both modality- and valence-dependent patterns. GAI models demonstrated performance comparable to human participants in vocal emotion recognition; Gemini Pro 2 achieved an accuracy rate of 47.14%, closely matching the human benchmark of 45.19%. In contrast, the models underperformed in the bodily gesture modality, where human participants achieved 77.08% accuracy, whereas Gemini Pro 2 reached 71.39%. A consistent valence-based performance bias was observed: all GAI models demonstrated significantly higher accuracy for positive emotions than for negative ones, particularly in the bodily modality. In contrast, human participants showed no significant difference in recognition accuracy for positive and negative emotions. All models consistently performed better with bodily gestures than with vocal expressions.Image 1:

A grouped bar chart comparing emotion recognition accuracy across three models against human benchmarks for body gesture and voice. See long description.Image 2:

A grouped bar chart comparing the accuracy of three models in recognizing positive and negative emotions from body gestures against human performance baselines. See long description.Image 3:

A grouped bar chart comparing the mean proportion of correct positive and negative emotion recognition in voice across three models. See long description.

Conclusions

This study provides detailed empirical evidence of GAI’s present capacity for nonverbal emotion recognition. While some models—particularly Gemini Pro 2—approach human performance in specific contexts, notable gaps remain, especially in recognizing negative affect conveyed through bodily gestures. These findings underscore the promise of GAI in socially embedded applications such as clinical support, while highlighting the need for continued development using ecologically valid, diverse, and multimodal datasets to enhance the emotional competence of artificial agents.

Disclosure of Interest

None Declared

Information

Type
Abstract
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s), 2026. Published by Cambridge University Press on behalf of European Psychiatric Association
Submit a response

Comments

No Comments have been published for this article.