Abstract
Speech-emotion models assign familiar labels such as “happy”, but those labels need not correspond to the distinctions listeners or speakers intend. We examine what drives these assignments using 546 CREMA-D recordings: 91 actors producing the same sentence in six acted emotional deliveries. Fourier, Oruk’s speech-emotion model, labels 247 recordings “happy”, including 74 of 91 acted-happy recordings and 71 of 91 acted-neutral recordings. Removing one internal direction estimated from other actors’ recordings reduces the mean happy score on the originally tagged clips from 0.660 to 0.315; 64 random edits matched in size leave it at 0.649 on average. A simple change to the input produces a comparable effect: adding 0.5 seconds of silence before the unchanged speech alters 161 of 546 happy-tag decisions, whereas adding the same silence afterwards alters 32. A separate WavLM analysis using different CREMA-D takes finds that representation trajectories distinguish acted deliveries most strongly in early layers, with separation decreasing in later layers. Together, these results show how internal interventions and controlled audio edits can reveal the cues underlying a model’s categories. They motivate checking padding, amplitude and segmentation before interpreting speech-emotion outputs, and comparing those outputs with listener judgements. The findings concern acted portrayals and model labels, rather than speakers’ felt emotions; overlap between Fourier’s training data and CREMA-D cannot be excluded.
