Highlights
-
• Linguistic and paralinguistic cues interact in bilingual emotion processing.
-
• For bilinguals, semantic valence consists of semantic knowledge and stimulus valence as distinct components.
-
• Regardless of learners’ semantic knowledge, semantics-prosody congruence aids emotion perception, whereas incongruence hinders it.
-
• Greater semantic knowledge amplifies semantics-prosody (in)congruence effects in L2 emotion processing.
-
• Semantic valence influences the accuracy and reaction time of emotional prosody perception in distinct ways for L2 learners.
1. Introduction
Emotional information in spoken language is conveyed through multiple channels, most prominently semantic valence and emotional prosody (Kotz & Paulmann, Reference Kotz and Paulmann2007; Nygaard & Tzeng, Reference Nygaard and Tzeng2021; Pell et al., Reference Pell, Jaywant, Monetta and Kotz2011; Perlovsky, Reference Perlovsky2013; Seydell-Greenwald et al., Reference Seydell-Greenwald, Chambers, Ferrara and Newport2020; Xiao; Reference Xiao2025). Semantic valence refers to the stored semantic knowledge about whether an event, object or concept is positive or negative (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017; Itkes & Kron, Reference Itkes and Kron2019). Emotional prosody refers to the modulation of the tone of voice to express positive or negative emotions, attitudes, and feelings (Kemmerer, Reference Kemmerer2014). When processing emotional speech, listeners decode not only the semantic valence of an utterance (what is said), but also the speaker’s emotional prosody (how it is said) (Cutler et al., Reference Cutler, Dahan and Van Donselaar1997; Wilson & Wharton, Reference Wilson and Wharton2006). For example, a listener may perceive an utterance with positive semantic valence (e.g., she is wise) differently from an utterance with negative semantic valence (e.g., she is arrogant), even if both are spoken with happy emotional prosody. Previous research has focused on the effects of semantic valence on word recognition (Estes & Adelman, Reference Estes and Adelman2008; Gao et al., Reference Gao, Shinkareva and Peelen2022; Kim & Sumner, Reference Kim and Sumner2017; Kuperman et al., Reference Kuperman, Estes, Brysbaert and Warriner2014; Yap & Seow, Reference Yap and Seow2014) and emotion judgments (Hauser & Schwarz, Reference Hauser and Schwarz2016; Lin et al., Reference Lin, Ding and Zhang2020; Xiao & Liu, Reference Xiao and Liu2025, Reference Xiao and Liu2026) among native speakers in their first language (L1). However, much less is known about how second language (L2) learners process semantic valence and emotional prosody.
In tonal languages such as Mandarin Chinese, the interplay between semantic valence and emotional prosody is even more complex because both rely on the same acoustic feature – pitch (Shen, Reference Shen, Lin and Wang1985; Xu, Reference Xu2005). In other words, the modulation of pitch does not merely convey emotional prosody; it also functions as lexical tone to differentiate word meanings. There are four lexical tones in Mandarin. For instance, the syllable “ma” can represent four distinct words depending on the lexical tone applied: mother (Tone 1), hemp (Tone 2), horse (Tone 3) and to scold (Tone 4) (Yip, Reference Yip2002). When processing emotional speech in tonal languages, semantic valence and emotional prosody coexist and interact (Chang et al., Reference Chang, Lee, Wang, Young, Li and Chu2023; Ip & Cutler, Reference Ip and Cutler2020; Ouyang & Kaiser, Reference Ouyang and Kaiser2015), raising an important question for English-Chinese bilinguals: Does greater knowledge of semantic valence influence the perception of emotional prosody? Therefore, this study investigates the role of semantic valence in Chinese emotional prosody perception for L1-English L2-Chinese learners.
1.1 Semantic knowledge and semantic valence
The Theory of Constructed Emotion proposes that language plays a fundamental role in supporting individuals in acquiring emotion categories initially and using this knowledge later to interpret emotion perception (Barrett, Reference Barrett2006; Barrett, Reference Barrett2017; Barrett et al., Reference Barrett, Lindquist and Gendron2007; Lindquist et al., Reference Lindquist, Satpute and Gendron2015; Lindquist & Gendron, Reference Lindquist and Gendron2013). Within this framework, semantic knowledge acts as the conceptual glue for emotion perception, binding otherwise variable sensorimotor and affective states into recognizable emotion categories (Barrett et al., Reference Barrett, Lindquist and Gendron2007). This reliance on semantic knowledge is evident in facial emotion perception. For example, when an emotion word was semantically satiated (e.g., by repeatedly saying anger to temporarily disrupt its meaning), participants became slower and less accurate in categorizing facial expressions into emotions (Lindquist et al., Reference Lindquist, Barrett, Bliss-Moreau and Russell2006). Furthermore, when semantic knowledge is chronically impaired, such as in patients with semantic dementia, individuals perceive emotional facial expressions merely as broad affective dimensions (e.g., pleasant or unpleasant) rather than as specific, discrete emotions (e.g., anger, sadness) (Lindquist et al., Reference Lindquist, Gendron, Barrett and Dickerson2014). Despite these findings in facial emotions, the role of semantic knowledge in perceiving vocal emotions, such as emotional prosody, is largely underexplored. This study therefore examines the effect of semantic knowledge in emotional prosody perception. Here, semantic knowledge refers narrowly to lexico-semantic knowledge, namely, the knowledge of individual word meanings.
The theoretical link between emotional prosody and semantic knowledge can be grounded in Russell’s Circumplex model (Russell, Reference Russell1980), which maps emotional states along two primary dimensions: valence (positive versus negative) and arousal (excited versus calm). Although both dimensions are theoretically relevant, the present study focuses on the dimension of valence, building on a conceptual framework of valence that distinguishes between the knowledge of valence and the feeling of valence (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017; Itkes & Kron, Reference Itkes and Kron2019). The knowledge of valence (semantic valence) is the stored semantic knowledge about whether an object or event is positive or negative, whereas the feeling of valence (affective valence) represents the immediate, subjective, and sensory-driven affective reaction elicited by a stimulus (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017; Itkes & Kron, Reference Itkes and Kron2019). Furthermore, they found that semantic valence and affective valence dissociate in both self-reported experience and physiological responses (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017; Itkes et al., Reference Itkes, Eviatar and Kron2019; Olteanu et al., Reference Olteanu, Salama, Kimchi and Kron2023). Specifically, knowledge-based perception of semantic valence showed less attenuation with repeated exposures, lower sensitivity to perceptual changes, and a higher likelihood of eliciting mixed emotions. In contrast, feeling-based perception of affective valence showed greater attenuation with repeated exposures to pleasant and unpleasant images, greater sensitivity to the perceptual details of the stimuli, and a lower likelihood of eliciting mixed emotions.
The work of Itkes and colleagues represents a notable advancement in the field of emotion perception by demonstrating that the effect of semantics, semantic valence specifically, is not a passive byproduct but a distinct system with its own behavioral profile. However, their research, and emotion perception research more broadly, has largely been conducted in native speakers, examining how semantic valence varies across contexts without considering how participants’ underlying semantic knowledge may itself vary across individuals. Notably, L2 learners’ semantic knowledge is often incomplete or variable. For example, an L2 English learner unfamiliar with the intended semantically positive word sleek might mistakenly interpret it as having a negative valence, as evidenced by phonaesthemes, which suggest that certain sound clusters (e.g., sl-) are often linked to particular affective associations (Bergen, Reference Bergen2004; Hutchins, Reference Hutchins1998).
Taken together, prior research shows that semantic knowledge provides a conceptual scaffold for perceiving and interpreting emotions (Barrett, Reference Barrett2006). Importantly, semantic knowledge of valence has been shown to operate independently from the affective feeling of valence (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017). In an L2 context, however, such semantic knowledge may be less stable and more variable than in an L1. Because of this inherent variability, L2 learners’ semantic knowledge cannot be treated as random variance. We propose that participants’ semantic knowledge, particularly their lexico-semantic knowledge, can also modulate emotional prosody perception in addition to the semantic valence of the stimuli. Therefore, this study extends the established research of semantic valence into the L2 domain by conceptualizing it as two components: the semantic knowledge of participants and the inherent semantic valence of stimuli. We expect that both components play distinct roles in bilinguals’ perception of emotional prosody.
1.2 Semantic valence in second language acquisition
L2 learners can identify a wide range of emotional prosody with above-chance accuracy at syllable, word, and sentence levels (e.g., Alm & Llorà, Reference Alm and Llorà2006; Altrov, Reference Altrov2013; Dromey et al., Reference Dromey, Silveira and Sandor2005; Laukka & Elfenbein, Reference Laukka and Elfenbein2021; Wei et al., Reference Wei, He, Kauschke, Scharinger and Domahs2022; Xiao, Reference Xiao2025; Xiao & Liu, Reference Xiao and Liu2026; Zhu, Reference Zhu2013). However, the effects of semantic valence on emotion perception in an L2 is distinct from that of an L1. For example, L2 learners generally perceive the emotionality in words and sentences (e.g., I love you) as having less emotional weight and greater emotional distance compared to their native language (Dewaele, Reference Dewaele2008; Keysar et al., Reference Keysar, Hayakawa and An2012; Pavlenko, Reference Pavlenko2008, Reference Pavlenko2012). Moreover, a learner’s linguistic and cultural background also influences their interpretation of L2 semantic valence. Jackson et al. (Reference Jackson, Watts, Henry, List, Forkel, Mucha, Greenhill, Gray and Lindquist2019) found that the semantic valence of stimuli varies significantly across language families. For example, the meaning of anxiety is closely associated with fear in Tai-Kadai languages, while it is associated with grief and regret in Austroasiatic languages. Therefore, it is necessary to systematically investigate the role of semantic valence in L2 emotion perception.
Current debates in the field of L2 emotional prosody perception have centered on the two components of semantic valence: the semantic knowledge of the participants and the inherent semantic valence of the stimuli. The first debate explores whether L2 learners’ semantic knowledge facilitates or interferes with their perception of emotional prosody in an L2. Some studies found a facilitation effect of semantic knowledge. For example, non-native speakers with semantic knowledge (i.e., L2 learners) outperformed non-native speakers without such knowledge (i.e., naïve listeners) in recognizing emotional prosody (Altrov, Reference Altrov2013; Xiao & Liu, Reference Xiao and Liu2026; Zhu, Reference Zhu2013). Additionally, L2 Japanese learners with more learning experience recognized emotional prosody more accurately compared to those with less learning experience (Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016). However, Bhatara et al. (Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016) found that L2 English learners with higher proficiency were less accurate in recognizing positive emotional prosody in their L2 English compared to those with lower proficiency, and they suggested this is due to top-down semantic interference, where highly proficient learners paid more attention to the lexical and semantic meaning rather than the speaker’s emotional tone.
Notably, in previous studies, researchers have typically measured L2 semantic knowledge by general proficiency levels (e.g., Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016), assuming that advanced learners possess more semantic knowledge (i.e., knowing more words). This general proficiency is a broad composite construct that serves as an indirect proxy for semantic knowledge. Meanwhile, other researchers have also demonstrated that vocabulary screening tests directly quantify lexico-semantic knowledge, a factor critical to emotion perception (Brysbaert et al., Reference Brysbaert, Lagrou and Stevens2017; Ferré et al., Reference Ferré, Guasch, Stadthagen-González, Hinojosa, Fraga, Marín and Pérez-Sánchez2024; Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020). Therefore, we first measured learners’ semantic knowledge by using general proficiency in the pilot experiment, aiming to contribute evidence from a tonal language to the existing debate on the semantic knowledge effect (e.g., Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016). In the main experiment, we then used a vocabulary screening test to assess participants’ semantic knowledge, examining whether greater knowledge of semantic valence facilitates the perception of emotional prosody in L2 Chinese.
The second debate addresses how the semantic valence of the stimuli influences emotional prosody perception in a second language. Building on evidence that native speakers perceive emotional prosody faster and more accurately in congruent conditions than in incongruent ones (e.g., Lin et al., Reference Lin, Ding and Zhang2020; Pell et al., Reference Pell, Jaywant, Monetta and Kotz2011), L2 research has also begun to explore this semantics-prosody congruence effect, but has yielded mixed results. Some researchers reported a congruence effect, where L2 Chinese learners recognized emotional prosody more accurately and quickly when the semantic valence and emotional prosody of the stimuli were congruent (Xiao & Liu, Reference Xiao and Liu2025). But others did not replicate this effect in L2 English learners (Cho & Dewaele, Reference Cho and Dewaele2021). One limitation of this line of work is that it only considers the semantic valence of stimuli, without accounting for the variability in L2 learners’ semantic knowledge. It is possible that the congruence effect emerges only in L2 learners with specific levels of semantic knowledge. To address this, the present study manipulated both components of semantic valence, namely, L2 learners’ semantic knowledge and the inherent semantic valence of stimuli, aiming to clarify the independent and interactive effects of the two components of semantic valence in L2 emotional prosody perception.
1.3 Semantic valence in L2 Chinese learners
As demonstrated above, previous studies on emotional prosody perception in L2 learners have yielded contrasting results regarding the role of semantic valence, specifically the effects of semantic knowledge (e.g., Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016; Xiao & Liu, Reference Xiao and Liu2026) and semantics-prosody congruence (e.g., Cho & Dewaele, Reference Cho and Dewaele2021; Xiao & Liu, Reference Xiao and Liu2025). These conflicting findings also suggest that the perception of emotional prosody may differ between tonal languages (e.g., Mandarin Chinese) and non-tonal languages (e.g., English). Specifically, tonal languages constrain the use of pitch for signaling emotional prosody because pitch is already occupied by lexical tone (Ross et al., Reference Ross, Edmondson and Seibert1986). Consequently, Chinese listeners may face a higher cognitive demand when integrating these overlapping pitch cues for semantic and prosodic information (Lin et al., Reference Lin, Ding and Zhang2020; Xiao, Reference Xiao2025; Xiao & Liu, Reference Xiao and Liu2025), or they may compensate by adopting secondary acoustic cues, such as intensity and speech rate, to differentiate emotional prosody (Wang et al., Reference Wang, Lee and Ma2018).
Mandarin Chinese, as a tonal language, uses tone of voice to encode both semantics and emotional prosody (Xu, Reference Xu2005). It has four lexical tones that differentiate word meanings: 1: high-level tone; 2: rising tone; 3: low-rising tone; 4: falling tone (Yip, Reference Yip2002). For example, ma1 means mother, ma2 means hemp, ma3 means horse, and ma4 means to scold. Semantic cues and prosodic cues coexist within the shared acoustic feature – pitch (Shen, Reference Shen, Lin and Wang1985), suggesting an interaction between semantic valence and emotional prosody in Chinese words and sentences (Lin et al., Reference Lin, Ding and Zhang2020; Xiao & Liu, Reference Xiao and Liu2025). Therefore, processing emotional prosody in Mandarin Chinese can be challenging for L2 learners, as it requires rapid integration of linguistic (semantic valence) and paralinguistic (emotional prosody) cues within a fraction of a second (Kao & Zhang, Reference Kao and Zhang2020; Taguchi, Reference Taguchi2008).
Prior research on Chinese emotional prosody perception has mainly compared L2 Chinese learners with native Chinese speakers from a cross-cultural perspective (e.g., Xiao & Liu, Reference Xiao and Liu2025, Reference Xiao and Liu2026; Zhu, Reference Zhu2013). Notably, these studies found that L2 Chinese learners, even with a brief learning experience (e.g., less than six months), achieved accuracy on par with native speakers in identifying emotional prosody in Chinese words and sentences, with performance improving as syllable length increases (Xiao & Liu, Reference Xiao and Liu2026; Zhu, Reference Zhu2013). Furthermore, much like native speakers, L2 Chinese learners demonstrated a significant semantics-prosody congruence effect, recognizing emotional prosody more accurately and faster when the semantic valence and emotional prosody of the stimuli were congruent (Xiao & Liu, Reference Xiao and Liu2025). Interestingly, L2 Chinese learners exhibited an even stronger reliance on the semantic valence of stimuli than native Chinese speakers: while both groups showed this congruence effect when prosodic information was limited in disyllable words, only L2 learners showed the influence of semantic valence when prosodic cues were ample in full sentences (Xiao & Liu, Reference Xiao and Liu2025).
However, in L2 emotional prosody perception, few studies have systematically manipulated the two components of semantic valence, namely, the semantic knowledge of participants and the semantic valence of the stimuli (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016; Xiao & Liu, Reference Xiao and Liu2025). Existing research has either examined the effects of the stimuli’s semantic valence while overlooking learners’ varying semantic knowledge (Cho & Dewaele, Reference Cho and Dewaele2021; Xiao & Liu, Reference Xiao and Liu2025) or manipulated the learners’ semantic knowledge without controlling for the stimuli’s semantic valence (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016). To clarify the role of semantic valence in L2 Chinese, the current study manipulated both learners’ semantic knowledge and the semantic valence of the stimuli to investigate the perception of emotional prosody in Chinese words and sentences. The findings of this study will provide insights into how L2 learners simultaneously process linguistic knowledge and paralinguistic information in a tonal language.
2. The current study
The overarching aim of this study was to examine the role of semantic valence in bilingual emotion processing. While semantic valence is traditionally defined as the stored semantic knowledge about whether an event, object, or concept is positive or negative (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017), we argue that it must be decoupled into two distinct components for bilinguals: the semantic knowledge of participants and the semantic valence of the stimuli. To isolate the independent and interactive effects of these two components, participants’ semantic knowledge was operationalized through learning experience (in the pilot experiment) and a vocabulary test (in the main experiment); the semantic valence of the stimuli was also manipulated across both experiments. This study investigated whether greater knowledge of semantic valence facilitated the perception of emotional prosody for L1-English L2-Chinese learners in Chinese words and sentences.
Specifically, we conducted a pilot experiment to examine whether novice and advanced learners differ in their perception of Chinese emotional prosody, contributing new evidence from a tonal language to the existing debate (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016). In the main experiment, we manipulated the two components of semantic valence (i.e., learners’ semantic knowledge and stimuli’s semantic valence) to investigate their specific effects and interactions during Chinese emotional prosody perception, extending the account of semantic valence (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017) to the field of second language acquisition.
We used G*Power (Faul et al., Reference Faul, Erdfelder, Buchner and Lang2009) to determine the a priori sample size. For both experiments, 40 participants were required to obtain 0.89 power to detect a medium effect size of 0.22 (obtained from Lin et al., Reference Lin, Ding and Zhang2020) at the standard 0.05 alpha error probability. All materials, code, data, and anonymized participant details are available on the Open Science Framework (OSF) repository at https://osf.io/fs4py/overview. Data were analyzed using R (R Core Team, 2024). To build the model, we used backward elimination starting with a maximal model that included all potential main effects and interactions, and we removed the non-significant variables one at a time based on model comparison (Barr et al., Reference Barr, Levy, Scheepers and Tily2013).
3. The pilot experiment
To examine the contrasting results regarding semantic knowledge in non-tonal languages (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016), the pilot experiment extended the investigation in a tonal language, Mandarin Chinese. We investigated whether greater semantic knowledge affects the perception of emotional prosody in Mandarin Chinese words and sentencesby comparing novice and advanced L1-English L2-Chinese learners. To isolate the specific impact of semantic knowledge, we controlled the semantic valence of the stimuli to be neutral (e.g., shou1yin1ji1, radio). We hypothesized that L2 learners’ semantic knowledge would significantly influence their perception of emotional prosody in Chinese words and sentences. Additionally, we predicted that the effect of semantic knowledge would interact with both emotional prosody and syllable length based on prior cross-cultural findings (e.g., Pell & Kotz, Reference Pell and Kotz2011; Xiao & Liu, Reference Xiao and Liu2026).
3.1 Methods
3.1.1 Participants
Participants were L1-English L2-Chinese learners enrolled in Mandarin courses at a public U.S. university. We excluded 12 heritage speakers of Mandarin Chinese or other tonal languages, resulting in a total of 40 participants in the current analysis. Following prior studies (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016), we categorized participants based on their length of L2 Chinese learning experience: 20 novice L1-English L2-Chinese learners (novice learners, learning experience < 6 months; 9 females; mean age = 19, SD = 0.65) and 20 advanced L1-English L2-Chinese learners (advanced learners, learning experience > 3 years; 13 females; Mean age = 21.05, SD = 0.94). The novice learners were also included in a study that addressed a different research question (Xiao & Liu, Reference Xiao and Liu2026). All participants had normal or corrected-to-normal vision, reported no hearing problems, and received course credit or monetary compensation. The study was approved by the Institutional Review Board of the University of South Carolina.
3.1.2 Stimuli
We selected and adapted a total of 144 Chinese stimuli from Shen (Reference Shen, Lin and Wang1985), controlling for lexical tone and neutral semantic valence (see Table 1 for examples). A female native Chinese speaker recorded these stimuli in four emotional prosodies: neutral, joy, anger, and sadness. Utterances were segmented and normalized using Praat (Boersma & Weenink, Reference Boersma and Weenink2023), and the intended emotional prosody of each stimulus was further validated by six native Chinese speakers prior to the experiment. Based on the combination of emotional prosody and lexical tone, the stimulus set included 16 monosyllabic words, 64 disyllabic words, 48 trisyllabic words, and 16 sentences. The acoustic parameters of the stimuli can be found in the supplementary materials (Supplementary Table S6).
Example stimuli from the pilot experiment, categorized by syllable length
Table 1. Long description
The table displays example stimuli of the pilot experiment.
Column headers from left to right are: Empty, Monosyllable, Disyllable, Trisyllable, and Sentence.
Row 1 (Pinyin):
- Monosyllable: xiu tone 1
- Disyllable: shou tone 1 yin tone 1
- Trisyllable: zhang tone 1 zhong tone 1 bin tone 1
- Sentence: zhang tone 1 zhong tone 1 bin tone 1 xing tone 1 qi tone 1 tian tone1 xiu tone 1 shou tone 1 yin tone 1 ji tone 1
Row 2 (Chinese characters):
- Monosyllable: 修
- Disyllable: 收音
- Trisyllable: 张中斌
- Sentence: 张中斌星期天修收音机
Row 3 (English translation):
- Monosyllable: repair
- Disyllable: Receive sound
- Trisyllable: Zhang Zhongbin
- Sentence: Zhang Zhongbin repairs the radio on Sunday.
Note: Pinyin represents the official romanization system used to transcribe Mandarin Chinese pronunciation, with numbers indicating the specific lexical tone. Chinese characters are the logographic symbols representing the distinct word for each syllable-tone combination, followed by their English translations.
3.1.3 Procedure
Participants first completed a brief language background questionnaire regarding age, gender, native languages, and L2 Chinese learning experience, and then were randomly assigned to one of the four versions of the emotional prosody judgment task. There are four blocks in each task: monosyllable, disyllable, trisyllable and sentence blocks. The four versions were used to counterbalance the order of the block across participants. Within each block, the order of emotional prosody was also counterbalanced. The emotional prosody judgment task was untimed, where participants were instructed to sit in a quiet environment, listen to a series of utterances, one at a time, and then judge the intended emotional prosody for each utterance using a 4-alternative forced-choice format (i.e., neutral, joy, anger, and sadness). Six filler utterances were included to monitor participant attention. Participants’ emotional prosody recognition responses for each trial were recorded. The experiment lasted about 30 minutes.
3.1.4 Analysis
The total number of trials in the data analysis was 5,760. For each trial, participants received a score of 1 if they recognized the emotional prosody correctly; they received a score of 0 if their judgment mismatched the intended emotional prosody of the utterance. The accuracy (ACC) of the participants’ responses was analyzed using a logistic mixed-effects model (Jaeger, Reference Jaeger2008). The model included three fixed factors: semantic knowledge group with two levels (novice learners, advanced learners); emotion prosody type with four levels (neutral, joy, anger, and sadness); and syllable length with four levels (monosyllable, disyllable, trisyllable, and sentence). The advanced group, neutral prosody, and monosyllable were treated as the reference levels, respectively. Dummy coding was used, and we entered intercepts for items and subjects, as well as a by-item random slope for the effect of emotion prosody type (Cunnings, Reference Cunnings2012).
3.2 Results
Both novice learners and advanced learners demonstrated very high ACC in recognizing emotional prosody, averaged across four types of emotional prosody (Figure 1). The mean (SD) ACC for novice learners was 95.9% (0.199), and the mean (SD) ACC for advanced learners was 91.8% (0.275). Overall, the mean ACC of emotional prosody judgment of L2 Chinese learners increased with syllable length: 87.5% for monosyllables, 93.0% for disyllables, 96.1% for trisyllables, and 96. 3% for sentences, respectively. However, the SD decreased with syllable length: 0.331 for monosyllables, 0.254 for disyllables, 0.193 for trisyllables and 0.190 for sentences. The mean ACC of the four types of emotional prosody was 95.7% for neutral, 90.9% for joy, 94.3% for anger, and 94.4% for sadness.
Mean accuracy of emotional prosody judgments across four types of emotional prosody and syllable length in novice and advanced groups. The black vertical lines show standard errors.
Figure 1. Long description
The graph consists of two side-by-side panels. The y-axis represents Mean accuracy ranging from 0.6 to 1.0. The x-axis lists four emotional prosody categories: neutral, joy, anger, and sadness. A legend on the right defines syllable length using four shades of blue: monosyllable (lightest), disyllable, trisyllable, and sentence (darkest).
* Novice learners panel (left): Accuracy generally increases with syllable length across all emotions. For neutral, joy, and sadness, monosyllables show the lowest accuracy (approximately 0.86 to 0.88) while sentences reach near 1.0. Anger shows a slightly different pattern with monosyllables starting higher at 0.96.
* Advanced learners panel (right): Accuracy trends are more varied. For neutral, accuracy peaks at disyllable (0.98) and decreases slightly for sentences. For joy and sadness, there is a sharp upward trend from monosyllables (lowest at 0.76 for sadness) to sentences (highest at 0.95). Anger remains relatively stable across all lengths between 0.90 and 0.95.
Black vertical error bars are present on every bar, indicating standard error.
As shown in Supplementary Table S1, a logistic mixed-effects model showed a significant main effect of emotion prosody type, where both novice and advanced learners showed lower ACC for joy (β = −1.789, SE = 0.396, z = −4.521, p < .001), anger (β = −1.104, SE = 0.340, z = −3.250, p < .01), and sadness (β = −0.947, SE = 0.296, z = −3.197, p < .01) compared to neutral emotional prosody. There was a significant main effect of syllable length such that both groups showed a higher ACC for disyllables (β = 0.859, SE = 0.325, z = 2.642, p < .01), trisyllables (β = 1.455, SE = 0.351, z = 4.143, p < .001) and sentences (β = 1.303, SE = 0.445, z = 2.927, p < .01) compared to monosyllables. This indicated that L2 Chinese learners, regardless of their semantic knowledge, perceived neutral prosody more accurately than positive and negative prosody, and their ACC of emotional prosody perception increased with syllable length.
We also observed significant interactions between semantic knowledge group and emotional prosody, and between semantic knowledge group and syllable length, in L2 emotional prosody processing. Specifically, novice learners perceived emotional prosody more accurately compared to advanced learners when processing joy (β = 1.527, SE = 0.377, z = 4.052, p < .001), anger (β = 1.127, SE = 0.399, z = 2.827, p < .01) and sadness (β = 1.235, SE = 0.405, z = 3.049, p < .01) compared to processing neutral prosody. Furthermore, a significant interaction between group and syllable length was also observed, where novice learners’ advantage over advanced learners was reduced when recognizing emotional prosody in disyllables (β = −0.712, SE = 0.356, z =−2.002, p < .05) compared to monosyllables. We further subset the data to include neutral prosody only, and we found that there was no significant difference in emotional prosody perception between novice and advanced learners. The results revealed that novice learners had an advantage over advanced learners in recognizing positive and negative prosody in Chinese words and sentences, but not in identifying neutral prosody.
To sum up, both emotional prosody and syllable length significantly influenced L2 emotional prosody perception, with learners’ semantic knowledge modulating these effects. Specifically, novice learners showed higher accuracy than advanced learners in identifying positive and negative prosody, whereas both groups performed similarly on neutral prosody. This advantage for novice learners was mitigated by syllable length, impacting shorter stimuli more than longer ones. These findings suggested a potential semantics-prosody incongruence effect where conflict between semantic valence and emotional prosody interferes with perception, and since advanced learners possessed more semantic knowledge, they may experience greater interference than novice learners. Importantly, while no congruence effect emerged when semantic valence and emotional prosody were both neutral, this effect may still occur in stimuli with positive or negative semantic valence. Because this pilot experiment only used stimuli with neutral semantic valence, we conducted the second experiment (the main experiment) to further investigate how the stimuli’s semantic valence interacts with learners’ semantic knowledge in L2 emotional prosody perception.
4. The main experiment
The main experiment investigated the independent and interactive effects of two components of semantic valence, namely, the participants’ semantic knowledge and the semantic valence of the stimuli, on the perception of emotional prosody in L2 Chinese learners. Building on the pilot study findings, we refined our measure of semantic knowledge based on learners’ performance on a target vocabulary test. To systematically manipulate the semantic valence of the stimuli, we selected words with positive, negative, and neutral semantic valence, matched for syllable length, lexical tone, and frequency. Furthermore, as the pilot study found no group differences for neutral prosody, the current experiment focused exclusively on positive and negative emotional prosody. Finally, because the pilot study showed that syllable length interacted with semantic knowledge, we divided the stimuli into word and sentence tasks.
Based on the pilot experiment and previous research (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Lin et al., Reference Lin, Ding and Zhang2020; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016; Xiao & Liu, Reference Xiao and Liu2025), we anticipated an effect of semantic knowledge on emotional prosody perception in L2 Chinese. We hypothesized that L2 Chinese learners would demonstrate a semantics-prosody congruence effect regardless of their semantic knowledge levels. Furthermore, we predicted an interaction between semantic knowledge and semantics-prosody congruence, such that learners with high semantic knowledge would show a larger congruence effect than those with low semantic knowledge.
4.1 Method
4.1.1 Participants
We recruited L1-English L2-Chinese learners enrolled in intermediate-level Chinese courses. These learners had studied Chinese for one to three years and were familiar with basic syllable-tone combinations and simple sentence structures in Mandarin Chinese, indicating a similar baseline of general proficiency. All learners were taught the meanings of the real target words used as experimental stimuli in class three times per week for two weeks. Notably, the experimental stimuli also included another subset of novel pseudowords, which were explicitly excluded from instruction to ensure they remained unknown to the participants. During this period, learners completed two practice vocabulary tests.
Based on previous research demonstrating that vocabulary tests can quantify learners’ semantic knowledge (Brysbaert et al., Reference Brysbaert, Lagrou and Stevens2017; Ferré et al., Reference Ferré, Guasch, Stadthagen-González, Hinojosa, Fraga, Marín and Pérez-Sánchez2024; Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020), we administered a timed vocabulary test immediately prior to the experiment. This vocabulary test allowed us to categorize learners of similar general proficiency into two groups based on their semantic knowledge of the target stimuli. Learners who scored above 90% on the final vocabulary test were categorized as having more semantic knowledge, while those who scored below 75% were categorized as having less semantic knowledge. We excluded two participants whose scores fell between 75% and 90%, as well as six participants who were heritage speakers of Mandarin Chinese or other tonal languages.
A total of forty participants were included in the analysis: 23 L1-English L2-Chinese learners with more semantic knowledge (mean vocabulary score = 98%; 12 females; mean age = 19.35, SD = 0.99) and 17 L1-English L2-Chinese learners with less semantic knowledge (mean vocabulary score = 57%; 5 females; mean age = 19.39, SD = 1.08). Twelve participants in the more semantic knowledge group were also included in a previous study that addressed a different research question (Xiao & Liu, Reference Xiao and Liu2025). All participants had normal or corrected-to-normal vision, reported no hearing problems, and received course credit or monetary compensation for their participation. The study were approved by the Institutional Review Board of the University of South Carolina.
4.1.2 Stimuli
The stimuli were divided into word and sentence tasks. The word task included 128 disyllabic Chinese words, and the sentence task included 128 Chinese sentences. The critical manipulation of the stimuli was the congruence of semantic valence (positive, negative, and neutral) and emotional prosody (positive and negative). All words with positive, negative, and neutral semantic valence were matched for syllable length and lexical tone. In the word task, real disyllables with positive and negative semantic valence were matched in word frequency (Sun et al., Reference Sun, Hendrix, Ma and Baayen2018). We also created novel pseudo-disyllables with neutral semantic valence using lexical gaps. A lexical gap in Chinese occurs when a syllable can combine with specific tones but not all four tones (Ryu et al., Reference Ryu2022). For example, chun4 represents a lexical gap in Mandarin Chinese, as the syllable “chun” can combine with tone 1 (chun1 means spring), tone 2 (chun2 means pure), and tone 3 (chun3 means stupid), but not tone 4 (no real word exists in chun4). In the sentence task, we embedded the word stimuli from the word task into a simple carrier sentence “ta1 hen3 (She is very) _____” to minimize syntax parsing and contextual information for L2 learners (Cho & Dewaele, Reference Cho and Dewaele2021). Table 2 provides example stimuli with positive semantic valence (1a, 2a), negative semantic valence (1b, 2b), and neutral semantic valence (1c, 2c).
Example stimuli for the word task and sentence task in the main experiment, categorized by semantic valence
Table 2. Long description
The table displays the example stimuli of the main experiment, including the word task and the sentence task. The is organized into three columns based on semantic valence: Positive, Negative, and Neutral.
Word Task Examples.
- Positive semantic valence (1a): Pinyin rui4zhi4, Chinese characters 睿智, means wise in English.
- Negative semantic valence (1b): Pinyin ao4man4, Chinese characters 傲慢, means arrogant in English.
- Neutral semantic valence(1c): Pinyin chun4pou4, written as chunpou, this is a pseudo-word.
Sentence Task Examples.
- Positive semantic valence (2a): Pinyin ta1hen3rui4zhi4, Chinese characters 她很睿智, means She is very wise in English.
- Negative semantic valence (2b): Pinyin ta1hen3ao4man4, Chinese characters 她很傲慢, means She is very arrogant in English.
- Neutral semantic valence (2c): Pinyin ta1hen3chun4pou4, written as 她很chunpou, means She is very chunpou in English.
The table includes Pinyin with tone numbers, logographic Chinese characters, and English translations for each entry.
Note: Pinyin (Row 2 and 5) represents the official romanization system used to transcribe Mandarin Chinese pronunciation, with numbers indicating the specific lexical tone. Chinese characters (Row 3 and 6) are the logographic symbols representing the distinct word for each syllable-tone combination, followed by their English translations (Row 4 and 7).
A male native Chinese speaker recorded all stimuli for the word and sentence tasks with both positive (i.e., happy) and negative (i.e., sad) emotional prosody in a soundproof studio (Cho & Dewaele, Reference Cho and Dewaele2021; Lin et al., Reference Lin, Ding and Zhang2020; Pell & Kotz, Reference Pell and Kotz2011). Background noise was removed, and peak volume was normalized using Audacity. The stimuli were segmented and normalized with the same amplitude using Praat (Version 6.3.10). The acoustic parameters of the stimuli can be found in the supplementary materials (Supplementary Table S7).
4.1.3 Procedure
All participants completed the final vocabulary test immediately prior to the emotional prosody judgment tasks in the language lab. The vocabulary test was administered online with a 10-minute time limit under a Chinese instructor’s supervision. For each target word, participants heard it spoken in neutral prosody and selected its correct meaning using a three-alternative forced-choice format. Following the vocabulary test, participants completed two emotional prosody judgment tasks, performing the word task before the sentence task. For each trial, participants listened to an audio clip and judged whether the emotional prosody of each utterance was positive or negative as quickly as possible without making any mistakes. Auditory stimuli were presented in a randomized order using the E-Prime 2.0 software (Psychology Software Tools, Pittsburgh, PA). Each trial lasted 5000 ms, and a new trial began immediately after the participant made a response. Participants’ ACC and reaction time (RT) were recorded for each trial. After completing the word and sentence tasks, participants provided language background information and completed a prosody rating task. As the present study focuses on semantic valence, the prosody rating data are not reported here. The experiment took approximately 30 minutes to complete.
4.1.4 Analysis
In the main experiment, we analyzed both ACC and RT. We excluded RTs of incorrect judgments or those greater than three standard deviations calculated based on each participant’s mean, resulting in 8,786 valid trials. We used logistic mixed-effects models (Jaeger, Reference Jaeger2008) to analyze ACC and linear mixed-effects models (Bates et al., Reference Bates, Mächler, Bolker and Walker2015) to analyze RT. The full model for ACC and RT analyses was the same in both tasks. Each model included three fixed factors: (1) semantic knowledge group with two levels (learners with more semantic knowledge, learners with less semantic knowledge), (2) semantics-prosody congruence with three levels (congruent, incongruent,and neutral) and (3) emotional prosody with two levels (positive, negative). The more knowledge group, neutral semantics-prosody congruence, and positive prosody valence were treated as the reference levels, respectively. Dummy coding was used, and we entered intercepts for items and subjects (Cunnings, Reference Cunnings2012). In addition, we examined the RT distributions in the two tasks separately using the diffusion model analysis (Brysbaert et al., Reference Brysbaert, Lagrou and Stevens2017; Myers et al., Reference Myers, Interian and Moustafa2022).
4.2 Results
4.2.1 ACC results
Overall, L2 Chinese learners exhibited a similar accuracy in perceiving emotional prosody in both word task (learners with more semantic knowledge: M = 84.4%, SD = 0.36; learners with less semantic knowledge: M = 84.6%, SD = 0.36) and sentence task (learners with more semantic knowledge: M = 89.1%, SD = 0.31; learners with less semantic knowledge: M = 89.8%, SD = 0.30). Compared to the neutral condition, both learner groups demonstrated higher accuracy in the semantics-prosody congruent condition (Word Task: Mean diff = 6.3%; Sentence Task: Mean diff = 3.3%) and lower accuracy in the incongruent condition (Word Task: Mean diff = 9.4%; Sentence Task: Mean diff = 5.8%) (Figure 2).
Mean accuracy of emotional prosody judgments in the word task (left) and the sentence task (right). The black vertical lines show the standard errors. Orange brackets indicate significant between-group differences in the congruent condition, and green brackets indicate significant between-group differences in the incongruent condition. ***p < .001, **p < .01, *p < .05.
Figure 2. Long description
The figure consists of two side-by-side bar graphs. Both graphs share a Y-axis labeled mean accuracy ranging from 0.6 to 1.0. A legend on the right identifies three congruence conditions: orange for congruent, green for incongruent, and white for neutral.
Left Panel: Word task. The X-axis is divided into two groups: learners with more semantic knowledge and learners with less semantic knowledge. Each group is further divided into mean accuracy of emotion judgments in negative and positive prosody.
* For learners with more semantic knowledge, congruent accuracy is high (approx. 0.94), neutral is moderate (approx. 0.85, and incongruent is lowest (approx. 0.73).
* For learners with less semantic knowledge, congruent accuracy is lower than the more semantic knowledge group (approx. 0.88), while incongruent accuracy is higher (approx. 0.85).
* Orange brackets with three asterisks indicate significant group differences for congruent conditions. A green bracket with one asterisk indicates a significant group difference for the incongruent condition .
Right Panel: Sentence task. Similar X-axis structure.
* For learners with more semantic knowledge, congruent accuracy remains high (approx. 0.95), while incongruent accuracy is around 0.82.
* For learners with less semantic knowledge, congruent accuracy is approx. 0.90.
* Orange brackets with three asterisks show significant group differences for congruent conditions. Green brackets with one asterisk show significant group differences for incongruent conditions .
For the word task (Supplementary Table S2), a logistic mixed-effects regression model indicated that learners with more semantic knowledge and learners with less semantic knowledge both showed semantics-prosody (in)congruence effects: compared to the neutral condition, their ACC of emotional prosody perception was significantly higher in the congruent condition (β = 1.200, SE = 0.202, z = 5.932, p < .001) and significantly lower than in the incongruent condition (β = −0.982, SE = 0.155, z = −6.331, p < .001). Furthermore, there were significant interactions between group and congruence, where learners with less semantic knowledge showed a smaller congruence effect (β = −0.864, SE = 0.251, z = −3.439, p < .001) and a smaller incongruence effect (β = 0.421, SE = 0.200, z = 2.111, p < .05) compared to learners with more semantic knowledge. In addition, a significant interaction between group and emotion prosody was found such that learners with less semantic knowledge were more accurate in recognizing positive emotional prosody compared to learners with more semantic knowledge (β = 0.770, SE = 0.177, z = 4.333, p < .001).
Similar patterns were also found in the sentence task (Supplementary Table S4). Both learner groups recognized emotional prosody perception more accurately in the congruent condition (β = 1.096, SE = 0.227, z = 4.822, p < .001), and less accurately in the incongruent condition (β = −1.026, SE = 0.175, z = −5.874, p < .001). Furthermore, there were significant interactions between group and congruence such that the effects of congruence (β = −1.117, SE = 0.296, z = −3.781, p < .001) and incongruence (β = 0.611, SE = 0.246, z = 2.484, p < .05) were reduced on the perception of emotional prosody for learners with less semantic knowledge. Additionally, we also found a significant interaction between group and emotion prosody, where learners with less semantic knowledge recognized positive prosody more accurately than those with more semantic knowledge (β = 0.763, SE = 0.217, z = 3.519, p < .001).
4.2.2 RT results
As shown in Figure 3, learners with more semantic knowledge were slower in their recognition of emotional prosody than learners with less semantic knowledge in both the word task (Mean diff = 194 ms) and the sentence task (Mean diff = 186 ms).
Mean reaction time in milliseconds for emotional prosody judgments in the word task (left) and the sentence task (right). The black vertical lines show the standard errors. Orange brackets indicate significant between-group differences in the congruent condition, and black brackets indicate significant between-group differences in the positive prosody. ***p < .001, **p < .01, *p < .05.
Figure 3. Long description
The figure consists of two side-by-side bar graphs. Both share a Y-axis labeled mean reaction time ranging from 400 to 1600 ms. A legend on the right identifies three congruence conditions: orange for congruent, green for incongruent, and white for neutral.
Left Panel: Word task. Divided into two sub-sections: Learners with more semantic knowledge and Learners with less semantic knowledge. Each sub-section contains two clusters of bars for the reaction times of emotional prosody judgments in negative and positive prosody. For learners with more knowledge, reaction times are consistently high, around 1600 milliseconds. For learners with less knowledge, reaction times are lower, between 1300 and 1500 milliseconds. A black bracket with three asterisks indicates a significant difference between the two groups for positive prosody.
Right Panel: Sentence task. Also divided into more and less semantic knowledge groups. For learners with more knowledge, reaction times for negative prosody are around 1700 milliseconds and positive prosody around 1550 milliseconds. For learners with less knowledge, times are lower, between 1300 and 1500 milliseconds. Orange brackets with one asterisks indicate significant group differences specifically for the congruent condition in both negative and positive prosody.
Linear mixed-effect models revealed that in the word task (Supplementary Table S3), there was a significant interaction between group and emotional prosody, indicating that learners with less semantic knowledge were quicker in recognizing positive emotional prosody than learners with more semantic knowledge (β = −132.945, SE = 36.806, t = −3.612, p < .001). However, no significant main effects of congruence, group, or their interaction were found on the RTs of emotional prosody perception in Chinese words.
In the sentence task (Supplementary Table S5), both learner groups recognized positive prosody significantly faster than negative prosody (β = −97.159, SE = 23.501, t = −4.134, p < .001) in Chinese sentences. Interestingly, both groups recognized emotional prosody in the congruent condition significantly slower compared to the neutral condition (β = 73.324, SE = 33.575, t = 2.184, p < .05). In addition, there was a significant interaction between group and congruence: learners with less semantic knowledge perceived emotional prosody in the congruent condition significantly faster than those with more semantic knowledge (β = −104.687, SE = 48.593, t = −2.154, p < .05).
We modeled the word and sentence tasks separately for the diffusion model analyses. For each participant, three parameters were estimated: boundary separation, indexing response caution; non-decision time, indexing encoding and motor execution; and drift rate, indexing information processing efficiency. Across both word and sentence tasks, we found no significant group differences in boundary separation (p > 0.19 in both tasks), indicating comparable response caution between learners with more and less semantic knowledge. Furthermore, significant effects were observed for the drift rate. Both learner groups showed a significantly reduced drift rate on the incongruent trials, suggesting less efficient integration of semantic and prosodic cues. Moreover, there were significant interactions between group and congruence: learners with less semantic knowledge exhibited reduced susceptibility to the semantics-prosody incongruence effects compared to learners with more semantic knowledge in the word task (B = 0.284, p = .033) and sentence task (B = 0.313, p = .005). Finally, in the sentence task only, learners with less semantic knowledge showed a significantly faster non-decision time (p = .017), suggesting more rapid sensory encoding and/or motor execution. Collectively, these results indicated that the observed group differences were primarily attributable to their distinct information processing efficiencies, rather than response caution.
5. Discussion
The influence of semantic valence, defined as the stored semantic knowledge about whether an object or event is positive or negative, on the perception of emotional prosody is well-established in native speakers (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017; Itkes & Kron, Reference Itkes and Kron2019). However, this account of semantic valence assumes a stable and relatively uniform baseline of semantic knowledge among participants, which is insufficient to account for emotion processing in L2 learners. Recent research in L2 processing has yielded divergent findings (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016; Cho & Dewaele, Reference Cho and Dewaele2021; Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016; Xiao & Liu, Reference Xiao and Liu2025), sparking a debate regarding whether semantic valence facilitates emotional prosody perception by providing multiple channels of information or interferes with it by increasing the processing load. This investigation into the role of semantic valence in emotional prosody perception is particularly relevant in a tonal language such as Mandarin Chinese, where the same acoustic feature (i.e., pitch) encodes both semantic and prosodic information (Xu, Reference Xu2005).
Therefore, the current study investigated the role of semantic valence in emotional prosody perception in L2 Chinese learners. In doing so, we extend the established account of semantic valence (Itkes et al., Reference Itkes, Kimchi, Haj-Ali, Shapiro and Kron2017; Itkes & Kron, Reference Itkes and Kron2019) to the field of second language acquisition and propose that in the L2 domain, semantic valence should be examined in terms of two distinct components: learners’ semantic knowledge and the inherent semantic valence of the stimuli.
We first conducted a pilot experiment to examine how learners’ semantic knowledge affects their perception of emotional prosody in Chinese words and sentences. Following prior research (Shochi et al., Reference Shochi, Brousse, Guerry, Erickson and Rilliard2016; Xiao & Liu, Reference Xiao and Liu2026), we used learning experience of L2 learners as an estimate of semantic knowledge and categorized them into novice and advanced learners. Building on this, the main experiment investigated the independent and interactive effects of both the learners’ semantic knowledge and the stimuli’s semantic valence on emotional prosody perception in Chinese words and sentences. To improve the measure of semantic knowledge, we trained and tested learners of similar general proficiency on the real words from the target stimulus set and then categorized them as learners with more and less semantic knowledge based on their vocabulary scores (Brysbaert et al., Reference Brysbaert, Lagrou and Stevens2017; Ferré et al., Reference Ferré, Guasch, Stadthagen-González, Hinojosa, Fraga, Marín and Pérez-Sánchez2024; Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020). We also manipulated the congruence of semantic valence and emotional prosody in the stimuli. Overall, our results from the two experiments indicated that L2 Chinese learners could recognize emotional prosody with high accuracy and rapid response time, regardless of their level of semantic knowledge. The following section discusses the role of semantic valence in emotional prosody perception in L2 Chinese learners.
First, our results underscored the critical role of semantic knowledge in the perception of emotional prosody in L2 contexts. Specifically, in the pilot experiment, novice learners outperformed advanced learners in recognizing positive emotional prosody. This pattern was replicated in the main experiment: learners with less semantic knowledge showed higher accuracy than those with more semantic knowledge in perceiving positive emotional prosody in both word and sentence tasks. These findings converged with prior research evidence that L2 Chinese learners outperformed native Chinese speakers in identifying positive emotional prosody (Xiao & Liu, Reference Xiao and Liu2026; Zhu, Reference Zhu2013) and that L2 English learners with lower proficiency outperformed those with higher proficiency in their positive emotional prosody perception (Bhatara et al., Reference Bhatara, Laukka, Boll-Avetisyan, Granjon, Anger Elfenbein and Bänziger2016). One interpretation is that this advantage for positive prosody in learners with less semantic knowledge reflects an asymmetry in bilingual emotion processing. Negative prosody often functions as a warning signal and tends to be acoustically more distinct and readily recognizable universally, whereas positive prosody is typically more subtle or culturally variable (Laukka & Elfenbein, Reference Laukka and Elfenbein2021; Sauter et al., Reference Sauter, Eisner, Ekman and Scott2010). When prosodic cues are less salient, as in positive emotions, learners with more semantic knowledge may weight semantic valence more heavily, which can bias their judgments if semantic and prosodic cues are incongruent. Learners with less semantic knowledge, by contrast, may rely more consistently on prosodic cues, resulting in a relative advantage in recognizing positive emotional prosody. These findings also provide novel support for the Theory of Constructed Emotion (Barrett, Reference Barrett2006, Reference Barrett2017; Barrett et al., Reference Barrett, Lindquist and Gendron2007; Lindquist et al., Reference Lindquist, Barrett, Bliss-Moreau and Russell2006, Reference Lindquist, Satpute and Gendron2015; Lindquist & Gendron, Reference Lindquist and Gendron2013) in a bilingual context, demonstrating that L2 learners actively draw on their semantic knowledge to interpret emotional prosody, and, importantly, that these interactions vary with specific type of emotional prosody.
We also found that the semantic valence of the stimuli, specifically, the (in)congruence between semantic valence and emotional prosody, significantly influenced the perception of emotional prosody for L2 Chinese learners. Building on previous evidence from native speakers (e.g., Cho & Dewaele, Reference Cho and Dewaele2021; Lin et al., Reference Lin, Ding and Zhang2020; Min & Schirmer, Reference Min and Schirmer2011; Xiao & Liu, Reference Xiao and Liu2025), we explored the semantics-prosody (in)congruence effects on the perception of emotional prosody in learners with more semantic knowledge and learners with less semantic knowledge. We found that both groups showed significant congruence and incongruence effects in perceiving emotional prosody in Chinese words and sentences. Specifically, when semantic valence and emotional prosody were congruent (e.g., semantically positive word rui4zhi4, wise, spoken in a joyful tone), semantic valence facilitated emotional prosody perception. However, when semantic valence and emotional prosody were incongruent (e.g., semantically negative word ao4man4, arrogant, spoken in a joyful tone), semantic valence interfered with emotional prosody perception. These findings indicate robust semantics-prosody (in)congruence effects in L2 learners, regardless of their levels of semantic knowledge.
Moreover, the semantic (in)congruence effects had a greater impact on L2 learners with more semantic knowledge than on those with less semantic knowledge. In the pilot experiment, when the semantic valence of the stimuli was controlled to be neutral, novice learners recognized positive and negative emotional prosody (i.e., semantic-prosody incongruent) even more accurately than advanced learners. We therefore speculated that the semantics-prosody incongruence may hinder emotional prosody perception more for advanced learners than for novice learners, which suggests an interaction between the two components of semantic valence: the learners’ semantic knowledge and the semantic valence of the stimuli. To confirm this hypothesis, in the main experiment, we further examined the interactions between the two components of semantic valence. We found that when semantic valence was congruent with emotional prosody, learners with more semantic knowledge received more facilitation from semantic valence than those with less semantic knowledge. However, increased knowledge of linguistic information did not always enhance the paralinguistic perception of emotional prosody: learners with more semantic knowledge also experienced greater interference than those with less semantic knowledge when semantic valence was incongruent with emotional prosody. The findings of (in)congruence effects can be further explained through the three-stage working model of emotional prosody processing (Schirmer & Kotz, Reference Schirmer and Kotz2006). This model posits three stages: an early sensory processing of low-level acoustic properties, an integration of emotionally significant acoustic cues, and a cognition stage involving evaluative judgments and semantic processing. The stronger congruence and incongruence effects observed in learners with more semantic knowledge may suggest their greater engagement of this final cognition stage. Namely, when semantic and prosodic cues are congruent, joint evaluation enhances emotion judgments, resulting in higher accuracy. When cues conflict, integration imposes additional cognitive demands to resolve competing signals, reducing accuracy. This computational cost is also reflected in the RT data: learners with more semantic knowledge are consistently slower in recognizing emotional prosody than those with less semantic knowledge.
How do semantic and prosodic cues interact in bilingual emotion processing? The observed group differences may reflect two possible accounts: a competition account (Lin et al., Reference Lin, Ding and Zhang2020), where semantics-prosody congruent cues facilitate while incongruent cues interfere with processing, or an integration account (Schirmer & Kotz, Reference Schirmer and Kotz2006; Taguchi, Reference Taguchi2008), where L2 learners, depending on their semantic knowledge, attempt to simultaneously integrate multiple cues. While the current study was not designed to adjudicate between these two accounts, we note that semantic information was processed implicitly in our emotion judgment task, as participants were not explicitly instructed to attend to word meaning. Furthermore, our results showed that learners with more semantic knowledge showed slower responses than those with less semantic knowledge regardless of congruence, and they also received greater facilitation and interference from the semantics-prosody (in)congruence effects. These findings may align better with the integration account, where L2 listeners with more semantic knowledge may be more automatic, albeit more effortful, in integrating semantic and prosodic cues during emotion processing than those with less semantic knowledge.
Furthermore, our findings showed that semantic valence influences the ACC and RT of the perception of emotional prosody in distinct ways for L2 learners. In terms of ACC, learners’ semantic knowledge can either facilitate or interfere with accurate emotional prosody recognition, depending on the congruence between semantic valence and emotional prosody in the stimuli. Intriguingly, in terms of RT, L2 learners with less semantic knowledge consistently recognized emotional prosody in words and sentences more quickly than those with more semantic knowledge. This divergence between ACC and RT suggests that semantic valence may act as a double-edged sword: enhancing or hindering ACC, while also introducing a processing cost when learners must integrate both linguistic and paralinguistic cues simultaneously. Subsequent diffusion model analyses further confirmed that the group differences were driven by differences in information processing efficiencies, rather than a simple speed-accuracy trade-off.
While learners with less semantic knowledge may bypass or activate semantic cues when judging emotional prosody, those with more semantic knowledge are likely to reach the third stage and thus be more attuned to L2 semantic information, as predicted by the three-stage model (Schirmer & Kotz, Reference Schirmer and Kotz2006). But why do L2 learners with more semantic knowledge show a slower response in their perception of emotional prosody? One explanation for the observed RT slowdown can be attributed to reduced automaticity in L2 semantic processing (Thoma & Baum, Reference Thoma and Baum2019). For bilinguals with more developed semantic knowledge, the effortful retrieval of lexical meaning and its integration with prosodic information increases cognitive load (Bylund et al., Reference Bylund, Antfolk, Abrahamsson, Olstad, Norrman and Lehtonen2023; Schmidtke, Reference Schmidtke2014), resulting in slower emotion prosody recognition. However, this is not the only explanation. According to the Revised Hierarchical Model (Kroll et al., Reference Kroll, Van Hell, Tokowicz and Green2010; Kroll & Tokowicz, Reference Kroll, Tokowicz and Nicol2001), learners with more semantic knowledge are expected to show slower RTs to access their L2 semantic information, as deeper semantic processing leads to longer response times. This holds true even after a relatively brief L2 exposure (Ferré et al., Reference Ferré, Sánchez‐Casas and Guasch2006; Talamas et al., Reference Talamas, Kroll and Dufour1999). Another alternative explanation draws on the Bilingual Interactive Activation plus model (Dijkstra & van Heuven, Reference Dijkstra and Van Heuven2002), which proposes that bilingual semantic processing involves the non-selective, parallel activation of linguistic representations. As learners with more semantic knowledge successfully access L2 semantic information alongside prosodic cues, they need to resolve the competition between these parallel streams, resulting in an RT slowdown. Notably, the results of the current study cannot adjudicate which account better explains the semantic processing in L2 emotional prosody perception, and future research manipulating these specific cognitive mechanisms is therefore needed.
In summary, this study reveals significant effects of learners’ semantic knowledge, stimuli’s semantic valence, as well as their interactions, on the perception of emotional prosody in Chinese words and sentences. Our findings indicate that while the two components of semantic valence are separable, they jointly modulate how L2 learners perceive emotional prosody. More broadly, this research highlights that individual variability in bilingual semantic knowledge is not merely background noise, but an important factor in affective speech perception. By investigating the role of semantic valence in L2 contexts, we provide new empirical evidence from a tonal language on how knowledge-based perception and feeling-based perception can be integrated in bilingual emotion processing.
Finally, several limitations of the present study warrant consideration. First, the use of pseudowords may inadvertently evoke affective impressions for L2 Chinese learners who are still actively mapping tones to meanings. Future research should systematically manipulate lexical tones and emotional prosody in pseudowords to disentangle these effects. Second, we only examined the perception of basic emotional prosody in L1-English L2-Chinese learners, which may limit the generalizability of the findings. Future investigations are needed to explore the role of semantic valence in learners from diverse L1 backgrounds (e.g., tonal and non-tonal languages) as well as in more nuanced emotional prosody types. Third, this study operationalized semantic knowledge using learning experience and a vocabulary test, yielding empirical evidence on the impact of learners’ semantic knowledge on emotional prosody perception. Building on these findings, future work may benefit from adopting more granular measures (e.g., vocabulary depth, valence-specific or arousal-specific semantic knowledge) to explore which dimensions of semantic knowledge drive these effects in bilingual emotion processing.
6. Conclusions
This study examined two components of semantic valence: learners’ semantic knowledge and the semantic valence of the stimuli, and their interactions in the perception of emotional prosody in a tonal language. Although L2 Chinese learners demonstrated high accuracy and speed in recognizing emotional prosody at word and sentence levels, semantic valence significantly modulated their performance. Specifically, learners’ semantic knowledge facilitated emotional prosody perception when the semantic valence and emotional prosody of the stimuli were congruent but interfered with emotional prosody perception under incongruent conditions. These semantics-prosody (in)congruent effects were more pronounced among learners with more semantic knowledge compared to those with less semantic knowledge. Interestingly, greater semantic knowledge was also associated with slower recognition of positive emotional prosody, suggesting a potential processing cost in bilingual emotion processing. The findings highlight that semantic valence influences emotional prosody perception, suggesting a dynamic link between linguistic knowledge and paralinguistic perception in an L2. This study advances our understanding of how bilinguals simultaneously process semantic and prosodic cues in a tonal language, and offers implications for L2 instruction and cross-cultural communication.
Data availability statement
The data that support the findings of this study are openly available through the Open Science Framework (OSF) repository at https://osf.io/fs4py/overview.
Funding statement
The research was supported by NSF grant BCS-2234914 and NIH/NIDCD grant R01DC017162.
Competing interests
The authors declare no competing interests.
Open access
