Highlights
-
• Develops the first, scalable vignette-based method for assessing productive, context-sensitive emotion vocabulary (EV) in adult English additional language (Lx) users.
-
• Moves EV assessment beyond decontextualised recognition formats by eliciting multiple emotion labels in context.
-
• Shows that 18 of 22 pictorial-text vignettes elicited the intended emotion or a close synonym as the dominant response, with substantial naming agreement.
-
• Demonstrates that accuracy and dominance-based scoring capture complementary dimensions of productive EV: breadth and contextual naming alignment.
-
• Identifies response order, English proficiency and lexical frequency as key predictors of context-appropriate emotion labelling, while context length has no overall effect.
1. Introduction
Emotions shape cognition and decision-making, and are intricately intertwined with language acquisition and use (Lindquist, Reference Lindquist2021; Richards, Reference Richards2020). Language learning is often pervaded by a wide range of emotions (Kanazawa, Reference Kanazawa2021; Pentón Herrera & Darragh, Reference Pentón Herrera and Darragh2024), making emotion words – lexical choices that directly express an emotional state (e.g., “relaxed” and “anxious”; Pavlenko, Reference Pavlenko2008) – crucial for learners. These words are essential for interpreting and communicating thoughts, forming social connections (Sabag-Shushan et al., Reference Sabag-Shushan, Katzir and Prior2023) and effectively navigating the learning process (e.g., Pentón Herrera & Darragh, Reference Pentón Herrera and Darragh2024). Therefore, emotion word recognition, processing and expression serve as a foundation and an outcome of additional language (Lx) learning. Yet most work foregrounds anxiety, motivation and enjoyment (Dewaele et al., Reference Dewaele, Chen, Padilla and Lake2019), leaving emotion vocabulary (EV) acquisition and use underexplored and restricting educators’ access to evidence-based tools for teaching and assessment.
The Common European Framework of Reference for Languages (CEFR; Council of Europe, 2020) explicitly includes EV, expecting Lx learners to be able to express varying degrees of emotions. Previous studies also indicate that low-proficiency learners tended to have limited lexical resources and struggled with producing emotional content (Lam & Marquardt, Reference Lam and Marquardt2023). Despite broad consensus on the crucial role of proficiency in expressing diverse emotions (Council of Europe, 2020), existing vocabulary assessments often overlook EV, concentrating on general or specialised words (Sabag-Shushan et al., Reference Sabag-Shushan, Katzir and Prior2023; Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024). Given the strong theoretical and empirical connections between language and emotions (Altarriba & Basnight-Brown, Reference Altarriba and Basnight-Brown2010; Barrett et al., Reference Barrett, Adolphs, Marsella, Martinez and Pollak2019; Pavlenko, Reference Pavlenko2008) and the effect of EV on meaning construction (Lindquist et al., Reference Lindquist, MacCormack and Shablack2015), differentiating emotion words from other vocabulary is essential. This is further underscored by the unique acquisition patterns of emotion words that influence both memory and language processing (Dewaele & Pavlenko, Reference Dewaele and Pavlenko2002; Kanazawa, Reference Kanazawa2021; Sabag-Shushan et al., Reference Sabag-Shushan, Katzir and Prior2023). Treating EV as a distinct category is therefore both a theoretical and practical imperative for understanding how Lx learners acquire, store, and retrieve these words (Lindquist et al., Reference Lindquist, MacCormack and Shablack2015).
Existing EV measures target only a limited range of emotions and are predominantly recognition-based (e.g., Bazhydai et al., Reference Bazhydai, Ivcevic, Brackett and Widen2019; Costa et al., Reference Costa, Faria and Takšić2011; Sturrock & Freed, Reference Sturrock and Freed2023), risking overestimation of productive knowledge and not capturing the full range of emotion words known by learners. There is also a lack of validation evidence for these tests in English Lx contexts. Additionally, emotion-related methodologies in applied linguistics heavily rely on self-report methods, including quantitative surveys, and do not aim to measure affective language knowledge (Dewaele et al., Reference Dewaele, Chen, Padilla and Lake2019).
Taken together, there is a need for a validated, context-rich, productive EV test that can assess a wide range of emotion words across cultures and languages. The aim of this exploratory study is to provide preliminary validity evidence for the Productive Emotion Vocabulary Test (PEVT), a pictorial and textual vignette-based free emotion word production task designed for research purposes.
2. Literature review
2.1. Measuring vocabulary knowledge
Within language testing, vocabulary knowledge is typically operationalised as breadth (size) and depth (quality) with receptive and productive formats (e.g., Meara & Miralpeix, Reference Meara and Miralpeix2017; Nation, Reference Nation2022; Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024). As a multidimensional and complex construct, vocabulary knowledge has spawned numerous tests and their psychometric and conceptual critiques are reviewed elsewhere (e.g., González-Fernández, Reference González-Fernández2025; Meara & Miralpeix, Reference Meara and Miralpeix2017; Nation, Reference Nation2022; Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024). Widely used breadth tests are largely receptive, assessing how many words are known at selected frequency bands (e.g., Vocabulary Size Test, VST; Nation & Beglar, Reference Nation and Beglar2007). Depth entails how well specific words are known and the extent to which learners can use them correctly and/or appropriately in different registers or contexts (Nation, Reference Nation2022). Measures of depth (e.g., Word Associates Test, WAT, Read, Reference Read1998) as the “woolliest, least definable and least operationalisable construct” (Schmitt & Schmitt, Reference Schmitt and Schmitt2014, p. 920), tend to largely focus on associative and collocational knowledge (but see González-Fernández, Reference González-Fernández2025). Productive measures include controlled (e.g., Productive Vocabulary Levels Test PVLT; Laufer & Nation, Reference Laufer and Nation1999) and free-association formats (e.g., Lex30, Meara & Fitzpatrick, Reference Meara and Fitzpatrick2000). These tests typically restrict responses to one word or use cue words or topics and require relatively large number of words to reliably compare high- and low-frequency words and discriminate between learners (Meara & Miralpeix, Reference Meara and Miralpeix2017).
Most of the above tests rely on decontextualised written cues and responses (Read, Reference Read1998), reflecting a strong tendency in vocabulary studies. The limited number of tests that utilise pictures as stimuli (Peabody Picture Vocabulary Test, PPVT, Dunn, Reference Dunn1959 and later versions; and others in Meara & Miralpeix, Reference Meara and Miralpeix2017) mostly employ receptive, multiple-choice formats and focus on concrete lexis rather than providing context. A cartoon-character test (Meara & Miralpeix, Reference Meara and Miralpeix2017) elicits mainly adjectives based on a character out of context (i.e., static character, no scenario). Although written decontextualised test formats assess learners’ construct-specific word knowledge (Read, Reference Read1998), which is useful for general and academic vocabulary testing, emotion labels are often contextually licensed categorisations (i.e., flexible, not universal, but context-dependent and constructed) and are closely tied to appraisal cues within contexts (e.g., scenarios or experiences). Consequently, decontextualised written prompts (e.g., meaning recognition or synonym fluency formats) with insufficient appraisal or situational information may under-sample the intended construct (e.g., productive/contextualised emotion labelling in English). Along these lines, the present study examined whether the diverse affordances of multimodal cues (e.g., facial expressions, posture and appraisal text) in the vignettes are sufficient to facilitate plausible label convergence. Furthermore, this enables observing lexical variability in labelling, thus potentially enhancing ecological validity relative to decontextualised formats.
The above-reviewed instruments have high utility for measuring general and academic vocabulary knowledge in English Lx populations, but they predominantly target a single construct, use decontextualised or otherwise constrained stimuli, and are not readily adaptable to domain-specific lexis such as emotion words. This study explores how the PEVT complements this portfolio by targeting productive EV in context. In breadth terms, it estimates how many context-appropriate emotion labels participants can produce. In depth terms, we focus on one narrow, operationalised facet: contextual naming alignment. This facet is linked to Nation’s (Reference Nation2022) dimensions of concepts and referents and constraints on use. In the PEVT, the vignette supplies the referential and appraisal context, and participants must retrieve labels that fit the inferred affective state, intensity and situation. In this study, depth therefore goes beyond conventional, oft-used operationalisations by targeting the use and appropriacy of the produced form in a specific context and its alignment with empirically derived in-sample naming patterns (response dominance), rather than with the general corpus-based frequency of a word considered outside context. Alignment with empirically observed naming patterns for a vignette is thus treated as an indicator of contextual appropriacy and use, not as a comprehensive measure of vocabulary depth. This operationalisation differs from traditional depth measures, such as associative or collocational tests, but follows the same broader logic of assessing the quality and contextual use of lexical knowledge rather than only the number of words known.
2.2. Conceptualising emotion vocabulary
Researchers frame emotions variably, influenced by their distinct theoretical frameworks and research objectives (Lindquist, Reference Lindquist2021). These differing, often conflicting, conceptualisations of emotion words commonly use two key approaches: the prototype and dimensional models (e.g., Grosse et al., Reference Grosse, Streubel, Gunzenhauser and Saalbach2021). Prototype models arrange emotion terms hierarchically: core labels (e.g., “anger”, “joy”) operate as prototypes for secondary emotions (e.g., “contempt”, “pride”; Shaver et al., Reference Shaver, Schwartz, Kirson and O’Connor1987; Watson & Stanton, Reference Watson and Stanton2017). Dimensional models, by contrast, locate emotions along valence-arousal continua (Warriner et al., Reference Warriner, Kuperman and Brysbaert2013). While these perspectives illuminate conceptual structure, they are less directly actionable for item sampling and scoring: prototype lists privilege a few largely high-frequency words (e.g., “sad”), and dimensional databases can intermingle emotion-denoting words with merely emotion-laden words (e.g., Affective Norms for English Words; Bradley & Lang, Reference Bradley and Lang1999).
To obtain a linguistically transparent sampling and scoring frame for emotion words, we adopt Pavlenko’s (Reference Pavlenko2008) tripartite taxonomy, distinguishing: (1) emotion words directly naming affective states or processes (“happy”, “to worry”), (2) emotion-related words describing behaviours or physiological concomitants (“tears”, “to scream”) and (3) emotion-laden words evoking affect through social or evaluative meaning (“cancer”, “jerk”, “darling”). However, in this study, we operationalise EV using only categories 1 and 2. The reason is twofold. First, the PEVT primarily targets felt-state labels. Emotion words naturally complete the “one feels…” frame, whereas emotion-related words frequently function as emotion descriptors in psychological literature and are often counted as proper emotion words in studies (Pavlenko, Reference Pavlenko2008). Second, research indicates that emotion-laden words are processed differently than the other two categories (Altarriba & Basnight-Brown, Reference Altarriba and Basnight-Brown2010; Zhang et al., Reference Zhang, Wu, Meng and Yuan2017). Because emotion-laden words encapsulate a wide range of emotion-invoking content words, often captured in general contexts (e.g., “spider”, “marriage”) and their interpretation depends heavily on socio-pragmatics, register and context, including them would introduce variance unattributable to the target construct. By excluding category 3 words, the PEVT maintains a precise focus on descriptors of internal states rather than general vocabulary and obtains a consistent framework for both item selection and scoring.
2.3. Measuring emotion vocabulary
Given the symbiotic relationship between emotions and words (Pentón Herrera & Darragh, Reference Pentón Herrera and Darragh2024), EV is paramount for emotional development, which in turn affects both emotional literacy – the ability to recognise, name and express emotions – and emotional intelligence (EI), which involves managing, responding to and communicating emotions effectively (Mayer et al., Reference Mayer, Caruso and Salovey2016; Sturrock & Freed, Reference Sturrock and Freed2023). Because precise emotional labelling is crucial for affective communication competence (Pentón Herrera & Darragh, Reference Pentón Herrera and Darragh2024) and EI (e.g., Lindquist et al., Reference Lindquist, MacCormack and Shablack2015; Pavlenko, Reference Pavlenko2008; Sturrock & Freed, Reference Sturrock and Freed2023), some EV tests have been used as a proxy for EI. These tests have notable limitations for the present purpose, however.
Most tools treat EV as a receptive construct embedded in EI batteries. For instance, the 35-item Vocabulary of Emotions Test (VET; Costa et al., Reference Costa, Faria and Takšić2011) uses form-meaning recognition of adjectives describing a state or mood without appraisal context, and the item pool is relatively small, comprising adjectives and dictionary definitions only. Similarly, the Mayer–Salovey–Caruso Emotional Intelligence Test “Understanding Emotions” branch (MSCEIT; Mayer et al., Reference Mayer, Caruso and Salovey2016) and the Geneva Emotion Recognition Test (GERT; Schlegel et al., Reference Schlegel, Fontaine and Scherer2019) require label selection from a short list. Importantly, such recognition formats primarily index receptive EV knowledge, whereas the present study targets productive emotion labelling in context. Across these instruments, the core limitations for our purpose include limited target words, fixed option sets that may limit sensitivity to gradations of knowledge, lack of contextual detail and scant validation with adult English Lx populations. This gap is further exacerbated by conventional Lx vocabulary tests, which largely employ recognition formats that primarily index receptive vocabulary knowledge and do not target domain-specific emotion lexis. Although these instruments are well suited for their intended use (e.g., receptive recognition with constrained options), the context-dependent nature of emotion words points to the need for tasks that assess productive EV knowledge and use in response to situational and appraisal cues. Arguably, a larger number of items with multiple responses may more effectively capture learners’ ability to express varying degrees of emotions as per CEFR descriptors (Council of Europe, 2020).
Productive alternatives (e.g., word-fluency or label-generation tasks; Bazhydai et al., Reference Bazhydai, Ivcevic, Brackett and Widen2019; Mavrou, Reference Mavrou2021) allow flexible responses to prompts like “happy” but depend on test-taker motivation to generate synonyms, use decontextualised cues and lack standardised scoring limiting comparability and ecological validity, especially in English Lx. The use of core emotion words only may also result in omission of early-acquired emotions (e.g., “love”, “hate”), essential for English Lx classrooms. This reemphasises the necessity for a more theoretically aligned, integrative test framework integrating both language and emotions within context.
The Theory of Constructed Emotions posits that emotions are contextually dependent and strongly shaped by personal experiences and other individual factors (e.g., linguaculture; Barrett, Reference Barrett2017a, Reference Barrett2017b; Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021). This contextuality is evident in situations where the same physical responses, such as crying, can occur when experiencing happiness, sadness, or grief, each indicating a distinct emotional state. Context-free test formats, in turn, further complicate EV assessment by largely tapping into linguistic processing, thereby underrepresenting learners’ ability to map language onto emotions (i.e., to appraise a situation based on context and express emotions).
The vignette methodology offers a promising solution (Goetze, Reference Goetze2023). Although vignettes may vary in their content (see Goetze, Reference Goetze2023), those of interest herein comprise scenarios with text and image to furnish ample contextual information for realistic emotional stimuli, thus closely resonating with constructed-emotion accounts (Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021). For instance, the Children’s Emotion-Specific Vocabulary Vignettes Test (CEVVT; Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020) uses 20 vignettes to elicit EV from German-speaking children via narrated scenarios depicting a protagonist in situations, physiological reactions and thoughts. It has been shown to effectively capture a wide range of emotion lexicon in children and adolescents (Grosse & Streubel, Reference Grosse and Streubel2024; Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020). This indicates its capacity to reliably assess both the number of different emotion words produced (vocabulary size) and the extent to which these align with the adolescent participants’ naming patterns (a narrow facet of vocabulary depth). Building on this, we adapt the CEVVT for adult English Lx users (see Figure 1), changing the format to written free recall, expanding the item bank to 22 vignettes and automating delivery for large-scale research. The design targets productive EV while retaining contextual authenticity.
Sample item from the PEVT: “pride”.
Note: Participants were presented with either (A) high- or (B) low-context version.
2.4. Emotion vocabulary and influencing factors
EV acquisition is also shaped by individual factors, including language proficiency, linguistic background and gender. Linguistic background herein is operationalised as self-reported English L1 vs. Lx, while English proficiency is controlled for. Studies have found that most individuals express emotions more freely in their L1, whereby earlier-acquired languages carry more emotional weight while Lx usage often lacks the same emotional depth (Ferré et al., Reference Ferré, Guasch, Stadthagen-Gonzalez and Comesaña2022; Pavlenko, Reference Pavlenko2008, Reference Pavlenko2012). Lx learners sometimes perceive Lx words as less intense or disembodied (even fake) compared to their L1 counterparts (Pavlenko, Reference Pavlenko2008). Consequently, Lx users may face constraints in identifying and articulating more specific or subtle emotions, resulting in the production of high-frequency (Pérez-García & Sánchez, Reference Pérez-García and Sánchez2020; Yeh et al., 2007) or less emotionally intense emotion words (Ferré et al., Reference Ferré, Guasch, Stadthagen-Gonzalez and Comesaña2022; Pavlenko, Reference Pavlenko2012). Ferré et al. (Reference Ferré, Guasch, Stadthagen-Gonzalez and Comesaña2022) found that emotionally charged words were also better retained than neutral words. Moreover, lower-proficiency learners often resort to generic terms with intensifiers (e.g., “very happy/sad”) rather than precise or lower-frequency alternatives (e.g., “thrilled”, “despair”; Pérez-García & Sánchez, Reference Pérez-García and Sánchez2020). This pattern aligns with the lexical entrenchment (Diependaele et al., Reference Diependaele, Lemhöfer and Brysbaert2013; Whitford & Titone, Reference Whitford and Titone2019) and lexical quality hypotheses (Baek et al., Reference Baek, Lee and Choi2023; Perfetti, Reference Perfetti2007) positing that high-frequency words or those used more often have higher quality representations in the mental lexicon compared to low-frequency words, thus resulting in faster and more accurate recognition and processing, and, by extension, production (Whitford & Titone, Reference Whitford and Titone2019). Taken together, EV testing faces a twofold challenge. First, EV tests need to be sensitive to differences in EV knowledge associated with proficiency and linguistic background. Second, because high-frequency words are commonly recalled first, the testing procedure must elicit alternative responses to capture learners’ more specific and differentiated emotional lexicon. These issues may be further exacerbated by other individual factors: higher EI has been associated with greater accuracy, and females typically outperform males in recognising and producing emotion words (Bazhydai et al., Reference Bazhydai, Ivcevic, Brackett and Widen2019; Chee et al., Reference Chee, Szabo and Ambrose2026; Mavrou, Reference Mavrou2021).
In addition to individual differences, test design features may influence productive EV responses in vignette-based tasks. Since emotion labels are closely tied to appraisal cues and situational information, varying the amount of contextual support (e.g., high vs. low context) enables examining how much information is needed for adult participants to converge on plausible labels, while also assessing the potential construct-irrelevant variance introduced by additional text. Moreover, response order may influence the produced labels as lexical retrieval may be affected by accessibility and frequency. Accordingly, vignettes may elicit more conventional and higher-frequency labels earlier, consistent with the least-effort account of lexical choice (Zipf, Reference Zipf1949) and frequency-driven accessibility in vocabulary use (Schmitt & Schmitt, Reference Schmitt and Schmitt2014). We therefore investigate context length as a planned design factor and treat response order as an exploratory factor when evaluating the EV produced.
In sum, despite the paramount role of emotions in language learning, research on EV acquisition is hindered by the scarcity of domain-specific tests for Lx contexts. This limits evidence-informed support for learners who struggle to recognise and verbally express their own and others’ emotions (Dewaele & Pavlenko, Reference Dewaele and Pavlenko2002; Pentón Herrera & Darragh, Reference Pentón Herrera and Darragh2024). To mitigate this gap, we examine how the vignette methodology performs with adult users of English Lx and present preliminary validation evidence for the PEVT. In this study, we treat validity as an argument about score interpretation and use, supported by multiple sources of evidence (Chapelle, Reference Chapelle2012; Kane, Reference Kane2012; Supplementary Table B2 in Supplementary Materials). We present preliminary validity evidence for the PEVT as a research instrument for assessing productive contextualised emotion labelling in adult English Lx users: retrieving context-appropriate emotion or emotion-related words in response to visual, situational and appraisal cues. This task necessarily involves interpreting the situation based on appraisal cues, mapping these cues to a plausible emotion category/affective state, retrieving context-appropriate lexical labels, selecting labels that match in intensity, context, inter alia and producing up to three labels in writing. Depending on the vignette and participants’ language proficiency, response production may involve additional processes such as cross-language mediation or translation, particularly for lower-proficiency Lx users. However, the exact nature of these processes is beyond the current scope. Accordingly, we interpret accuracy scores as a measure of context-appropriate productive EV breadth: the number of acceptable emotion or emotion-related labels produced. We interpret dominant scores as an index of contextual naming alignment: the extent to which a response matches empirically observed naming patterns for the vignette. Following the vocabulary-depth rationale outlined above, this is treated as a narrow facet of depth related to concepts and referents and constraints on use, rather than as a comprehensive measure of vocabulary depth. Accordingly, we do not claim that the PEVT provides a comprehensive measure of emotion recognition ability and it is not intended as a direct measure of EI or emotional literacy. Importantly, we also acknowledge that it is not linguaculturally independent and cross-linguistic norming is a key goal of ongoing work.
3. The present study
This study examines the new format of the test, item elicitation, scoring comparisons and whether context, response order, language proficiency and frequency affect the results. We address the following research questions:
RQ1: To what extent do the vignettes elicit the target emotion (or close synonyms) as dominant responses?
RQ2: Do context length (high vs. low) and response order affect response accuracy?
RQ3: To what extent are accuracy and dominant scores associated with language proficiency (LexTALE), and does this association hold while accounting for produced-word frequency?
Across these RQs, context length and proficiency are treated as planned predictors; response order and produced-word frequency are included as exploratory predictors to probe response-process influences on accuracy and unweighted-dominant responses. RQ1 provides content- and response-process evidence by evaluating whether vignettes elicit intended targets and by quantifying naming agreement. RQ2 provides response-process evidence by testing whether context length and response order systematically affect responses. RQ3 provides preliminary evidence of associations between PEVT scores and LexTALE, alongside comparisons of scoring specifications.
3.1. The development of the PEVT
To meet the call for a validated EV test for adult English Lx users, we developed the PEVT, a vignette-based form recall test, assessing a wide range of emotion words. We adapted the CEVVT vignette methodology (Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020) for its favourable productive format and close alignment with the Theory of Constructed Emotions (Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021). The PEVT presents contextual information and the protagonist’s thoughts in written form (Figure 1). Unlike the CEVVT’s interview format, participants write down up to three emotion words per vignette, allowing straightforward adaptation to a digital survey format and scalability. This makes testing more cost- and time-efficient compared to interviews.
The PEVT contains 22 items (six basic and 16 complex emotions), comprising eight positive and 14 negative scenarios, while holding the format constant. We do not claim the current bank exhaustively samples the full EV domain (e.g., across all elicitor types, i.e., including social-interpersonal vs. achievement emotions, or affective dimensions – valence, arousal and dominance ratings); rather, it provides an initial, theory-informed item set for instrument-development and preliminary validity evidence. A more robust item-sampling criterion is applied in Chee et al. (Reference Chee, Szabo and Ambrose2025), which expands the current item bank. Item attributes for the current study are summarised in Supplementary Table A3. Twenty items are English translations of the German words from the CEVVT, selected from a 52-vignette pool using two criteria: the targeted emotion was named (1) by at least half of an adult norming group and (2) more frequently (p < .05) than the other targets in the final set (Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020).
The vignettes were first redrawn digitally to standardise different characteristics (emotion-typical thoughts and actions and physiological expressions) to enable creating a larger item bank (see Chee et al., Reference Chee, Szabo and Ambrose2025). Two novel pilot items (“bored”, “anticipation”) were added, ensuring these depict distinct emotions. Vignettes were approved by five raters involved in the project, as well as an external senior researcher with expertise in experimental psychology, particularly in image and word processing studies to establish face validity and consistency across the 22 items.
The vignettes provide visual context by depicting a protagonist with emotion-specific facial expressions and bodily postures alongside textual context. Children may benefit from more contextual information, but to examine the amount of contextual information needed for optimal elicitation in adults, two parallel versions of PEVT were created. Each participant saw 11 high-context (situation and thought/appraisal) and 11 low-context (thought/appraisal only) vignettes in random order to control for question-order effects. The item-context condition was counterbalanced across the two versions, so each vignette appeared for ~50% of participants. As an exploratory open question, we also examined whether the response order has an impact on the scoring method used.
The PEVT employed a free production format with up to three responses per vignette aiming to capture not only high-frequency words but also more specific emotion words (e.g., “happy” vs. “content”). Given that emotion recognition and expression are not universal (Barrett, Reference Barrett2017a; Reference Barrett2017b), the three-response format appeared reasonable to verify target response elicitation and naming agreement. Moreover, it balances capturing varied emotion labels with practical considerations for low-proficiency participants. In turn, this approach not only circumvents the limitations associated with receptive or restrictive productive tasks, such as single-word translation tasks, but also enhances the granularity of measurement, allowing for greater discrimination in assessing the construct.
3.2. Scoring the PEVT
Positioned within established vocabulary constructs, the PEVT uses two complementary scoring approaches. First, accuracy scoring indexes productive EV breadth by counting the number of context-appropriate emotion or emotion-related labels produced. Second, dominance-based scoring indexes contextual naming alignment: the extent to which responses correspond to the labels most commonly produced for the same vignette. We treat this latter score as a narrow facet of depth because it concerns how well a lexical item fits a specific referential and appraisal context, linking it to concepts and referents and constraints on use in Nation’s (Reference Nation2022) vocabulary knowledge framework. It does not assess broader depth components such as collocational, morphological, polysemous or register knowledge. The advantage of the vignette-based methodology with a recall format adopted here is that it allows for a systematic and methodical examination of both aspects of vocabulary knowledge.
As the PEVT targets productive contextualised emotion labelling, not spelling or part of speech, minor spelling errors were corrected and normalised to corresponding headwords (e.g., “happiness” and “happier” were coded as “happy”). For each vignette, we calculated dominant scores by aggregating the responses for each vignette and identifying the three most frequent headwords essentially reflecting naming agreement (Supplementary Material A). Under dominant scoring, three sub-scores were calculated: top-dominant, rewarding only the most frequent response, unweighted-dominant, treating the three most dominant responses equally and weighted-dominant, grading responses according to their rank. Internal consistency was moderate (Cronbach’s α = .66, .71, .74 for top-, unweighted-, weighted-dominant, respectively).
Although dominant scores enable evaluating how well individual responses align with the larger population and item performance (i.e., determining how well target responses are elicited), these may not fully capture lexical richness. For a productive test, it could be too restrictive, and even counterintuitive, to penalise appropriate unique or low-frequency responses. Therefore, accuracy scoring disregards response frequency and relies on rater judgement based on the criteria: an emotion or emotion-related word based on the emotion word list (Szabo et al., forthcoming), which follows Pavlenko’s definition, a close synonym of the expected/dominant response, or in the case of complex emotions, the corresponding basic emotion category and an appropriate emotion for the vignette. Two online dictionaries, the Encyclopaedia Britannica (2024) and Merriam-Webster Dictionary (2024), were consulted. Inter-rater reliability for accuracy coding was estimated via Cohen’s κ at the item level (binary: accurate vs. inaccurate). Overall κ = .38 indicated fair-to-moderate agreement (69%; N = 1,910), while per-item κ averaged .40 (range .15–.68; 9/22 ≥ .50). The gap between κ and raw agreement indicates that accuracy scoring involves some challenges due to the subjective nature of emotions and contextualised emotion labelling: several labels might be plausible for the same vignette but differ in intensity, specificity or appraisal fit. Additionally, vague or circular dictionary definitions (e.g., “overjoyed: feeling great joy”, Merriam-Webster, 2024) also required raters to consider alternative senses and uses of the word in different contexts. Disagreements were primarily resolved through discussion between the raters, considering the correct application of the scoring criteria, dictionary definitions, potential contextual use, emotional intensity and appraisal fit for the vignette. After two rounds of evaluation and adjudication, raters reached 87.39% agreement overall (per-item range 79.78%–94.29%). Only four responses out of 1,910 required third-rater adjudication. These results indicate that future iterations require more explicit scoring rubrics, including clearer decision rules for near-synonyms, basic-category labels, intensity mismatches and contextually plausible but non-dominant responses. Internal consistency for accuracy scoring was moderate to high (α = .79).
Considering the inseparable relationship between vocabulary knowledge and frequency (Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024), we additionally examined whether weighting the scores by item difficulty or frequency yields a more granular score, thereby enhancing discrimination. To achieve this, scores were multiplied by reverse-coded difficulty indices (i.e., 1-P, if 80% of participants produced that response, it was multiplied by [1–.80 =] .20). Similarly, lower frequency items were weighted by a higher frequency index (i.e., inverse Zipf’s value [1/zipf] sourced from SUBTLEX-UK; van Heuven et al., Reference van Heuven, Mandera, Keuleers and Brysbaert2014). In essence, this meant that lower-frequency or more difficult items obtained a higher weight (See in Supplementary Table A2). Because these weighted variants are sample- and item-bank dependent, we treat them as instrument-development comparisons and report them alongside unweighted scores, including correlations with LexTALE as convergent validity evidence. For the main inferential analyses (context/response-order effects and relations with proficiency), we use accuracy as the primary outcome because it is directly interpretable as the number of context-appropriate emotion labels produced and is most readily transferable to new samples and settings. Additional test characteristics are summarised using Read and Chapelle’s (Reference Read and Chapelle2001) vocabulary assessment framework in Supplementary Material B.
3.3. Participants
An a priori G*Power analysis using linear multiple regression as a participant-level approximation indicated that 68 participants would provide 80% power to detect a medium overall effect,
$ {f}^2=.15 $
, for the two planned predictors, context and LexTALE, at
$ \alpha =.05 $
(Faul et al., Reference Faul, Erdfelder, Lang and Buchner2007). Using snowball sampling, we shared the Qualtrics link via university networks and social media, encouraging participants to share with their connections. As a preliminary validation of items and format, we targeted adult, high-proficiency English users (L1 and Lx). One hundred sixty-one participants took part in this study, with 107 completing all tasks and retained for analyses. Besides incomplete responses, no further data were excluded. Participants (N = 107, Mage = 28.95, SD = 12.18) were primarily students (55%) and predominantly Malaysians (64%) with diverse linguistic backgrounds (Table 1).
Summary of participant demographics
Table 1. Long description
The table consists of five columns with the primary data in the final column titled Participant n (%).
* Gender: Male 38 (36%), Female 69 (64%).
* Age: 18 to 24 60 (56%), 25 plus 47 (44%).
* Language background: English L 1 36 (34%), English L x 71 (66%).
* L 1 s: English 36 (34%), Chinese 35 (33%), Malay 13 (12%), 14 others (e.g., Arabic, Creole and Hindi) 2 3 (22%).
* Country of origin: Malaysia 69 (64%), China 8 (6%), UK 6 (5%), Romania 5 (4%), Indonesia 5 (4%), 13 others (e.g., Sri Lanka, Vietnam and South Korea) 14 (12%).
* Students: 59 (55%).
Note: L 1 English equals English as first language, L x English equals English as an additional language.
Note: L1 English = English as first language, Lx English = English as an additional language.
3.4. Instruments
3.4.1. Lexical test for advanced learners of English (LexTALE)
Language proficiency is closely linked to vocabulary knowledge and, by extension, to EV. Given variability even among L1 speakers, we used LexTALE (Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012) to ascertain that our sample had high language proficiency for validation. This widely used, standardised, untimed visual lexical decision task comprises 40 words and 20 non-words. It has been shown effective in discriminating between advanced English speakers, with completion taking around four minutes. We employed averaged-percent-correct scoring, correcting for the unequal distribution of words and non-words (Table 2). The reliability analysis yielded a Cronbach’s alpha of .86.
Language proficiency distribution based on LexTALE
Table 2. Long description
The table consists of three columns: Proficiency Level, Range, and N (sample size).
* The first row under the header indicates the Advanced level, defined by a Range greater than or equal to 80, with an N of 82.
* The second row indicates the Upper intermediate level, defined by a Range of 60 to 79, with an N of 22.
* The third row indicates the Lower intermediate level, defined by a Range less than 60, with an N of 3.
A note at the bottom states that these bins were used for descriptive purposes only and that analyses used continuous scores.
Note: Bins used for descriptive purposes only; analyses used continuous scores.
3.4.2. Language history questionnaire (LHQ3)
Language History Questionnaire (LHQ3) (Li et al., Reference Li, Zhang, Yu and Zhao2020) features the most common questions from 41 published questionnaires including self-assessment of L1/L2 proficiency, bilingual history and the home-language environment. Given our focus, only the questions related to language proficiency were utilised, requiring participants to rate their proficiency levels on a scale of 1–7 in listening, speaking, reading and writing. Internal consistency of the four-item scale was α = .93. Aggregated scoring was then used to calculate self-rated proficiency (Li et al., Reference Li, Zhang, Yu and Zhao2020).
3.5. Procedures
Ethical approval was granted at faculty level at the University of Nottingham Malaysia (FASS/RE/EDU/2023-03/0006). Participants provided electronic informed consent and completed the study online via Qualtrics, following the order: LexTALE, PEVT, and LHQ3 (Figure 2).
Visual flowchart for procedures, study design and counterbalancing.
Figure 2. Long description
The flowchart begins at the top with Participants N = 107 adults. An arrow points down to Lex T A L E, an English proficiency measure. From here, the path splits into two parallel groups.
On the left is P E V T Test 1 with n = 53. It includes 22 vignettes in randomized order, divided into Set A n = 11 High-context and Set B n = 11 Low-context.
On the right is P E V T Test 2 with n = 54. It includes 22 vignettes in randomized order, divided into Set A n = 11 Low-context and Set B n = 11 High-context.
Arrows from both test boxes point to a central gray box stating: Counterbalanced across sets: each vignette appears in either High or Low context across participants.
An arrow points down to a box labeled Per vignette: 3 emotion labels Response 1-3.
The final box at the bottom is L H Q 3, the Language History Questionnaire.
3.6. Data analysis
Analyses were organised by RQs and aligned with the validity evidence sources targeted in this study. To address RQ1 (item evaluation), we compared the intended emotion words to empirically dominant headwords elicited by each vignette and quantified naming agreement as dominant response rates and unique headword counts (Supplementary Material A). In addition, to probe scoring fairness and vignette performance across participant subgroups, we contrasted the most dominant headwords produced by (1) English L1 vs. Lx (LHQ3 self-reports) and (2) higher vs. lower proficiency participants (LexTALE median-split).
To compare candidate scoring specifications for instrument-development purposes, we examined internal consistency, corrected item-total correlations and associations among PEVT scores and proficiency measures (LexTALE and self-rated proficiency) and summarised these in correlation panels. To address RQ2 and RQ3, we analysed response-level accuracy as a binary variable using a pre-specified logistic mixed-effects model, which accounts for the repeated-measure structure of the task (three responses per vignette and multiple vignettes per participant). Fixed effects included context condition (high vs. low) and English proficiency (LexTALE scores z-scored) as planned variables, while response order (1–3) and produced-word frequency (Zipf values z-scored) were treated as exploratory response-process covariates, capturing lexical accessibility of the emotion words produced. Random intercepts were specified for participants and vignettes (items). In addition, because each participant provided three responses per item, we included a participant-by-item random intercept to account for within-item dependence across the three responses. Following the item evaluation, we fitted analogous models for accuracy and unweighted-dominant responses with the 18 retained vignettes. Vignettes 14, 15, 17 and 20 were excluded as they were flagged as problematic for target elicitation and potential redesign in future iterations.
For these Generalised Linear Mixed Models (GLMMs), we adapted Dunn’s guidelines (Dunn, Reference Dunn2024) for random-person-random-item (RPRI) Rasch modelling and specified the accuracy model within lme4 (Bates et al., Reference Bates, Mächler, Bolker and Walker2015) package in R as: saccuracy ~ context + resp_num + Lextale + resp_zipfvalue + (1|participant) + (1|stimuli) + (1|pid_item). The model for unweighted dominance was specified as: unweighted_dom ~ context + resp_num + Lextale + resp_zipfvalue + (1 | participant) + (1 | stimuli) because the participant-by-vignette random intercept produced a singular fit. Data analyses were performed in RStudio 2021.09.2 (R Core Team, 2021) using tidyverse (Wickham et al., Reference Wickham, Averick, Bryan, Chang, McGowan, François, Grolemund, Hayes, Henry, Hester, Kuhn, Pedersen, Miller, Bache, Müller, Ooms, Robinson, Seidel, Spinu and Yutani2019), performance (Lüdecke, Reference Lüdecke2021), emmeans (Lenth, Reference Lenth2024), pROC (Robin et al., Reference Robin, Turck, Hainard, Tiberti, Lisacek, Sanchez and Müller2011), DHARMa (Hartig, Reference Hartig2022) and gtsummary (Sjoberg et al., Reference Sjoberg, Whiting, Curry, Lavery and Larmarange2021) packages. The scaled, long dataset is available in the Open Science Framework repository: https://osf.io/7xzyv/overview?view_only=33c3d8ebe05848e2aab185b7f663421d
4. Results
Across 22 vignettes, participants produced 7,062 responses, amounting to 611 unique headwords. The 10% trimmed mean showed participants spent 27.8 seconds per vignette (M = 37.74, SD = 70.20).
4.1. Target elicitation and naming agreement
To address RQ1, we examined whether the vignettes elicited the target response (including a close synonym or corresponding basic category) as the most dominant response and quantified naming agreement. Overall, 18/22 vignettes (82%) elicited the target response, a close synonym or the basic emotion category as the most dominant headword, indicating that most vignettes performed as designed.
To examine the quality of the vignettes, first we used item difficulty to determine the proportion of participants producing a dominant response. Across items, participants produced dominant responses at a relatively high rate (M = 67.3%, median = 70.09%, SD = 16.77%, Table 3). Item-21 and item-15 emerged as the easiest and most difficult items with 90.65% and 20.56% of participants providing dominant responses, respectively (Supplementary Table C1). Importantly, no other items fell below 30%.
Descriptive statistics for responses by context and naming agreement
Table 3. Long description
The table contains 10 columns: Variable, N, Mean, S D, Median, Min, Max, Range, Skew, and Kurtosis. All variables have an N of 22.
* High underscore dom underscore count: Mean 36.55, S D 8.51, Median 37.50, Min 14.00, Max 49.00, Range 35.00, Skew minus 0.94, Kurtosis 0.61.
* Low underscore dom underscore count: Mean 36.05, S D 8.30, Median 36.50, Min 19.00, Max 48.00, Range 29.00, Skew minus 0.45, Kurtosis minus 0.57.
* Dom underscore sum: Mean 71.73, S D 17.94, Median 75.00, Min 22.00, Max 97.00, Range 75.00, Skew minus 1.05, Kurtosis 0.80.
* Dom underscore percent: Mean 67.03, S D 16.77, Median 70.09, Min 20.56, Max 90.65, Range 70.09, Skew minus 1.05, Kurtosis 0.80.
* High underscore unique: Mean 27.09, S D 6.56, Median 27.00, Min 15.00, Max 41.00, Range 26.00, Skew 0.03, Kurtosis minus 0.54.
* Low underscore unique: Mean 27.68, S D 5.94, Median 28.00, Min 14.00, Max 38.00, Range 24.00, Skew minus 0.47, Kurtosis minus 0.38.
* Total underscore unique: Mean 36.82, S D 9.64, Median 37.50, Min 19.00, Max 57.00, Range 38.00, Skew minus 0.21, Kurtosis minus 0.44.
* Total underscore count: Mean 262.00, S D 28.90, Median 261.50, Min 203.00, Max 309.00, Range 106.00, Skew minus 0.56, Kurtosis minus 0.31.
Note: High/Low_dom_count = the number of responses in high or low contexts, Dom_sum = the sum of dominant responses, Dom_percent = the percentage of dominant responses, High/Low_unique = the number of unique responses, Total_unique = the total number of unique responses across all 22 items, Total_count = the total number of responses across all 22 items.
While item difficulty examines the proportion of participants producing the dominant response, naming agreement explores the number of unique responses for each item (in Supplementary Table C1). The higher this number is, the more diverse the responses are, indicating less agreement. Item-17 elicited the fewest unique responses (19), indicating high convergence, whereas item-20 elicited the most (57), indicating greater lexical diversity and lower agreement. Importantly, the dominant-response counts and unique-response counts were highly similar across context conditions.
Second, 18 items (82%) successfully elicited the target label, a close synonym or the corresponding basic emotion category as the most dominant response, indicating high item facility and successful elicitation of targets. For example, item-1 (“happiness/joy”) elicited “happy” from 72% of the participants. Similarly, item-18 (“envy”) elicited “jealous” (68%) and “envy/envious” (52%). Three items (item-5, item-6 and item-11) elicited close synonyms (“remorse” –> “regret”; “shame” –> “embarrassed”; “fear” –> “scared”), thus warranting renaming in future iterations. Considering the nuanced semantic differences, “regret” is indeed a better fit for the vignette and the mismatch between target and elicited response might be a symptom of direct (dictionary) translations in the original German CEVVT (Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020). A similar issue was observed in item-6 and item-11. Item-22 (“anticipation”) predominantly elicited “excited” (89%). Four items (item-20, item-17, item-15 and item-14) were identified as problematic for eliciting the intended targets reliably or because the dominant response became redundant relative to other items, suggesting a need to redesign or replace these items in subsequent iterations.
Third, dominant headwords were largely consistent across context conditions and only two items elicited different dominant responses across contexts (Supplementary Material A). Item-7 in low context elicited “angry” (49%) and “annoyed” (40%) as the most dominant responses, the target “frustrate” (38%) emerged only as the third most dominant response. Conversely, item-10 (“disappoint”) was successfully elicited in both contexts; the naming agreement was much higher in the high context (85%) than in the low context (66%), where the dominant response “sad” reached 68%. All the other low-context items either performed similarly or better than their high-context counterparts in eliciting dominant responses. The effect of context is further examined in Section 4.3.
To probe whether dominant responses were driven by a particular subgroup, we examined the most dominant headwords across (1) English L1 vs. Lx and (2) higher vs. lower proficiency groups based on a LexTALE median-split. Dominant responses converged strongly across these subgroups: for the L1–Lx groups, the most dominant headword matched for 19/22 (86%) and for high–low groups 20/22 (91%). Because second and third dominant responses were more variable and difficulty differences were small, we limited this analysis to the most dominant headwords. Divergences across both comparisons were limited to a small set of already flagged items: 4, 7 and 15. For example, item-7 targeting “frustrate” was successfully elicited from both the L1 and high groups, but the Lx and low groups produced “angry” as the top headword, thus still successfully identifying the corresponding emotion category. Relative to overall norms (N =107), high group matched the top headword in 22/22 cases (100%), L1 matched 20/22 (91%); Lx 21/22 (95%) and low 20/22 (91%). This indicates that across all items, the dominant scoring is not driven by lower-proficiency responses (in Supplementary Table D5).
4.2. Scoring the PEVT and language proficiency
Having established vignette performance and identified items flagged for redesign (RQ1), here we examine how alternative scoring specifications (Section 3.2) behave and how they relate to proficiency. Moreover, as part of instrument-development and validity evidence we also explore whether different weightings and the number of responses materially change the associations with language proficiency. Item facility and corrected item-total correlations are provided in Table 4 and a fuller Item Response Theory analysis will be reported in (Szabo et al., forthcoming). Item-total correlations show that accuracy and weighted-dominant scoring provide the strongest and most even discrimination across items. Top-dominant scoring is informative of naming agreement and face validity but is weakly discriminating.
Item facility and item-total correlations across scoring methods
Table 4. Long description
A table with 6 columns and 22 rows of data. The columns are labeled: Item, Facility, r underscore Top underscore dom, r underscore U W underscore dom, r underscore W underscore dom, and r underscore Acc.
Key data points include:
* Item 1: Facility 0.72, r_Top_dom 0.34, r_UW_dom 0.25, r_W_dom 0.33, r_Acc 0.36.
* Item 4: Facility 0.35, r_Top_dom 0.33, r_UW_dom 0.35, r_W_dom 0.36, r_Acc 0.43.
* Item 6: Facility 0.81, r_Top_dom 0.42, r_UW_dom 0.29, r_W_dom 0.36, r_Acc 0.36.
* Item 15: Facility 0.39, r_Top_dom 0.24, r_UW_dom 0.11, r_W_dom 0.14, r_Acc 0.39.
* Item 21: Facility 0.91, r_Top_dom 0.07, r_UW_dom 0.42, r_W_dom 0.33, r_Acc 0.31.
* Item 22: Facility 0.89, r_Top_dom 0.25, r_UW_dom 0.25, r_W_dom 0.35, r_Acc 0.34.
Facility represents the proportion of participants producing a dominant response. The r values are corrected point-biserial correlations for Top-Dominant, Unweighted-Dominant, Weighted-Dominant, and Accuracy scores. According to the note, values greater than or equal to 0.30 are acceptable, 0.20 to 0.29 are marginal, and less than 0.20 are weak.
Note: Facility = proportion of participants producing any of the top-three dominant headwords. r_Top_dom, r_UW_dom, r_W_dom and r_Acc are corrected point-biserial correlations for the Top-, Unweighted-, Weighted-Dominant and Accuracy scores, respectively. Values ≥ .30 = acceptable; .20–.29 = marginal; < .20 = weak.
Figure 3 shows that LexTALE and self-rated proficiency yielded a moderate correlation (r = .34). Only accuracy scoring related significantly to self-rated proficiency. Conversely, all scores except unweighted-dominant correlated with LexTALE. Dominant scores correlated strongly with each other (r = .85–.97), indicating a consistent measure of the same construct. Accuracy correlated moderately (r = .46–.59) with dominant scores and modestly with LexTALE (r = .29), supporting convergent validity. We next examine whether weighting and the number of responses materially change these proficiency associations for instrument-development purposes.
Dominant and accuracy scores correlation panel.
Figure 3. Long description
A matrix of 36 panels organized in 6 rows and 6 columns.
Diagonal Panels (Top-Left to Bottom-Right):
These panels contain teal histograms with black density curves for each variable: Lextale, Self_Rated_prof, Unweighted_Dominance, Top_Dominant, Weighted_Dominance, and Accuracy.
Upper Triangle (Numerical Correlations):
This section displays Pearson correlation coefficients. Significant values include:
- Unweighted_Dominance and Top_Dominant: 0.85 with three asterisks.
- Unweighted_Dominance and Weighted_Dominance: 0.97 with three asterisks.
- Top_Dominant and Weighted_Dominance: 0.94 with three asterisks.
- Accuracy correlates with Lextale at 0.29 with two asterisks and with Weighted_Dominance at 0.53 with three asterisks.
Lower Triangle (Scatter Plots):
This section shows bivariate scatter plots with red linear regression lines.
- The plots for Unweighted_Dominance, Top_Dominant, and Weighted_Dominance against each other show tight, positive linear clusters, reflecting the high correlation coefficients seen in the upper triangle.
- Plots involving Accuracy show a more dispersed positive linear trend.
- Plots involving Lextale and Self_Rated_prof show the highest variance with flatter regression lines.
Figure 4 compares the associations between LexTALE and the different participant-level PEVT scores (dominance-based, accuracy and weighted variants). This enables comparing scoring specifications and their convergent relations with proficiency for instrument-development purposes. All plotted values are computed as total scores per participant across the 22 vignettes (see Section 3.2 for scoring definitions). Descriptive statistics for LexTALE and all plotted comparisons are provided in Supplementary Table D6.
LexTALE and weighted scores correlation panel.
Figure 4. Long description
A multi-panel attribute space diagram arranged in a 12 by 12 grid.
Diagonal Panels: Running from top-left to bottom-right, these panels contain teal histograms with density curves for individual variables. Labels include Lextale, W underscore Dom underscore diff, Top underscore Dom underscore diff, UnW underscore Dom underscore diff, Accuracy underscore diff, W underscore Dom underscore freq, Top underscore Dom underscore freq, UnW underscore Dom underscore freq, Accuracy underscore freq, W underscore dom underscore FreqxDiff, Top underscore dom underscore FreqxDiff, and UnW underscore dom underscore FreqxDiff.
Lower-Left Triangle: Contains scatter plots for every pair of variables. Most plots show a strong positive linear correlation with data points clustered tightly around a red regression line, particularly between the different ‘Dom’ and ‘Freq’ weighted scores. The Lextale variable shows a more dispersed, weaker linear relationship with other metrics.
Upper-Right Triangle: Displays numerical Pearson correlation coefficients corresponding to the scatter plots. Values range from -0.15 to 0.98. High correlations (e.g., 0.98, 0.97) are marked with three asterisks indicating high statistical significance. Low correlations (e.g., -0.03, 0.06) appear near the far right column associated with Accuracy underscore FreqxDiff.
Axes: The outer perimeter of the grid contains numerical scales for the variables, with values ranging from 0 to 90 for Lextale and smaller decimal scales for weighted difference scores.
These comparisons indicate that frequency-only and frequency-difficulty weighting yielded modest changes in the correlation with LexTALE. In particular, frequency-weighted accuracy showed the strongest association among the weighted variants (r = .33), while difficulty-only weighting did not improve dominant scores and only slightly increased accuracy (r = .24). We therefore also examined whether the LexTALE–accuracy association varies as a function of the number of responses (first-only, first two and all three responses), again using participant-level total scores (Figure 5). This comparison informs test-format decisions by showing whether restricting responses changes the proficiency association.
LexTALE and accuracy weighted scores correlation panel by response number.
Figure 5. Long description
A multi-panel attribute space diagram arranged in a 12 by 12 grid.
* Diagonal Panels: From top-left to bottom-right, these panels contain teal histograms with density curves for each variable. Labels include Lextale, Accuracy, Accuracy underscore diff, Accuracy underscore freq, Accuracy underscore FreqxDiff, Accuracy underscore R 1, Accuracy underscore diff underscore R 1, Accuracy underscore freq underscore R 1, Accuracy underscore FreqxDiff underscore R 1, Accuracy underscore R 2, Accuracy underscore diff underscore R 2, Accuracy underscore freq underscore R 2, and Accuracy underscore FreqxDiff underscore R 2.
* Lower Triangle: Contains scatter plots for every pair of variables. Each plot includes a red linear regression line showing a positive correlation trend. The density of data points varies, with some showing tight linear clustering and others showing more dispersed distributions.
* Upper Triangle: Displays the Pearson correlation coefficients corresponding to the scatter plots. Values range from 0.13 to 0.94. Most values are accompanied by asterisks indicating statistical significance, such as 0.88 triple asterisk or 0.24 single asterisk.
* Axes: The outer perimeter of the grid contains numerical scales for the variables, with values such as 30 to 90 for Lextale and 1 to 5 for frequency-based scores.
Accuracy correlated more strongly with LexTALE when based on the first (r = .36) or the first two responses only (r = .36) compared to all three responses (r = .29). For frequency-weighted accuracy, the correlation was highest with two responses (r = .40), followed by one (r = .35) and three (r = .33). Taken together, accuracy and weighted-dominant scores showed the strongest discrimination, while frequency-weighted accuracy based on two responses showed the strongest association with LexTALE in this sample. In the main mixed-effects analyses, however, we retain the full response-level data and model response order and produced-word frequency directly, rather than redefining the primary outcome based on these comparisons.
4.3. The effect of context, response order, proficiency and produced-word frequency
Descriptive comparisons showed no significant differences in overall participant-level LexTALE and PEVT scores between English L1 and Lx participants, nor between the two counterbalanced test versions (Supplementary Tables D1–D4). This reduces the likelihood that any observed context effects are attributable to pre-existing proficiency or test-version differences. Supplementary Table D4 and Figure 6 provide a descriptive overview of how scores vary across context condition and response order, indicating a gradual decline in scores from first to third response and substantial overlap between contexts.
The effect of context and response order on dominant and accuracy scores.
Figure 6. Long description
A two-panel figure displaying raincloud plots. Each plot includes a jittered scatter plot, a box plot, and a half-violin density plot for two context levels: H (purple) and L (light blue).
Left Panel: Accuracy Response Score.
* The y-axis ranges from 10 to 60.
* The x-axis shows Response Number 1, 2, and 3.
* For Response 1, scores are highest, clustered around 50, with low variance.
* For Response 2, scores decrease to a median around 45 with increased spread.
* For Response 3, scores are lowest, with a median near 38 and the widest distribution.
Right Panel: Top 3 Dominant Response Score.
* The y-axis ranges from 0 to 150.
* The x-axis shows Response Number 1, 2, and 3.
* Response 1 shows the highest scores, with medians near 100 and a tall density curve.
* Response 2 shows a significant drop, with medians below 50.
* Response 3 shows the lowest scores, with medians around 25.
In both panels, context H and L show similar distributions, though L often exhibits slightly lower medians or wider tails in the density plots.
We next fitted a logistic mixed-effects model with response-level accuracy as the binary outcome. The model included context and LexTALE as planned predictors, and response order and produced-word frequency as exploratory covariates with random intercepts for participant, vignette and participant-by-vignette. The model showed conditional R2 = .41 and marginal R2 = .21, indicating that the full model accounted for a meaningful amount of variation in response accuracy, with fixed effects accounting for a moderate 21% and the remaining variation captured by the random effects associated with participants, items and participant-by-vignette combinations. Context condition did not affect accuracy, whereas relative to first responses, the odds of an accurate response were substantially lower for second responses and third responses (Table 5). Higher LexTALE scores were associated with higher odds of producing a context-appropriate emotion label. Produced-word frequency was also positively associated with accuracy, indicating that context-appropriate responses tended to involve higher-frequency labels in this item set. Model diagnostics based on DHARMa indicated no evidence of overdispersion (dispersion = 0.99, p = .916) or zero inflation (ratioObsSim = 1.02, p = .810), although the residual-uniformity test suggested a small deviation from the simulated reference distribution (KS D = 0.022, p = .006). Because of this small deviation, we conducted a sensitivity analysis by refitting the model without the participant-by-vignette random intercept. The fixed-effect pattern was unchanged: context remained non-significant, while response order, LexTALE and produced-word frequency remained significant in the same directions. The sensitivity model showed acceptable DHARMa diagnostics, with no evidence of residual-uniformity violations (KS D = 0.013, p = .287), overdispersion (dispersion = 0.99, p = .956) or zero inflation (ratioObsSim = 1.03, p = .778). Thus, the substantive conclusions do not depend on including the participant-by-vignette term. Given the theoretically justified random-effects structure and the stability of the fixed effects across specifications, we retained the participant-by-vignette random-intercept specification for the primary model. The predictive discrimination of the mixed-effects model was evaluated using the area under the receiver operating characteristic curve (AUC), based on fixed-effects predicted probabilities. The model showed moderate discrimination (AUC = 0.75). Model-based predicted effects from the final logistic mixed-effects model are shown in Figure 7, illustrating the response order by context pattern alongside the marginal effects of LexTALE and produced-word frequency on response accuracy.
Fixed and random effects from the logistic mixed-effects model predicting response-level accuracy
Table 5. Long description
The table is organized into columns for Predictors, O R (Odds Ratio), S E (Standard Error), z-score, 95% C I (Confidence Interval), and p-value.
Fixed Effects:
* (Intercept): O R 23.23, S E 4.83, z 15.14, C I 15.46 to 34.90, p less than 0.001.
* Context (low/high): O R 0.99, S E 0.09, z minus 0.13, C I 0.84 to 1.17, p 0.900.
* Response order (2/1): O R 0.32, S E 0.04, z minus 9.76, C I 0.25 to 0.40, p less than 0.001.
* Response order (3/1): O R 0.17, S E 0.02, z minus 15.26, C I 0.14 to 0.21, p less than 0.001.
* Response order (3/2): O R 0.54, S E 0.17, z minus 7.08, C I 0.43 to 0.66, p less than 0.001.
* LexTALE: O R 1.29, S E 0.10, z 3.34, C I 1.11 to 1.51, p 0.001.
* Produced-word frequency (Zipf; z): O R 1.94, S E 0.08, z 15.43, C I 1.78 to 2.11, p less than 0.001.
Random Effects and Model Statistics:
* sigma-squared: 3.29.
* tau sub 00 pid_item: 0.20.
* tau sub 00 participant: 0.46.
* tau sub 00 stimuli: 0.43.
* I C C: 0.25.
* N participant: 107.
* N stimuli: 18.
* N pid_item: 1926.
* Observations: 5778.
* Marginal R-squared / Conditional R-squared: 0.215 / 0.410.
Note: OR = Odds Ratio, CI = Confidence Interval, Response-order pairwise contrasts are Tukey-adjusted from emmeans. * p<0.05 ** p<0.01 *** p<0.001.
Model-based predicted effects on response accuracy from the final logistic mixed-effects model.
Note: Model-based predicted effects from the final logistic mixed-effects model. Panel A shows predicted accuracy across response order separately for high- and low-context vignettes. Panels B and C show the marginal effects of LexTALE and produced-word frequency (Zipf; z-scored), respectively, on predicted accuracy. Shaded bands indicate 95% confidence intervals around the model-based predictions.
Figure 7. Long description
Panel A, Response order by context. The y-axis is Predicted accuracy from 80 percent to 90 percent. The x-axis is Response order with values 1, 2, and 3. Data points for High context in red and Low context in blue show a downward trend. At response order 1, accuracy is near 95 percent. At order 2, it drops to approximately 88 percent. At order 3, it falls to 80 percent. Vertical error bars represent confidence intervals.
Panel B, Lex T A L E. The y-axis is Predicted accuracy from 88 percent to 96 percent. The x-axis is Lex T A L E (z) from negative 3 to 1. A black line shows a steady linear increase in accuracy as Lex T A L E scores increase, starting around 91 percent and ending near 97 percent. A grey shaded band indicates the confidence interval.
Panel C, Produced-word frequency. The y-axis is Predicted accuracy from 75 percent to 100 percent. The x-axis is Produced-word frequency (Zipf; z) from negative 2 to 2. The data shows an asymptotic curve, starting at 82 percent accuracy at negative 2 and rising sharply before leveling off near 99 percent accuracy at a z-score of 2. A grey shaded band indicates the confidence interval.
For the unweighted-dominance model, adding the participant-by-vignette random intercept produced a singular fit because its variance was estimated at zero; this term was therefore omitted from the reported model. The GLMM fitted for unweighted-dominant responses showed the same overall pattern: context remained non-significant (p = .067), whereas response order, LexTALE and produced-word frequency were significant in the expected directions. Therefore, the results are reported in Supplementary Material E.
5. Discussion
This study aimed to provide preliminary validity evidence for the PEVT by examining item performance, scoring and the effects of context, response order, proficiency and produced-word frequency on productive emotion labelling. The following discussion addresses each of these in turn.
5.1. PEVT item and format evaluation
This study focused on the design and preliminary validation of the PEVT for English Lx speakers. Unlike typical vocabulary tests, the PEVT employs a vignette-based productive word elicitation task (Goetze, Reference Goetze2023). This enables eliciting multiple plausible labels in context and comparing individuals’ performance with similar backgrounds, experiences or language proficiency on specific items instead of restricting responses to limited options as in recognition tasks. This directly addresses the concern raised earlier that decontextualised prompts may under-sample productive/contextualised emotion labelling, whereas appraisal cues and multimodal context may better support plausible label convergence (Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021; Goetze, Reference Goetze2023). Eighteen of twenty-two vignettes elicited the target emotion, demonstrating high item quality and substantial naming agreement.
The L1 diversity of the participant pool, encompassing a range of different age groups and high English proficiency, supports potential generalisability of the findings. The subgroup convergence analyses of the top-dominant responses indicated largely stable naming patterns across English L1–Lx and high–low proficiency groups. This partially mitigates the risk of same-sample norming. Notably, the few mismatches coincide with items already flagged for redesign. Nonetheless, given the inherent complexities in recognising and expressing emotions across individuals, cultures and languages (Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021), as well as the variation in vocabulary knowledge across L1s (Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024), future work should explore dominant responses across linguacultures.
Some item-level issues also emerged. In a few cases, highly proficient English speakers consistently produced close synonyms rather than the original target labels, suggesting that some targets reflected translation choices that were less natural in English. This is consistent with the aforementioned challenge that emotion labels are not fully interchangeable across linguacultures (Pavlenko, Reference Pavlenko2008; Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021). This was particularly evident for pairs such as “remorse–regret”, “fear–scared” or “shame–embarrassed” where participants demonstrated awareness of the distinct semantic nuances. The high naming agreement and close synonymy observed for these items warrant renaming in future iterations rather than removal.
Four items, however, elicited dominant responses other than the target response (item-20), produced low naming agreement (item-15) or had dominant response that became redundant compared to the other items (item-14 and item-17). Specifically, higher frequency and corresponding core emotions (i.e., more general labels) such as “happy” and “sad” outcompeted lower-frequency and more precise alternatives (“content” and “lonely”). These patterns are consistent with previous work showing that Lx users tend to rely on higher-frequency, less emotionally intense emotion labels (e.g., Ferré et al., Reference Ferré, Guasch, Stadthagen-Gonzalez and Comesaña2022; Pavlenko, Reference Pavlenko2012; Pérez-García & Sánchez, Reference Pérez-García and Sánchez2020), and underscore the challenge of eliciting lower-frequency and more nuanced EV in vignette-based tasks (Chee et al., Reference Chee, Szabo and Ambrose2025). Future validation work will therefore need to refine or replace these items, or introduce stop-words (e.g., “do NOT use happy/sad”) and continue expanding the item bank towards lower-frequency targets, potentially ameliorating the problem of eliciting low-frequency and complex emotion words.
To evaluate the test format, we also explored the effect of context on the responses and observed encouraging results. Twenty items performed equally in both contexts, a strong indication that for adult participants the combination of visual and the protagonist’s appraisal cue (thought) was often sufficient to support appropriate emotion labelling. This aligns with the Appraisal Theory of Emotions (Scherer, Reference Scherer2005), suggesting that additional information provided in high-context conditions is unlikely to uniformly affect participants’ answers. This is illustrated in Supplementary Material F where the protagonist’s appraisal of the event has been negatively manipulated with the corresponding visual information, further demonstrating the flexibility of controlled vignette design. However, two items performed distinctly and suboptimally in the low context, eliciting different dominant responses with significantly lower naming agreement compared to high-context counterparts. These items may require redesign or retention of the high-context version in future iterations.
As a first systematic attempt to assess productive EV via vignettes in adult Lx users, these results are promising but preliminary. Nevertheless, the presented evidence supports the current item set and format as a basis for future development of the PEVT, while also identifying clear priorities for refinement, especially in relation to target-label precision and naturalness, lower-frequency emotion terms and cross-linguacultural norming.
5.2. Scoring the PEVT
In the ongoing development phase of the PEVT, we have adopted a methodical and robust approach to evaluating alternative scoring methods. Dominant scoring served a dual purpose. First, it functioned as an item-evaluation tool for examining how consistently each vignette elicited the target response or a close synonym. Second, it provided an index of contextual naming alignment, understood here as a narrow facet of depth (Nation, Reference Nation2022; Schmitt & Schmitt, Reference Schmitt and Schmitt2014): capturing whether participants retrieve labels that fit the specific referential, affective and appraisal context of a vignette in a way that aligns with population naming patterns. In Nation’s (Reference Nation2022) vocabulary knowledge framework, this relates to concepts and referents and constraints on use because the task requires mapping a scenario to an emotion concept and selecting a label appropriate to the situation, intensity and inferred affective state.
This operationalisation is necessarily selective, as are other depth measures. For instance, the WAT, as an associative depth test, focuses on paradigmatic and syntagmatic relations (Read, Reference Read1998), whereas the PEVT focuses on contextual appropriacy and naming alignment in productive EV. Dominance-based scoring should therefore not be interpreted as a comprehensive measure of vocabulary depth. Within its narrower scope, however, it is valuable because it is empirically grounded in the response behaviour of the target population and can support efficient, reproducible scoring. In the current sample, subgroup convergence analyses suggested that dominant responses were relatively stable across English L1/Lx and higher/lower proficiency groups, indicating that the observed response patterns were not driven by a single subgroup. Nevertheless, cognitive appraisals of emotions are individually, cross-culturally and lexically variable, and future work requires more clearly defined norming groups and cross-linguacultural validation. If broader vocabulary depth is of interest, additional indicators such as emotion collocations, multiword expressions and register shifts should also be included.
Accuracy scoring, by contrast, captures the range of conceptually appropriate emotion words produced by each participant, even when these are unique or less frequent, and therefore gauges productive EV breadth more directly. Accordingly, it retains the contextual support that decontextualised fluency tasks typically lack (e.g., Mavrou, Reference Mavrou2021; Meara & Fitzpatrick, Reference Meara and Fitzpatrick2000). The scoring comparisons revealed that dominance-based and accuracy scores were closely related but not interchangeable: dominance-based scores were most informative for naming agreement and item quality, whereas accuracy and weighted-dominant scores showed stronger discrimination across items and higher associations with LexTALE.
Weighting by frequency or item difficulty yielded modest improvements in some score–proficiency associations, especially for frequency-weighted accuracy, and restricting scoring to the first one or two responses also slightly strengthened the LexTALE correlations. This is consistent with the earlier observation that frequency is inseparable from vocabulary knowledge (Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024), but the sample-dependent nature of these weighted scores also underscores why they are better treated as instrument-development tools than as default inferential outcomes. In the main mixed-effects analysis, we therefore retained full response-level accuracy and modelled response order and produced-word frequency directly. This preserves interpretability and transferability while still allowing score comparisons to inform future refinement of the PEVT.
5.3. Productive EV knowledge and other factors
Our findings showed no significant overall effect of context condition on productive emotion labelling. This suggests that, for relatively proficient adult users of English, the combination of visual information and the protagonist’s appraisal cue was generally sufficient to support productive emotion labelling, without consistent added benefit from longer scenario descriptions (high context). This fits the earlier constructed-emotion and appraisal-based rationale for the vignette format, according to which emotion labelling depends on how participants use situational and appraisal cues to categorise the scenario (Barrett, Reference Barrett2017a; Barrett, Reference Barrett2017b). This pattern was not uniform across all items, however. Only two vignettes (item-7 and item-10) showed different dominant responses across contexts and lower naming agreement in low context, but these alternatives still reflected the intended or corresponding basic emotion category. Thus, low context appeared in these cases to reduce naming precision rather than emotion recognition itself. For test development, this is encouraging because shorter prompts may reduce construct-irrelevant variance associated with reading speed and comprehension, cognitive load and general task difficulty, while preserving the effectiveness of the vignette format for most items.
Results showed a strong response-order effect on performance. Compared with second responses, first responses were over three times more likely to be accurate, and compared with third responses they were nearly six times more likely to be accurate. The same tendency held for unweighted-dominant responses: first responses were nearly four times more likely than second responses and almost eight times more likely than third responses to align with the dominant response pattern. This supports the view that earlier responses reflect more accessible and readily retrievable emotion labels, consistent with least-effort accounts of lexical choice (Zipf, Reference Zipf1949) and with research showing that Lx users tend to rely on more entrenched, higher-frequency vocabulary in production (e.g., Pérez-García & Sánchez, Reference Pérez-García and Sánchez2020; Yeh et al., 2007).
The positive associations between produced-word frequency and both accuracy and dominance further reinforce this interpretation. The magnitude of the frequency effect was substantial: for each one-standard-deviation increase in produced-word Zipf value, the odds of an accurate response increased by 94%, and the odds of a dominant response increased by 127%. Higher-frequency emotion labels were therefore considerably more likely both to emerge as contextually fitting and to align with in-sample naming patterns. This effect should also be interpreted in light of the current item bank, however. Targets and dominant responses are concentrated largely in the 1k–3k bands. As such, the present study cannot determine whether target-word frequency itself systematically influences the frequency of the labels produced. Addressing this question would be valuable both for test development and for research, because a more systematically frequency- and affect-rating-balanced item bank (see Chee et al., Reference Chee, Szabo and Ambrose2025, Reference Chee, Szabo and Ambrose2026) could show whether the vignettes reliably elicit lower-frequency, more nuanced emotion labels rather than mainly capturing the most lexically accessible ones. This warrants further, potentially more granular analysis of the emotion word types produced in future studies.
LexTALE, used here as a proxy for general English proficiency, also emerged as a significant positive predictor. In the accuracy model, each one-standard-deviation increase in LexTALE was associated with a 29% increase in the odds of producing a context-appropriate emotion label, while in the unweighted-dominance model, the odds of producing an unweighted-dominant response increased by 13%. This suggests that productive EV performance in the PEVT is partly constrained by broader Lx lexical proficiency, even though the task itself targets contextualised emotion labelling rather than decontextualised receptive knowledge. In this sense, the finding is consistent with previous work, showing that lower-proficiency Lx users tend to rely on more generic, higher-frequency and less differentiated EV, whereas higher proficiency supports access to more precise and contextually appropriate labels (e.g., Ferré et al., Reference Ferré, Guasch, Stadthagen-Gonzalez and Comesaña2022; Pavlenko, Reference Pavlenko2008, Reference Pavlenko2012; Pérez-García & Sánchez, Reference Pérez-García and Sánchez2020). At the same time, the LexTALE effect should be interpreted cautiously because LexTALE is a receptive measure, and it cannot on its own establish that productive vocabulary knowledge is driving performance. It is also possible that participants’ relatively high English proficiency attenuated differences typically observed between L1/Lx speakers, substantiating the value of measuring language proficiency directly rather than relying on arbitrary L1/Lx labels as proxy (Brysbaert et al., Reference Brysbaert, Stevens, Mandera and Keuleers2016).
5.4. Limitations and future directions
Given the effectiveness of the vignette methodology (Chee et al., Reference Chee, Szabo and Ambrose2025, Reference Chee, Szabo and Ambrose2026; Grosse & Streubel, Reference Grosse and Streubel2024; Streubel et al., Reference Streubel, Gunzenhauser, Grosse and Saalbach2020), the present findings provide a promising basis for further development of the PEVT. Nevertheless, some limitations should be acknowledged. First, although the PEVT is designed to assess contextualised productive emotion labelling, task performance is likely to reflect not only EV knowledge but also participants’ ability to infer emotions from the scenarios and process written prompts. Consequently, some variance may be attributable to emotion inference and reading demands in addition to lexical knowledge. The finding that low-context prompts were largely sufficient for most items is encouraging in this respect, but future validation should include complementary measures that more directly separate these components.
A further limitation concerns the reliability of accuracy coding. Although agreement was high after adjudication, initial inter-rater reliability was only fair-to-moderate with notable item-level variability. Because accuracy was used as the primary outcome in the main mixed-effects analyses, this constrains the strength of the validity argument for accuracy scores and suggests that contextualised emotion labelling is inherently subjective and judgement-sensitive: multiple labels may be plausible but differ in intensity, specificity or appraisal fit. Future work should therefore use more detailed scoring rubrics, with clearer decision rules for near-synonyms, basic-category labels, intensity mismatches and contextually plausible alternatives.
Third, the current item bank remains a pilot bank rather than an exhaustive sample of the EV domain. Although the 22 vignettes provide initial coverage across basic and complex emotions and positive and negative scenarios, the bank was not designed to systematically balance target-word frequency or broader affective dimensions. In the present data, target and dominant responses were concentrated largely in the 1k–3k frequency bands, and the strong effect of produced-word frequency suggests that lexical accessibility is an important determinant of performance. Future test development should therefore expand the item bank to include lower-frequency and more nuanced emotion labels and examine how target-word frequency shapes the labels participants subsequently produce (see Chee et al., Reference Chee, Szabo and Ambrose2025). Relatedly, the PEVT deliberately excludes emotion-laden words in Pavlenko’s (Reference Pavlenko2008) sense, such as socially evaluative, taboo or affectively charged words that evoke emotion without directly naming a felt state. This exclusion was intended to support construct precision and scoring consistency, but it potentially narrows the scope of the instrument. The PEVT should therefore be interpreted as a measure of productive contextualised emotion labelling, not as a measure of the full range of affective or emotion-laden language use in English.
Fourth, the current validity evidence is constrained by the composition of the sample: predominantly highly proficient users of English, largely Malaysian, with relatively limited age variation and a gender imbalance. These characteristics may have attenuated group differences and restrict generalisability. Conversely, deriving dominant responses from within a specific sample is also a methodological strength as it allows dominant scoring to reflect naming patterns of the population under study, rather than imposing norms derived from WEIRD (Western, Educated, Industrialised, Rich, and Democratic) comparison groups (Plonsky, Reference Plonsky2023). In this sense, dominant scoring is intentionally localised and linguaculturally sensitive. However, given the in-sample dominant responses, future work is needed to examine whether similar naming patterns hold across diverse linguacultural populations and whether score interpretations are transferable across populations. Considering extant evidence that emotions are contextually dependent and shaped by personal and linguistic experience (Barrett & Westlin, Reference Barrett, Westlin and Meiselman2021; Pavlenko, Reference Pavlenko2008; Schmitt et al., Reference Schmitt, Dunn, O’Sullivan, Anthony and Kremmel2024), cross-linguacultural norming remains an important next step.
Finally, LexTALE enables quick, reliable testing and comparisons across studies and served as a meaningful proxy for language proficiency in the present study, but it remains a receptive measure. Although its association with PEVT performance supports the view that productive EV is partly constrained by broader lexical proficiency, future validation should include an independent productive vocabulary measure to determine more directly how far PEVT performance reflects general lexical knowledge vs. more domain-specific emotion-word knowledge.
Taken together, these limitations highlight that test validation should be treated as a localised, iterative and continuous process (Schmitt et al., Reference Schmitt, Nation and Kremmel2020). Future validation should therefore include broader linguacultural and proficiency samples and younger English Lx users, expand the item bank and continue refining scoring and norming procedures. This opens the enticing possibility of broader application of the PEVT across diverse populations while ensuring that score interpretations remain appropriately evidence based. Finally, the present findings provide preliminary content, response-process, internal-structure and relations-to-other-variables evidence in support of the intended interpretation of PEVT scores, while also identifying clear priorities for further validation work (Chapelle, Reference Chapelle2012; Kane, Reference Kane2012).
6. Conclusion
This study presented the development and preliminary validity evidence for the PEVT as a vignette-based measure of productive EV in adult users of English as an Lx. The findings showed that vignettes effectively elicited a wide range of emotion words and intended target emotions or close synonyms within this sample. Distinct item formats and scoring methods have been empirically evaluated, clarifying how these capture complementary aspects of productive EV: the number of context-appropriate labels produced and the extent to which responses align with context-specific naming patterns, offering a solid foundation for further development. Mixed-effects analyses further showed that response order, language proficiency and the lexical frequency of produced labels meaningfully shaped test performance, whereas context length had no consistent overall effect.
The main contribution of the PEVT is methodological: it offers a scalable way to assess productive, context-sensitive EV, a construct that has been difficult to capture with existing receptive or decontextualised instruments. In doing so, it provides a research tool for examining how bilingual and multilingual speakers access, select and produce emotion labels in response to situational and appraisal cues. This is important for applied linguistics and bilingualism research because emotion words are central to communication, yet remain underrepresented in vocabulary assessment.
As the first known productive EV test for English Lx users, the PEVT stands out for its adaptability and expandability to larger, better controlled item banks, alternative contexts and other languages. It offers a novel and innovative approach against the prolific obsession with multiple-choice formats commonly used in educational assessments, especially in vocabulary studies. The vignette-based format enhances authenticity by closely simulating real-world scenarios where the ability to interpret emotional cues and articulate emotions in social interactions is imperative. Consequently, the PEVT goes beyond decontextualised vocabulary assessment by situating emotion-word production at the interface of linguistic and socio-emotional interpretation, which is critical for affective communication competence and closely related to EI (Chee et al., Reference Chee, Szabo and Ambrose2026; Pentón Herrera & Darragh, Reference Pentón Herrera and Darragh2024). Overall, the PEVT offers a promising platform for further research, presenting clear priorities and opportunities to refine and expand its application across different linguacultures. In this sense, while the PEVT advances contextually grounded assessment of EV, it should not be viewed as a finished instrument. Rather, it provides an evidence-based foundation for further validation, including item-bank expansion, clearer scoring rubrics, cross-linguacultural norming and testing with broader proficiency groups. With these refinements, the vignette-based approach has the potential to support more ecologically valid assessment of productive EV across multilingual populations.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S1366728926101680.
Data availability statement
This article earned Open Data a badge for transparent practices. The R codes and materials are available upon request from the corresponding author. The anonymised, scaled dataset is available on the Open Science Framework (OSF): https://osf.io/7xzyv/overview?view_only=33c3d8ebe05848e2aab185b7f663421d.
Acknowledgments
The authors would like to express their sincerest gratitude to Shanna Lee Zai Dee, Toh Jia Qian, Drs Sharimila Ambrose and Jessica Price for their assistance in designing, developing, and reviewing the vignettes and Drs Imma Miralpeix and Pengchong Anthony Zhang for advanced review of earlier versions of the paper. We also thank the three anonymous reviewers and the editor for their invaluable suggestions.
Competing interests
The authors have declared no conflict of interest.
Open access
