1. Introduction
The impressive development of Transformers, along with the rapid increase in computing capabilities and the accessibility of vast amounts of data, has revolutionised natural language processing (NLP) in recent years. Large language models (LLMs) have excelled in tasks such as transcription, translation, or summarisation, among others. However, LLMs present well-known challenges, including the opacity of training data and the lack of clarity in their behaviour. In conversational systems, a major concern is their tendency to hallucinate, generate misinformation, and lack reasoning. Furthermore, while LLMs could recognise lexical and structural patterns, they struggle with semantic inferences and extralinguistic reasoning essential for real-world knowledge tasks (Lappin Reference Lappin2024).
Conversational systems should be able to generate diverse language forms to interact fluently and accurately with a variety of users. In this context, natural language generation (NLG) plays a crucial role in dialogue, directly influencing user perception. In open-domain dialogues, the NLG must be able to express opinions and emotions, convey facts, provide information, ask contextual questions (who, what, where, when, why), confirm user inputs, or give instructions. Conversely, closed-domain dialogues must align with the specific task, objectives, and conversational context. In both cases, generated sentences should be semantically controlled, accurate, and free of hallucinations.
Although end-to-end architectures have advanced in the design of dialogue systems (Ji et al. Reference Ji, Chen, Fang, Zuo, Lu, Wang, Jiang, Zhou, Liu, Cheng, Yang, Wang, Yang, Li, Jiang, He, Chu, Xu and Zhao2024; López Zorrilla and Torres Reference López Zorrilla and Torres2022), they still have difficulty achieving dialogue coherence and interpretability. In this context, the concepts of dialogue acts (DAs), which represent a sentence’s intentionality or purpose, and meaning representation (MR), which adds specific attributes and entities in the shape of slot–value pairs, remain essential for managing conversational flow. Specifically, NLG generates MR-based sentences, that is, it takes the MR as input and produces sentences as output. When combined with LLMs, both closed- and open-domain dialogue systems (Wang et al. Reference Wang, Xiao, Li, He, Li, Wang and Song2024) benefit from improved adaptability to complex tasks (Galland et al. Reference Galland, Pelachaud and Pecune2024), as well as enhanced coherence and response accuracy (Hu et al. Reference Hu, Dai, Yan, Zhang, Guo, Qiu and Zhang2022; Griol and Callejas Reference Griol and Callejas2024; Ohashi and Higashinaka Reference Ohashi and Higashinaka2025).
In this work, we aim to explore different ways of representing meaning for sentence generation. Our objective is to analyse whether providing the generator model with additional information, in the form of a task demonstrator, enhances generation quality. This task demonstrator is a sample from the original dataset consisting of an MR–sentence pair and enriches the input both at training and inference time. Specifically, we investigate whether these one-shot representations improve generation compared to inputs containing only the MR.
The comparison requires evaluation metrics. In NLG, classic metrics, some originally from translation or language model evaluation, do not always capture relevant differences in generated sentences. Nevertheless, each metric assesses a different aspect of generation, making their comparison essential for an in-depth study like ours. In this work, we use reference-based metrics such as BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002), BLEURT (Sellam et al. Reference Sellam, Das and Parikh2020), and LABSE (Feng et al. Reference Feng, Yang, Cer, Arivazhagan and Wang2022), as well as reference-free metrics like Slot Accuracy (Li et al. Reference Li, Yao, Qin, Che, Li and Liu2020), complemented by the more recent Dialogue Act Accuracy (DAC) (Ramirez et al. Reference Ramirez, Agarwal, Juraska, Garg and Walker2023). Our goal is to analyse each metric’s ability to identify differences in sentences generated from the proposed MR formats.
Four corpora have been selected for this study: E2E (Novikova et al. Reference Novikova, Dušek and Rieser2017), VIGGO (Juraska et al. Reference Juraska, Bowden and Walker2019), MULTIWOZ (Budzianowski et al. Reference Budzianowski, Wen, Tseng, Casanueva, Ultes, Ramadan and Gasic2018), and EMPATHIC (Torres et al. Reference Torres, Olaso, Montenegro, Santana, Vázquez, Justo, Lozano, Schlögl, Chollet, Dugan, Irvine, Glackin, Pickard, Esposito, Cordasco, Troncone, Petrovska-Delacrétaz, Mtibaa, Hmani and Gonzalez-Pinto2019). All four are dialogue-oriented NLG corpora in which each sample consists of a DA-based MR and its corresponding example sentence. These datasets differ significantly in domain, size, lexicon, input variability, and acquisition process, among other features. We will analyse the suitability of each MR and assess the discriminative power of each metric based on the characteristics of each corpus.
In summary, this work aims to study the impact of MRs on language generation in dialogue tasks. To this end, we thoroughly analyse the relevance of the tasks, the characteristics of the corpora, and the aspects each metric evaluates in the experimental results. For this comparative study, we selected GPT-2 (Radford et al. Reference Radford, Wu, Child, Luan, Amodei and Sutskever2019), which is computationally more accessible and manageable than recent state-of-the-art LLMs such as GPT-4 (Achiam et al. Reference Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, Avila, Babuschkin, Bajali, Balcom, Baltescu, Bao, Bavarian, Belgum, Bello and Zoph2023), Gemini (Team et al. Reference Anil, Borgeaud, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth, Millican, Silver, Johnson, Antonoglou, Schrittwieser, Glaese, Chen, Pitler, Lillicrap, Lazaridou, Firat and Vinyals2023), and LLaMA 3 (Dubey et al. Reference Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, Goyal, Hartshorn, Yang, Mitra, Sravankumar, Korenev, Hinsvark, Rao, Zhang and Ma2024). In addition, recent work has shown that enriched inputs with task demonstrators can substantially enhance the generation capabilities of medium-sized models (Liu et al. Reference Liu, Fang, Cheng, Pan, Liu and Gao2023).
The main contributions can be outlined as follows:
-
• Analysis of the impact of prompt forms on the quality of generated sentences, as well as their zero-shot capabilities.
-
• Comparative evaluation of each metric’s ability to assess different quality aspects of generated sentences.
-
• Assessment of the suitability of classifying generated sentences into DAs as an evaluation metric for generation quality.
-
• Analysis of the influence of corpus design and characteristics, such as the number of references and attributes, on the quality of sentence generation.
-
• Cross-analysis of the distribution of the resulting quality scores in terms of representations, corpora, and metrics.
-
• Human qualitative analysis of the generated sentences.
To the best of our knowledge, this is the first in-depth study on NLG in the context of dialogue, implementing a comparative analysis of the impact of MRs on the generation quality. This thorough analysis involves multiple domains, tasks, and corpus characteristics, as well as the metrics used to evaluate the generated sentences.
Overall, our experiments yield several practical insights. At the representation level, our proposed inputs are particularly helpful in complex domains and small datasets with high variability in terms of MRs, as well as the lexicon and semantics of their sentences. They also show zero-shot capabilities across domains. In terms of metrics, we observe that semantic metrics capture the quality of the generated outputs more accurately than lexical ones based on n-gram overlap. In addition, when comparing semantic metrics, those trained with human ratings can detect omissions and other subtle semantic issues that embedding-based metrics often miss. Finally, the evolution of metric scores during fine-tuning, together with strong results on Slot Accuracy and DAC, indicates that the models adapt quickly to the task and are robust at the semantic and communicative-intent levels.
This work has also served as the basis for two subsequent extensions. Specifically, Vázquez and Torres (Reference Vázquez and Torres2025) explores whether enriched MRs improve robustness and adaptability across different languages, contributing to the development of more inclusive and generalised dialogue systems. In turn, as part of a broader study, Vázquez (Reference Vázquez2026) includes an analysis of the impact of these representations on the outputs produced by LLMs in zero-shot settings.
The remainder of the paper is structured as follows. Section 2 explores the related work. Section 3 introduces the corpora used in our experiments and highlights their key differences. Section 4 explains the explored MRs, including the novel enriched representations and the methodology for selecting task demonstrators. Section 5 defines the evaluation metrics. Section 6 details the experimental results and discusses their key findings. Section 7 summarises the work with the main conclusions.
2. Related work
2.1 NLG for dialogue tasks
Task-oriented dialogue systems commonly adopt modular pipeline architectures due to the higher understandability and controllability of the process than end-to-end systems (Algherairy and Ahmed Reference Algherairy and Ahmed2024). Among these modules, NLG is crucial, transforming structured inputs—MRs—into coherent and fluent natural language outputs. These MRs typically consist of a DA indicating the communicative intent accompanied by slot–value pairs that encode the semantic content to be expressed. Work on task-oriented dialogue emphasises the relevance of structured semantic information, task-specific knowledge, and in-context examples for improving response generation, controllability, and adaptability (Reed et al. Reference Reed, Li, Ramirez, Wu and Walker2022; Ramirez et al. Reference Ramirez, Agarwal, Juraska, Garg and Walker2023).
This field experienced a revolution driven by Transformer-based architectures and LLMs. The Transformers (Vaswani Reference Vaswani2017) significantly improved the handling of linguistic long-range dependencies, that is, relationships between tokens or discourse elements that are far apart in the input sequence, and allowed for efficient parallelisation. These architectures laid the foundation for the first LLMs, such as GPT-2 (Radford et al. Reference Radford, Wu, Child, Luan, Amodei and Sutskever2019), T5 (Raffel et al. Reference Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li and Liu2020), and BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019), which have proven highly effective for NLG tasks. Models like DialoGPT (Zhang et al. Reference Zhang, Sun, Galley, Chen, Brockett, Gao, Gao, Liu and Dolan2020) and SC-GPT (Peng et al. Reference Peng, Zhu, Li, Li, Li, Zeng and Gao2020) demonstrate the adaptability of GPT-2 for dialogue tasks. On the one hand, DialoGPT, a tunable response generation model trained with conversations, can be integrated into open-domain or domain-specific systems with small-scale fine-tuning. On the other hand, SC-GPT is pretrained on a large annotated NLG corpus, acquiring the capability to controllably transform MRs into natural language outputs. Finally, this model, with a small amount of fine-tuning, has also demonstrated the ability to generate adequate responses for spoken dialogue systems in noisy environments (Mousavi et al. Reference Mousavi, Roccabruna, Alghisi, Rizzoli, Ravanelli and Riccardi2024).
Newer LLMs—GPT-4 (Achiam et al. Reference Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, Avila, Babuschkin, Bajali, Balcom, Baltescu, Bao, Bavarian, Belgum, Bello and Zoph2023), Gemini (Team et al. Reference Anil, Borgeaud, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth, Millican, Silver, Johnson, Antonoglou, Schrittwieser, Glaese, Chen, Pitler, Lillicrap, Lazaridou, Firat and Vinyals2023), and LLaMA 3 (Dubey et al. Reference Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, Goyal, Hartshorn, Yang, Mitra, Sravankumar, Korenev, Hinsvark, Rao, Zhang and Ma2024)—which are much larger than their precursors, have further advanced the state of the art by showcasing capabilities to perform various NLG tasks with minimal or no fine-tuning. ChatGPTFootnote a represents this evolution, demonstrating the ability to handle multi-turn dialogues with coherent, fluent, and logical responses. Additionally, Ramirez et al. (Reference Ramirez, Agarwal, Juraska, Garg and Walker2023) have shown that these models can produce outputs with near-perfect semantic accuracy without fine-tuning through prompt learning techniques that enrich the input with demonstrators of the task to perform. Similarly, Bowden and Walker (Reference Bowden and Walker2025) used ChatGPT to produce realistic synthetic dialogue corpora, showing its potential for generating high-quality conversational data while controlling aspects such as quality, safety, lexical diversity, and topic consistency.
However, the computational requirements of these new LLMs pose significant challenges, such as hardware requirements, memory constraints, or real-time performance. Consequently, some authors still prefer using smaller models, such as GPT-2 (López Zorrilla et al. Reference López Zorrilla, Torres and Cuayáhuitl2023). Liu et al. (Reference Liu, Fang, Cheng, Pan, Liu and Gao2023) and Jiang et al. (Reference Jiang, Zhang, Luo, Yuan, Zhao and Liu2024) have demonstrated that enriched inputs with task demonstrators enhance the generation capabilities of these models. Inspired by them, we perform a detailed study of the impact of enriched MRs in the performance of GPT-2 models, with the innovation of fine-tuning with these representations—a relatively underexplored approach.
2.2 MRs for NLG
NLG inputs are wide and varied, as are their tasks. In the bibliography, we find works about describing information given in a table, summarising an article from the news or generating text expressing the meaning encapsulated in a graph (Moryossef et al. Reference Moryossef, Goldberg and Dagan2019; Parikh et al. Reference Parikh, Wang, Gehrmann, Faruqui, Dhingra, Yang and Das2020; Gehrmann et al. Reference Gehrmann, Adewumi, Aggarwal, Ammanamanchi, Anuoluwapo, Bosselut, Chandu, Clinciu, Das, Dhole, Du, Durmus, Dušek, Emezue, Gangal, Garbacea, Hashimoto, Hou, Jernite and Zhou2021), among others. However, DA-based MRs remain the predominant input type used in most dialogue tasks (Du et al. Reference Du, Oraby, Perera, Shen, Narayan-Chen, Chung, Venkatesh and Hakkani-Tur2020; Wang et al. Reference Wang, Zhang, Guo, Dai, Chen and Luo2022; Wu et al. Reference Wu, Gung, Shu and Zhang2023).
These representations became more important with the rise of neural models and Transformer architectures, which resulted in more annotated datasets (Novikova et al. Reference Novikova, Dušek and Rieser2017; Budzianowski et al. Reference Budzianowski, Wen, Tseng, Casanueva, Ultes, Ramadan and Gasic2018; Rastogi et al. Reference Rastogi, Zang, Sunkara, Gupta and Khaitan2020). For instance, the well-known E2E dataset, annotated in terms of DA, was created as part of the E2E NLG challenge (Novikova et al. Reference Novikova, Dušek and Rieser2017). This challenge demonstrated the capability of diverse models and architectures to generate fluent and varied outputs using DA-based MRs. Peng et al. (Reference Peng, Zhu, Li, Li, Li, Zeng and Gao2020) also showed that models capable of transforming these inputs to sentences in one domain can adapt to new domains with a small amount of fine-tuning. Furthermore, these MRs have been employed with relevant contributions in works that employed methodologies like continual learning (Mi et al. Reference Mi, Chen, Zhao, Huang and Faltings2020) or reinforcement learning (Wang et al. Reference Wang, Zhang, Kim and Gu2021). Finally, MRs are a practical way to encapsulate information for the NLG module in dialogue system architectures (Wang et al. Reference Wang, Zhang, Guo, Dai, Chen and Luo2022; Liu et al. Reference Liu, Fang, Vandyke and Collier2024; Ohashi and Higashinaka Reference Ohashi and Higashinaka2025).
Efforts to improve traditional DA-based MRs have led to significant advancements. For instance, Du et al. (Reference Du, Oraby, Perera, Shen, Narayan-Chen, Chung, Venkatesh and Hakkani-Tur2020) enriched inputs with descriptions about the nature of DAs and slots, while Tsai et al. (Reference Tsai, Oraby, Perera, Kao, Du, Narayan-Chen, Chung and Hakkani-Tur2021) incorporated additional style-control information into structured NLG inputs, showing that complementary input signals can improve controllability while preserving semantic adequacy. These strategies enhanced semantic accuracy, robustness, and diversity. In a different approach, Abstract meaning representations (AMRs) emerged as a novel way to capture the meaning of a sentence by abstracting its syntax, focusing on relationships between concepts and their roles (Banarescu et al. Reference Banarescu, Bonial, Cai, Georgescu, Griffitt, Hermjakob, Knight, Koehn, Palmer and Schneider2013; Yang et al. Reference Yang, Tang and Lin2024; Hryhoryeva Reference Hryhoryeva2025). They have also produced inspiring advances in tasks like machine comprehension, machine translation, and summarisation (Tohidi and Dadkhah Reference Tohidi and Dadkhah2022; Jin et al. Reference Jin, Chen, Adauto, Liu, Zhang, Michael, Schölkopf and Diab2024; Baptista et al. Reference Baptista, Reis, Dias and Santos2024). In dialogue, although AMRs are predominantly used in natural language understanding modules (Brutti et al. Reference Brutti, Donatelli, Lai and Pustejovsky2022; Bonial et al. Reference Bonial, Foresta, Fung, Hayes, Osteen, Arkin, Hedegaard and Howard2023), some studies highlight their potential in improving NLG as well (Yang et al. Reference Yang, Tang and Lin2024; Hryhoryeva Reference Hryhoryeva2025).
LLMs have revolutionised the field by enabling most NLP tasks to be tackled effectively through prompt-based learning techniques (Brown et al. Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Hebert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter and Amodei2020; Gao, Fisch, and Chen, Reference Gao, Fisch and Chen2021; Gu et al. Reference Gu, Han, Liu and Huang2022). These methodologies involve augmenting inputs with task-specific instructions, explanations, or task demonstrators to leverage LLMs’ zero-shot and few-shot learning capabilities. Specifically, in this study, we explore new enriched representations that include a task demonstrator in the input. Building on prior work that demonstrated their benefits for a specific domain (Vázquez et al. Reference Vázquez, López Zorrilla, Olaso and Torres2023), we present a novel comparison that explores their impact on multiple domains and datasets.
2.3 NLG datasets for dialogue
High-quality datasets have facilitated progress in the research on NLG for dialogue systems. They provide a basic tool to train and evaluate models. Therefore, over the years, researchers have created a variety of datasets tailored to different dialogue tasks, ranging from single-domain scenarios to multi-domain and knowledge-grounded interactions.
Wen et al. (Reference Wen, Gašić, Mrkšić, Su, Vandyke and Young2015) and Novikova et al. (Reference Novikova, Dušek and Rieser2017) were pioneers in creating annotated datasets designed to generate task-specific sentences. Wen et al. (Reference Wen, Gašić, Mrkšić, Su, Vandyke and Young2015) define a series of DAs and slots to develop dialogue systems that provide information about restaurants and hotels. Similarly, the E2E dataset (Novikova et al. Reference Novikova, Dušek and Rieser2017), focused on restaurant recommendations, was created to address the problems of classic NLG approaches.
Following these early works, multi-domain datasets were developed to address the growing need for systems capable of handling more diverse and complex dialogues. The Schema-Guided Dialogue dataset (Rastogi et al. Reference Rastogi, Zang, Sunkara, Gupta and Khaitan2020) introduced a unified schema for intents and slots that enhances scalability and generalisation across domains, even unseen ones. Similarly, MultiWOZ (Budzianowski et al. Reference Budzianowski, Wen, Tseng, Casanueva, Ultes, Ramadan and Gasic2018) is a large-scale dataset that offers richly annotated dialogues in domains like restaurants, hotels, and transportation. This dataset is an evaluation benchmark for tasks such as MR-to-text generation or dialogue state tracking.
Recently, the need to create systems that generate knowledge-grounded outputs has resulted in a new trend of datasets. These datasets incorporate information from external sources in their MRs. For instance, ViGGO (Juraska et al. Reference Juraska, Bowden and Walker2019), a dataset in the domain of video games, includes the information extracted from WikiData in its MRs as the content of their attribute–value pairs. In an extension of this work, Vazquez et al. (Reference Vazquez, Ramirez, Pullabhotla, Qiang, Zhang, Walker and Torres2024) develop a systematic methodology that combines the ViGGO DAs with WikiData knowledge graphs to create a similar dataset, WikiDialog, with seven new domains. Similarly, OpenDialKG, which obtains external information from the database Freebase, generates knowledge-rich dialogues across domains like music, movies, and sports (Moon et al. Reference Moon, Shah, Kumar and Subba2019).
In this study, we compare datasets spanning single-domain (E2E), multi-domain (MultiWOZ), and knowledge-grounded dialogues (ViGGO). Additionally, we have incorporated the EMPATHIC corpus (Torres et al. Reference Torres, Olaso, Montenegro, Santana, Vázquez, Justo, Lozano, Schlögl, Chollet, Dugan, Irvine, Glackin, Pickard, Esposito, Cordasco, Troncone, Petrovska-Delacrétaz, Mtibaa, Hmani and Gonzalez-Pinto2019; Montenegro et al. Reference Montenegro, López Zorrilla, Mikel Olaso, Santana, Justo, Lozano and Torres2019; Olaso et al. Reference Olaso, Vázquez, Ben Letaifa, De Velasco, Mtibaa, Hmani, Petrovska-Delacrétaz, Chollet, Montenegro, López-Zorrilla, Justo, Santana, Tenorio-Laranga, González-Fraile, Fernández-Ruanova, Cordasco, Esposito, Gjellesvik, Johansen and Schlögl2021; Vázquez et al. Reference Vázquez, López Zorrilla, Olaso and Torres2023). This corpus is extracted from health coaching sessions where a virtual coach helps users change their unhealthy habits in the domain of nutrition. Therefore, we add this scarcely explored and particularly challenging task, coaching, to our already novel and thorough comparison of the impact of an alternative enriched representation across a diverse range of datasets.
2.4 NLG metrics
The evaluation metrics for NLG have advanced significantly in recent years (Sai et al. Reference Sai, Mohankumar and Khapra2022). However, metrics originally developed for machine translation or summarisation tasks keep being the most widely used for NLG (Schmidtová et al., Reference Schmidtová, Mahamood, Balloccu, Dušek, Gatt, Gkatzia, Howcroft, Plátek and Sivaprasad2024; Celikyilmaz et al. Reference Celikyilmaz, Clark and Gao2020; Sai et al. Reference Sai, Mohankumar and Khapra2022). Metrics such as ROUGE (Lin Reference Lin2004), METEOR (Banerjee and Lavie Reference Banerjee and Lavie2005), NIST (Doddington Reference Doddington2002), and mainly BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002)—which rely on n-gram overlap—are almost universally present in evaluations of NLG systems (Ramirez et al. Reference Ramirez, Agarwal, Juraska, Garg and Walker2023; Wu et al. Reference Wu, Gung, Shu and Zhang2023) and included in the most prominent NLG benchmarks and challenges (Colin et al. Reference Colin, Gardent, M’rabet, Narayan and Perez2016; Novikova et al. Reference Novikova, Dušek and Rieser2017; Gehrmann et al. Reference Gehrmann, Adewumi, Aggarwal, Ammanamanchi, Anuoluwapo, Bosselut, Chandu, Clinciu, Das, Dhole, Du, Durmus, Dušek, Emezue, Gangal, Garbacea, Hashimoto, Hou, Jernite and Zhou2021). Consequently, we selected BLEU for our study as a standard metric in NLG and to include a widely used lexical perspective in our evaluation, which complements the other metrics and evaluation approaches considered in this work.
Metrics that evaluate semantic accuracy are also widely used in the NLG community (Schmidtová et al. Reference Schmidtová, Mahamood, Balloccu, Dušek, Gatt, Gkatzia, Howcroft, Plátek and Sivaprasad2024; Reed et al. Reference Reed, Li, Ramirez, Wu and Walker2022; Ramirez et al. Reference Ramirez, Agarwal, Juraska, Garg and Walker2023; Ruiz et al. Reference Ruiz, Torres and del Pozo2023), as they tend to align more closely with human ratings. Among these, BLEURT (Sellam et al. Reference Sellam, Das and Parikh2020) is particularly relevant due to its strong correlation with human evaluations, as it has been explicitly trained with human ratings. In this work, along with BLEURT, we introduce a new dimension to the analysis using LaBSE (Feng et al. Reference Feng, Yang, Cer, Arivazhagan and Wang2022). LaBSE, a language-agnostic embedding model, computes semantic similarity as the cosine similarity of sentence embeddings. While conceptually distinct, BLEURT and LaBSE have not yet been compared in detail regarding their performance in prior research.
NLG metrics typically evaluate outputs based on ground-truth references, but metrics that compare the outputs with their corresponding input have gained importance in recent years (Schmidtová et al. Reference Schmidtová, Mahamood, Balloccu, Dušek, Gatt, Gkatzia, Howcroft, Plátek and Sivaprasad2024). In dialogue systems, where MRs containing slots are prevalent, slot error rate or Slot Accuracy is often used to evaluate a generator’s robustness (Du et al. Reference Du, Oraby, Perera, Shen, Narayan-Chen, Chung, Venkatesh and Hakkani-Tur2020; Li et al. Reference Li, Yao, Qin, Che, Li and Liu2020; Wang et al. Reference Wang, Zou, Fan, Aw and He2023). This metric determines how many values, usually entities, in the input are present in the output. Recently, some studies have adopted an alternative approach to analyse DA-based input–output alignment: the DAC (Reed et al. Reference Reed, Li, Ramirez, Wu and Walker2022; Ramirez et al. Reference Ramirez, Agarwal, Juraska, Garg and Walker2023). In these works, DA classifiers are proposed to assess output quality, where a generated sentence is considered acceptable if the classifier can predict its source DA. Based on the findings of these works, we incorporated the underexplored DAC along with Slot Accuracy to broaden the scope of our analysis.
In summary, our study evaluates NLG engines and representation approaches using a diverse range of automatic metrics, including underexplored dimensions. Additionally, unlike previous works that primarily validate models and methodologies through metrics, we explore metrics’ focus and provide a detailed analysis of how these metrics interact with features of representations, datasets, and tasks to shape outcomes.
3. Corpora
The datasets define the domains and tasks in which we validate the proposed representations. We selected four dialogue NLG corpora that differ in domain, size, MR variability, lexical variability, and acquisition process, among other features. In particular, we have included E2E (Novikova et al. Reference Novikova, Dušek and Rieser2017), ViGGO (Juraska et al. Reference Juraska, Bowden and Walker2019), MultiWOZ (Budzianowski et al. Reference Budzianowski, Wen, Tseng, Casanueva, Ultes, Ramadan and Gasic2018), and EMPATHIC (Torres et al. Reference Torres, Olaso, Montenegro, Santana, Vázquez, Justo, Lozano, Schlögl, Chollet, Dugan, Irvine, Glackin, Pickard, Esposito, Cordasco, Troncone, Petrovska-Delacrétaz, Mtibaa, Hmani and Gonzalez-Pinto2019; Montenegro et al. Reference Montenegro, López Zorrilla, Mikel Olaso, Santana, Justo, Lozano and Torres2019; Olaso et al. Reference Olaso, Vázquez, Ben Letaifa, De Velasco, Mtibaa, Hmani, Petrovska-Delacrétaz, Chollet, Montenegro, López-Zorrilla, Justo, Santana, Tenorio-Laranga, González-Fraile, Fernández-Ruanova, Cordasco, Esposito, Gjellesvik, Johansen and Schlögl2021; Vázquez et al. Reference Vázquez, López Zorrilla and Torres2023). Despite their differences, all four datasets consist of MRs paired with corresponding example sentences. Moreover, they all use DA-based MRs, where the DA primarily represents the communicative intent of the sentence, while the slotFootnote b–value pairs encapsulate the semantic content. This representation serves as the baseline of our experiments, as explained in Section 4. Tables 1–4 show that this representation structure is shared across all datasets.
Samples of the E2E dataset
Table 1. Long description
The table presents samples from the E2E dataset, showing meaning representations (MR) and corresponding sentences. It has two columns labeled MR and Sentence. Row 1: MR: inform (name = The Vaults; eatType = pub; priceRange = more than 30; customerRating = 5 out of 5; near = Caf Adriatic). Sentence: The Vaults pub near Caf Adriatic has a 5 star rating. Prices start at 30. Row 2: MR: inform (name = The Cambridge Blue; eatType = pub; food = English; priceRange = cheap; near = Caf Brazil). Sentence: Close to Caf Brazil, The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of 10.50. Delicious Pub food.
Samples of the ViGGO corpus
Table 2. Long description
A table comparing dialogue NLG corpora samples. The table has two columns: MR and Sentence. Row 1: inform (name = The Forest of Doom; release_year = 2014; genres = role-playing, text adventure; has_multiplayer = no), The Forest of Doom is a role-playing text adventure game released in 2014. It is a single-player only game. Row 2: give_opinion (name = Undertale; esrb = E (for Everyone); rating = excellent), Undertale is a downright excellent game, and it’s rated E so everyone can enjoy it. Row 3: request_attribute (has_multiplayer = ?), Do you prefer to game with others, I mean, multiplayer?
3.1 E2E datasetFootnote c
E2E dataset (Novikova et al. Reference Novikova, Dušek and Rieser2017) is a well-known corpus in the domain of restaurants obtained through crowdsourcing. Although it was created for dialogue tasks, it was collected through crowdsourcing, with participants writing sentences for predefined MRs without any contextual dialogue. Table 1 shows that all the MRs of this dataset use the same DA, inform, with multiple combinations of attributes and values. The two samples in the table inform about two different restaurants, sharing one slot–value pair (
$eatType=pub$
) and multiple attributes such as priceRange or near, but also presenting attributes (customerRating and food) and values (“The Vault” vs “The Cambridge Bleu” or “more than £30” vs “cheap”) that are different.
Samples of the MultiWOZ corpus
Table 3. Long description
A table with two rows and two columns. The first row contains the headers MR and Sentence. The first row data: MR: SYSTEM_Booking_Book (bookday = Wednesday; bookpeople = five; booktime = 11 am; name = this restaurant; ref = DU8IWQZ2) & SYSTEM_Restaurant_Inform (address = 2 Rose Crescent City Centre; name = The Gardenia). Sentence: The Gardenia is located at 2 Rose Crescent City Centre. I have booked your table at this restaurant for five people, Wednesday at 11 am. Your reference number is DU8IWQZ2. The second row data: MR: USER_general_thank(). Sentence: No that will be all, thank you.
Samples of the EMPATHIC corpus
Table 4. Long description
The table presents samples from four dialogue natural language generation corpora: E2E, ViGGO, MultiWOZ, and EMPATHIC. It has two columns labeled MR and Sentence. The table contains two rows of data for each corpus. Row 1: inform (name = The Vaults ; eatType = pub ; priceRange = more than 30 ; customerRating = 5 out of 5 ; near = Caf Adriatic), The Vaults pub near Caf Adriatic has a 5 star rating. Prices start at 30. Row 2: inform (name = The Cambridge Blue ; eatType = pub ; food = English ; priceRange = cheap ; near = Caf Brazil ), Close to Caf Brazil, The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of 10.50. Delicious Pub food. Row 3: inform (name = The Forest of Doom ; release_year = 2014 ; genres = role-playing, text adventure ; has_multiplayer = no ), The Forest of Doom is a role-playing text adventure game released in 2014. It is a single-player only game. Row 4: give_opinion (name = Undertale ; esrb = E (for Everyone) ; rating = excellent ), Undertale is a downright excellent game, and its rated E so everyone can enjoy it. Row 5: request_attribute (has_multiplayer = ? ), Do you prefer to game with others, I mean, multiplayer?
3.2 ViGGO corpusFootnote d
ViGGO corpus (Juraska et al. Reference Juraska, Bowden and Walker2019) is a dataset in the domain of video games collected with the same protocol as E2E. However, ViGGO contains more variability in terms of DA. Table 2 presents samples of three of the nine different communicative intentions: inform, give an opinion, or ask about a preference for a specific video game characteristic. The attributes and values appear in multiple slot–value pair combinations in the MRs. The attributes are domain-specific, such as release_year or esrb, in the samples of Table 2.
3.3 MultiWOZ corpusFootnote e
MultiWOZ corpus (Budzianowski et al. Reference Budzianowski, Wen, Tseng, Casanueva, Ultes, Ramadan and Gasic2018) is a large-scale dataset that contains dialogues covering seven domains: restaurants, hotels, attractions, taxis, trains, hospitals, and police. It includes annotated complete dialogues between a user and an assistant. In these dialogues, the user asks for information in one of these domains, and the assistant, thought to be replaced by an automatic system, provides this information. They are simulated conversations since they were obtained by employing crowdsourcing in a Wizard-of-Oz (WOZ) framework, where the crowd workers wrote the next turn for a given dialogue context previously created by other workers. As the complete dialogues were annotated, Table 3 presents turns of users (USER in DAs) and assistants (SYSTEM). Unlike the other datasets, MultiWOZ contains MRs that combine simple DAs, like the first example of Table 3. In these cases, the DA is the union of simple DAs, so in this example of the table, the DA is the union of SYSTEM_Booking_Book and SYSTEM_Restaurant_Inform. This DA is domain-specific, but MultiWOZ also presents open-domain DAs, such as the DA of the second example of the table: USER_General_Thank. Finally, the attributes are domain-specific, like bookday or address, and are the entities that appear in the dialogues.
3.4 EMPATHIC corpusFootnote f
The EMPATHIC corpus was gathered during the EMPATHIC project (Torres et al. Reference Torres, Olaso, Montenegro, Santana, Vázquez, Justo, Lozano, Schlögl, Chollet, Dugan, Irvine, Glackin, Pickard, Esposito, Cordasco, Troncone, Petrovska-Delacrétaz, Mtibaa, Hmani and Gonzalez-Pinto2019; Montenegro et al. Reference Montenegro, López Zorrilla, Mikel Olaso, Santana, Justo, Lozano and Torres2019; Olaso et al. Reference Olaso, Vázquez, Ben Letaifa, De Velasco, Mtibaa, Hmani, Petrovska-Delacrétaz, Chollet, Montenegro, López-Zorrilla, Justo, Santana, Tenorio-Laranga, González-Fraile, Fernández-Ruanova, Cordasco, Esposito, Gjellesvik, Johansen and Schlögl2021; Vázquez et al. 2023). This project aims to change users’ unhealthy habits through questions and consequent reflection. For this end, a dialogue system was built to carry out health coaching sessions following a behaviour model, the GROW model (Leach Reference Leach2020).
During this project, the EMPATHIC corpus was collected through WoZ experiments. These experiments were simulated coaching sessions where a real user spoke with a wizard who played the role of a virtual coach. From these spoken and complete interactions, only the coach turns were annotated in terms of DAs. Table 4 shows four samples of the EMPATHIC corpus, where the ad-hoc MRs designed for this corpus can be observed. This corpus presents two similarities between this dataset and MultiWOZ. First, it contains open-domain DAs such as Gen_Hello and close-domain DAs such as GSQ or RQ. This second group is highly bound to the GROW behavioural model based on questions, so GSQ and RQ stand for Goal Set Question and Reality Question, respectively. Second, the attributes, which are close-domain, were the entities found in the previously acquired conversations.
3.5 Corpora features
Table 5 summarises the characteristics of the four datasets and shows how different they are. The first two columns have already been mentioned in the definition of the datasets. So, in terms of domains, only E2E and MultiWOZ share a domain, restaurants. Meanwhile, only E2E and ViGGO employed the same protocol for the acquisition processes.
Characteristics of the four selected corpora. Domains stands for the topics the datasets cover. Acquisition process means the methodology to obtain the datasets. Numbers of DAs, attributes, and MRs quantify the diversity of inputs. Corpus size stands for the number of MR–sentence pairs. Running words include all the words in the corpus, whereas Vocabulary is the number of different words. In the Number of DAs for MultiWOZ, 798 represents the total number of DAs that appear as a combination of the 52 simple DAs defined for the corpus
Table 5. Long description
A table comparing characteristics of four datasets. The table has 5 columns and 4 rows. Column headers are Dataset, Domains, Acquisition process, Number of DAs, Number of attributes, Number of MRs, Corpus size, Running words, and Vocabulary. Row 1: E2E, Restaurants, Crowd, 1, 8, 6.0k, 51k, 1.0M, 5.1k. Row 2: ViGGO, Video games, Crowd, 9, 14, 2.3k, 6.9k, 150k, 4.4k. Row 3: MultiWOZ, 7 domains, WOZ + Crowd, 798 (52), 25, 54k, 140k, 1.9M, 30k. Row 4: EMPATHIC, Coaching, WOZ, 78, 21, 1.2k, 2.3k, 13k, 1.7k.
The number of DAs and attributes in relation to the number of MRs also makes a difference between the corpora. While E2E only employ one DA and eight attributes for 6k MRs, EMPATHIC employ many more DAs (78) and attributes (21) for a reduced quantity of MRs (1.2k). These numbers represent a big difference in the variability of the MRs. Only the large-scale MultiWoZ contains higher variability than EMPATHIC in the representations, with 52 simple DAs combined for a total of 798 DAs, due to its special MRs.
Another important difference among the corpora is their size, that is, the number of samples. In this regard, MultiWOZ and E2E are much bigger than the other two datasets.
The datasets also differ in the number of references. The references are the sentences associated with the same MR in the corpus. This feature does not appear in the table but is highly bound with the Corpus size and the Number of MRs. The first feature is higher than the second for all the datasets, that is, all the datasets present multiple sentences for the same MR. In Section 6.4, we analyse the importance of this dataset characteristic.
Finally, while Running words are the total words in the corpus, Vocabulary stands for the number of different words. In this regard, E2E combines low MR variability, with only one DA, with low lexical variability in its sentences: although it contains around 1 M running words, its vocabulary includes only 5.1k different words. By contrast, the remaining datasets show greater lexical variability. For instance, ViGGO’s vocabulary size is very close to E2E’s despite having one order of magnitude fewer samples and running words.
4. Meaning representations
In this section, we explore different ways to represent the input to feed NLG systems. First, we chose the original MRs of the corpora (Tables 1–4) as Baseline representations. As we described in Section 3, these MRs consist of a DA and a set of slot–value pairs. Next, building on the promising research in Ramirez et al. (Reference Ramirez, Agarwal, Juraska, Garg and Walker2023); Vázquez et al. (Reference Vázquez, López Zorrilla and Torres2023), we propose including a task demonstrator in the input connected to the Baseline MR. This Baseline representation is also referred to as input MR because it is the MR for the generation in all these representations. Meanwhile, the task demonstrator is a sample extracted from the corpora, consisting of an MR (demonstrator MR) and its corresponding sentence (demonstrator sentence). These enriched representations change both the input at inference time and the training data.
Meaning representations overview. One Baseline representation and three Prompt representations. The Baseline representation, which is the MR found in the original datasets, is the input MR (orange boxes) for the generation in all the representations. Prompt representations are created by connecting a task demonstrator with the input MR. This task demonstrator includes an MR, the demonstator MR (green), and its corresponding example sentence, the demonstrator sentence (grey). The colours of the boxes, the letters n and m for the number of the attributes, and the use (or not) of apostrophes inside Input and Demonstrator MR boxes are used to indicate which elements are necessarily identical (or not) among the representations.
Figure 1 shows a schema of the four representations of the meaning studied in this work. The figure shows that Prompt 1, Prompt 2, and Prompt 3 follow the same structure, but they differ in how similar demonstrator MR is to the input MR. In this regard, the conditions to select the demonstrator for each Prompt representation are as follows:
-
• Prompt 1: The DA of the demonstrator MR must be the same as the input MR, as it is indicated with the light blue boxes in Figure 1.
-
• Prompt 2: Demonstrator MR must include the same DA (light blue boxes) and number of attributes (n) as the input MR
-
• Prompt 3: The Demonstrator MR must contain the same DA (light blue boxes) and attributes (light yellow boxes) as the input MR.
These conditions for selecting a demonstrator make task demonstrators from Prompt 3 more specific to the Baseline than those from Prompt 2, and in turn, those from Prompt 2 more specific than those from Prompt 1. Table 6 presents real examples, one for each dataset, of input MRs (first row of each dataset) with their corresponding demonstrator for the three Prompt representations (next rows). They illustrate that the conditions stated above are satisfied and show the specificity of the demonstrators for each proposed representation. In MultiWOZ, for instance, the DA for the Baseline representation and for the Demonstrator MR of the three enriched representations is SYSTEM_Booking_Book & SYSTEM_general_reqmore. Next, the number of attributes of the Input MR in this dataset is 2, which matches the number of them in the corresponding Demonstrator MR of Prompt 2 and Prompt 3. Finally, bookstay and ref (highlighted in red and orange in all the rows of this dataset) are the attributes for the Baseline representation and the Demonstrator MR of Prompt 3. By contrast, Prompt 1 and Prompt 2 present attributes that do not appear in the original representation (highlighted in brown). The same restrictions are followed in the other datasets. Prompt 1 was discarded for E2E because the selection methodology, which we explain next, would assign the same task demonstrator for all the E2E MRs.
Examples of task demonstrators linked to an input MR for each dataset. DAs and attributes are in bold and coloured to highlight identical elements between the demonstrator MRs and the input MR. Note that brown colour is employed for those attributes that do not appear in the input MR, but they do in any demonstrator MR. Prompt 1 was discarded for E2E because the selection methodology would assign the same task demonstrator for all the E2E MRs
Table 6. Long description
Table with five rows and four columns. Columns are labeled Dataset, Example type, and Examples. The table presents examples of task demonstrators linked to an input MR for each dataset. DAs and attributes are in bold and colored to highlight identical elements between the demonstrator MRs and the input MR. Note that brown color is employed for those attributes that do not appear in the input MR, but they do in any demonstrator MR. Prompt 1 was discarded for E2E because the selection methodology would assign the same task demonstrator for all the E2E MRs. Row 1: Dataset: E2E; Example type: Input MR; Examples: inform (name = Zizzi; eatType = coffee shop; customer rating = high; near = Burger King). Row 2: Dataset: E2E; Example type: Prompt 2 demonstrator; Examples: inform (name = Loch Fyne; eatType = restaurant; familyFriendly = yes; food = English) + A child friendly restaurant that has English food in Loch Fyne. Row 3: Dataset: E2E; Example type: Prompt 3 demonstrator; Examples: inform (name = Wildwood; near = Café Rouge; customer rating = low; eatType = restaurant) + Wildwood, a restaurant with a low customer rating, is near Café Rouge. Row 4: Dataset: ViGOO; Example type: Input MR; Examples: give_opinion (name = Super Mario World; player_perspective = side view; rating = excellent; has_multiplayer = yes). Row 5: Dataset: ViGOO; Example type: Prompt 1 demonstrator; Examples: give_opinion (name = Guitar Hero: Smash Hits; rating = poor; developer = Beenox) + Guitar Hero: Smash Hits is just not my kind of game. I find Beenox games are often not very fun. Row 6: Dataset: ViGOO; Example type: Prompt 2 demonstrator; Examples: give_opinion (name = Worms: Reloaded; player_perspective = side view; rating = average; available_on Steam = yes) + Worms: Reloaded is an average quality side view game you can at least find on Steam. Row 7: Dataset: ViGOO; Example type: Prompt 3 demonstrator; Examples: give_opinion (name = Portal 2; player_perspective = first person; has_multiplayer = yes; rating = excellent) + I’ve been having an absolute blast playing Portal 2. First person games are always my favorite, but the multiplayer makes it even better. Row 8: Dataset: MultiWOZ; Example type: Input MR; Examples: SYSTEM_Booking_Book (bookstay = 2; ref = VH33JKKF) SYSTEM_general_reqmore (). Row 9: Dataset: MultiWOZ; Example type: Prompt 1 demonstrator; Examples: SYSTEM_Booking_Book (ref = GS2CO4IV) SYSTEM_general_reqmore () + Your table is booked and will be reserved for 15 minutes. Your reference number is GS2CO4IV. May I help you with anything else today?. Row 10: Dataset: MultiWOZ; Example type: Prompt 2 demonstrator; Examples: SYSTEM_Booking_Book (name = Cambridge Belfry; ref = LJOY6H2) SYSTEM_general_reqmore () + Your booking at the Cambridge Belfry was successful. The reference number is LJOY6H2. Can I help you with anything else today?. Row 11: Dataset: MultiWOZ; Example type: Prompt 3 demonstrator; Examples: SYSTEM_Booking_Book (bookstay = 4; ref = 3VUMBFZ0) SYSTEM_general_reqmore () + Great, you’re booked for 4 nights with reference number 3VUMBFZ0. May I help you with anything else?. Row 12: Dataset: EMPATHIC; Example type: Input MR; Examples: RQ_curr_sit (action = cook). Row 13: Dataset: EMPATHIC; Example type: Prompt 1 demonstrator; Examples: RQ_curr_sit (action = tell; freq = daily) + Can you tell me about your daily eating habits?. Row 14: Dataset: EMPATHIC; Example type: Prompt 2 demonstrator; Examples: RQ_curr_sit (food = water) + And do you think you drink enough water?. Row 15: Dataset: EMPATHIC; Example type: Prompt 3 demonstrator; Examples: RQ_curr_sit (action = change) + Would you like to change your eating habits?.
In this proposal, we also aim to reduce the number of different demonstrators in training, making our approach extensible even to datasets with reduced size. Thus, our selection methodology of demonstrators assigns the same demonstrators to all MRs that satisfy identical conditions. In other words, all the MRs from the original dataset with the same DA are connected to the same demonstrator for Prompt 1, similarly, those with the same DA and number of attributes for Prompt 2, and the same DA and exact attributes for Prompt 3. Therefore, the number of different demonstrators in the training set matches the number of DAs for Prompt 1, the combination of DAs and the number of attributes for Prompt 2, and the combinations of DAs and attributes for Prompt 3.
Table 7 shows the number of demonstrators for each dataset in each Prompt representation. This table confirms that the number of demonstrators for Prompt 1 matches the number of DAs (Table 5). Consequently, the number of different demonstrators for Prompt 1 in E2E would be 1. So, we discarded this representation for this dataset since all input MRs would have been linked to the same demonstrator. In addition, as expected, the more specific the demonstrators, the higher the number of them, that is, Prompt 3 always contains more than Prompt 2, which, in turn, contains more than Prompt 1. Lastly, the corpora with more MR variability, such as MultiWOZ and EMPATHIC, employ more demonstrators than, for instance, E2E, with a very low variability in this regard.
Number of different demonstrators in each dataset for the three Prompt representations
Table 7. Long description
The table presents the number of different demonstrators for each dataset across three prompt representations. It has four rows and four columns. The columns are labeled Prompt 1, Prompt 2, and Prompt 3. The rows are labeled with the names of the datasets: E2E, ViGOO, MultiWOZ, and EMPATHIC. Row 1: E2E, Prompt 1: blank, Prompt 2: 6, Prompt 3: 120. Row 2: ViGOO, Prompt 1: 9, Prompt 2: 21, Prompt 3: 473. Row 3: MultiWOZ, Prompt 1: 798, Prompt 2: 2281, Prompt 3: 11644. Row 4: EMPATHIC, Prompt 1: 78, Prompt 2: 197, Prompt 3: 537.
aThere is no Prompt 1 representation for E2E, because E2E only has one DA, so all the MRs from the original corpus would be connected to the same demonstrator for this representation.
5. Metrics
The metrics play an important role in this work. We aim to explore what each metric can detect and understand how the features of tasks, corpora, and input representations affect the model evaluation with each metric. In addition, our selection of metrics is designed to cover complementary representation spaces—ranging from widely adopted, standard NLG metrics to more recent and less commonly used measures—so that we can assess multiple aspects of the outputs (e.g., lexical similarity, semantic similarity, and input–output coherence). These selected metrics can be grouped into reference-driven and reference-less metrics and are as follows:
-
• Reference-driven metrics: Metrics whose scores are obtained by comparing each generated sentence with the multiple references found in the corpus. We have selected three reference-driven metrics that cover lexical and semantic aspects as well as, to some extent, human criteria.
-
– BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002): Metric universally employed in the NLG-related works, although it was originally created for the translation task. It is a good indicator of lexical accuracy as its evaluation is based on n-gram overlapping. In this work, we employ BLEU-4 (with n-grams from uni-grams to 4-grams) with a smoothing function (Chen and Cherry Reference Chen and Cherry2014).
-
– BLEURT (Sellam et al. Reference Sellam, Das and Parikh2020): Machine-learned metric that aims to replicate human evaluation to some extent, thanks to its training process. This process employed a BERT model trained with a public collection of 430k sentence pairs and ratings based on human perception for each pair. As a result, the model returns a similarity value for each pair of sentences.
-
– LaBSE (Feng et al. Reference Feng, Yang, Cer, Arivazhagan and Wang2022): Metric based on sentence embeddings. They are obtained with a BERT dual encoder characterised by being a language-agnostic model that supports more than 100 languages. The metric scores are the cosine similarity between the sentence embeddings of generated sentences and references.
-
• Referenceless metrics: They can also be named source-based metrics because they compare the generated sentence (the output) with the MR (the input). In this case, we have selected two metrics that evaluate the coherence of the generated sentences in terms of semantic content and communicative intention.
-
– Slot Accuracy (Li et al. Reference Li, Yao, Qin, Che, Li and Liu2020): Metric that focuses on the values associated with the slots in the MRs. It evaluates whether the model includes them in the generated sentences. So, the score is the percentage of values in the input that appear in the output, providing an indication of the robustness of the NLG in terms of how much input information is communicated in the output. Note that this metric does not include generated sentences obtained from MR without attributes. Therefore, the evaluation set could be smaller than the one used in other metrics.
-
– Dialogue Act Accuracy (Ramirez et al. Reference Ramirez, Agarwal, Juraska, Garg and Walker2023): The DAC is an underexplored metric that quantifies the accuracy of classifiers to predict the source DA of a generated sentence. The scores are the results of classification problems where the inputs are the generated sentences, and the outputs are the DAs. In this work, four corpora were selected that differ in the number and typology of DAs. So, different types of DA classifiers were trained. There was no classifier for E2E because it only has one DA. Then, ViGGO and EMPATHIC classifiers are multiclass, as only one DA can be assigned to each generated sentence. By contrast, the MultiWOZ classifier is multilabel because its MR accept DAs that are a combination of simple DAs. For the classifiers, we employed DistilBERT models (Sanh Reference Sanh2019) because they have demonstrated great training capabilities despite their reduced size. In addition, we split the original datasets in a proportion of 95/5 for training and validation, where the validation set gave us reference values to determine when a generated output is reasonably good in this regard.
6. Results & discussion
This section presents a thorough comparative study that defines the objective of our work. This comparison allows us to analyse three different aspects: (a) the MR that best leverages the capabilities of the generative models, (b) the impact of these MRs on generation quality across datasets, and (c) the influence of dataset and task features on model evaluation with each metric. Therefore, we obtain a comprehensive view of how these three elements—representations, datasets, and metrics—interact and affect the results within the GPT-2 generative models considered in this work.
In general terms, the GPT-2 experiments reported in this study have shown that enriched representations perform particularly well on small datasets involving complex tasks and high variability in MRs, as well as the lexicon and semantics of the sentences. However, they can act as distractors if the baseline MR is already long and contains a lot of semantic information. These enriched representations are also effective in zero-shotFootnote g settings, obtaining better results than the Baseline before training. The analysis of the metrics reveals that those that evaluate semantic aspects capture output quality more accurately than those that focus on the lexicon. Finally, the fine-tuned GPT-2 models show rapid adaptation to the analysed tasks and robustness in handling the input at both semantic and communicative intention levels.
To explain this detailed study with the highest clarity, we structure the section as follows. First, we present the experimental framework for training the models and generating the sentences in Section 6.1. Next, we show and discuss the results obtained in the evaluation of the generated sentences. The analysis of the results is split into results with reference-driven (Section 6.2) and referenceless metrics (Section 6.3). Next, Section 6.4 goes deeper into evaluating the importance of dataset features in the results, exploring specific corpus features such as the number of references and attributes per MR. Finally, Section 6.5 describes a human study of the generated sentences that allows us to determine the role of metrics and representations in the results.
6.1 Experimental framework
We selected one of the pretrained GPT-2 models (Radford et al. Reference Radford, Wu, Child, Luan, Amodei and Sutskever2019) due to their acceptable computational requirements and their demonstrated suitability for fine-tuning and prompt-based learning approaches (Peng et al. Reference Peng, Zhu, Li, Li, Li, Zeng and Gao2020; Liu et al. Reference Liu, Fang, Cheng, Pan, Liu and Gao2023). In particular, we perform experiments with the GPT-2 Medium model to validate the proposed MRs and compare their impact on the generated outputs across domains. Accordingly, the empirical findings reported in the following sections should be interpreted as evidence for this model and experimental set-up, rather than as claims about all contemporary generative architectures.
These experiments involve fine-tuning procedures with fivefold cross-validation, ensuring that the models’ performances are rigorously evaluated across different subsets of the data. For consistency with the pretraining set-up, we utilise the same byte-pair tokeniser (Sennrich et al. Reference Sennrich, Haddow and Birch2016) during the fine-tuning process. In training, we perform over five epochs using a learning rate scheduler with a linear warm-up that starts 5e-5, a batch size of 8, and an Adam optimiser with weight decay (Kingma and Ba Reference Kingma and Ba2014). For the generation phase, we generate five distinct outputs per MR. Each output is constrained to a maximum length of 80 tokens, whereas we set the temperature to 1.0, stimulating output variability.
6.2 Analysis of the results with reference-driven metrics
To assess how well the different representations support generation quality across domains, we begin by analysing the scores obtained for the reference-driven metrics, namely BLEU, BLEURT, and LaBSE. These metrics calculate their scores by comparing the generated outputs, five per MR, with the references related to such MRs in the original corpora. This analysis offers insight into the behaviour of enriched and baseline representations across different datasets, using as evaluation criteria metrics that rely on such predefined outputs.
To obtain the results for each metric, first, a score (generation score) is computed for each one of the five generated outputs per input by comparing them separately with all the references. The scores for each MR (MR scores) are the average of the five generation scores. Finally, the average scores for each metric are the average over all the MR scores. These scores are calculated for each combination of metrics, representations, and datasets. Note that in this analysis, we focus on the average scores after fine-tuning (Table 8) and the evolution of these scores before training, epoch 0, and after each epoch (Figure 2).
Average scores after fine-tuning for each reference-driven metric, representation, and dataset. Base corresponds to the baseline representation, whereas Pi is the Prompt i representation with i = 1,2,3. The datasets (Data) are E2E, ViGGO (ViG), MultiWOZ (MWOZ), and EMPATHIC (EMP). No Prompt 1 for E2E. The best score for each dataset is denoted in bold. When present, the symbol
$^\dagger$
indicates that the best-performing representation is significantly different from all the remaining representations for the same dataset and metric according to paired two-sided t-tests over the MR-level scores, with Holm–Bonferroni correction for multiple comparisons (
$p\lt 0.05$
)
Table 8. Long description
The table presents average scores after fine-tuning for each reference-driven metric, representation, and dataset. It is divided into three sections: (a) BLEU, (b) BLEURT, and (c) LaBSE. Each section lists scores for different datasets (E2E, ViG, MWOZ, EMP) and representations (Base, P1, P2, P3). Panel A: BLEU scores. Row 1: E2E, Base 0.15, P1 N/A, P2 0.12, P3 0.13. Row 2: ViG, Base 0.21, P1 0.15, P2 0.17, P3 0.19. Row 3: MWOZ, Base 0.16, P1 0.14, P2 0.14, P3 0.17. Row 4: EMP, Base 0.17, P1 0.19, P2 0.20, P3 0.26. Panel B: BLEURT scores. Row 1: E2E, Base 0.60, P1 N/A, P2 0.54, P3 0.54. Row 2: ViG, Base 0.65, P1 0.58, P2 0.61, P3 0.63. Row 3: MWOZ, Base 0.60, P1 0.57, P2 0.58, P3 0.59. Row 4: EMP, Base 0.55, P1 0.56, P2 0.57, P3 0.60. Panel C: LaBSE scores. Row 1: E2E, Base 0.76, P1 0.69, P2 0.70, P3 N/A. Row 2: ViG, Base 0.82, P1 0.73, P2 0.77, P3 0.79. Row 3: MWOZ, Base 0.68, P1 0.66, P2 0.67, P3 0.67. Row 4: EMP, Base 0.66, P1 0.68, P2 0.68, P3 0.70.
6.2.1 Average scores after fine-tuning
Table 8(a), (b), and (c) show the scores obtained after fine-tuning the models with each representation–dataset pair for BLEU, BLEURT, and LaBSE, respectively. These tables provide an initial overview of the models’ performance differences across metrics and allow us to identify which representation obtains the best results (denoted in bold) for each dataset. As reported in Vázquez (Reference Vázquez2026), these scores generally arise from Gaussian distributions across datasets and metrics, suggesting a relatively homogeneous behaviour around the average values. The main exception is EMPATHIC, where the scores show bimodal tendencies and are distributed over a wider part of the score range in line with the particular characteristics of this dataset.
Low BLEU results. Table 8 shows very low results for BLEU. Its values (no higher than 0.3) are much lower than the results obtained with the other two metrics (no lower than 0.5). This discrepancy with the other two metrics suggests that lexical similarity based on n-gram overlap may not adequately quantify the output quality.
Demonstrators effective in small datasets with complex tasks. Table 8 shows that Prompt representations are effective in datasets with the special characteristics of EMPATHIC. It seems that the demonstrators take more relevance when the dataset is small, with a high variety in the input and the output, and a complex task. For E2E, ViGGO, and MultiWOZ, the results with the enriched representations are satisfactory, but the models perform better with the Baseline MR.
Prompt 3: The best prompt. Prompt representations obtain better results when they include more specific task demonstrators in their input, across all datasets. In other words, Prompt 3 consistently outperforms Prompt 2, which in turn is better than Prompt 1. These more specific representations require a higher number of demonstrators during training, as shown in Table 7 of Section 4. This may suggest that good performance depends on having a sufficient number of demonstrators. However, the results obtained with Prompt 1 for EMPATHIC show that enriched representations can still be effective even when the number of training demonstrators is low.
Evolution of the average score of each reference-driven metric during the fine-tuning for each representation in each dataset.
6.2.2 Evolution of the scores
To complement the analysis of average scores, we examine how model performance evolves throughout the fine-tuning process. This dynamic perspective helps us understand not only the final outcomes but also how different representations support the model’s progressive adaptation to the task. In particular, tracking the evolution of scores across epochs allows us to assess the zero-shot effectiveness of the representations and their impact on the model’s learning.
Figure 2 presents learning curves for each combination of metric, dataset, and representation, showing the average score before training (zero-shot) and after each of the five epochs. Each row in the figure corresponds to a specific dataset (E2E, ViGGO, MultiWOZ, and EMPATHIC, in that order), and each column corresponds to a specific metric (BLEU, BLEURT, and LaBSE, respectively). Within each subplot, the evolution of the average scores for each representation is plotted separately.
Prompt-representations trigger model zero-shot capabilities. All the learning curves present higher values for the Prompt representations than for the Baseline before training (epoch 0), except for BLEU (the first column of plots), whose curves start very close to 0.0 across all datasets and representations. In other words, the zero-shot capabilities of the generative models are activated by the enriched representations. Even in datasets that do not benefit from these representations after fine-tuning, this improvement is observed in the zero-shot setting. Therefore, we show that basic enriched inputs with a single task demonstrator can improve generation quality before training, even in moderately sized models like GPT-2.
Good task adaptability. The curves generally show most of their improvement during the first epoch, reaching reasonably high scores—except for BLEU. Thus, we confirm that a capable generative model, once fine-tuned, can be easily adapted to multiple tasks and domains.
MultiWOZ: Slow adaptability with overly general demonstrators. The curves for Prompt 1 and Prompt 2 in MultiWOZ exhibit a more gradual learning process compared to the general behaviour. Although the final scores after five epochs are comparable, the curves do not show most of the improvement in the first epoch. This behaviour may be explained by two factors: a) the limited number of demonstrators for Prompt 1 and Prompt 2 (Table 7) compared to the size of MultiWOZ (Table 5), and b) the high output variability of MultiWOZ. These two factors might result in generic task demonstrators that fail to cover the output variability in MultiWOZ. As a result, the demonstrators seem not to be informative enough for all MR, making the adaptation process more challenging for the generative model.
Slot Accuracy average scores after fine-tuning for each representation and dataset. Base corresponds to the Baseline representation, whereas Pi is the Prompt i representation with i= 1,2,3. The datasets (Data) are E2E, ViGGO (ViG), MultiWOZ (MWOZ), and EMPATHIC (EMP). No Prompt 1 for E2E. The best score for each dataset is denoted in bold. When present, the symbol
$^\dagger$
indicates that the best-performing representation is significantly different from all the remaining representations for the same dataset and metric according to paired two-sided t-tests over the MR-level scores, with Holm–Bonferroni correction for multiple comparisons (
$p\lt 0.05$
)
Table 9. Long description
The table presents slot accuracy average scores after fine-tuning for each representation and dataset. It has four rows and five columns. The columns are labeled Data, Base, P1, P2, and P3. The row labels are E2E, ViG, MWOZ, and EMP. Row 1: E2E, Base 0.79, P1 -, P2 0.65, P3 0.67. Row 2: ViG, Base 0.80, P1 0.69, P2 0.72, P3 0.72. Row 3: MWOZ, Base 0.94, P1 0.91, P2 0.91, P3 0.91. Row 4: EMP, Base 0.88, P1 0.86, P2 0.87, P3 0.90. The best score for each dataset is denoted in bold. The symbol indicates that the best-performing representation is significantly different from all the remaining representations for the same dataset and metric according to paired two-sided t-tests over the MR-level scores, with HolmBonferroni correction for multiple comparisons.
6.3 Analysis of the results with referenceless metrics
Reference-driven metrics need scenarios with predefined references to compare, and they work better as long as these references are representative enough for the given inputs. In addition, their outcomes can be affected by the number of references associated with the MRs, as we will present in Section 6.4. In this work, we decided to employ referenceless metrics to overcome this reference necessity and widen the perspective with two completely different metrics: Slot Accuracy and DAC. We discuss the results of these two metrics separately because Slot Accuracy can be evaluated as a generation problem, while DAC is a classification problem.
6.3.1 Slot Accuracy
Slot Accuracy measures how accurately the generated sentence expresses the slot values specified in the input. To analyse the scores for this metric, Table 9 presents the average scores after fine-tuning, while Figure 3 depicts the evolution of these average scores over training. Note that, unlike reference-driven metrics, a generation score is computed by comparing such generated sentence directly with the source MRs, rather than with multiple reference sentences. After that, the process to calculate the MR scores and the average scores is the same as for the reference-driven metrics.
High semantic coherence. Table 9 presents excellent results mainly for EMPATHIC and MultiWOZ. These two datasets obtain around 90% of accuracy for all the MRs. E2E and ViGGO also obtain good results, mainly with the Baseline representation. These results mean that the fine-tuned models are semantically coherent with the input.
Evolution of the average scores of Slot Accuracy during the fine-tuning for each representation in each dataset.
Slot Accuracy confirms the model behaviour. Slot Accuracy confirms the model behaviour with the different representations and datasets. First, we observe the benefit of the Prompt representations for datasets like EMPATHIC (Table 9). Next, the performance improvement in zero-shot settings across all the datasets with the enriched inputs is even more evident for this metric (Figure 3). Finally, this figure also depicts the fast adaptability of all the models to the new tasks, except for the representations with very generic demonstrators for MultiWOZ.
6.3.2 Dialogue Act Accuracy
DAC scores are the results of the classification of the generated sentences in terms of DAs. To discuss them, Table 10 presents the average scores after fine-tuning, while Figure 4 depicts the evolution of those scores during the fine-tuning process. Note that E2E is excluded from this analysis, as it contains only one DA, that is, only one possible classification output. The analysis also does not present the distribution of MR scores because 0 (incorrect prediction DA) and 1 (correct) are the only possible outcomes of the classifiers. In addition, the table and figure show the results obtained using the validation partition as a reference for a good accuracy value in each dataset. The sentences evaluated in this validation set are from the corpus, whereas the rest of the evaluation is over sentences generated by the model.
Dialogue Act Accuracy average scores after fine-tuning for each representation in each dataset. Val and Base correspond to the Validation dataset and Baseline representation, respectively, whereas Pi are the Prompt i representations with i = 1,2,3. The datasets (Data) are ViGGO (ViG), MultiWOZ (MWOZ), and EMPATHIC (EMP). The best score for each dataset is denoted in bold. When present, the symbol
$^\dagger$
indicates that the best-performing representation is significantly different from all the remaining representations for the same dataset and metric according to paired two-sided t-tests over the MR-level scores, with Holm–Bonferroni correction for multiple comparisons (
$p\lt 0.05$
)
Table 10. Long description
The table presents Dialogue Act Accuracy average scores after fine-tuning for each representation in each dataset. It has four rows and five columns. The columns are labeled as Val, Base, P1, P2, and P3. The rows are labeled as ViG, MWOZ, and EMP. Row 1: ViG, Val: 0.98, Base: 0.99, P1: 0.65, P2: 0.89, P3: 0.94. Row 2: MWOZ, Val: 0.85, Base: 0.86, P1: 0.52, P2: 0.68, P3: 0.83. Row 3: EMP, Val: 0.68, Base: 0.65, P1: 0.72, P2: 0.70, P3: 0.70. The best score for each dataset is denoted in bold. The symbol indicates that the best-performing representation is significantly different from all the remaining representations for the same dataset and metric according to paired two-sided t-tests over the MR-level scores, with Holm-Bonferroni correction for multiple comparisons.
Evolution of the Dialogue Act Accuracy average scores during the fine-tuning for each representation in each dataset. No E2E as it only contains the inform DA.
High coherence with communicative intentions. The results in Table 10 illustrate the excellent classification capabilities of the selected models and the coherence of the generated sentences with the communicative intentions given by the DAs. First, the Validation column proves the exceptional performance of the DistilBERT models as classifiers. Ninety-eight per cent with 9 DAs for ViGGO, 85% with 52 DAs in a multilabel classification problem for MultiWOZ, and 68% with 78 DAs for EMPATHIC are excellent results, considering the number of DAs for each dataset. Second, the results obtained with the generated outputs for the different representations (last four columns) prove that these models can generate coherent sentences regarding communicative intention. Some results have even exceeded those obtained with the validation set.
Same model behaviour across all the metrics. Figure 4 confirms some features observed for the model behaviour in the interaction with the datasets and representations, such as the zero-shot capabilities with the Prompt representations, the fast (or slow in some specific cases) model adaption to the task, or the prompt influence in the generation for each dataset. This is very relevant because the metrics evaluate different aspects of the generated sentences.
6.4 Impact of the corpus characteristic on the results
The datasets used in this work are highly diverse, as described in Section 3. In this section, we analyse whether this diversity in corpus characteristics significantly affects the results. Specifically, we focus on two features shaped by the design and acquisition protocols of the datasets: the number of references and the number of attributes per MR. The number of references determines how many reference sentences are available for comparison in reference-driven metrics, potentially influencing these scores. In turn, the number of attributes is considered a measure of the complexity of the input. We examine the impact of these two corpus features on the three reference-driven metrics computed over the fine-tuned models: BLEU, BLEURT, and LaBSE. The section first describes the distribution of these features across datasets and then analyses their impact on the results separately.
6.4.1 Relevant corpus characteristics for generation
For the study of the influence of these two features, we first examine the distribution of MRs in each dataset by number of attributes and references. (Figure 5). The plots of this figure show the percentage of MRs that contain the number of references or attributes indicated on the x-axis of each plot. In the case of the number of attributes (Figure 5b), the numbers go from 0 to 8, whereas the x-axis of Figure 5a presents both specific numbers of references and intervals, such as from 6 to 10 (6–10) or more than 1000 (>1000).
MR distributions in terms of the number of references/attributes. The number of references/attributes is given on the x-axis with specific numbers of references/attributes or intervals. The distributions are in percentages.
Figure 5. Long description
Panel A: A line graph shows the percentage of MRs on the y-axis and the number of references on the x-axis. The x-axis includes intervals such as 1, 2, 3, 4-5, 6-10, 11-20, 21-100, 101-1000, and >1000. The y-axis ranges from 0 to 100 percent. Four datasets are represented: E2E, ViGGO, MultiWOZ, and EMPATHIC, each with distinct line styles and markers. Panel B: Another line graph displays the percentage of MRs on the y-axis and the number of attributes on the x-axis. The x-axis includes values 0, 1, 2, 3, 4, 5, 6, 7, and 8. The y-axis ranges from 0 to 40 percent. The same four datasets are represented with the same line styles and markers as in Panel A.
Figure 5a shows that all the MRs of ViGGO have 3 references,Footnote h while E2E, acquired with the same protocol, has been designed to have MRs that principally contain between 6 and 20 references. These references in both datasets generally express very similar semantic content and often present limited lexical variation, reflecting the controlled acquisition process of these datasets. For instance, inform (name = Loch Fyne ; eatType = restaurant ; food = English ; familyFriendly = yes ) is an E2E MR associated with several similar references such as “A child friendly restaurant that has English food is Loch Fyne.”, “A good kid-friendly, English restaurant is Loch Fyne.”, or “Loch Fyne serves English food. It’s a family friendly restaurant.”. By contrast, EMPATHIC and MultiWOZ were built by annotating previously created conversations. Consequently, half of the MultiWOZ and EMPATHIC MRs have only one reference, but these datasets also have a significant number of MRs with more than 100 references or even more than 1000 in the case of MultiWOZ. For example, Gen_Open_question (action = cook ; food = fish ) is an EMPATHIC MR linked to a single, highly specific sentence: “What’s your favourite fish to cook?”. Meanwhile, SYSTEM_general_bye () is a MultiWOZ MR associated with more than 5,000 references, including “I hope you enjoy your stay. Contact us anytime. Goodbye.”, “Thanks for using our system today!”, and “You have a good day too.”, which show notable lexical and surface variability.
The design of the datasets also influences the distribution of the number of attributes per MR (Figure 5b), with similarities between the same datasets. E2E and ViGGO were created to train NLG engines that can manage inputs with multiple attributes. Consequently, these two datasets always present at least one attribute and usually more than three, with a predominance of MRs between 3 and 4 attributes in ViGGO and between 5 and 6 in E2E. This tendency is illustrated by the previous E2E example, which contains four attributes: name, eatType, food, and familyFriendly. By contrast, the attributes for MultiWOZ and EMPATHIC are the entities found in the conversations during the annotation process. As a result, more than half of the MRs of these two datasets present no attributes or only one, whereas the proportion for MRs with more than four attributes is almost irrelevant. The previous EMPATHIC and MultiWOZ examples also reflect this behaviour: the EMPATHIC MR contains only two attributes, action and food, while the MultiWOZ MR SYSTEM_general_bye () contains no attributes.
Results for BLEU (top), BLEURT (middle), and LaBSE (bottom) scores in terms of the number of references per MR with the fine-tuned models. The x-axis establishes specific numbers of references or intervals of numbers of references.
6.4.2 Influence of the number of references
Figure 6 shows the distribution of scores in terms of the number of references across datasets. This figure includes three subplots per dataset, showing BLEU (top), BLEURT (middle), and LaBSE (bottom) scores obtained with the fine-tuned models. Note that the ViGGO dataset is not included in this analysis because all its MRs have three references. Each point in the plots represents the average score over all MRs that share the same number (or interval) of references as indicated on the x-axis. For example, a point over the label 6–10 corresponds to the average score computed across all MRs that have between 6 and 10 references.
No meaningful differences across metrics and representations. The results for BLEU, BLEURT, and LaBSE in each dataset mainly differ in their absolute values, while the overall trends remain consistent across the three metrics. In the comparison between representations, the differences observed in the results mainly appear in the MRs with one reference. Therefore, we can assume that the impact of the representations on the model is not affected by the number of references.
Higher impact in datasets with lexically and semantically different references. The comparison across datasets reveals a distinct effect of the number of references on the scores. In E2E, the scores remain very similar regardless of the number of references. By contrast, MultiWOZ and EMPATHIC show a clearer decrease in their scores as the number of references increases. We hypothesise that this behaviour is related to the output variability of the datasets: low in E2E and higher in MultiWOZ and EMPATHIC, as discussed in Section 3 and illustrated in the examples above. In MultiWOZ and EMPATHIC, MRs with many references are often associated with sentences that differ substantially in both lexicon and semantics. This reduces the likelihood that a generated sentence will obtain a high generation score against all the references, producing a lower average. In contrast, in E2E, even MRs with a high number of references tend to contain semantically similar sentences with limited lexical variation. Consequently, the number of references has little impact on this dataset.
6.4.3 Influence of the number of attributes
Results for BLEU (top), BLEURT (middle), and LaBSE (bottom) scores in terms of the number of attributes per MR with the fine-tuned models.
Figure 7. Long description
Four sets of line graphs depict the performance of different models across various datasets. Each set contains three line graphs labeled BLEU, BLEURT, and LaBSE. Panel A: E2E. Panel B: ViGGO. Panel C: MultiWOZ. Panel D: EMPATHIC. Each graph shows the performance scores on the y-axis and the number of attributes on the x-axis. The lines represent different prompts: Baseline, Prompt 1, Prompt 2, and Prompt 3. The graphs illustrate how the performance metrics vary with the number of attributes for each dataset and prompt.
Figure 7 shows the effect of the distribution of the number of attributes over the four datasets. The figure is the equivalent of Figure 6 but for the number of attributes. In this case, the ViGGO dataset is included in the study as its MRs do present variability in the number of attributes (Figure 5b).
Increasing impact of the MR with the number of attributes. Figure 7 shows consistent trends for BLEU, BLEURT, and LaBSE: the higher the number of attributes, the larger the differences among the representations. Therefore, the structure of the MRs becomes more relevant when the input includes a high number of attributes.
Issue with artificial perfect scores. Figure 7d shows scores of 1.0 for EMPATHIC MRs with six attributes in both metrics. Although these values may look highly positive, they actually reflect an undesired behaviour. Due to the small size of EMPATHIC and the very limited number of MRs with six attributes (Figure 5b), we found that these perfect scores come from MRs with a single reference that also appears in the task demonstrator of the Prompt 3 representation. In these cases, the model simply copies the demonstrator sentence, which matches the only reference used for evaluation, leading to artificially perfect scores. For example, the MR RQ_curr_sit_freq (action = think ; action = eat ; number = five ; food = fruits ; food = vegetables ; freq = per day ) is the only MR with this DA and these attributes. Therefore, its reference and the Prompt 3 demonstrator are identical: “And do you think you eat five fruits and vegetables per day?”. Moreover, the five generated samples are exactly the same sentence. Note that the effect on the average scores is negligible, since this happens in less than 1% of the samples. Still, these observations suggest that datasets that include complex MRs with few references may benefit from preprocessing steps such as paraphrasing or other data augmentation techniques to reduce this behaviour.
Direct relation with Slot Accuracy results. The number of attributes has a direct impact on the possible Slot Accuracy values. This metric is computed as the proportion of correctly generated slot values; therefore, MRs with fewer attributes lead to coarser score values. As shown in Figure 5b, MultiWOZ and EMPATHIC tend to have fewer than three attributes in their MRs, so the possible Slot Accuracy values are usually limited to 0, 50, or 100%. Conversely, E2E and ViGGO usually present inputs with between three and eight attributes, which allows for a wider range of possible Slot Accuracy values.Footnote i In addition, the size of the evaluation set for Slot Accuracy is drastically reduced for MultiWOZ and EMPATHIC, since inputs with no attributes are not included in their evaluation, which accounts for a substantial portion of these two datasets.
6.5 Qualitative analysis of the generated sentences
To complement the quantitative evaluation with a more in-depth understanding of the model outputs, we provide a qualitative analysis of the generated sentences. For this human evaluation, a manual selection of examples from ViGGO and EMPATHIC was carried out. We have selected these two datasets because they present notable differences in both design and performance. In this assessment, we first analyse why BLEU scores are much lower than those obtained with the other two reference-driven metrics: BLEURT and LaBSE. Then, we carry out a similar and novel comparison between these two semantic reference-based metrics. Finally, we study the opposing effect of the Prompt 3 representation in ViGGO and EMPATHIC, which was selected as the best among the enriched representations.
For all the comparisons, a set of samples was selected from the outputs of the fine-tuned models. The metric comparisons include generated sentences that show the largest discrepancies between the metrics analysed. Meanwhile, the comparison between the Baseline and Prompt 3 focuses on those MRs that, on average, show the largest differences in reference-based metric scores between the two representations, that is, the MRs with the highest average absolute difference in MR scores across the three reference-based metrics.
From this qualitative inspection, we confirm that metrics focusing on lexical aspects do not reliably reflect generation quality. In contrast, the human-trained BLEURT metric appears to capture omissions and semantic nuances that go undetected by LaBSE. Finally, the comparison of representations sheds light on the different impact of the enriched representations on ViGGO and EMPATHIC outputs. Specifically, adding a demonstrator to some long ViGGO inputs can induce a forgetting effect in the models, while in EMPATHIC, the demonstrator helps fine-tuned models to learn sentence structures principally.
Examples to compare BLEU with BLEURT and LaBSE. Slot Accuracy (Slot acc) is also included. Each example contains, from top to bottom: MR, references, a generated sentence, as well as the dataset and scores for all the metrics in the last row. Relevant elements are highlighted in bold and different colours as follows: orange for the headers related to the inputs, purple for those related to the outputs, blue for the datasets, red for BLEU scores, and green for BLEURT and LaBSE scores
Table 11. Long description
A table with four rows and five columns comparing BLEU, BLEURT, LaBSE, and Slot Accuracy scores across datasets. Column headers are MR, References, Generated sentence, and Dataset. Row 1: MR: inform (name = Crysis ; release_year = 2007 ; esrb = M (for Mature) ; rating = good ; genres = action-adventure, shooter ; has_multiplayer = yes). References: 3 references. Generated sentence: Crysis is an action-adventure shooter released in 2007. It has received good reviews. Dataset: ViGGO, BLEU 0.06, BLEURT 0.66, LaBSE 0.81, Slot acc 0.80. Row 2: MR: Gen_Open_question (action = cook ; food = fish). References: 1 reference: What’s your favourite fish to cook?. Generated sentence: Do you like to cook fish?. Dataset: EMPATHIC, BLEU 0.07, BLEURT 0.71, LaBSE 0.83, Slot acc 1.00. Row 3: MR: Gen_Hello (). References: 25 references. Generated sentence: Hello!. Dataset: EMPATHIC, BLEU 0.05, BLEURT 0.64, LaBSE 0.59.
6.5.1 Analysis of metric scores
Tables 11 and 12 present representative generation examples that show the different behaviour of the metrics. Each table contains ViGGO and EMPATHIC examples. These examples consist of the MR, the references (with the number of them) linked to this MR, the generated sentence, the dataset, and the generation scores for all the metrics, except DAC, whose individual scores are irrelevant. In the tables, the headers of the input elements are orange, while those of the output are purple. For the compared metrics, we highlighted the lowest values in red and the highest ones in green.
Examples to compare BLEURT with LaBSE. BLEU and Slot Accuracy (Slot acc) are also included. Each example contains, from top to bottom: MR, references, a generated sentence, as well as the dataset and scores for all the metrics in the last row. Relevant elements are highlighted in bold and different colours as follows: orange for the headers related to the inputs, purple for those related to the outputs, blue for the datasets, red for BLEURT scores, and green for LaBSE scores
Table 12. Long description
The table compares different metrics for evaluating generated sentences. It has two main sections: one for ViGGO and one for EMPATHIC examples. Each section includes MR, references, generated sentence, dataset, and scores for BLEURT, LaBSE, BLEU, and Slot Accuracy. The headers for input elements are in orange, and those for output elements are in purple. The lowest values are highlighted in red, and the highest ones in green. Row 1: MR, info (name = Tony Hawk’s Pro Skater 3, release_year = 2001, esrb = T (for Teen), genres = sport, player_perspective = third person); References, 3 references: Tony Hawk’s Pro Skater 3 was released in 2001. It is a third person sports game that is rated T (for Teen). Tony Hawk’s Pro Skater 3 was a third person sport game released in 2001. It was rated T (for Teen). Another third person perspective sport game is Tony Hawk’s Pro Skater 3. It came out in 2001 and is rated T (for Teen). Generated sentence, The 2001 sport game, Tony Hawk’s Pro Skater 3, was a third person game. It has a t (for teen) rating and is rated t (for physical). Dataset, ViGGO; BLEU, 0.34; BLEURT, 0.61; LaBSE, 0.92; Slot acc, 1.00. Row 2: MR, Gen_Agreement (); References, 91 references: Absolutely - Of course. - Yeah - Right - I totally understand. Generated sentence, Sure. Dataset, EMPATHIC; BLEU, 0.02; BLEURT, 0.09; LaBSE, 0.56; Slot acc, -.
BLEU is not adequate for evaluation of output quality. Table 11 shows three examples (one for ViGGO and two for EMPATHIC) of good generated outputs with very low BLEU (in red in the table) and reasonably high BLEURT and LaBSE scores (in green). All three outputs accurately reflect their corresponding MR. The ViGGO generation example, “Crysis is an action-adventure shooter released in 2007. It has received good reviews,” contains two omissions (esrb and has_multiplayer), but it is a good inform sentence that conveys nearly all the intended semantic content. Regarding the EMPATHIC generated sentences, “Do you like to cook fish?” and “Hello!” are perfectly acceptable outputs for an open question about cooking fish and a greeting, respectively. Thus, the poor BLEU scores in these examples confirm that it may not be suitable for assessing output quality. In addition, in the first EMPATHIC example, the generated sentence contains the n-gram “to cook fish?”, which uses the same words as the reference n-gram “fish to cook?”. However, this does not lead to a high score, presumably due to the different word order. Therefore, BLEU appears to be highly sensitive to lexical order.
BLEURT detects omissions and semantic nuances. Table 12 shows representative examples of the comparison performed between BLEURT and LaBSE. The two examples (one for each dataset) present higher scores for LaBSE. However, the human-trained BLEURT seems to capture more subtle aspects than LaBSE. For instance, the ViGGO example presents a hallucination in its generated output, where the slot esrb = T (for Teen) is generated twice, and one of them contains incorrect information: “It has a t (for teen) rating and is rated t (for physical).” This error goes unnoticed by LaBSE, which gives an almost perfect score of 0.92, whereas BLEURT produces a more adequate score of 0.61. In the EMPATHIC example, the generated sentence “Sure.” can be considered good for expressing agreement and obtains 0.09 from BLEURT and 0.56 from LaBSE. We attribute this difference to the fact that while all references are appropriate to convey agreement, not all express it with the same degree of emphasis. These semantic nuances in the meanings of “Yeah”, “I totally understand”, or the generated “Sure” appear to influence BLEURT scores, which may provide evidence of BLEURT’s sensitivity to such distinctions.
Examples of the generated sentences obtained with the Baseline and Prompt 3 representations. Each example contains from top to bottom: MR, references, the task demonstrator linked to the MR in the Prompt 3 representation, generated sentences obtained with the Baseline representation, generated sentences obtained with the Prompt 3 representation, and dataset and scores for all the metrics for both representations (Baseline/Prompt 3) in the last row. The highest score for each metric is highlighted in bold
Table 13. Long description
Table with 13 rows and 5 columns comparing generated sentences from Baseline and Prompt 3 representations. Columns are MR, References, Prompt 3 task demonstrator, Baseline generated sentences, Prompt 3 generated sentences, and Dataset. Each row presents data for different tasks. The table includes scores for metrics like ViGGO BLEU, BLEURT, LaBSE, and Slot accuracy for both Baseline and Prompt 3 representations. The highest scores for each metric are highlighted in bold.
6.5.2 Analysis of the influence of Prompt representation for different datasets
Table 13 shows two representative examples of generated sentences for Baseline and Prompt 3 inputs (one for ViGGO and one for EMPATHIC). In these examples, the input elements (with headers in orange in the table) are the selected Input MR (labelled MR), the references (including their number), and the task demonstrator used in the Prompt 3 representation. For the output (headers in purple), the table shows both the Baseline and Prompt 3 generated sentences (including their number) and the MR scores separately (Baseline/Prompt 3). The highest MR scores for each metric are highlighted in bold. Finally, the dataset is specified (in blue). In this comparison, although metric scores were used to select the examples, we focus on comparing the generated sentences to understand why enriched representations have an opposite effect across the two datasets.
Prompt representations produce omissions in ViGGO. ViGGO generated outputs with Prompt 3 tend to be shorter and contain omissions when the inputs contain high semantic context, that is, a high number of attributes. The ViGGO example of Table 13 illustrates it perfectly: all the Prompt 3 generated sentences included in the table contain multiple omissions and are much shorter than the references, whereas only the second Baseline output omits has_linux_release and hallucinates (incorrect generation of has_mac_release ). As a consequence, we may assume that concatenating the MR and the task demonstrator in the Prompt 3 representation results in an excessively long input, which makes the model omit some information.
Model learns from the demonstrator in EMPATHIC. The inspection of the EMPATHIC MRs shows Prompt 3 generated outputs where the model seems to learn from the task demonstrator, especially when it fails to correctly generate the MR with the Baseline representation. In the EMPATHIC example of Table 13, the generated sentences “Let’s talk about travel.” and “Let’s talk about travelling” clearly adopt the structure of Let’s talk about cinema from the task demonstrator. Although the other Prompt 3 generated output “We can have a trip to the islands.” is incorrect for Gen_Open_topic (hobbies = travel ), the Prompt 3 outputs are more accurate overall than the Baseline ones, all of which are incorrect. Therefore, we can assume that the proposed representations provide useful guidance in cases where fine-tuning alone is insufficient with the Baseline MRs.
In conclusion, the enriched representation proves effective in smaller datasets where fine-tuning on the original MRs is insufficient to capture the generation patterns. However, in datasets with inputs containing a high density of semantic content, such as ViGGO, the resulting enriched MRs tend to become excessively long, leading the model to omit relevant information.
7. Conclusions
In this work, we have analysed in depth the impact of different MR structures on the performance of GPT-2 models across multiple dialogue datasets. In particular, we focus on enriched representations that integrate a task demonstrator in the MRs. To validate our approach, we assess them across four datasets using five distinct metrics, offering a detailed analysis of the interactions between representation, domains, and metric features. The research has been performed by fine-tuning GPT-2 models, prioritising computational and energy efficiency to enable reproducibility on modest hardware and faster iteration.
Our study shows that, in the GPT-2 experiments, the proposed representations of meaning are especially effective for small datasets, complex tasks, and high variability in their MRs and sentences. In this context, the demonstrator enriches the input for MRs where the reduced fine-tuning is insufficient for an accurate generation. Conversely, in other datasets, the models perform better when they are fine-tuned with the original MRs. These datasets usually present long inputs with high semantic content, and the extra information of the task demonstrator might produce a forgetting effect in the models that leads to short generated outputs with omission. Another positive effect of the enriched representations appears in zero-shot settings, that is, before training the models. We have observed that all the models without training perform better with the Prompt representations, even in those datasets where these enriched inputs have not been effective after fine-tuning. Finally, the comparison between enriched representations proves that more specific demonstrators in the input are more beneficial for the models.
The thorough analysis of the metrics has also outlined relevant findings. First, metrics that evaluate lexical aspects based on n-gram overlapping do not seem to accurately capture output quality. In this regard, semantic metrics are more accurate. In a novel comparison of semantic metrics, we have observed that those metrics trained with human ratings detect omissions and semantic nuances not measurable by other metrics based on sentence embeddings. Finally, the reference-less metrics have confirmed the effect of the representations and datasets on the model’s behaviour and revealed the model’s coherence with the input in terms of communicative intention and semantics.
This study also highlights the suitability of GPT-2 models for generating language in the context of dialogue. Particularly, they present good adaptation capabilities. The excellent performance observed after just one training epoch, regardless of the representation or dataset, confirms this feature.
8 Limitations
The main limitation of this study is that the experiments were conducted only with GPT-2 models. This choice allowed us to perform a controlled and computationally accessible comparison across representations, datasets, and metrics, but it also limits the current generalisability of the findings. In particular, the results should not be interpreted as direct evidence that the same behaviours will necessarily hold for more recent medium-sized or large generative architectures, especially given the rapid evolution of language models.
Nevertheless, the conclusions of this work have been reinforced by subsequent experiments reported in Vázquez (Reference Vázquez2026). In particular, Section 3.8 of that doctoral thesis analyses the impact of these representations on outputs produced by more recent LLMs in zero-shot settings. Those later experiments suggest that several of the tendencies observed in this paper, especially the relevance of enriched MRs for more complex tasks and datasets such as EMPATHIC, persist beyond GPT-2. However, these results belong to a broader subsequent study and do not replace the need for future work that systematically evaluates the proposed representations with more contemporary architectures of comparable size to GPT-2 within the same experimental framework.
Funding statement
This work was supported by the CRYSTAL HORIZON-MSCA-SE (grant number 101182965), Spanish MCIU (BRAINS project grant number PID2024-155948OB-C51), and the Basque Government (A.V. grant number PRE 2020 1 0274).
Competing interests
The authors declare none.
†
p<0.05
†
p<0.05
†
p<0.05
Open access
