In the realm of data-to-text generation tasks, the use of large language models (LLMs) has become common practice, yielding fluent and coherent outputs. Existing literature highlights that the quality of in-context examples significantly influences the empirical performance of these models, making the efficient selection of high-quality examples crucial. We hypothesize that the quality of these examples is primarily determined by two properties: their similarity to the input data and their diversity from one another. Based on this insight, we introduce a novel approach, Double Clustering-based In-Context Example Selection, specifically designed for data-to-text generation tasks. Our method involves two distinct clustering stages. The first stage aims to maximize the similarity between the in-context examples and the input data. The second stage ensures diversity among the selected in-context examples. Additionally, we have developed a batched generation method to enhance the token usage efficiency of LLMs. Experimental results demonstrate that, compared to traditional methods of selecting in-context learning samples, our approach significantly improves both time efficiency and token utilization while maintaining accuracy.

Lexical semantic change (LSC) detection investigates changes in word meaning over time, focusing on language use at the lexical-semantic level from a diachronic perspective. The field has made significant progress over the past two decades, driven by the increased availability of multilingual benchmarks, notable performance improvements, and growing interdisciplinary applications. In this paper, we review the evolution of LSC models and benchmark constructions within the context of popular shared tasks. By categorizing LSC models into generations defined by key components, our investigation suggests that performance breakthroughs have been largely driven by advances in semantic representations, transitioning from count-based models to recent transformer-based approaches. Notably, transformer models have established themselves as state-of-the-art by integrating Word-in-Context tasks, which emphasize semantic proximity in context. Furthermore, we review substantial studies that primarily leverage diachronic word embeddings to explore political, social, and cultural contexts beyond the linguistic domain. Our work provides valuable insights for future model development and encourages further interdisciplinary exploration within digital humanities and social sciences.

Sentimental analysis (SA) of movie reviews has become an essential means of supporting audiences, filmmakers, and investors to facilitate data-driven marketing, support audience engagement, and maximize audience development returns. Nevertheless, current SA models are yet to overcome potentially daunting problems such as noisy data, ambiguity of language, limited handling of negations, and a loss of rich semantic information during mirror preprocessing that ultimately decreases classification performance. To overcome this hurdle, a unique two-stage strategy entitled Hierarchical Attention-Guided Sentiment Network (HASN) is proposed that enables reliable and interpretable sentiment predictions. The first stage includes a clearly defined preprocessing pipeline of the IMDb dataset that features text cleaning, lemmatizing, removing stop words, negation, TF-IDF filtering of low-frequency terms, and lastly, Word2Vec embeddings, which create semantically consistent and contextually rich representations of potential inputs. The second stage integrates a tri-stream attention recurrent extractor (TARE), which reconstructs and merges previously learned outputs with the learned syntactic and semantic contexts with the benefits of stacked Bi-LSTMs, combines learned syntactic and semantic contexts with multi-head attention to identify the tokens that drive sentiment, and ends with a residual-normalized GRU (ResNorm-GRU), which sequentially refines the sequential process of intake. The attentive conditional random field (Attn-CRF) serves as a classifier by boosting coherence in label predictions through attention weighting with modeling of the sequential dependencies. The experiments with the IMDb dataset demonstrated that the best results detected were the following: 98.09, 98.47, 97.70, and 98.08%. In summary, the HASN framework is a precise, scalable, and interpretable model for analyzing sentiments in movie reviews.

Living languages continuously evolve to reflect the cultural changes of human societies. This evolution manifests through neologisms (new words) or the semantic change of existing words (new meanings for existing words). Understanding the meaning of words is vital for interpreting texts from different cultures (regionalisms or slang), domains (e.g., technical terms), or time periods. In computer science, this phenomenon is relevant to computational linguistics tasks such as machine translation, information retrieval, and question answering. Semantic change can impact the performance of these applications, making it important to understand and characterize these changes formally. This problem has recently attracted significant attention from the computational linguistics community. Several approaches can detect semantic changes with good precision, but more effort is needed to characterize how word meanings change and to determine how to mitigate the impact of this phenomenon. This survey provides a comprehensive overview of existing approaches to the characterization of semantic change. We also formally define three classes of characterization: change in dimension (whether a word’s meaning becomes broader or narrower), change in orientation (whether a word acquires a more pejorative or ameliorative sense), and change in relation (whether a word is used in a new figurative context, such as a metaphor or metonymy). We demonstrate the applicability of this formalism on existing corpora, summarize the key aspects of selected publications, and discuss current needs and trends in research on semantic change characterization.

This survey provides an overview of the challenges of misspellings in natural language processing (NLP). Misspellings are ubiquitous in digital communication, and even if humans can generally interpret misspelt text, NLP models frequently struggle to handle it: this causes a decline in performance in common tasks like text classification and machine translation. In this paper, we reconstruct a history of misspellings as a scientific problem. We then discuss the latest advancements to address the challenge of misspellings in NLP. Main strategies to mitigate the effect of misspellings include data augmentation, double step, character-order agnostic, and tuple-based methods, among others. This survey also examines dedicated data challenges and competitions to spur progress in the field. Critical safety and ethical concerns are also examined, for example, the voluntary use of misspellings to inject malicious messages and hate speech on social networks. The survey also explores psycholinguistic perspectives on how humans process misspellings, potentially informing innovative computational techniques for text normalisation and representation. Additionally, the survey explores the challenges that misspellings pose in multilingual contexts. Finally, the misspelling-related challenges and opportunities associated with modern large language models are also analysed, including benchmarks, datasets and performances of the most prominent language models against misspellings. This survey provides a comprehensive review of recent research on misspellings and aims to serve as a valuable resource for researchers seeking to get up to speed on this problem within the rapidly evolving landscape of NLP.

Semantic textual similarity (STS) is to measure semantic equivalence between sentences; it plays an important role in natural language processing (NLP) tasks. The major core of STS is text representation. This paper studies how to obtain a sentence skeleton for text representation in STS. Unlike most existing syntax models, we propose a skeleton-based reinforcement learning method to identify the skeleton and construct sentence representations. Parallel networks are adopted to extract features of different dimensions in the sentence. In the framework of parallel networks, two sentence representation models are designed: context constrained LSTM (CC-LSTM) and Adorned skeleton LSTM (AS-LSTM). CC-LSTM builds the sentence representation by constraining the word context. AS-LSTM constructs the sentence representation through using the identified skeleton and its qualifiers. Our approach achieves good results without using external resources. Especially AS-LSTM, which outperforms the state-of-the-art without using external resources in the SICK dataset.

Controllable summarization models are typically limited only to a short text, such as a topic mention, a keyword, or an entity, to control the output summary. At the same time, existing models for controllable summarization are prone to generate artificial content, resulting in unreliable summaries. In this work, we propose a method for controllable abstractive summarization that can exploit arbitrary textual context from a short text to a collection of documents to direct the focus of the generated summary. The proposed method incorporates a sentence BERT model to extract an embedding-based representation of the given context, which is then used to tag the most representative words of the input document towards this context. In addition, we propose an unsupervised metric to evaluate the faithfulness of the topic-oriented sentences of the generated summaries with respect to the input document. Experimental results under different zero-shot setups demonstrate that the proposed method surpasses both state-of-the-art large language models (LLMs) and controllable summarization methods. The generated summaries are both reliable and relevant with respect to the input document.

Automatic named entity recognition (NER) is essential for many natural language processing applications, particularly in low-resource languages and those with corpora restricted to a single domain, where the lack of diverse data may hinder cross-domain generalisation. In this context, we attempt to contribute to the Galician NER field by extending manually annotated resources and exploring several approaches to improve the performance of automatic classifiers. To this end, we enrich two new datasets with NER annotations in Galician, which we release in this paper. Through experimental evaluation, we investigate the benefits of fine-tuning transformers on NER data by comparing various monolingual and multilingual BERT models and developing a new extra-small DeBERTa model for Galician. Furthermore, we assess the quality of the NER model by expanding the initial training corpus with additional data from related varieties, including Portuguese and Spanish. For evaluation, we use five Galician corpora, including our two newly annotated datasets and three adapted from existing resources, ensuring assessment across multiple domains. Our findings reveal (1) a positive impact of enhanced cross-lingual training data on improved model performance across five Galician test datasets; (2) similar performance of monolingual and multilingual models depending on the training data; and (3) the competitive effectiveness of extra-small BERT models for Galician NER tasks compared to larger models.

The study of the attribution of input features to the output of neural network models is an active area of research. While numerous Explainable AI (XAI) techniques have been proposed to interpret these models, the systematic and automated evaluation of these methods in sequence-to-sequence (seq2seq) models is less explored. This paper introduces a new approach for evaluating explainability methods in transformer-based seq2seq models, building upon the forward simulation of XAI methods. We use teacher-derived attribution maps as a structured side signal to guide a student model, and quantify the utility of different attribution methods through the student’s ability to simulate targets. Using the Inseq library, we extract attribution scores over source–target sequence pairs and inject these scores into the attention mechanism of a student transformer model under four composition operators (addition, multiplication, averaging, and replacement). Across three language pairs (de–en, fr–en, ar–en) and attributions from Marian-MT and mBART models, Attention, Value Zeroing, and Layer Gradient $\times$ Activation consistently yield the largest gains in BLEU (and corresponding improvements in chrF) relative to baselines. In contrast, other gradient-based methods (Saliency, Integrated Gradients, DeepLIFT, Input $\times$ Gradient, GradientShap) lead to smaller and less consistent improvements. These results suggest that different attribution methods capture distinct signals and that attention-derived attributions better capture the alignment between source and target representations in seq2seq models. Finally, we introduce an Attributor transformer that, given a source–target pair, learns to reconstruct the teacher’s attribution map. Our findings demonstrate that the more accurately the Attributor can reproduce attribution maps, the more useful an injection of those maps is for the downstream task. The source code can be found on GitHub.a

Crawling parallel texts—texts that are mutual translations—from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up leading to actual parallel content. In this work, we propose a smart crawling method that guides the crawl towards finding parallel content more rapidly. We follow a neural approach that consists in adapting a pre-trained multilingual language model based on the encoder of the Transformer architecture by fine-tuning it for two new tasks: inferring the language of a document from its Uniform Resource Locator (URL) and inferring whether a pair of URLs link to parallel documents. We evaluate both models in isolation and their integration into a crawling tool. The results demonstrate the individual effectiveness of both models and highlight that their combination enables us to address a practical engineering challenge: the early discovery of parallel content during web crawling in a given language pair. This leads to a reduction in the amount of downloaded documents deemed useless and yields a greater quantity of parallel documents compared to conventional crawling approaches.

Results obtained by transformer-based token classification models are now considered to be a benchmark for the Automatic Terminology Extraction (ATE) task. However, the unsatisfactory results (they rarely exceed 0.7 of the F1 value) raise the question of whether this approach is correct and of what text features are being remembered or inferred by the model trained on this type of annotation. In the paper, we describe a number of experiments using the fine-tuned RoBERTa base model on the ACTER data, RD-TEC, and three Wikipedia articles, which proved that the results of the ATE task obtained by such models depend considerably on the type of texts being processed and their relationship to the training data. While the results are relatively good for some texts with highly specialized vocabulary, the poor results seem to correlate with the high frequency (in general English texts) of tokens that are part of terms in a particular domain. Another property that affects the results is the degree of overlap between the vocabulary of the test data and the vocabulary of terms from the training data. Words that have been labeled as terms in the training data are usually labeled as terms in other, unrelated domains as well. Moreover, we show that the results obtained by these models are unstable—models trained on more data do not include all the items identified by models trained on a smaller dataset and can present substantially lower performance.

This paper presents a comparative study of neural machine translation (MT) models and large language models, emphasizing translation literalness, lexical diversity, and translation quality. We evaluate open-source neural machine translation (NMT) models, LLaMa 3.1 Instruct, and GPT-4o across nine language pairs in pretrained, fine-tuned, and few-shot prompting settings. This comprehensive analysis underscores the differential strengths of each model, contributing valuable insights into their respective utility in MT. Our findings in the literary and news domains show that fine-tuning reduces translation literalness, allowing freer translations across models, most strongly for LLaMa, which produces the least literal translations across domains. The effect of fine-tuning on lexical diversity is model-dependent: it decreases consistently for LLaMa but remains stable or is improved for NMT. Few-shot prompting, on the other hand, enhances GPT-4o’s lexical diversity across domains and significantly boosts its translation quality in the literary domain, yielding the highest overall COMET scores. Furthermore, our correlation analysis reveals a distinct domain effect: literary translation evaluation rewards diversity, whereas news translation evaluation favors literalness. Overall, the results highlight complementary strengths: few-shot GPT-4o for high-quality, lexically rich translations with low error rates, and fine-tuned LLaMa for freer, less literal translations.

In the digital information age, artificial intelligence is increasingly being applied to national governance and judicial decision-making assistance. Existing studies lack case studies and empirical analyses of the effectiveness of large models in aiding judicial decisions. To address this research gap, this study designs a comprehensive evaluation framework encompassing five core task dimensions: Task-oriented Information Extraction, Legal Article Citation, Event Extraction, Judicial Decision Generation, and Legal Opinion Generation. By using carefully crafted prompts to activate the legal reasoning capabilities of the models, we conducted extensive testing on 13 mainstream large language models (LLMs). The experimental results demonstrate that large models perform excellently in processing legal texts and providing preliminary legal opinions, but still exhibit shortcomings in complex legal reasoning and precise decision-making. On this basis, we applied a weakly supervised learning strategy to fine-tune the LLMs for targeted improvements. The results indicate that introducing a small amount of task-specific learning can significantly enhance the performance of LLMs in judicial tasks. This further underscores the critical role of data and the acquisition of domain-specific knowledge in applying AI technology to judicial tasks. Additionally, this study briefly discusses the issue of the boundaries of AI’s involvement in judicial activities, aiming to provide theoretical foundations and practical guidance for the deep integration of AI technology with legal practice.

This study investigates the nuanced challenges of fine-grained word sense disambiguation (WSD) tasks with regular polysemy detection (RPD) of the named entity, focusing on evaluating the trade-offs between encoder and decoder-based model performance and computational efficiency. The datasets, including Chinese Wordnet 2.0 (CWN) as sense inventory, the Social Media Corpus (PTT) for user-generated content, and the Academia Sinica Balanced Corpus (ASBC) for formal linguistic data, were chosen to provide a diverse and representative framework for evaluating both common nouns and proper nouns with regular polysemy in Taiwan Mandarin. This analysis evaluated ten encoder- and decoder-based models, assessing their performance on two tasks. The encoder-based models demonstrate comparable accuracy to the decoder-based models on WSD tasks (77.5% vs. 78.5%), and similarly strong performance in RPD tasks (84.2% vs. 83.8%). On a large-scale all-words WSD task, the encoder model not only outperformed the decoder model but also generated substantially lower carbon emissions – an eight-fold reduction. These differences underscore the trade-offs between model architecture and task-specific performance, highlighting the necessity for balancing performance and energy efficiency in the design and application of language models, advocating for sustainable and eco-friendly practices in natural language processing development.

Twitter is one of the most widely used social networks globally but is also notorious for harbouring significant levels of hate speech. This paradox has garnered the attention of companies and governments concerned about how digital hate can fragment communities and incite real-world violence. While extensive research has focused on detecting hate speech, there is a lack of comprehensive analysis of the actors involved, their characteristics, and their interactions within the online environment. Common assumptions categorise users into merely two groups—haters and non-haters—overlooking the existence of other groups that may more accurately represent the dynamics of hate dissemination. Additionally, existing user classification models often rely on large volumes of tweet data, a limitation given the restricted access to the Twitter API. Social networks are also frequently visualised using graphs where edges represent only superficial relationships. This study addresses these gaps through five research questions. We propose formal user clustering methods and develop a classifier that uses exclusively profile attributes—information more readily obtainable from the network. We also introduce a more nuanced definition of interaction for graph edges, based on ideological support or opposition between users. To conduct our analysis, we have extracted two complementary datasets: (i) a keyword-based corpus of 3.3M Spanish tweets containing hate-associated terms, of which 1.6M unique tweets were retained after filtering and (ii) a user-based corpus comprising timelines of $\approx$3,000 users linked to hate speech, totalling over 3 M tweets. Our results reveal the existence of three primary user classes—haters, upstanders, and neutrals—in contrast to the conventional binary classification. We demonstrate that profile attributes are reliable indicators for automatically classifying users and find significant statistical differences between these classes. Finally, we develop a graph visualisation tool to assist authorities in analysing interactions among different user types, providing a useful exploratory tool to support the analysis of online hate and inform potential mitigation strategies.

Soft prompt learning methods offer parameter-efficient tuning of pre-trained language models for few-shot scenarios. This study explores the integration of supervised contrastive learning (SCL) into two leading soft prompt tuning models: DifferentiAble pRompT (DART) and PTuning. By incorporating SCL as an auxiliary task, we observe consistent performance enhancements across 13 few-shot natural language understanding tasks, including benchmarks such as SST-2, TREC, MNLI, and real-world datasets such as Overruling, TC, and ADE. We also delve into SCL’s impact in label-imbalanced settings, introducing a novel approach called balanced batch in SCL (BBSCL). BBSCL employs balanced mini-batches, sampling the majority class proportionally to the minority class to stabilize SCL calculations. Our results indicate that SCL and BBSCL significantly boost the performance and robustness of soft prompt learning models, especially on datasets with intricate label spaces. Experimentally, DART + SCL and PTuning + SCL outperform their base models by an average of $2.1\%$ across the 13 tasks. Additionally, we find that SCL’s contribution is more substantial in scenarios with complex and less separable label spaces. Compared to large language models such as GPT-3.5 and OpenChat, our enhanced soft prompt learning models with SCL and BBSCL extensions exhibit superior performance in both balanced and imbalanced few-shot settings. This research not only improves the effectiveness of few-shot tuning techniques but also deepens our understanding of this area.

Rapid advancements in education assessment and natural language processing have fundamentally transformed the methodologies employed in multi-dimensional writing assessment. Neural network approaches have shown remarkable effectiveness, obviating the need for handcrafted feature engineering. However, the potential of pre-trained language models for constructing multi-dimensional automated essay scoring (AES) models still needs to be explored. The present study addresses critical challenges, including the resilience of AES models to subtle input perturbations leading to prediction inaccuracies and the elucidation of the intricate mechanisms within deep learning frameworks. We propose an integrated approach leveraging multi-task learning and the pre-trained model Bidirectional Encoder Representations from Transformers (BERT), augmented with adversarial training via the Fast Gradient Sign Method (FGSM), aiming to enhance scoring accuracy and stability in multi-dimensional AES tasks. Findings reveal that BERT-Integrated-MTL-FGSM significantly outperforms existing baselines. Integrating Explainable Artificial Intelligence (XAI) elucidates the complex evaluation processes within the deep learning black box, offering potentially diagnostic feedback for students and assisting educators in delivering individualized instruction guided by word-level attributions across diverse analytic rating scales. These insights enhance the robustness, generalizability, and interpretability of multi-dimensional AES models, thus paving an interdisciplinary pathway for multi-dimensional writing assessment and instruction.

We propose an unsupervised, corpus-independent method to extract keywords from a single text. It is based on the spatial distribution of words and the response of this distribution to a random permutation of words. Our method has three advantages over existing unsupervised methods (such as YAKE). First, it is significantly more effective at extracting keywords from long texts in terms of precision and recall. Second, it allows inference of two types of keywords: local and global. Third, it extracts basic topics from texts. Additionally, our method is language-independent and applies to short texts. The results are obtained via human annotators with previous knowledge of texts from our database of classical literary works. The agreement between annotators is moderate to substantial. Our results are supported via human-independent arguments based on the average length of extracted content words and on the average number of nouns in extracted words. We discuss relations of keywords with higher-order textual features and reveal a connection between keywords and chapter divisions.

Named Entity Recognition (NER) is a natural language processing task that traditionally relies on supervised learning and annotated data. Acquiring such data is often a challenge, particularly in specialized fields like medical, legal, and financial sectors. Those are commonly referred to as low-resource domains, which comprise long-tail entities, due to the scarcity of available data. To address this, data augmentation techniques are increasingly being employed to generate additional training instances from the original dataset. In this study, we evaluate the effectiveness of two prominent text augmentation techniques, Mention Replacement and Contextual Word Replacement, on two widely used NER models, Bi-LSTM + CRF and BERT. We conduct experiments on three datasets from low-resource domains, and we explore the impact of various combinations of training subset sizes and the number of augmented examples. We not only confirm that data augmentation is particularly beneficial for smaller datasets, but we also demonstrate that there is no universally optimal number of augmented examples, i.e., NER practitioners must experiment with different quantities in order to fine-tune their projects.