2.1 NER methods and architectures

NER emerged in the 1990s, notably through evaluation campaigns such as the MUC (Message Understanding Conferences)Footnote ^a and later the Conference on Computational Natural Language Learning (CoNLL) shared tasks (Tjong Kim Sang Reference Tjong Kim Sang2002; Tjong Kim Sang and De Meulder Reference Tjong Kim Sang and De Meulder2003), which established the ENAMEX category (PER, person; LOC, geographical locations; ORG, organisations; and MISC, miscellaneous entities) alongside other entity types such as the NUMEX numerical expressions (Grishman and Sundheim Reference Grishman and Sundheim1996). These initiatives also released benchmark datasets that helped standardise the task and enabled systematic comparison of approaches.

Early NER systems relied on a combination of supervised learning methods and rule-based techniques, often incorporating handcrafted features and external linguistic resources such as gazetteers and lexicons (Keraghel et al. Reference Keraghel, Morbieu and Nadif2024). More recently, there have been significant advancements across various types of NER architectures and approaches, including knowledge-based systems (Wang et al. Reference Wang, Shen, Cai, Wang, Wang, Xie, Huang, Lu, Zhuang and Jiang2022), unsupervised strategies (Nadeau, Turney, and Matwin Reference Nadeau, Turney and Matwin2006), feature-engineered supervised systems (Agerri and Rigau Reference Agerri and Rigau2016), and neural network models (Yan et al. Reference Yan, Deng, Li and Qiu2019). While each approach offers its own advantages, they also have distinct limitations. Knowledge-based NER systems rely on handcrafted rules, lexicons, and orthographic features so that do not require extensive annotated training data and typically leverage domain-specific resources to identify named entities. However, their dependency on manually crafted rules limits their adaptability to new domains. Unsupervised NER methods minimise the need for large labelled datasets by relying on patterns and statistical methods for data extraction. While effective at reducing annotation effort, these methods may be less accurate than supervised approaches. Feature-engineered supervised systems take a more holistic approach by employing machine learning algorithms with carefully crafted features. These systems integrate domain-specific knowledge, orthographic features, and other linguistic information. Before the rise of neural networks, feature-engineered systems were the popular method for the NER task.

As in other NLP tasks, NER has also benefited from advances in pre-trained word embeddings (Levy et al. Reference Levy, Goldberg and Dagan2015; Habibi et al. Reference Habibi, Weber, Neves, Wiegandt and Leser2017). Therefore, modern neural architectures for NER classify words using representations based on different linguistic units, such as words, characters, sub-word units, or combinations of these Wang et al. (Reference Wang, Jiang, Bach, Wang, Huang, Huang and Tu2021a). These models offer greater flexibility and adaptability compared to earlier approaches, making them the current standard in NER development.Footnote ^b Nevertheless, and despite its conceptual simplicity when compared to more elaborate architectures, fine-tuning transformer-based models remains a highly competitive approach, delivering strong performance that is often only marginally below the state of the art (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019; He, Gao, and Chen Reference He, Gao and Chen2021a).

More recently, LLMs have been explored for NER due to their ability to operate in zero-shot or few-shot settings and to support new entity definitions without extensive annotation (Ashok and Lipton Reference Ashok and Lipton2023; Keraghel et al. Reference Keraghel, Morbieu and Nadif2024). These properties make them attractive for low-resource scenarios, but they still underperform task-specific models in several cases (Wang et al. Reference Wang, Sun, Li, Ouyang, Wu, Zhang, Li, Wang and Guo2025). In addition, as generative rather than sequence-labelling models, LLMs may produce characteristic errors such as hallucinated entities, extraneous text, or incomplete coverage of the input (Ashok and Lipton Reference Ashok and Lipton2023; Merdjanovska, Aynetdinov, and Akbik Reference Merdjanovska, Aynetdinov and Akbik2024).

2.2 Datasets and domains

While several unsupervised approaches exist, the major improvements in NER model development and evaluation have been driven primarily by accessible annotated corpora (Yadav and Bethard Reference Yadav and Bethard2018). In this regard, recent advancements have led to the development of a wide variety of datasets covering multiple languages, domains, and entity types (Keraghel et al. Reference Keraghel, Morbieu and Nadif2024; Bade et al. Reference Bade, Kolesnikova and Oropeza2024). Among the most prominent datasets are those created for the mentioned CoNLL 2002 and 2003, which released datasets derived from newswire articles in Spanish, Dutch, English, and German. NER datasets have also been created for social media platforms, particularly Twitter/X, where the data present unique challenges compared to well-formed text. These challenges include inconsistent spelling, informal and abbreviated language, creative punctuation, and incomplete or ungrammatical sentences. One of the earliest and most influential datasets in this domain is the Twitter NER corpus introduced by Ritter et al. (Reference Ritter, Clark and Etzioni2011), which annotated ENAMEX entities in noisy microblog text. Subsequent efforts further expanded the scope of social media NER, notably the WNUT 2016 and WNUT 2017 shared task datasets on emerging and rare entities (Strauss et al. Reference Strauss, Toma, Ritter, de Marneffe and Xu2016; Derczynski et al. Reference Derczynski, Nichols, van Erp and Limsopatham2017), which include entity types that are less common in traditional newswire corpora. These datasets feature a broader range of entity types, including people, companies, facilities, bands, sports teams, movies, and TV shows, reflecting the dynamic, domain-specific nature of social media content. Together, these advancements underscore the growing diversity and specialisation of NER datasets designed to address the linguistic variability and domain shift inherent in social media platforms. The diversity of these datasets reflects the evolving nature of NER tasks and highlights the need for specialised approaches tailored to different textual contexts.

A particularly interesting case in the context of NER are BioNER tasks, which focus on the extraction of entities from biomedical texts, such as proteins, DNA, RNA, or cell attributes (Thiyagu et al. Reference Thiyagu, Manjula and Shruthi2019). In this context, the 2010 i2b2 NER task (Uzuner et al. Reference Uzuner, South, Shen and DuVall2010) targets clinical data, focusing on entities related to clinical problems, tests, and treatments. Other notable examples include the Drug NER shared task (Segura-Bedmar, Martínez, and Herrero-Zazo Reference Segura-Bedmar, Martínez and Herrero-Zazo2013), which concentrates on entities like drugs, brands, chemicals, and related terms. Crucially, BioNER typically distinguishes between flat NER, which considers single, non-overlapping entity spans, and nested NER, which allows entities to overlap or be embedded within others (Park et al. Reference Park, Son and Rho2024). While flat entities are relevant in biomedical scenarios, a substantial portion of the BioNER literature focuses on nested structures, which require specialised annotation schemes and modelling approaches. In this paper, however, we follow a flat annotation scheme, consistent with the standard paradigm for ENAMEX entity recognition.

2.3 Languages and cross-lingual NER

Most resources and studies for NER have been developed for English, covering domains such as medical texts, news, social media, and Wikipedia (Keraghel et al. Reference Keraghel, Morbieu and Nadif2024). For less-resourced languages, early approaches leveraged Wikipedia data to perform automatically annotation of multilingual data (Kim, Toutanova, and Yu Reference Kim, Toutanova and Yu2012). Recent studies have shown the benefits of implementing cross-lingual knowledge transfer methods (Feng et al. Reference Feng, Feng, Qin, Feng and Liu2018), while others simply rely on the fine-tuning of multilingual transformer models, obtaining competitive results even in zero-shot scenarios (Pires, Schlinger, and Garrette Reference Pires, Schlinger and Garrette2019). A widely adopted strategy for several downstream tasks involves exploiting annotated data from closely related languages, either directly via cross-lingual transfer or through adaptation methods that account for structural and lexical differences between the source and target languages (Garcia et al. Reference Garcia, Gómez-Rodríguez and Alonso2018; Torge et al. Reference Torge, Politov, Lehmann, Saffar and Tao2023).

In the case of Galician, Portuguese and Spanish are the most commonly employed source languages and have been shown to yield the best results in cross-lingual tasks (Sarymsakova, Sánchez-Rodríguez, and Garcia Reference Sarymsakova, Sánchez-Rodríguez and Garcia2024). Portuguese is particularly effective due to its strong linguistic proximity to Galician, while Spanish also proves beneficial given its long-standing coexistence with Galician within the same territory, which has resulted in significant linguistic influence (see Section 3.1 for details about Galician language).

Regarding NER studies for Portuguese, the earliest systematic evaluations were conducted within the HAREM shared tasks (Santos et al. Reference Santos, Seco, Cardoso and Vilela2006), which introduced a substantially finer-grained entity classification than the traditional ENAMEX scheme, making direct adaptation to standard NER settings not always straightforward. Subsequent evaluation campaigns, such as IberLEF 2019 (Collovini et al. Reference Collovini, Santos, Consoli, Terra, Vieira, Quaresma, Souza, Claro and Glauber2019), as well as independent studies, have largely followed approaches comparable to those adopted for other languages. A comprehensive overview is provided by Albuquerque et al. (Reference Albuquerque, Souza, Gomes, de C. Pinto, Filho, Costa, de M. Lopes, da Silva, de Carvalho and Oliveira2023), who survey manually and automatically annotated Portuguese NER corpora spanning multiple domains and venues and report competitive results across a range of models, including BiLSTM+Conditional Random Field (CRF) and transformer-based architectures, in line with contemporary NER benchmarks for high-resource languages. Reported performance varies considerably across datasets and domains, with F1-scores ranging approximately from 67 to 97 for BiLSTM+CRF models and from 75 to 95 for BERT+CRF approaches, reflecting differences in annotation schemes, corpus size, and domain specificity.

Regarding NER research for Spanish, the CoNLL-2002 shared task dataset remains one of the most widely adopted benchmarks, consisting of manually annotated newswire texts labelled using the BIO scheme with the ENAMEX categories (Carreras et al. Reference Carreras, Màrquez and Padró2002). In addition, the AnCora corpus provides a large-scale manually annotated resource that, although originally designed as a linguistic treebank, has been successfully exploited for NER (Taulé et al. Reference Taulé, Martí and Recasens2008). Recent Spanish NER systems predominantly rely on fine-tuning pre-trained transformer models on these datasets, achieving strong performance comparable to other high-resource languages (Gutiérrez-Fandiño et al. Reference Gutiérrez-Fandiño, Armengol-Estapé, Pàmies, Llop-Palao, Silveira-Ocampo, Carrino, Armentano-Oller, Rodriguez-Penagos, Gonzalez-Agirre and Villegas2022). Reported results vary depending on the dataset and evaluation setup, with fine-tuned Spanish-specific and multilingual transformer models, such as BETO (Cañete et al. Reference Cañete, Chaperon, Fuentes, Ho, Kang and Pérez2020) and XLM-RoBERTa (Conneau et al. Reference Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer and Stoyanov2019), typically reaching F1-scores in the 88–90 range on CoNLL-style benchmarks, confirming the effectiveness of transformer-based approaches for Spanish NER.

As for Galician, the object of our study, several studies have focused on developing NER resources. Garcia et al. (Reference Garcia, Gayo and González López2012) presented a rule-based (knowledge-based) NER model, achieving a precision of 69.9, a recall of 62.05, and an F1-score of 60.42. Building on these results, Agerri et al. (Reference Agerri, Gómez Guinovart, Rigau and Solla Portela2018) proposed a semi-supervised approach for NER, improving performance to 83.85 for precision, 84.54 for recall, and 84.19 for F1. More recently, Garcia (Reference Garcia2024) explored the effectiveness of syntax-based vector models across various NLP tasks, including NER.

The aforementioned study evaluates several training techniques for NER models in Galician, comparing architectures like BiLSTM-CNN-CRF initialised with static embeddings, and fine-tuned BERT models such as mBERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019) and XLM-RoBERTa. Among these, fine-tuned BERT models demonstrated the best performance. Specifically, the multilingual XLM-RoBERTa-large achieved an F1-score of 86.93, outperforming multilingual mBERT and other approaches. However, monolingual BERT models yielded even better results for Galician, achieving a precision of 87.69, a recall of 89.7, and an F1-score of 88.68. This NER system was trained and evaluated using these methods and a comprehensive, manually reviewed corpus, SLI NERC (Agerri et al. Reference Agerri, Gómez Guinovart, Rigau and Solla Portela2018), which is described in detail in Section 3. These and other previously mentioned findings emphasise (i) the superior performance of NER models trained on large, meticulously curated corpora and (ii) the superior suitability of monolingual BERT models for Galician NER tasks compared to recurrent networks, with fastText obtaining the highest results among static embeddings. Nonetheless, apart from the SLI NERC dataset, no other datasets have been used to evaluate these recently proposed NER models for Galician. Therefore, a comprehensive assessment is essential to understand how these models perform across different Galician corpora.

In summary, this section has presented an overview of state-of-the-art methods and resources for NER, encompassing both monolingual and cross-lingual settings and accounting for different annotation schemes, including flat, nested, continuous, and discontinuous entities. English serves as a reference point, with consistently strong results, while transformer-based architectures also constitute the dominant and most effective approach for Portuguese and Spanish. On CoNLL-like datasets with flat annotations, state-of-the-art performance is typically high, often exceeding 94% F1, but degrades notably across datasets, domains, and more complex annotation schemes. In contrast, results for Galician remain inconclusive, highlighting the need for further evaluation across multiple corpora.

In light of these findings, our contributions, as formulated in Section 1, aim to (i) expand the resources available for training and evaluating NER systems in Galician using flat annotation schemes, (ii) evaluate the use of various pre-trained BERT-based word embeddings for training Galician NER systems on five different datasets, and (iii) analyze and confirm the impact of incorporating cross-lingual data into the NER system training process.

3.1 Galician

Modern Galician (ISO code gl) originated from mediaeval Galician-Portuguese and retains many features in common with Portuguese, being currently spoken by approximately 2 million people, primarily in the Spanish Autonomous Region of Galicia (Monteagudo et al. Reference Monteagudo, Loredo and Vázquez-Grandío2024). Although its dialects have historically been regarded as part of the same language as Portuguese (Lindley Cintra and Cunha Reference Lindley Cintra and Cunha1984; Mira Mateus et al. Reference Mira Mateus, Brito, Duarte, Hub Faria, Frota, Matos, Oliveira, Vigário and Villalva2003), a combination of sociolinguistic, political, and identity factors, along with the substantial linguistic influence of Spanish, is used to support its recognition as a separate language (Fernández Rei Reference Fernández Rei1991).

Officially standardised since 1983, Galician now coexists with Spanish as an official language and employs a Spanish-based orthography (Samartim Reference Samartim2012). Therefore, in computational contexts, differences in spelling have posed challenges for resource sharing with Portuguese, yet several studies have successfully adapted Portuguese resources to develop NLP tools for Galician (Garcia et al. Reference Garcia, Gómez-Rodríguez and Alonso2018; Sarymsakova et al. Reference Sarymsakova, Sánchez-Rodríguez and Garcia2024). Although Galician remains a low-resource language (Mateo and Rodríguez Reference Mateo and Rodríguez2012; Sánchez et al. Reference Sánchez-Rodríguez, Sarymsakova, Castro and Garcia2024), initiatives such as the Proxecto Nós (de Dios-Flores et al. Reference de Dios-Flores, Magariños, Vladu, Ortega, Pichel, García, Gamallo, Fernández Rei, Bugarín-Diz, González González, Barro and Regueira2022) have contributed valuable datasets and models, further enriching the language’s representation in NLP and complementing its prior inclusion in widely used multilingual models, such as mBERT or XLM-RoBERTa.

3.2 Galician NER resources

We have used the SLI NERC corpus as our primary training and evaluation resource (Agerri et al. Reference Agerri, Gómez Guinovart, Rigau and Solla Portela2018). We rely on splits of 161,853 tokens for training and 40,482 tokens for testing, released by Garcia (Reference Garcia2024).Footnote ^c This corpus was encoded in CoNLL format, specifically created, annotated, and manually reviewed using Galician texts for NER. This dataset originates from the CTG Galician Technical Corpus,Footnote ^d which encompasses a wide range of domains, including GALEX (Galician legal texts), XIGA (Galician texts on computing and telecommunications), AUGA (Galician texts on ecology and environmental sciences), ACHEGA (Galician economy texts), SOGAL (Galician texts on sociology and social sciences), and MEDIGAL (Galician texts on medicine). It is important to note that the currently available Galician NER model released by Garcia (Reference Garcia2024) was trained and evaluated on this dataset. We aim to enhance this model’s performance by leveraging the same baseline resources.

Additionally, other annotated datasets, such as LREC (Garcia and Gamallo Reference Garcia and Gamallo2014) and CorNER (Garcia et al. Reference Garcia, Gayo and González López2012), were adapted and employed as evaluation resources in our study. The LREC dataset comprises journalistic and encyclopaedic (Wikipedia) texts in Galician, with approximately 35% journalistic and 65% Wikipedia content, totalling around 47,000 tokens. This dataset was primarily designed for person entity recognition tasks in journalistic and encyclopaedic documents. After processing the data, two linguists, both native Galician speakers, manually reviewed the NER annotation and added coreferential information to the corpus, ensuring linguistic accuracy and consistency.

The CorNER dataset (Garcia et al. Reference Garcia, Gayo and González López2012), consisting of around 20,000 tokens, is focused on named entities in Galician, including ENUMEX types such as dates, numerals, quantities, and proper nouns (ENAMEX). It was partially created through a random selection from the Gari-Coter project (Barcala et al. Reference Barcala, Domínguez, Gamallo, López, Moscoso, Rojo, Santalla and Sotelo2007), a textual corpus focused on economic press and supplements in daily newspapers. This corpus comprises 601 news articles totalling 235,175 tokens. Among others, the sources include newspapers and magazines such as ‘O Correo Galego’, ‘Galicia Confidencial’, and ‘Tempos Novos’. Beyond the Gari-Coter project, the dataset also incorporates encyclopaedic texts (including Wikipedia articles) and articles from the ‘Galicia Hoxe’ newspaper, spanning various sections. The corpus was preprocessed using FreeLing (Padró and Stanilovsky Reference Padró and Stanilovsky2012), and a subset of 20,000 tokens (containing nearly 1,000 proper names) was manually annotated to evaluate the system’s ability to identify ENAMEX entities.

All of the resources described, covering a wide range of domains, have enabled researchers to tackle various NLP tasks in Galician. Many of these tasks have already been successfully implemented and reported in the previously cited works, yielding promising results and positive performance metrics. However, only a few of these corpora have been used for NER evaluation in Galician to date. Specifically, the SLI NERC corpus has been used as an evaluation resource in Agerri et al. (Reference Agerri, Gómez Guinovart, Rigau and Solla Portela2018) and Garcia (Reference Garcia2024). The other resources, such as LREC, CorNER, and others, have not been used in current NER models for Galician. As such, our goal is to adapt and include these resources for training and evaluating NER models.

3.3 Galician PUD and TreeGal NER datasets

In the following subsections, we detail the procedure for processing, annotating, and the results obtained with two newly developed NER corpora in Galician. Both corpora are released in the Universal Dependencies (UD) format (de Marneffe et al. Reference de Marneffe, Manning, Nivre and Zeman2021). These datasets aim to contribute to the Universal Named Entity Recognition (UNER) framework (Alves et al. Reference Alves, Kuculo, Amaral, Thakkar and Tadic2020), addressing the multilingual challenges in NLP by providing high-quality NER datasets across multiple languages, all using unified guidelines.

3.3.3 Annotation of named entities

The TreeGal and Galician PUD corpora were annotated by two experts: a native speaker with a strong linguistic background and a postdoctoral researcher in Linguistics with advanced proficiency in Galician. To ensure consistency and alignment with the annotation guidelines, both annotators initially worked on a test dataset (400 sentences for TreeGal, then 200 for Galician PUD) from the original split of each corpus. To ensure quality in the annotation process, the initial annotated sets underwent a manual review by both annotators, who cross-examined each other’s results through exchanges and comparisons of annotations. The results of the inter-annotator agreement for this review task (Modality A) are presented in Table 1.

Table 1.

Results of inter-annotator agreement (IAA) for the Galician PUD and TreeGal NER labelling

Table 1. Long description

A table with three columns and three rows. The columns are labeled Modality, Dataset, and IAA. The rows present data for different modalities and datasets. Row 1: Modality A, Dataset TreeGal (400), IAA 0.959. Row 2: Modality A, Dataset PUD (200), IAA 0.969. Row 3: Modality B, Dataset PUD (800), IAA 0.958.

For the further annotation process, the training datasets (along with the Galician PUD development set) were divided and assigned to each expert for semi-automatic annotation. After predicting the NER tags for each token using the aforementioned BERT-based pre-trained model, the outputs for both TreeGal and Galician PUD were reviewed by the annotators (Modality B). During the annotation phase, regular follow-up meetings with additional language experts were held to resolve any ambiguities or questions that emerged during the review and curation phases. It is worth noting that, for the machine-assisted annotation review, the training dataset from Galician PUD was chosen and evaluated by both annotators, since the inter-annotator agreement for this corpus in the Modality A review task is higher than that of TreeGal.

This enabled us to enhance the quality of the manually annotated NER corpus for Galician at the final stage. By implementing this methodology, we mitigated the potential for mutual influence or bias among annotators during the review process conducted under Modality A. Consequently, we were able to evaluate whether the agreement for the machine-generated annotations remained consistent. As demonstrated by Modality B in Table 1, the results confirm that the annotation process was devoid of bias among annotators.

3.4 Results

Building on the previous works cited in Section 3 and leveraging our two newly annotated Galician NER corpora, we compiled five datasets for the NER task. The named entity statistics for each dataset are provided in Tables 2 and 3. The per-token distribution of annotations ranges from 1.82% to 7.02%, meaning that across datasets, between 1.82% and 7.02% of all tokens are annotated as named entities. The TreeGal dataset has the fewest annotated entities (488, or 1.82% of the corpus). It is worth mentioning that the distribution of the four named entity classes in TreeGal and Galician PUD is roughly similar, indicating a consistent annotation approach across both datasets, which are nearly identical in size.

The combined total of annotated entities across the five datasets is 11,039 (see Table 16 in the Appendix). The majority of these are classified as organisation entities, accounting for 31.25% (#3,450), followed by location entities at 28.05% (#3,096). Person entities make up 22.29% (#2,461), and miscellaneous entities constitute 18.41% (#2,032). The distribution of named entities across the datasets is relatively balanced, with a slight predominance of organisation entities. The proportional frequency of these entities is beneficial for the fine-tuning and improvement process of the existing Galician NER model, as it provides a solid representation of one of the most frequent entity types.

4.1 Data

To enhance the performance of the existing Galician NER model (Garcia Reference Garcia2024), we conducted a series of experiments using datasets comprising the SLI NERC, TreeGal, Galician PUD, CorNER, and LREC corpora. We used the SLI NERC splits provided by Garcia (Reference Garcia2024), which divided the SLI NERC test dataset into ‘dev’ and ‘test’ splits. Therefore, SLI NERC was divided into training (70%, 145,582 tokens), development (10%, 16,254 tokens), and test (20%, 40,405 tokens) sets, as shown in Table 3.

Table 2.

Distribution of entity classes in Galician PUD, TreeGal, CorNER, and LREC datasets

Table 2. Long description

The table presents the distribution of entity classes in four datasets: Galician PUD, TreeGal, CorNER, and LREC. It includes columns for the number, percentage, and average length of each entity class. The entity classes are PER, LOC, ORG, and MISC. Row 1: PER, 420, 33.71 percent, 1.44, 141, 28.89 percent, 1.88, 121, 13.10 percent, 1.83, 1184, 35.68 percent, 1.88. Row 2: LOC, 399, 32.02 percent, 1.44, 110, 22.54 percent, 1.15, 280, 30.30 percent, 1.14, 543, 16.37 percent, 1.42. Row 3: ORG, 276, 22.15 percent, 1.89, 150, 30.74 percent, 1.57, 438, 47.40 percent, 1.66, 805, 24.26 percent, 1.94. Row 4: MISC, 151, 12.12 percent, 2.28, 87, 17.83 percent, 2.08, 85, 9.20 percent, 2.08, 786, 23.69 percent, 2.74. Row 5: Overall, 1246, 5.30 percent, -, 488, 1.82 percent, -, 924, 4.41 percent, -, 3318, 7.02 percent, -. Row 6: Total tokens, 23509, 26817, 20932, 47254. The table shows the number and percentage of each entity class in each dataset, along with the average length of the entities.

Percentages for each entity are calculated over the total number of tags (see “Overall”), and overall percentages for annotated entities per dataset are computed over a total number os tokens. The average token length per entity type is shown in the “len” column.

Table 3.

Distribution of entity classes in SLI NERC dataset

Table 3. Long description

A table comparing the distribution of entity classes across three different datasets: Train, Dev, and Test. The table has 6 rows and 9 columns. Column headers are Tag, Train #, Train %, Train len, Dev #, Dev %, Dev len, Test #, Test %, and Test len. Row labels are PER, LOC, ORG, MISC, Overall, and Total tokens. Row 1: PER, 303, 0.21, 1.90, 79, 0.49, 1.92, 213, 0.53, 2.07. Row 2: LOC, 1247, 0.86, 1.44, 78, 0.48, 1.34, 439, 1.09, 1.49. Row 3: ORG, 1095, 0.75, 1.92, 130, 0.80, 1.77, 556, 1.38, 2.15. Row 4: MISC, 538, 0.37, 2.36, 25, 0.15, 2.36, 360, 0.89, 2.53. Row 5: Overall, 3183, 2.19, -, 312, 1.92, -, 1568, 3.88, -. Row 6: Total tokens, 145582, -, -, 16254, -, -, 40405, -, -. The table shows the number, percentage, and length of different entity classes in each dataset.

In line with our contributions outlined in Section 1—particularly the evaluation of a newly trained model with additional data integration—we incorporated the TreeGal training split into the training dataset. The TreeGal training split comprises 60% of TreeGal, amounting to 16,705 tokens. As Table 4 shows, we obtained the final splits to perform the evaluation. Consequently, the final training set combined the SLI NERC and TreeGal datasets, resulting in a total of 162,287 tokens. The development set, exclusively from the SLI NERC dataset, remained at 16,254 tokens. The test splits consist of the following datasets: the Galician PUD corpus (23,509 tokens), the TreeGal test set (10,112 tokens), the CorNER corpus (20,932 tokens), the LREC corpus (47,254 tokens), and the SLI NERC corpus (40,405 tokens).

Table 4.

Distribution of entity classes in final training, development, and test dataset

Table 4. Long description

A table with four rows and three columns. The columns are labeled Set, Corpus, and Tokens. The rows are labeled Train, Dev, and Test. The Train set includes SLI NERC (train) with 145,582 tokens and TreeGal (train) with 16,705 tokens. The Dev set includes SLI NERC (dev) with 16,254 tokens. The Test set includes Galician PUD with 23,509 tokens, TreeGal (test) with 10,112 tokens, CorNER with 20,932 tokens, LREC with 47,254 tokens, and SLI NERC (test) with 40,405 tokens.

4.2 Architecture selection

To train the NER models, we used the standard fine-tuning approach of pre-trained transformer models. To do so, we took advantage of the transformers libraryFootnote ^g provided by HuggingFace (Wolf et al. Reference Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac, Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu, Le Scao, Gugger, Drame, Lhoest and Rush2020). For each model, we explored the different batch sizes {8, 16, 32} and learning rates { $1e^{-5}, 2e^{-5}, 3e^{-5}, 4e^{-5}, 5e^{-5}$ } and selected the best checkpoint by its F1-score in the development set after five epochs. To assess the performance across datasets, and taking into account their different sizes, we compute both micro and macro average F1-scores.

We compare the performance of the following monolingual and multilingual pre-trained models (see Table 17 in Appendix A.2 for the details of each model):

Monolingual . We compare both “small” and “base” variants of Bertinho (Vilares, Garcia, and Gómez-Rodríguez Reference Vilares, Garcia and Gómez-Rodríguez2021) and BERT-gl Garcia (Reference Garcia2021) models. Additionally, we evaluate the new DeBERTa-xsmall model, described in Section 4.3.

Multilingual . Regarding multilingual models, we tested the official multilingual BERT base (mBERT, Devlin et al. (Reference Devlin, Chang, Lee and Toutanova2019)), the ‘base’ and ‘large’ versions of XLM-RoBERTa (Conneau et al. Reference Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer and Stoyanov2019), and the multilingual DeBERTa_v3 model (mDeBERTa_v3-base, He et al. (Reference He, Gao and Chen2021a)).

Baseline . As a baseline, we use the best model provided by Garcia (Reference Garcia2024), which consists of a BERT-base-gl fine-tuned with the CTG-NER corpus and available at the HuggingFace Hub.Footnote ^h Therefore, this is a strong baseline with state-of-the-art performance on NER for Galician, which only differs in having used slightly less training data than the models we are fine-tuning in this study.

4.3 A new Galician DeBERTav3 xsmall

Available monolingual encoder models for Galician (Vilares et al. Reference Vilares, Garcia and Gómez-Rodríguez2021; Garcia Reference Garcia2021) take advantage of the BERT architecture (without the next sentence prediction training objective) and appear at ‘small’ and ‘base’ sizes, having 6 and 12 hidden layers, about 44 and 86 million backbone parameters, and occupying approximately 300 and 600 megabytes, respectively. The standard BERT architecture was improved by DeBERTa models (He et al. Reference He, Liu, Gao and Chen2021b), which use disentangled attention (differentiating content and position embeddings) instead of self-attention and, in its version 3, substitutes masked language modelling with replaced token detection (He et al. Reference He, Gao and Chen2021a). This sample-efficient pre-training task allows smaller models to obtain better results than larger models from previous architectures.

To verify how well this architecture performs on NER for Galician and to compare it with larger available models, we trained a new extra-small (xsmall) DeBERTa_v3 model for Galician. To do so, we did a continual pre-training of the original English DeBERTA-xsmallFootnote ⁱ on CorpusNÓS, an open Galician corpus composed of 2 billion words (de Dios-Flores et al. Reference de Dios-Flores, Suárez, Pérez, Outeiriño, Garcia and Gamallo2024). We trained the model using the original tokeniser on a server with two 80 GB NVIDIA Hopper H100 GPUs for one epoch over the corpus. The final model, here dubbed deberta-gl-xsmall, has 22 M backbone parameters, occupies 142 megabytes, and is freely available on HuggingFace.Footnote ^j

4.4 Results

We present the results on the five evaluation datasets, starting with the baseline and then the monolingual and multilingual models.

Baseline . As shown in Table 5, the baseline model achieves a micro average accuracy of 84.70% and macro average of 85.24%, with its highest prediction score (90.13%) observed on the CorNER test dataset, and the lowest in the Galician PUD corpus (79.84%). The F1-scores reported in Table 5 are averaged across datasets. This difference of more than 10 points across datasets reinforces the need for improved cross-domain NER systems for Galician.

Table 5.

Accuracy of the baseline model trained with custom splits (70/10/20%) of SLI NERC

Table 5. Long description

A table comparing the performance of a baseline model across different test datasets. The table has five rows and six columns. The columns are labeled Model, Test dataset, F1, P, R, Micro F1, and Macro F1. The row labels are Baseline, SLI NERC test, Treegal test, PUD, CorNER, and LREC. Row 1: Model, Did; Test dataset, D N A; F1, 88.68; P, 87.69; R, 89.69; Micro F1, 84.70; Macro F1, 85.24. Row 2: Model, Did; Test dataset, Treegal test; F1, 84.36; P, 85.06; R, 83.67; Micro F1, D N A; Macro F1, D N A. Row 3: Model, Did; Test dataset, PUD; F1, 79.84; P, 79.43; R, 80.26; Micro F1, D N A; Macro F1, D N A. Row 4: Model, Did; Test dataset, CorNER; F1, 90.13; P, 90.87; R, 89.39; Micro F1, D N A; Macro F1, D N A. Row 5: Model, Did; Test dataset, LREC; F1, 83.20; P, 81.88; R, 84.57; Micro F1, D N A; Macro F1, D N A.

4.4.1 Monolingual models

We started by searching for the best hyperparameters of the BERT-base-gl model, whose results are plotted in Figure 1. A comparison of both results (Table 5 and Figure 1) reveals an improved performance (84.70% vs. 85.07% in micro average F1). The red bar in Figure 1 highlights the best performance, obtained with a learning rate of 4e-05 and a batch size of 16. This preliminary finding, combined with the studies discussed in Section 2, supports the further experiments comparing additional models and data to enhance accuracy for the NER task in Galician.

Figure 1.

Results for NER Galician model trained with BERT-base-gl embeddings and diverse hyperparameters.

A summary of the results for the best checkpoint of each monolingual model is shown in Table 6. These results show that BERT-base-gl outperformed all other models at both micro and macro F1 levels. This model, trained with 5 epochs, a batch size of 16, and a learning rate of $4e^{-5}$ , achieved the best results among the monolingual models and delivered the highest F1-scores across all five test datasets. Among these, the best accuracy was observed in the SLI and CorNER evaluation datasets, and the lowest metrics were observed in the PUD and LREC corpora.

Table 6.

The results on NER task trained with BERT-based monolingual models and evaluated on five different datasets

Table 6. Long description

The table presents the performance metrics of various BERT-based monolingual models evaluated on five different datasets. It includes columns for the model name, test dataset, F1 score, precision (P), recall (R), MicroF1, MacroF1, batch size (BS), and learning rate (LR). The models listed are BERT-base-gl, BERT small gl, DeBERTa xsmall gl, Bertinho base, and Bertinho small. Each model’s performance is detailed across the SLI NERC test, Treegal test, PUD, CorNER, and LREC datasets. Notable trends include BERT-base-gl outperforming other models in both micro and macro F1 levels, with the highest F1-scores observed in the SLI and CorNER datasets and the lowest in the PUD and LREC corpora.

BS—batch size; LR—learning rate. All the provided models were trained using a configuration of 5 epochs.

The performance of the other available models follows relatively similar trends as those reported in other studies, here with better results for both Bertinho models than for BERT-small-gl. Regarding the new DeBERTa-xsmall-gl, it performs better than the ‘small’ models, despite having half their backbone parameters (22M vs. 44M) and size.

4.4.2 Multilingual models

For the multilingual models, the results in Table 7 indicate that the mDeBERTa_v3-base NER model outperforms the monolingual BERT-base-gl model (85.47 vs. 85.07 µ F1). Trained with 5 epochs, a batch size of 16, and a learning rate of $4e^{-5}$ , the model achieved the highest F1-scores (micro F1 of 85.47 and macro F1 of 86.34) across all five test datasets, outperforming other multilingual models: 87.78% for the SLI test set, 85.28% for TreeGal, 85.01% for PUD, 90.40% for CorNER, and 83.22% for LREC. Among these, the highest accuracy was observed in the SLI and CorNER evaluation datasets. At the same time, the lowest scores were observed for the PUD and LREC corpora, consistent with the results from the monolingual BERT-base-gl model.

Table 7.

The results on NER task trained with BERT-based multilingual models and evaluated on five different datasets

Table 7. Long description

The table presents the performance metrics of various BERT-based multilingual models evaluated on five different datasets. It includes columns for the model name, test dataset, F1 score, precision (P), recall (R), micro F1, macro F1, batch size (BS), and learning rate (LR). The models compared are mDeBERTa_v3-base, mBERT, XLM-RoBERTa-base, and XLM-RoBERTa large. Each model’s performance is detailed across the SLI NERC test, Treegal test, PUD, CorNER, and LREC datasets. Notable trends include the mDeBERTa_v3-base model achieving the highest F1 scores across all datasets, particularly excelling in the SLI and CorNER datasets. The lowest scores are observed for the PUD and LREC corpora across all models.

BS—batch size; LR—learning rate. All the provided models were trained using a configuration of 5 epochs.

Among the other multilingual models, XLM-RoBERTa-large achieved competitive performance, matching the best monolingual model. At the same time, both mBERT and XLM-RoBERTa-base performed similarly to Bertinho-base, as also noted by previous research (Vilares et al. Reference Vilares, Garcia and Gómez-Rodríguez2021).

In conclusion, regarding transformer architectures, our initial results on monolingual corpora differ slightly from trends reported in other evaluations across different tasks, for example, de Dios-Flores and Garcia (Reference de Dios-Flores and Garcia2022). While monolingual models trained on smaller, language-specific datasets perform well in our study, multilingual models trained on larger datasets seem to outperform them providing better accuracy metrics. These results suggest that multilingual models may be more effective for training NER models in contrast to other downstream tasks and intrinsic evaluations, as NER is less dependent on linguistic structure than, for example, PoS-tagging or targeted syntactic evaluation. Crucially, for both best monolingual and multilingual models, and despite using the same development set, the variation across datasets was lower than that of the baseline model, making them more suitable for generic NER labelling in Galician.

4.5 Augmentation of training data

To achieve our contributions as formulated in Section 1, we aim to incorporate supplementary data from closely related linguistic varieties to evaluate their effect on the performance of Galician Named Entity Recognition models. For this purpose, we have incorporated the Spanish AnCora dataset (Taulé et al. Reference Taulé, Martí and Recasens2008) converted to IOB format from its UD version,Footnote ^k and the Portuguese Bosque datasetFootnote ^l as released by Pires (Reference Pires2017).Footnote ^m

Using data from languages related to Galician allows us to test the model’s sensitivity to subtle linguistic contrasts, its capacity to generalise across related varieties, and its robustness in handling language continua rather than clearly separated systems.

These datasets are comparable in size and annotation standards to SLI NERC, which constitute most of our training data. Specifically, the Portuguese NER dataset contains 11,832 annotated entities, while the Spanish Ancora dataset contains 34,427 NER-labelled tokens. However, the Spanish AnCora corpus is significantly larger, with 589,548 tokens. Table 8 provides an overview of the named entity distribution across these datasets.

Table 8.

Distribution of entity classes in Spanish AnCora, Portuguese Bosque, and PUD datasets

Table 8. Long description

A table comparing the distribution of entity classes in Spanish AnCora and Portuguese NER datasets. The table has two main sections: AnCora and Portuguese NER, each with columns for Tokens, Tag, #, and %. AnCora section: 589,548 tokens; PER tag: 8,450 (#), 1.4 (%); LOC tag: 5,408 (#), 1 (%); ORG tag: 11,522 (#), 2 (%); MISC tag: 9,046 (#), 1.5 (%); Total: 34,426 (#), 6 (%). Portuguese NER section: 240,215 tokens; PER tag: 4,230 (#), 2 (%); LOC tag: 2,768 (#), 1 (%); ORG tag: 3,396 (#), 1.5 (%); MISC tag: 1,438 (#), 0.5 (%); Total: 11,832 (#), 5 (%).

To explore the impact of the data augmentation, we have selected the best monolingual and multilingual models evaluated in the previous section. In each case, we performed three experiments: (1) fine-tuning of the model with the Galician and Spanish data; (2) fine-tuning using Galician and Portuguese data; and (3) fine-tuning with the training data of the three varieties, Galician, Spanish, and Portuguese (Mixed).

The results of cross-lingual data integration are presented in Tables 9 and 10, for the monolingual and multilingual models, respectively. In the first cross-lingual training split, including Spanish data, the performance of the monolingual BERT-base-gl model decreases (84.65 µ F1, and 85.18 macro F1 vs. the previous 85.07/85.88). For the multilingual model, results also decline (from 85.47 to 83.91 for micro and from 86.34 to 84.27 for macro), indicating that including Spanish data negatively impacts the model’s performance.

Table 9.

Results for cross-lingual training data augmentation using BERT-base-gl embedding model, compiling Spanish and Portuguese NER corpus

Table 9. Long description

The table presents results for cross-lingual training data augmentation using the BERT-base-gl embedding model, compiling Spanish and Portuguese NER corpus. It has three main sections: AnCora, Portuguese NER, and Mixed, each with different test datasets and performance metrics. The columns include Test dataset, F1, P, R, MicroF1, MacroF1, BS, and LR. Row 1: AnCora, SLI NERC test, 87.48, 86.50, 88.49, 84.65, 85.18, 8, 2e-05. Row 2: AnCora, Treegal test, 85.54, 85.37, 85.71. Row 3: AnCora, PUD, 85.11, 84.84, 85.39. Row 4: AnCora, CorNER, 83.04, 81.51, 84.63. Row 5: AnCora, LREC, 84.65, 83.63, 85.70. Row 6: Portuguese NER, SLI NERC test, 88.90, 87.88, 89.95, 86.30, 86.72, 8, 3e-05. Row 7: Portuguese NER, Treegal test, 84.81, 84.73, 84.90. Row 8: Portuguese NER, PUD, 82.07, 81.87, 82.26. Row 9: Portuguese NER, CorNER, 92.71, 92.56, 92.86. Row 10: Portuguese NER, LREC, 85.09, 84.02, 86.20. Row 11: Mixed, SLI NERC test, 88.81, 87.76, 89.89, 86.30, 87.03, 32, 4e-05. Row 12: Mixed, Treegal test, 86.38, 86.03, 86.73. Row 13: Mixed, PUD, 83.82, 83.45, 84.19. Row 14: Mixed, CorNER, 91.51, 91.46, 91.56. Row 15: Mixed, LREC, 84.60, 83.29, 85.96.

BS—batch size; LR—learning rate. All the provided models were trained using a configuration of 5 epochs.

Table 10.

Results for cross-lingual training data augmentation using mDeBERTa_v3-base embedding model, compiling Spanish and Portuguese NER

Table 10. Long description

The table presents performance metrics for cross-lingual training data augmentation using the mDeBERTa_v3-base embedding model, focusing on Spanish and Portuguese Named Entity Recognition (NER). It has three main sections: AnCora, Portuguese NER, and Mixed, each with different test datasets and metrics. The columns include Test dataset, F1, P, R, MicroF1, MacroF1, BS, and LR. Row 1: AnCora, SLI NERC test, 86.88, 85.94, 87.85, 83.91, 84.27, 16, 5e-05. Row 2: AnCora, Treegal test, 83.65, 83.23, 84.08. Row 3: AnCora, PUD, 84.26, 83.24, 85.31. Row 4: AnCora, CorNER, 84.19, 83.65, 84.74. Row 5: AnCora, LREC, 82.34, 81.03, 83.69. Row 6: Portuguese NER, SLI NERC test, 87.07, 85.75, 88.42, 85.96, 86.32, 8, 2e-05. Row 7: Portuguese NER, Treegal test, 84.40, 84.32, 84.49. Row 8: Portuguese NER, PUD, 81.87, 81.31, 82.42. Row 9: Portuguese NER, CorNER, 93.00, 92.51, 93.51. Row 10: Portuguese NER, LREC, 85.24, 84.27, 86.23. Row 11: Mixed, SLI NERC test, 86.77, 85.13, 88.49, 85.35, 85.70, 8, 3e-05. Row 12: Mixed, Treegal test, 84.72, 85.15, 84.29. Row 13: Mixed, PUD, 84.08, 84.04, 84.11. Row 14: Mixed, CorNER, 88.60, 88.46, 88.74. Row 15: Mixed, LREC, 84.34, 83.34, 85.35.

BS—batch size; LR—learning rate. All the provided models were trained using a configuration of 5 epochs.

In the case of integrating a more closely related variety (Portuguese), the performance of both models, especially the monolingual one, improved. For BERT-base-gl, it achieved a micro F1-score of 86.30 and a macro F1-score of 86.72, while the mDeBERTa_v3-base model demonstrated improved performance in the micro F1 scenario (85.96) and a small drop of 0.02 in the macro F1-score. The multilingual model improves only when trained on the Portuguese NER corpus, whereas training on the cross-lingual dataset with Spanish AnCora slightly decreases performance. A deeper investigation into this case is left for future work.

Finally, in the Mixed scenario (with a total corpus of 991,436 tokens), Table 9 exhibits that the macro F1-score for BERT-base-gl increased to 87.03, while keeping the same values as for the previous experiment (86.30) In contrast, the performance of the mDeBERTa_v3-base model slightly declined, obtaining a micro F1-score of 85.35 and a macro F1-score of 85.70 (Table 10). Notably, across the evaluation datasets, both monolingual and multilingual models trained on the Mixed dataset achieved the highest accuracy on the SLI and CorNER datasets. At the same time, the lowest scores were recorded for the PUD and LREC corpora. This observation was previously noted in Section 4.4.

These final experiments show that labelled Portuguese data is beneficial for training NER models in Galician, while incorporating Spanish data has, in general, adverse effects (or little macro F1 in the Mixed scenario for the monolingual model). Furthermore, it is worth noting that when augmenting the training data with cross-lingual examples, improvements are higher in the monolingual model than in the multilingual one, with BERT-base-gl achieving the best macro and micro F1-scores.

4.6 Comparison with LLMs

Although not the primary objective of our study, we include a comparison between few-shot methods using LLMs and our best supervised model, BERT-base-gl fine-tuned with the Mixed cross-lingual data. It is worth noting that generative LLMs are known to present challenges when applied to sequence labelling tasks, as their outputs may exhibit structural mismatches with gold-standard annotations, including token misalignment, missing or duplicated tags, and invalid label sequences (Zhou et al. Reference Zhou, Zhang, Gu, Chen and Poon2024).Footnote ⁿ

We compared four state-of-the-art open-weight instruction-tuned models with coverage for Galician: (i) Qwen2.5-7B-Instruct (Qwen Team 2024),Footnote ^o selected as the best-performing LLM for Galician in the IberBench leaderboard (González et al. Reference González, Borrego Obrador, lvaro Romo Herrero, Sarvazyan, Chinea-Ríos, Basile and Franco-Salvador2026)Footnote ^p ; (ii) Qwen3-4B-Instruct-2507Footnote ^q and (iii) Qwen3-8BFootnote ^r (Qwen Team 2025), included as successor models; and (iv) Llama-3.2-3B-InstructFootnote ^s as a representative of the Llama 3.2 family (Grattafiori et al. Reference Grattafiori, Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Vaughan, Yang, Fan, Goyal, Hartshorn, Yang, Mitra, Sravankumar, Korenev, Hinsvark, Rao, Zhang, Rodriguez, Gregerson, Spataru, Roziere, Biron, Tang, Chern, Caucheteux, Nayak, Bi, Marra, McConnell, Keller, Touret, Wu, Wong, Ferrer, Nikolaidis, Allonsius, Song, Pintz, Livshits, Wyatt, Esiobu, Choudhary, Mahajan, Garcia-Olano, Perino, Hupkes, Lakomkin, AlBadawy, Lobanova, Dinan, Smith, Radenovic, Guzmán, Zhang, Synnaeve, Lee, Anderson, Thattai, Nail, Mialon, Pang, Cucurell, Nguyen, Korevaar, Xu, Touvron, Zarov, Ibarra, Kloumann, Misra, Evtimov, Zhang, Copet, Lee, Geffert, Vranes, Park, Mahadeokar, Shah, van der Linde, Billock, Hong, Lee, Fu, Chi, Huang, Liu, Wang, Yu, Bitton, Spisak, Park, Rocca, Johnstun, Saxe, Jia, Alwala, Prasad, Upasani, Plawiak, Li, Heafield, Stone, El-Arini, Iyer, Malik, Chiu, Bhalla, Lakhotia, Rantala-Yeary, van der Maaten, Chen, Tan, Jenkins, Martin, Madaan, Malo, Blecher, Landzaat, de Oliveira, Muzzi, Pasupuleti, Singh, Paluri, Kardas, Tsimpoukelli, Oldham, Rita, Pavlova, Kambadur, Lewis, Si, Singh, Hassan, Goyal, Torabi, Bashlykov, Bogoychev, Chatterji, Zhang, Duchenne, Çelebi, Alrassy, Zhang, Li, Vasic, Weng, Bhargava, Dubal, Krishnan, Koura, Xu, He, Dong, Srinivasan, Ganapathy, Calderer, Cabral, Stojnic, Raileanu, Maheswari, Girdhar, Patel, Sauvestre, Polidoro, Sumbaly, Taylor, Silva, Hou, Wang, Hosseini, Chennabasappa, Singh, Bell, Kim, Edunov and Ma2024). Regarding the prompting strategy, we evaluated both zero-shot and few-shot ( $k=3$ ) settings and report the latter, as it consistently yielded better results (see Section A.3 in the Appendix for further details).

Due to their generative nature, LLMs can produce outputs that are incompatible with the gold standard, so an evaluation strategy must be defined to maximise the alignment between both annotations. Following recent studies on sequence labelling (Paolini et al. Reference Paolini, Athiwaratkun, Krone, Ma, Achille, Anubhai, dos Santos, Xiang and Soatto2021) and LLM-based NER (Santos et al. Reference Santos, Cameron, Olival, Farrica and Vieira2024), we first apply the Needleman–Wunsch global sequence alignment algorithm (Needleman and Wunsch Reference Needleman and Wunsch1970) between each LLM output and the corresponding test set. Then, we project the predicted tags onto the gold tokens as follows: gold tokens aligned to gaps are assigned the tag O, LLM output tags are repeated when multiple gold tokens align to a single LLM token, and tokens inserted by the LLM that do not align with any gold token are discarded. Finally, the performance at the entity level is computed using a fuzzy span-matching criterion, in which any overlap between predicted and gold entity spans is considered a match (Zhou et al. Reference Zhou, Zhang, Gu, Chen and Poon2024).

Table 11.

LLM performance (F1-score) across NER datasets under fuzzy span evaluation

Table 11. Long description

The table presents the F1-scores of different language models (Qwen2.5-7B, Qwen3-4B, Qwen3-8B, Llama-3.2-3B, BERT-gl) across several NER datasets (CORGA, LREC, PUD, SLI, TreeGal) under fuzzy span evaluation. The table has 6 rows and 6 columns, including the dataset names and the performance scores for each model. Row 1: Dataset, Qwen2.5-7B, Qwen3-4B, Qwen3-8B, Llama-3.2-3B, BERT-gl. Row 2: CORGA, 62.3, 60.0, 65.2, 18.0, 92.3. Row 3: LREC, 63.7, 0.620, 62.4, 16.1, 88.1. Row 4: PUD, 60.6, 58.7, 60.8, 19.2, 88.3. Row 5: SLI, 58.6, 55.7, 53.1, 15.3, 93.2. Row 6: TreeGal, 58.0, 58.7, 55.0, 16.3, 91.8. Row 7: Macro avg, 60.6, 59.0, 59.3, 17.0, 90.7. The table shows that BERT-gl consistently outperforms the other models across all datasets.

Bold values mark the highest score achieved by the evaluated LLMs on each of the test datasets.

Table 11 reports the F1-scores of the evaluated LLMs (see Table 18 in Section A.3 of the Appendix for the complete results). As shown, the performance of the LLMs is substantially lower than that of our best model, the baseline, and the previously proposed supervised models. These findings are consistent with recent work showing that, even in controlled settings, LLMs tend to produce noisy NER annotations (Merdjanovska et al. Reference Merdjanovska, Aynetdinov and Akbik2024). However, as mentioned above, it should be noted that one of the main factors underlying this low performance is the nature of the models’ outputs and their varying degrees of adherence to the provided instructions. In particular, the Llama model exhibited especially poor performance, largely due to frequent deviations from the input sentences, such as generating tags for empty tokens and assigning a large number of named entity labels to common words (e.g., determiners, prepositions, and verbs). More extensive prompt engineering could potentially mitigate these issues. The reported values for Llama are therefore included for completeness and should not be interpreted as a fair or direct comparison.

4.7 Significance tests

To assess whether our best-performing models perform significantly better than the baseline, we have carried out a series of statistical significance tests.

First, we compared the overall performance of BERT-base-gl and mDeBERTa_v3-base, both trained on monolingual data, using the Friedman Aligned Ranks test. This choice was made because the data were not normally distributed and did not meet the homoscedasticity assumption. Following the approach of Rodriguez-Fernandez et al. (Reference Rodriguez-Fernandez, Canosa, Mucientes and Bugarin2015), we obtained significant results with $p=0.028$ , which is below the significance threshold of $\alpha =0.05$ .

Next, we examined whether our best-performing model (BERT-base-gl trained with mixed data) significantly outperformed the baseline across datasets. To do so, we applied the Wilcoxon Signed-Rank Test (Wilcoxon Reference Wilcoxon1992) using F1-scores from the five datasets. The results confirmed statistically significant performance gains, with a consistent improvement validated by the Wilcoxon Signed-Rank Test ( $z=-2.023$ , $p\lt 0.05$ ).

Overall, the new models trained in this study demonstrate statistically significant improvements over the previous state-of-the-art NER model for Galician.

5.1 Performance across entity types

In order to obtain qualitative data on the performance of the best models, we applied the approach proposed by Tjong Kim Sang and Buchholz (Reference Tjong Kim Sang and Buchholz2000) to calculate the F1-score for each named entity type, with the results compiled in Table 12. As observed, the highest accuracy among the annotated named entities was achieved for person entities, with scores of 95.34 for the baseline model, 95.45 for BERT-base-gl, 95.75 for mDeBERTa_v3-base, and 96.07 for BERT-base-gl trained with cross-lingual data integration. In contrast, the lowest metrics were recorded for miscellaneous entities, with scores of 72.95 for the baseline model, 72.09 for BERT-base-gl, 73.82 for mDeBERTa_v3-base, and 74.82 for BERT-base-gl trained with cross-lingual data integration.

Table 12.

Qualitative analysis micro average results for Galician NER models trained with BERT-based embeddings and evaluated on five Galician datasets. Values in bold are the best F1 results for each category and for each metric (bottom row)

Table 12. Long description

A table comparing the performance of Galician NER models trained with BERT-based embeddings. The table has five rows and four columns. Column headers are Baseline, BERT-base-gl, mDeBERTa_v3-base, and BERT-base-gl (cross-lingual). Row labels are PER, LOC, ORG, MISC, and Total. Row 1: PER, 95.34, 95.30, 95.39, 95.45, 94.62, 96.30, 95.75, 95.12, 96.40, 96.07, 94.60, 97.60. Row 2: LOC, 86.11, 83.06, 89.39, 87.25, 85.37, 89.22, 86.58, 84.95, 88.26, 86.90, 86.01, 87.81. Row 3: ORG, 81.41, 79.66, 83.24, 82.36, 80.17, 84.67, 82.55, 79.01, 86.43, 84.01, 81.29, 86.92. Row 4: MISC, 72.95, 75.64, 70.45, 72.09, 72.51, 71.67, 73.82, 76.34, 71.47, 74.82, 77.45, 72.35. Row 5: Total, 84.70, 84.02, 85.54, 85.07, 83.91, 86.41, 85.47, 84.32, 86.69, 86.30, 85.42, 87.23.

The numbers in Table 12 show consistent improvements across entity types for both monolingual and multilingual models trained with Galician and cross-lingual data. As shown in the final row of Table 12, the overall F1-score for named entities reached 86.30% on the mixed evaluation dataset, which compiles five Galician corpora, using the BERT-base-gl monolingual model and an augmented cross-lingual training dataset. This represents an improvement over the initial 84.70% F1-score achieved by the baseline model, and our BERT-base-gl NER model outperforms the previous state of the art by nearly two points across all named entity classes.

As for the span-wise evaluation of the BERT-base-gl model evaluated on the Mixed scenario, Table 13 reveals substantial variation in performance across entity types. The model achieves the highest accuracy for person entities, with an F1-score of 96.2, and a minimal error rate of 7.74%. Locations show similarly robust performance (87.4 F1). In contrast, organisational and miscellaneous entities pose greater challenges, with F1-scores of 84.9 and 75.9, respectively, and error rates of 30.88% and 45.81%. The lower recall for MISC entities indicates frequent omissions, while the elevated false-positive rates for ORG suggest occasional over-prediction or boundary errors. Overall, the model attains a micro-averaged F1-score of 87%, corresponding to an aggregate error rate of 26.11%, underscoring strong generalisation capabilities in cross-lingual settings while highlighting the need for targeted improvements in specific categories, such as MISC and ORG.

Table 13.

Span-wise metrics per entity type from Mixed dataset for the best-performing BERT-base-gl (cross-lingual) model

Table 13. Long description

A table comparing performance metrics for different entity types in a model. The table has 5 rows and 9 columns. Column headers are Entity, TP, FP, FN, Errors, Error%, Precision, Recall, and F1. Row labels are PER, ORG, MISC, LOC, and Overall. Row 1: PER, 2026, 107, 54, 161, 7.74, 94.98, 97.40, 96.18. Row 2: ORG, 1933, 395, 292, 687, 30.88, 83.03, 86.88, 84.91. Row 3: MISC, 1062, 266, 407, 673, 45.81, 79.97, 72.29, 75.94. Row 4: LOC, 1553, 232, 217, 449, 25.37, 87.00, 87.74, 87.37. Row 5: Overall, 6574, 1000, 970, 1970, 26.11, 86.80, 87.14, 86.97.

5.2 Performance across datasets

5.2.1 Best-performing model outcomes

Across the evaluation datasets, a consistent trend is observed: the SLI and CorNER datasets achieve the highest accuracy, while the PUD and LREC corpora consistently yield the lowest results. This pattern was previously noted in Sections 4.4 and 4.5, during the experiments with monolingual and multilingual models and cross-lingual data augmentation.

Building on these findings, it becomes crucial to conduct a qualitative analysis of NER task performance for both the BERT-base monolingual model and the mDeBERTa multilingual model. This analysis should focus on their performance within the PUD and LREC datasets, which consistently exhibited the lowest accuracy scores. Additionally, it is essential to extend this evaluation to the Combined Galician NER datasets, which aggregate all five Galician corpora, to understand better how these models perform when trained on a more diverse and comprehensive dataset.

In order to address the disparity in F1-scores observed between the PUD and LREC datasets, a qualitative analysis was conducted using the BERT-base-gl model’s predictions, which achieved higher performance (86.30% for micro and 86.72% for macro average F1) and was trained on cross-lingual data. The results are presented in Table 14.

Table 14.

NE labels errors for BERT-base-gl predictions evaluated on five Galician datasets and trained with cross-lingual data

Table 14. Long description

A table comparing error rates of NE labels for BERT-base-gl predictions across five Galician datasets. The table has 12 rows and 11 columns. Column headers are SLI NERC, TreeGal, PUD, CorNER, and LREC, each with # and % sub-columns. Row labels include O, B-LOC, I-LOC, B-PER, I-PER, B-ORG, I-ORG, B-MISC, I-MISC, Total #, and Total tokens. Row 1: O, 95, 28; 37, 32; 93, 29; 8, 6; 233, 22. Row 2: B-LOC, 29, 8; 13, 11; 110, 34; 8, 6; 54, 5. Row 3: I-LOC, 22, 6; 9, 8; 38, 12; 4, 3; 30, 3. Row 4: B-PER, 10, 3; 6, 5; 23, 7; 7, 6; 50, 5. Row 5: I-PER, 9, 3; 2, 2; 16, 5; 7, 6; 48, 5. Row 6: B-ORG, 36, 10; 11, 10; 20, 6; 35, 28; 234, 22. Row 7: I-ORG, 45, 13; 11, 10; 8, 2; 9, 7; 234, 22. Row 8: B-MISC, 45, 13; 8, 7; 12, 4; 20, 16; 89, 9. Row 9: I-MISC, 54, 16; 18, 16; 5, 2; 27, 22; 72, 7. Row 10: Total #, 345, 0.85; 115, 1.07; 325, 1.38; 125, 0.60; 1,044, 2.21. Row 11: Total tokens, 40,405; 10,112; 23,509; 20,932; 47,254.

Regarding the SLI corpus, comprising 40,405 tokens, 345 tokens (0.85% of the total dataset) were identified as erroneous. Among these errors, the most significant proportion (95 tokens, or 28% of the total errors) were O-tagged tokens, indicating they were either missed or not identified as named entities. As for the TreeGal corpus, consisting of 10,112 tokens, the model predicted 115 erroneous tokens (1.07% of the total corpus). Similarly, the majority of these errors were concentrated in missed or unidentified named entity tokens.

The PUD results show a slightly different trend, with a total of 23,509 tokens and 325 erroneous tokens (1.38% of the entire corpus). The majority of these errors are found in location entities, accounting for 110 cases, or 34% of the total errors. The CorNER corpus, consisting of 20,932 tokens, contains 125 erroneous tokens (0.60% of the total corpus), with most errors occurring within the organisation class, comprising 35 cases (28% of all errors). Finally, the LREC dataset, which includes 47,254 tokens, contains 1,044 erroneous tokens, representing 2.21% of the corpus—the highest error rate among the five datasets. In LREC, the most frequent errors (22%) occur across two classes: O-tagged tokens and organisation entities.Footnote ^t

Although errors related to O-tagged tokens are commonly observed across the datasets (with four out of five exhibiting this pattern), the notably higher error rate in organisation-tagged entities for the LREC dataset predictions is particularly striking, as shown in the heatmap presented in Figure 2. The performance of BERT-base-gl drops significantly on the LREC test dataset, with the highest number of erroneous tokens occurring in O-tagged tokens (233 cases) and organisation entities (234 cases each for B-ORG and I-ORG). These results are followed by the PUD predictions, which show 110 errors in location entities and 93 in unidentified named entities. These findings suggest that BERT-base-gl likely struggles to resolve ambiguity between location and organisation entities, as explained in Subsection 3.3.2.

5.2.2 Best-performing monolingual versus multilingual model outcomes

Additionally, to investigate the differences in micro and macro F1-scores observed in the monolingual and multilingual models trained with the cross-lingual data (Mixed scenario), we focused our analysis on the final training split. This split included the SLI and TreeGal datasets, as well as cross-lingual data from the AnCora and Portuguese NER corpora, and was evaluated on the Mixed test dataset. The Mixed test dataset comprises all five Galician corpora, as detailed in Tables 2 and 3.

Figure 2.

Distribution of the erroneously annotates NE for BERT-base-gl predictions among five Galician datasets.

As shown in Figure 3, the BERT-base-gl monolingual model has predicted 1,952 erroneous tokens, while the mDeBERTa-v3-base model shows a slightly higher total of 2,115 errors. This suggests that, despite its more advanced architecture, the mDeBERTa multilingual model is prone to more errors when performing NER in Galician. As for the highest error count across both models, it occurs in O-tagged tokens, which account for 24% of total errors in the BERT monolingual model and 21% in the mDeBERTa multilingual model. This may indicate that both models tend to miss named entities or fail to identify them accurately. Regarding other errors within NE classes, B-LOC and I-LOC failed tags are more prominent in mDeBERTa multilingual predictions, with 254 (12%) and 125 (6%) cases, respectively, compared to BERT-monolingual, with 214 (11%) and 103 (5%) errors. This suggests that mDeBERTa struggles slightly more with identifying the boundaries and continuity of location entities.

Figure 3.

NE labels errors for BERT-base-gl fine-tuned with cross-lingual data evaluated on five Galician test dataset.

mDeBERTa also exhibits higher errors for B-PER and I-PER tags, with 149 (7%) and 106 (5%) errors, respectively, compared to BERT’s 95 (5%) and 81 (4%) errors. This suggests that a multilingual transformer-based model may struggle to distinguish individual names accurately. The errors in B-ORG and I-ORG tags are substantial for both models, though the numbers are similar. For BERT-base-gl, there are 336 (17%) and 307 (16%) errors, while mDeBERTa has 347 (16%) and 326 (15%) errors.

This finding supports the observed tendency identified during our analysis of the distribution of erroneously annotated named entities in the BERT monolingual predictions across five Galician datasets. Specifically, it suggests that both monolingual and multilingual models may be having difficulty distinguishing between ambiguous cases involving location and organisation entities. Finally, errors for B-MISC and I-MISC tags are comparable across the models. BERT monolingual predictions exhibit 174 (9%) and 176 (9%) errors, while mDeBERTa multilingual shows 178 (8%) and 180 (9%) errors. Compared to other entity types, miscellaneous entities do not pose a significant challenge for both monolingual and multilingual BERT-based models in Galician.

Overall, both models exhibit the highest error rates on O-tagged tokens and organisation entities, suggesting that both the BERT-based monolingual and multilingual models may struggle to distinguish between ambiguous cases involving location and organisation entities, as previously discussed. The multilingual mDeBERTa model tends to have slightly higher error rates across most categories, particularly for person and location entities, compared to the BERT monolingual Galician model.

Regarding the annotation of named entity tags and their associated error rates, an analysis was conducted on the predictions made by the BERT monolingual Galician model across five datasets to explore potential discrepancies in relation to annotation errors, especially for the PUD and LREC datasets, which showed the highest error rates. The results, both in terms of error percentages and the most common NE tag errors, suggest that revising the NER tagging guidelines may be considered. This is particularly evident from the high rates of O-tagged token errors observed in four of the five datasets, as shown in Figure 2.