Problem statement

As stated in the “Introduction” section, the alignment process is the first required step for comparing two divergent witnesses, that is, two handwritten material artifacts that preserve a specific version of a text. These alignments then enable the detection of variant readings, such as substitutions, omissions, insertions, and transpositions, which are essential for understanding how the text was transmitted across centuries of copying.

The goal of this article is to automate this process by formalizing it as a computational text alignment problem. The core challenge is thus to determine the correct set of correspondences between the words in a reference text and those in a compared text, accounting for complex textual phenomena, such as synonymous substitutions, metathesis, and semantically equivalent but lexically distinct expressions.

Formalizing the alignment problem

Formally, within our computational framework, we consider the alignment problem as the task of aligning a reference witness $(r_i)_{1 \leq i \leq n}$ , an ordered sequence of n words, with a compared witness $(c_j)_{1 \leq j \leq k}$ of k words.

This alignment is represented as a binary matrix $ A \in \{0, 1\}^{n \times k} $ , where

$$ \begin{align*}a_{ij} = \left\{ \begin{array}{@{}ll} 1,\ \text{if word}\ r_i\ \text{is aligned with word}\ c_j,\\ 0,\ \text{otherwise}. \end{array} \right. \end{align*} $$

By allowing multiple 1s in a row or a column, this framework allows many-to-one and one-to-many alignments, frequent in the cases of glosses. An example of an alignment matrix for the same verse taken from two Old Latin manuscripts that will be used as our benchmarking test sets is shown in Figure 1. Several alignment challenges can be seen in this example: lucet is transposed in the compared witness, eam/eum is a gender variant, and compraehenderunt/conpraehenderunt is a spelling variant.

Figure 1.

Example of an alignment matrix of John 1:5 in Codex Corbeiensis Secundus (reference witness) and Codex Palatinus (compared witness).

Integration within the editorial workflow

Automatic alignment tools, including ALMA, are designed to be used at a specific stage in the editorial workflow, as illustrated in Figure 2.

Figure 2.

Integration of ALMA within the editorial workflow.

Figure 2 Long description

The flowchart begins on the far left with a dark blue box labeled Manuscript, with the subtitle Physical witness. An arrow leads into a large light gray box titled Transcription phase. Inside this box, the path splits into two parallel options: Manual transcription on top and H T R plus post-correction on the bottom. Both paths converge into a box labeled Diplomatic representation.

An arrow labeled input exits the Transcription phase and points to a red box with a dashed border labeled ALMA alignment, which is topped with the text This paper. From here, an arrow labeled alignment points to a green box labeled Variant selection. Finally, an arrow labeled selection points to a purple box on the far right labeled Critical apparatus.

As mentioned in the introduction, the workflow begins with cataloging the witnesses available for the text under study (recensio) and producing transcriptions. These transcriptions can be done manually or using HTR/OCR pipelines, followed by a post-correction stage to ensure the quality of the pipeline’s input: regardless of the chosen method, the transcription must constitute a faithful diplomatic representation of the manuscripts. ALMA is therefore downstream of this transcription phase: its goal is to accelerate the first step of the collatio phase, namely, the alignment of the different transcriptions. It is this systematic alignment that allows the expert to detect variants significant to understanding the text’s history, which are then noted in the critical apparatus associated with the edition. ALMA thus necessarily takes high-quality transcriptions as input, which are then used to produce the critical edition itself.

Sequence-based collation algorithms

Dom Froger was one of the first scholars to understand the potential of computational approaches for philology, in his seminal book on computational philology research, La critique des textes et son automatisation (Froger Reference Froger1968), where he describes a possible use of automatic tools for collation (Froger Reference Froger1966; Nury and Spadini Reference Nury and Spadini2020). This algorithm aimed to simulate the philologist’s manual approach: a main reference text is selected, and other texts are compared against it. The texts are then compared word by word until a difference is found. The algorithm then continues comparing words using a sliding window until a match between the texts is found.

Following this proposal, automatic collation approaches began to rely on sequence-analysis algorithms, inspired by protein sequence alignment, as early as the late 1980s (Robinson Reference Robinson1989). This class of algorithms aligns words using string similarity between their raw or lemmatized forms to minimize the total cost of sequence alignment. Among the existing algorithms, we can mention the Needleman–Wunsch algorithm (Needleman and Wunsch Reference Needleman and Wunsch1970), progressive multiple alignment (Spencer and Howe Reference Spencer and Howe2004), and, more recently, Dekker’s algorithm (Dekker and Middell Reference Dekker, Middell and Maegaard2011). Dekker’s algorithm, which will be compared to ALMA, identifies the smallest set of longest-matching tokens that accounts for all variants between two texts.Footnote ² Tokens can then either be considered as exact matches if the tokens are the same, or similar matches if their string similarity, such as the Levenshtein distance (Levenshtein Reference Levenshtein1966), falls below a set threshold.

While these approaches are computationally very efficient and widely used, they present several drawbacks. First of all, this class of algorithms is fundamentally based on sequence proximity and string identity (or identity of pre-processed forms, such as lemmas), and its alignment strategy remains agnostic to syntactic function and semantic meaning. This makes them inherently ill-suited to languages with freer word order, such as Latin and Greek, where meaning is conveyed by inflectional morphology rather than linear order. Secondly, because they rely entirely on token-level comparisons, they cannot generalize to cross-language alignment and fail to realize the potential of automatic multilingual collation. Finally, they are very vulnerable to alignment ambiguity caused by high-frequency function words, such as conjunctions, which are omnipresent in Greek and Latin texts and often function as punctuation. When multiple tokens in both witnesses share identical words, the algorithm lacks the syntactic or semantic context to choose the correct correspondence. This often results in a “first-available” match error based on positional proximity, which then propagates, skewing the entire subsequent alignment: an incorrect early alignment of a common conjunction can force the misalignment of subsequent content words, as the algorithm becomes locked into a locally coherent but globally erroneous path.

Syntax analysis-based collation algorithms

To address the limitations of sequence-based alignment models presented in the previous section, models that employ modern NLP tools to incorporate grammatical, syntactic, and semantic information during the alignment process have been developed. Among the most efficient approaches for monolingual alignment, Sultan, Bethard, and Sumner (Reference Sultan, Bethard and Sumner2014) have proposed an algorithm that relies on a language model to identify lemmas, dependencies, named entities, and content words, and that aligns words with similar functions from the reference and the compared witnesses. The algorithm begins by aligning similar word sequences by lemma, then uses syntactic dependencies and nearby context to align the remaining words, using a similarity measure based on a paraphrase database (PPDB). Finally, stop-words are aligned using a predefined list and contextual cues. Although this aligner accounts for lemmas, allowing it to match inflected forms unlike Dekker’s algorithm, it cannot detect semantically equivalent words not present in the PPDB and thus cannot function properly for languages without a well-annotated PPDB. Additionally, it restricts alignments to unique token pairs, thereby preventing the capture of one-to-many substitutions. In this study, due to its strong performance, the algorithm proposed by Sultan et al. has been selected for our benchmarking and is referred to as “POS.”

Deep learning-based collation algorithms

Rapid advances in NLP, following the development of novel neural network architectures such as transformers (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, Guyon, von Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett2017), have led to the development of new alignment models. While these models were primarily intended for machine translation via multilingual alignment, they can be easily adapted to the task of monolingual alignment. Methods can be separated into supervised approaches (Nagata, Chousa, and Nishino Reference Nagata, Chousa, Nishino, Webber, Cohn, He and Liu2020; Stengel-Eskin et al. Reference Stengel-Eskin, Su, Post, Van Durme, Inui, Jiang, Ng and Wan2019) that require training a model to perform the alignments, and unsupervised approaches, able to learn directly from the co-occurrence of terms between the source and the target sentence (Dou and Neubig Reference Dou, Neubig, Merlo, Tiedemann and Tsarfaty2021; Sabet et al. Reference Sabet, Dufter, Yvon, Schütze, Cohn, He and Liu2020; Zenkel, Wuebker, and DeNero Reference Zenkel, Wuebker, DeNero, Jurafsky, Chai, Schluter and Tetreault2020). When it comes to the study of these models for the alignment of ancient languages, the related literature is considerably sparser: other than the major studies performed by Palladino, Shamsian, and Yousef (Reference Palladino, Shamsian and Yousef2022) focusing on aligning Greek and Latin translations, little work has been done for the alignment of ancient languages, and no systematic benchmarking has been performed.

Of particular importance to us due to its performance, versatility across a wide range of languages, and ability to be fine-tuned on a given dataset is the SimAlign algorithm proposed by Sabet et al. (Reference Sabet, Dufter, Yvon, Schütze, Cohn, He and Liu2020), which we selected for our comparison with ALMA. This unsupervised method compares sentences via cosine similarity computed between token embeddings. To match two tokens using this similarity matrix, the authors propose a new algorithm called itermax, which performs the matching iteratively, in the case of words where the similarity argmaxes do not correspond in the source and target sentence: the alignment between tokens is then performed with a discounting factor $\alpha $ , until a match is found. A pruning approach ensures that similarities are meaningful by using an entropy measure before matching, and a proximity criterion is used to minimize distortion. While this approach has proven to be effective, especially when the model is fine-tuned on the dataset to align, it suffers from two key limitations: (1) The method tends to overmatch words by relying on greedy iterative matching, which is ill-suited for texts containing insertions or paraphrased material in either the reference or the compared witness. This often results in false positive alignments, especially in contexts where textual variation or expansion is present; (2) The alignment process prioritizes semantic similarity without accounting for the syntactic function of the matched words. As a result, it may incorrectly align words from different grammatical categories (e.g., verbs with nouns) as long as their semantic content is related.

When comparing this algorithm to ALMA, we use the most effective version presented in the article for benchmarking: we retrieve the embeddings from the eighth layer of the neural embeddings and employ the itermax algorithm.Footnote ³ The two models tested by the original authors, namely, mBERT and XLM-RoBERTa, are fine-tuned on our domain-specific dataset, as detailed in the “Neural models training” section.

Limits of the current state of the art

To overcome the current limitations of the state of the art, we introduce the ALMA pipeline, which integrates the core principles of existing text alignment paradigms. It moves beyond a single-method approach by jointly optimizing lemma alignments, grammatical coherence, and semantic similarity, thereby leveraging the strengths of each existing approach while mitigating their weaknesses. To systematically evaluate the performance of our new suggested algorithm, we compare it to three algorithms of the state of the art, each representing a dominant paradigm: Dekker’s algorithm (sequence-based) as implemented in CollateX, Sultan et al.’s algorithm (grammar-based), and Sabet et al.’s algorithm SimAlign (deep learning-based).

General workflow

The ALMA pipeline implements a deterministic workflow (see Figure 3 for a schematic view of the pipeline and a step-by-step example of the alignment process is provided in the Appendix) to compute word-level alignments between two input witnesses. The pipeline is designed to integrate linguistic information hierarchically, progressing from lexical and semantic similarity to syntactic structure, thereby maximizing alignment quality while reducing the potential errors of each existing alignment approach. Each step operates only on tokens not matched in previous steps.

Figure 3.

Overview of the multi-stage ALMA matching pipeline.

Figure 3 Long description

The flowchart begins at the top with two green boxes labeled Source and Reference. Arrows from both boxes point downward into a central vertical column of processing stages.

1. Stage 1 is a pink box labeled 1 Lemma matcher.

2. A dashed arrow points down to Stage 2, a pink box labeled 2 Semantic matcher.

3. A dashed arrow points down to Stage 3, a pink box labeled 3 P O S matcher.

4. A dashed arrow points down to Stage 5, a purple box labeled 5 Enforcement of unique matches.

5. A dashed arrow points down to Stage 4, a pink box labeled 4 Dependency matcher.

6. A dashed arrow points down to another purple box labeled 5 Enforcement of unique matches.

7. A dashed arrow points down to Stage 6, a yellow box labeled 6 Enforce matches proximity.

8. A final dashed arrow points down to the terminal green box at the bottom labeled Alignment matrix.

The general workflow is designed as follows and details regarding the inner workings of each component are provided in the “Detail of each component of the pipeline” section.

1. 1. Lemma-based matching: The first component matches tokens based on a strict lemma identity to match identical lexical correspondences, similarly to sequence-based approaches. Word order is not taken into account at this step to provide resilience for languages with a freer word order.

2. 2. Semantic similarity matching: To account for lexical variation such as synonymy, the algorithm then matches the different tokens based on their similarity in terms of embedding, as detailed in the “Component 2: Semantic matcher – Detecting synonyms using embeddings” section. We perform semantic matching before syntactic matching to establish simpler alignments, such as exact matches, transpositions, and synonymic substitutions, before moving on to more complex grammatical correspondences. Additionally, this approach is reasonable from a philological viewpoint: vocabulary and meaning often remain stable throughout textual transmission, whereas syntactic structures are more likely to diverge across witnesses. Additionally, because semantic matching precedes syntactic analysis, it helps minimize error propagation from potential syntactic parsing inaccuracies, a particularly significant consideration for resource-constrained languages.

3. 3. POS-based matching: The next step matches tokens that share the same POS tag among tokens that have not yet been matched, to capture words with the same function in the reference and compared witnesses that were not considered as close by the semantic matcher.

4. 4. Dependency syntactic matching: This step incorporates deeper syntactic structure by matching tokens that share the same grammatical dependency relation, using universal dependency tags, to capture words that play an identical functional role in their respective syntactic trees.

Two additional filters, whose efficiency will be discussed in the ablation study of the “Hyperparameter tuning and pipeline optimization using ablation study” section, are applied to enhance the robustness of the pipeline:

• • Enforcing uniqueness of matches using similarity: A uniqueness (injective) constraint is applied to the alignments proposed by the POS and Dependency matchers. This constraint is not applied after the Semantic matcher, allowing ALMA to capture genuine one-to-many phenomena, such as paraphrases, at that stage. The rationale for this selective injectivity is a calculated risk management strategy. While the POS and Dependency matchers effectively identify tokens that share a grammatical role, they are agnostic to meaning and thus highly prone to proposing multiple, spurious alignments for common function words or syntactically similar but semantically distinct content words (e.g., aligning every noun in the reference with every noun in the target). Enforcing semantic injectivity at these later stages serves as a safeguard against syntactic and grammatical overmatch, which would otherwise cascade through the pipeline. As discussed in the ablation study presented in the “Hyperparameter tuning and pipeline optimization using ablation study” section, this constraint improves precision without significantly reducing recall, since the majority of genuine one-to-many alignments are already captured at the semantic step.

• • Proximity-based pruning: A final post-alignment pruning step enforces a proximity constraint by ensuring that matches from the reference and compared witnesses are performed within the same word neighborhood. While the initial matching pipeline operates without word-order constraints to handle free-word-order languages, a final proximity filter ensures topological plausibility by pruning long-distance alignments and ensuring contextually coherent alignments.

Detail of each component of the pipeline

Component 2: Semantic matcher – Detecting synonyms using embeddings

The semantic matcher then matches together words considered synonyms within the same embedded space. From an embedding matrix of the two segments to be aligned, the cosine similarity between each pair of words is calculated to create a similarity matrix $ S \in \mathbb {R}^{n \times k} $ . For each vector embedding of word $r_i,\ \mathbf {r_i},$ in the reference witness and of word $c_j,\ \mathbf {c_j},$ in the compared witnesses, the coordinate element ( $i,j$ ) of S corresponds to

$$ \begin{align*}S_{ij} = \cos(\mathbf{r}_i, \mathbf{c}_j) = \frac{\mathbf{r}_i \cdot \mathbf{c}_j}{\|\mathbf{r}_i\| \|\mathbf{c}_j\|}. \end{align*} $$

Each row of the S matrix is then normalized by transforming it into $ z $ -normalized scores:

$$ \begin{align*}z_{ij} = \frac{S_{ij} - \mu_i}{\sigma_i}, \end{align*} $$

where $ \mu _i $ and $ \sigma _i $ denote the mean and standard deviation of the similarities of row $ i $ .

This transformation allows us to assess the homogeneity of similarity scores and avoid treating words as synonyms when their similarity does not stand out from the rest of the sentence. To ensure this, we consider only tokens with a z-score above a predefined threshold as potential matches. For each unaligned token in the source sentence, the target tokens with similarity scores exceeding the threshold are added as alignments in the binary matrix.

Formally, for each reference word $ r_i $ that has not yet been aligned, the compared word $ c_{l^*} $ is considered as a match if $ z_{i j^*}> \delta $ , where $ \delta $ is a fixed threshold, that corresponds to the standard deviation from the mean of the retained similarity. Because of this, if two words are selected as synonyms, it implies that their similarity is at least $\delta $ standard deviations above the mean $\mu_i$ . This allows us to capture only words that stand out in terms of semantic proximity, instead of matching words using a simple argmax in terms of similarity, which might be due to the random fluctuation of a flat distribution of semantic proximity in the case of a compared and target verse presenting no synonym in common. The coordinate element $(i, l^*)$ of the alignment matrix $ A \in \{0, 1\}^{n \times k} $ is then set to 1. The impact of the value of this score is evaluated in the “Hyperparameter tuning and pipeline optimization using ablation study” section.

The same procedure is then applied column by column to address the missing alignments from the perspective of the compared sentence.

Component 5: Enforcing unique matches using the semantic similarity matrix

After the POS and dependency matchers, we enforce matching uniqueness to avoid matches of words with similar grammatical functions but with different semantics.

To ensure the uniqueness of alignments between the two witnesses, an injective alignment constraint is applied to the binary alignment matrix $ A \in \{0,1\}^{n \times k} $ , as alignment outputted at each step may contain multiple alignments per row or column. We select in each row $ i $ (if $ \sum _{j} A_{ij} > 1 $ , i.e., there is more than one match for word i in the reference segment) the column $ j^\ast $ that maximizes the similarity:

$$\begin{align*}j^\ast = \arg\max_j S_{ij}, \end{align*}$$

all other alignments $ A_{ij} $ for $ j \neq j^\ast $ are set to 0.

A symmetric operation is then applied to the columns: for any column $ j $ such that $ \sum _{i} A_{ij}> 1 $ , only the row $ i^\ast $ satisfying

$$\begin{align*}i^\ast = \arg\max_i S_{ij} \end{align*}$$

is retained. Because of this transformation, the grammatical matching, based on POS or the syntactic tree, always selects the option with the highest similarity in terms of embedding and thus of context.

Component 6: Enforcing local proximity between word matches

Finally, the last step of the alignment process involves enforcing local proximity between word matches. This ensures that aligned words are present in the same window in both the reference and the compared witnesses, thereby avoiding matches that are too far apart in the segment. While the SimAlign paper relies on the distortion correction proposed in the IBM 2 alignment model (Brown et al. Reference Brown, Della Pietra, Della Pietra and Mercer1993), we adopt a more straightforward approach by applying a window-based constraint, measuring proximity along the diagonal of the alignment matrix.

For each match pair at coordinates $(i,j)$ , we compute a relative distortion measure:

$$\begin{align*}\delta_{ij} = \frac{i}{n} - \frac{j}{k}, \end{align*}$$

that is, the difference between the normalized relative positions in each sentence, where segments can have very different lengths. If the absolute distortion $ |\delta _{ij}| $ exceeds a fixed threshold $ \tau \in [0,1] $ corresponding to the percentage of the lengths of the aligned segments, the alignment is removed, that is, $ A_{ij}$ is set to 0. Different values of the proximity window are tested in the evaluation section. This operation retains only alignments located at a reasonable proportional distance from the main diagonal of the matrix, which helps the correspondences to be constrained to follow an approximate linear order.

Implementation and reproducibility

All the code used in this article is available as a Python package on GitHub.Footnote ⁴ We use the Python port of CollateX and the authors’ implementations for the POS algorithm and SimAlign. ALMA is implemented alongside each algorithm and accessed via a straightforward wrapper API. The scripts for training the models, obtaining sample ground-truth data from the manuscript transcriptions, and benchmarking are also available, enabling all results presented in this article to be reproduced.

For the end-user, ALMA can be used in two complementary modes depending on the user’s technical background. For computational researchers who require programmatic access or batch processing, ALMA exposes a Python API that is fully documented. For users with limited programming experience, ALMA is distributed with a web application that can be deployed locally from the source code. The interface allows the user to paste or upload the two witnesses to be aligned, select the desired algorithm and configure its hyperparameters, and visualize the resulting alignment as a list of matches and non-matches.

Extending to other languages

Although our experiments have only been conducted on Greek and Latin so far, our model can be immediately extended to languages that have a SpaCy or Stanza parser, and its implementation, both via the Python API and via the web interface, allows for this. Users can also use their own trained model, whether with the SpaCy API or the Stanza API.

The main problem lies in the desire to use ALMA for languages that still lack lemmatizers and taggers. Our implementation anticipates this case by allowing complete disabling of the tagger and, therefore, of linguistic analysis. While this necessarily reduces performance, we show in the ablation study (see the “Hyperparameter tuning and pipeline optimization using ablation study” section) that this inability to use a tagger does not result in an extreme drop in performance, and ALMA remains a very good candidate for automatic alignment. This dependency on neural models as a current limitation of our work is discussed in the “Dependence on neural model” section.

Selected codices

To demonstrate ALMA’s effectiveness on alignment tasks representative of philological work, we evaluate the algorithm on two monolingual alignment tasks, each drawn from a pair of manuscripts of the Gospel of John. This text from the New Testament was selected as relevant to our study for three main reasons: (1) the New Testament is one of the most copied texts in Western history, with nearly 6,000 witnesses in Greek and 10,000 witnesses in Latin. This large number of witnesses has prevented an exhaustive study, as such an analysis is beyond the capabilities of manual-only methods. High-performing automatic alignments thus have a strong potential for the discovery of relationships between manuscripts that have not yet been explored; (2) this text has been transmitted through several well-identified textual families, which has made it possible to choose, for this study, manuscripts from different textual traditions. This ensures that our selected texts present variants relevant to evaluating the performance of our algorithm; (3) high-quality transcriptions and manuscript scans are available thanks to numerous transcription projects, enabling us to use error-free transcriptions for our dataset. It must additionally be noted that we are faithful to the existing transcriptions and do not supply missing words or letters due to damage.

Building the alignment dataset

Annotation protocol and guidelines

All alignments have been performed by domain experts using the UGARIT tagging platform (Yousef et al. Reference Yousef, Palladino, Shamsian and Foradi2022). The annotation was initially done by the first annotator directly on the platform, and 40 percent of the annotated verses in each language were randomly selected and commented on by the second annotator to evaluate inter-annotator agreement and discuss any potential annotation conflicts. We established the following annotation rules for the different possibilities of observed matches and how to resolve them:

• • Sure match: A match is considered sure when it is possible to find an unambiguous correspondence between the two texts, particularly in the case of an exact lexical match, grammatical or morphological variants, or clear synonymic substitutions.

• • Omissions and insertions: Tokens that are present in only one of the two witnesses and have no correspondence in the other are left unaligned when the annotator determines that there is no strict semantic equivalent in the other witness. Notably, structural omissions, such as an absent subject, are treated as omissions.

• • Conjunctive discourse markers: When a discourse marker (e.g., δέ, καί, and autem) is found in only one witness, and there is no structural counterpart in the other, the marker is treated as an insertion and left unaligned. When two witnesses use distinct conjunctions occupying the same syntactic position, alignment is governed by semantic equivalence: functionally synonymous markers are aligned as sure matches, whereas markers with distinct discursive functions (coordination vs. opposition, for example) are left as unaligned and counted as an insertion/omission.

• • One-to-many alignments: When a word in one witness is semantically rendered by several words in the other witness, the word is associated with the several other words. A typical example encountered when annotating the dataset is when one of the parallel words is negated, such as non sciebat and nesciebat.

• • Repetitions: One difficulty encountered concerns the alignment of words that are repeated in one witness but not in the other: for example, if a conjunction is repeated several times, it is difficult to determine which conjunction each alignment refers to. In these cases, we apply a “nearest counterpart” rule: the target token nearest in the linear sequence to the source token is selected for alignment.

Inter-annotator agreement was measured using Cohen’s $\kappa $ computed at the token-pair level, and we computed $\kappa _{\text {Latin}} = 0.86$ and $\kappa _{\text {Greek}} = 0.94$ . This corresponds to almost perfect agreement on the Landis–Koch scale (Landis and Koch Reference Landis J. and Koch1977), showing a strong reliability regarding our tagging choices. Most of our inter-annotator disagreements were linked to the possible alignment of ambiguous discourse markers. Additionally, we observe that $\kappa $ is lower for Latin ( $\kappa =0.86$ ) than for Greek ( $\kappa =0.94$ ), consistent with the higher proportion of non-identity matches in the Latin dataset, as described in Table 1. All remaining disagreements were resolved by adjudication before the gold standard was finalized.

Table 1.

Characteristics of the monolingual gold-standard alignment.

Table 1 Long description

The table consists of three columns. The first column lists the metrics, while the second and third columns provide data for Latin and Greek respectively.

* Compared codices: Latin uses Palatinus forward slash Corbeiensis Secundus. Greek uses Bezae forward slash Vaticanus.

* Total tokens: Latin has 4,880. Greek has 5,078.

* Tokens with matches (percentage): Latin is 87.9. Greek is 89.9.

* Tokens without matches (percentage): Latin is 12.1. Greek is 10.1.

* Non-identity matches (percentage): Latin is 25.1. Greek is 10.6.

Evaluation score

We evaluate alignment performance using the standard alignment error rate (AER) metric, designed for evaluating the quality of alignment algorithms (Och and Ney Reference Och and Ney2000), which considers precision and recall against a gold standard containing both “Sure” (S) and “Possible” (P) alignments. However, in our context of aligning manuscript variants within the same language, the distinction between “Sure” and “Possible” alignments is unnecessary, as the manual alignment is monolingual and the variations are due to scribal changes on the same textual tradition, with considerably less ambiguity than when comparing translations. Consequently, for our gold-standard alignments, we define the set of sure alignments S to be identical to the set of possible alignments P. Under the condition that $S = P$ , the AER simplifies to

$$\begin{align*}\text{AER}(A, S) = 1 - \frac{2 \times |A \cap S|}{|A| + |S|} \end{align*}$$

with S as the set of all gold-standard alignments and A as the set of alignments proposed by an algorithm. We report the alignment score (AS), defined as $(1 - AER) \times 100$ , to return a performance that equals 100 percent in the case of perfect matching, alongside precision and recall to indicate the algorithm’s tendency to propose incorrect links (affecting Precision) and its tendency to miss valid ones (affecting Recall). We thus define

$$\begin{align*}\text{Precision} = \frac{|A \cap S|}{|A|}, \quad \text{Recall} = \frac{|A \cap S|}{|S|}. \end{align*}$$

All scores are calculated per verse, and we report the mean over verses within each dataset.

Compared algorithms and hyperparameters testing

We compare the three algorithms from each category of existing alignment algorithms described in the “Automatic collation and current state of the art” section to ALMA: the Dekker algorithm as implemented in CollateX (sequence-based), the POS algorithm (grammar-based), and SimAlign (deep learning-based). We then apply all the algorithms to the Latin and Greek ground-truth data and measure the evaluation scores for each algorithm, as defined in the previous section. The statistical significance of the performance difference is evaluated using Welch’s t-test (Welch Reference Welch1947), considering each verse as an individual and evaluating the variation in AS depending on the considered algorithms. We use a 95 percent confidence level, treating p-values below 0.05 as significant. When commenting on the significance of the results, we specify the value of the p-value itself. This complies with recent criticism of only reporting statistical significance rather than the value of the p-value itself, as this value is informative about the probability of obtaining a test statistic at least as extreme as the one observed under $H_0$ (Andrade Reference Andrade2019).

The hyperparameters tested for each algorithm are available in Table 2.

Table 2.

Summary of tested hyperparameters for each algorithm.

Table 2 Long description

The table consists of two columns.

* Row 1: Algorithm Dekker. Hyperparameters: Use Levenshtein distance in the set containing True or False.

* Row 2: Algorithm P O S. Hyperparameters: No hyperparameters.

* Row 3: Algorithm SimAlign. Hyperparameters: Model in the set containing bert-cased, xlm-roberta, bert-cased-fine-tuned, xlm-roberta fine-tuned, laberta, laberta fine-tuned, greberta, and greberta fine-tuned.

* Row 4: Algorithm A L M A. Hyperparameters: Model in the set containing bert-cased, xlm-roberta, bert-cased-fine-tuned, xlm-roberta fine-tuned, laberta, laberta fine-tuned, greberta, and greberta fine-tuned.

* Row 5: Algorithm A L M A (continued). Hyperparameters: Z-score threshold in the set containing 1, 2, 3.

* Row 6: Algorithm A L M A (continued). Hyperparameters: Distortion window in the set containing 10 percent, 30 percent, 70 percent.

Neural models training

Computing word embeddings

To build the similarity matrix, we trained several models and extracted contextual embeddings from them, which we then used for both the SimAlign model and ALMA. We used several different types of models for the embedding: (1) two non-specialized pre-trained multilingual models, Bert Multilingual Cased (Devlin et al. Reference Jacob, Chang, Lee and Toutanova2019) and XLM Roberta (Conneau et al. Reference Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer, Stoyanov, Jurafsky, Chai, Schluter and Tetreault2020) in their base version, as in the original paper proposing the SimAlign algorithm; (2) two specialized models for Ancient Greek and Latin, taken from the work of Riemenschneider and Frank (Reference Riemenschneider, Frank, Rogers, Boyd-Graber and Okazaki2023): LaBERTa for the Latin embeddings and GreBERTa for the Greek; and (3) each of these models, but fine-tuned with the witnesses from each considered tradition: for our Old Latin alignments, we used data from seven complete witnesses of the Gospel of John (Gryson Reference Gryson1999), and for our alignment in Greek, four major codices containing the Gospel text.

The total number of tokens for each trained model, as well as the complete elapsed time for the training, is available in Table 3.

Table 3.

Size of the fine-tuning corpus and elapsed training time for each model in Latin and Greek.

Table 3 Long description

The table consists of four columns: Language, Total words in training set, Models, and Elapsed time in seconds.

Under the Latin language category with 29,447 total words in the training set:

* La B E R T a model took 1,495 seconds.

* m B E R T model took 413 seconds.

* X L M-Ro B E R T a model took 1,988 seconds.

Under the Greek language category with 163,415 total words in the training set:

* Gre B E R T a model took 497 seconds.

* m B E R T model took 91 seconds.

* X L M-Ro B E R T a model took 556 seconds.

The models were trained using the masked language modeling objective, an unsupervised approach initially used to train the BERT model (Devlin et al. Reference Jacob, Chang, Lee and Toutanova2019). This involves randomly masking a proportion of tokens and training the model to predict the masked tokens using all tokens in the sequence. By learning the relationship between the different tokens in the sentence, the model can obtain their representations and thus their semantic proximity in the newly generated space.

In this study, we performed training by randomly selecting 15 percent of the input tokens to be predicted from their bidirectional context. We then use a partial data corruption procedure, replacing 80 percent of the selected tokens with the mask token, replacing another 10 percent with a random token drawn from the entire vocabulary, and leaving the remaining 10 percent unchanged. We then used the trained model to retrieve the embedding of each token to be collated, specifically from the eighth transformer hidden layer, as demonstrated to be the most effective in the SimAlign paper. Since the embeddings are performed at the token level, rather than directly on the word, we choose, like the authors of SimAlign, to average the representations of the subtokens corresponding to each word.

Impact and selection of hyperparameters

Impact of model tuning

Table 4 reports ALMA’s performance across embedding models, showing that fine-tuning consistently affects the quality of the alignments. In both cases, the specialized models LaBERTa and GreBERTa provided the best out-of-the-box AS score compared to the more generic multilingual models XLM-RoBERTa and mBERT, suggesting that domain-specific pretraining is a key factor for alignment quality. However, the differences between the different selected embedding models are not statistically significant (p > 0.3 in all considered cases), indicating that even non-specialized models perform very efficiently and that our solution need not be restricted to languages with existing specialized models.

Table 4.

Impact of model fine-tuning in Latin and in Greek, ranked by AS score within each language.

Table 4 Long description

The table contains four columns: Language, Base model, Tuned, and A S.

Under the Latin section:

- Base model La B E R T a: Tuned True has an A S of 89.17 (highlighted in red); Tuned False has an A S of 88.04.

- Base model X L M-Ro B E R T a: Tuned True has an A S of 88.78; Tuned False has an A S of 87.48.

- Base model m B E R T: Tuned True has an A S of 88.41; Tuned False has an A S of 87.92.

Under the Greek section:

- Base model Gre B E R T a: Tuned True has an A S of 92.08 (highlighted in red); Tuned False has an A S of 91.67.

- Base model m B E R T: Tuned True has an A S of 91.85; Tuned False has an A S of 91.26.

- Base model X L M-Ro B E R T a: Tuned True has an A S of 91.05; Tuned False has an A S of 89.12.

A note indicates that results are averaged over other hyperparameter combinations and best results are in red.

Note: The results are averaged over all other possible hyperparameter combinations. Best results are indicated in red.

Fine-tuning systematically improved the performance of all the considered embedding models. For Latin, fine-tuning the LaBERTa model improved AS by 1.13 points relative to the non-fine-tuned version. For Greek, fine-tuning GreBERTa provided more modest gains of 0.41 AS points. In all cases, fine-tuning models on the data improved their performance, regardless of the model considered. Given the demonstrated importance of fine-tuning, ALMA provides a simple command-line interface for model training that requires only the parallel texts as input. Incorporating fine-tuning into the pipeline thus becomes straightforward for the users, including those coming from a humanities background.

Z-score threshold and distortion filter

The impact of the threshold used to determine whether two words are synonyms and the distortion filter indicating the percentage of distance allowed between matched words in the reference and the compared sentences are shown in Figure 4.

Figure 4.

Impact of hyperparameters on AS. The impact of hyperparameters has been measured on the fine-tuned model.

Figure 4 Long description

The figure consists of two horizontal panels. Both panels share a Y-axis ranging from 82.5 to 97.5 and an X-axis labeled Distortion ratio with values 0.1, 0.3, and 0.7. A legend on the right identifies three Threshold levels: 3 (light green), 2 (teal), and 1 (dark blue).

Left Panel: Latin Corpus.

At Distortion ratio 0.1, the boxes are clustered low, with Threshold 3 at approximately 82.5 and Thresholds 2 and 1 near 85.0.

At Distortion ratio 0.3, performance increases significantly. Threshold 3 is near 89.5, while Thresholds 2 and 1 are higher, clustered around 92.0.

At Distortion ratio 0.7, performance remains stable or slightly higher than 0.3, with Threshold 3 near 90.0 and Thresholds 2 and 1 near 92.5.

Right Panel: Greek Corpus.

At Distortion ratio 0.1, Threshold 3 is at 85.0, while Thresholds 2 and 1 are clustered near 89.5.

At Distortion ratio 0.3, Threshold 3 rises to 90.0, while Thresholds 2 and 1 jump to approximately 95.0.

At Distortion ratio 0.7, Threshold 3 stays near 90.5, while Thresholds 2 and 1 reach their peak near 95.5.

Across both panels, lower Threshold values (1 and 2) consistently yield higher A S scores than Threshold 3, and performance generally improves as the distortion ratio increases from 0.1 to 0.3 before plateauing.

The negative impact of an excessively low distortion parameter is evident, as it significantly reduces the pipeline’s AS performance on average by $-6.98$ points in Latin (p-value < 0.001) and $-5.33$ in Greek (p-value < 0.001) for each tested threshold when using a strict distortion filter of 10 percent. Using a window of 70 percent leads to a slight performance increase of 0.31 (p-value = 0.52) for the Latin and of 0.32 for the Greek (p-value = 0.73), compared to a window of 30 percent.

Regarding the impact of the semantic threshold, we find that, as expected, a threshold that is too high prevents matching because it treats no words as synonyms. Choosing the high value of 3 thus significantly reduces, on average, the AS by 2.54 points (Latin, p-value = 0.01) and 4.86 points (Greek, p-value < 0.001) across all configurations for each language. The advantage of using threshold 1 over 2 is slight, with a non-significant improvement of 0.03 (p-value = 0.98) for the Latin and of 0.33 (p-value = 0.74) for the Greek. Globally, as indicated by the boxplot distribution, the impact of the two parameters is very similar across languages. There is also little interaction between the hyperparameters, with the threshold having a similar effect across distortion ratios. The best combination of threshold 1 and a distortion ratio of 70 percent yields significant improvements of 3.87 points for Latin (p-value < 0.001) and 4.27 points for Greek (p-value < 0.001) compared to all other averaged configurations. We can thus conclude that, aside from extreme values, the hyperparameters, including the selected models, have a moderate impact on the pipeline’s performance when selected within a reasonable range. We retain the best-performing semantic similarity threshold of 1, a distortion filter of 0.7, and the specialized model for each language for the remainder of the ablation study and the article.

Design of the ablation study

To evaluate the importance of each of the components shown in Figure 3, select the best-performing architecture, and better understand the internal workings of the matching pipeline, we evaluate the performance of each component by asking the following questions: (RQ1) What is the performance of the pipeline when removing the components one by one? (RQ2) What is the performance of the pipeline when removing the pruning filters? and (RQ3) What is the performance of each component alone?

The performance is evaluated using the AS, and we set the hyperparameters to the best-performing values from the previous section ( $\tau =0.7$ for the distortion filter, $1$ for the z-score, and GreBERTa and LaBERTa for Greek and Latin, respectively). The summary of the tested experimental variants of the pipeline is available in Table 5, and the results of the ablation tests are available in Table 6.

Table 5.

Experimental protocol for the evaluation of the components of the ALMA pipeline.

Table 5 Long description

The table is organized into two columns: Model variant and Description.

* Standard A L M A: Full pipeline presented in Figure 3.

* R Q 1: Ablation of each component:

- A L M A minus Lemma: The lemma-based matching step is removed.

- A L M A minus Semantic: The semantic similarity matching step is removed.

- A L M A minus P O S: The part-of-speech matching step is removed.

- A L M A minus Dependency: The dependency relation matching step is removed.

- A L M A minus (P O S plus Dependency): Both syntactic matching steps (P O S and Dependency) are removed.

* R Q 2: Impact of removal of pruning filters:

- A L M A minus Uniqueness: The uniqueness constraint applied after the P O S and Dependency matching steps is omitted.

- A L M A minus Proximity: The final proximity-based pruning step is omitted.

* R Q 3: Performance of stand-alone components:

- Lemma only: Only the lemma-based matching step is active.

- Semantic only: Only the semantic similarity matching step is active.

- P O S only: Only the part-of-speech matching step is active.

- Dependency only: Only the dependency relation matching step is active.

Table 6.

Results of the ablation study.

Table 6 Long description

The table contains three columns: Model variant, A S-Latin, and A S-Greek.

* Baseline: Standard ALMA scores 94.12 in Latin and 95.77 in Greek.

* R Q 1: Ablation of each component.

- ALMA minus Lemma: 94.61 (Latin), 95.05 (Greek).

- ALMA minus Semantic: 91.42 (Latin), 87.56 (Greek), both with triple asterisks.

- ALMA minus P O S: 94.14 (Latin), 95.29 (Greek).

- ALMA minus Dependency: 92.57 (Latin), 95.34 (Greek).

- ALMA minus (P O S plus Dependency): 92.55 (Latin), 95.31 (Greek).

* R Q 2: Pruning filter ablation.

- ALMA minus Uniqueness: 82.55 (Latin), 85.93 (Greek), both with triple asterisks.

- ALMA minus Proximity: 94.03 (Latin), 95.36 (Greek).

* R Q 3: Individual component analysis.

- Lemma only: 68.19 (Latin), 73.58 (Greek), both with triple asterisks.

- Semantic only: 86.71 (Latin), 89.61 (Greek), both with triple asterisks.

- P O S only: 46.04 (Latin), 30.49 (Greek), both with triple asterisks.

- Dependency only: 66.57 (Latin), 75.45 (Greek), both with triple asterisks.

Triple asterisks indicate a p-value less than 0.001.

Note: Hyperparameters are set using the value determined in the previous section. Best performances are indicated in red, and asterisks indicate the significance of the p-value, with the following ruleset: $^{***}$ (p-value $\ <$ 0.001), $^{**}$ (p-value < 0.01), and $^*$ (p-value < 0.05).

Contribution of each component (RQ1)

Removing the lemma-matching component, the first in the pipeline, results in a non-significant gain in AS for Latin (+0.49 points, p-value = 0.62) and a non-significant loss for Greek ( $-0.72$ , p-value = 0.79). This asymmetry confirms that lemma-matching plays a slightly different role across languages: for Latin, where the embedding model has been pre-trained on richer data, lemma filtering introduces marginal noise; for Greek, it provides a small but non-significant benefit by pre-filtering exact matches. The most dramatic performance drop ( $-2.70$ in AS, p-value < $0.001$ for Latin, $-8.21$ , p-value < $0.001$ for Greek) occurs when we remove the semantic component, highlighting its crucial role in ALMA’s performance. This is particularly evident in Greek, where the linguistic annotation pipeline is less effective for POS and syntactic dependency analysis. As for the POS/Dependency component, removing both syntactic matchers costs 1.57 points of AS for Latin but only 0.46 points for Greek. This contrast reflects the quality of the underlying morpho-syntactic parser: the Latin in our witnesses has characteristics similar to those on which the tagging model was trained, so grammatical matching provides genuine signal, whereas for Greek the syntactic annotation pipeline is hampered by the lack of diacritics. Neither loss reaches statistical significance. We thus see a trade-off in ALMA’s architecture: combining semantic and grammatical information, when the embedding and linguistic annotation pipelines perform well, increases the pipeline’s matching capacity. However, when these pipelines are of lower quality, they negatively affect performance, and the particularities of the considered corpus, especially the ratio of differing tokens, should be taken into account when deciding which component to activate.

Contribution of the pruning filters (RQ2)

When we consider the impact of filtering on performance, we see the strong usefulness of the uniqueness filter, which is missing in SimAlign, with a very significant improvement of $+11.57$ (p-value $< 0.001$ ) for Latin, and an improvement of $+9.84$ (p-value $< 0.001$ ) for Greek: this is the component with the most positive impact on performance. Despite its simplicity, the proposed filter is highly effective in ensuring high-quality matching and mitigating the risk of overmatching arising from analysis of semantic proximity and grammatical role alone. The proximity filter, as observed during hyperparameter evaluation, exhibits small and non-significant gains in both languages: removing the filter yields a marginal loss of $-0.09$ (p-value = 0.89) for Latin and a slightly larger but not statistically significant loss of $-0.41$ for Greek (p-value = 0.56).

Performance of each component (RQ3)

Each component considered individually performs significantly worse than the pipeline as a whole, demonstrating the effectiveness of ALMA’s well-rounded approach, which leverages the strengths of existing approaches. As expected, the approach using only the semantic component, which is very similar to SimAlign, yields the pipeline closest to ALMA’s performance, with decreases of $-7.41$ (p-value $= 0.00$ ) for Latin and $-6.16$ (p-value $= 0.00$ ) for Greek. This result is very consistent with SimAlign’s good performance in the state of the art and the strong potential of neural approaches for alignment. It also strengthens our claim that good performance is achievable when working with languages that currently still lack lemmatizers and taggers. Using POS alone for matching yields the lowest performance, with $-48.08$ (p-value $= 0.00$ ) for Latin and $-65.28$ (p-value $= 0.00$ ) for Greek, which is understandable given the potential ambiguity of matching based solely on parts of speech. Using dependencies alone, which is slightly more accurate for comparing the reference sentence and the compared sentence, yields a smaller performance loss, with $-27.55$ (p-value $= 0.00$ ) for Latin and $-20.32$ (p-value $= 0.00$ ) for Greek. For exact matching using the lemma, as in string-matching approaches such as CollateX, performance is much lower than that of the entire pipeline, with losses of $-25.93$ (p-value $=0.00$ ) for Latin and $-22.19$ (p-value $=0.00$ ) for Greek. We can therefore see the importance of combining the different approaches available in the state of the art, as each component of the pipeline interacts complementarily and performs much less efficiently on its own.

Final optimized pipeline

Because removing no component produced a statistically significant improvement, we retained the complete pipeline, as the marginal gain from removing the lemma-matching component in Latin (+0.49) does not justify altering the full pipeline. All the results presented in the ‘‘Comparison with the state of the art’’ section are computed using the full pipeline of Figure 3. In practice, we recommend that ALMA users first verify the advertised performance of their linguistic annotator to avoid performance degradation. Regarding the grammatical components, given their negligible impact on the Greek corpus, we retained them for our final evaluation. Similarly, given the negligible impact of the resulting performance drop, we have chosen to keep the proximity filter for the remainder of the article, using a window of 70 percent. Our final selected optimized workflow is thus the same as shown in Figure 3.

Global alignment scores

The values of the scores selected to assess the performance of the different algorithms in Latin are presented in Table 7 and in Greek in Table 8. The asterisks in the table indicate the statistical significance of the performance improvement when using ALMA rather than the other algorithms tested.

Table 7.

Mean performance scores on Latin.

Table 7 Long description

The table contains five columns: Model, Hyperparameters, A S, Precision, and Recall.

* A L M A (LaBERTa-tuned) with Hyperparameters Thresh=1, Ratio=0.7: A S 94.12, Precision 94.27, Recall 94.03. These values are highlighted as the best performance.

* SimAlign (X L M-RoBERTa-tuned): A S 92.34 super ***, Precision 92.60, Recall 92.17.

* SimAlign (X L M-RoBERTa): A S 91.24 super ***, Precision 91.21, Recall 91.38.

* SimAlign (LaBERTa-tuned): A S 90.65 super ***, Precision 90.94, Recall 90.42.

* SimAlign (mBERT): A S 89.57 super ***, Precision 89.18, Recall 90.07.

* SimAlign (LaBERTa): A S 89.26 super ***, Precision 89.50, Recall 89.07.

* SimAlign (mBERT-tuned): A S 88.80 super ***, Precision 87.70, Recall 90.05.

* P O S: A S 76.28 super ***, Precision 72.14, Recall 81.19.

* CollateX (LevDist = False): A S 67.92 super ***, Precision 62.48, Recall 74.70.

* CollateX (LevDist = True): A S 67.92 super ***, Precision 62.48, Recall 74.70.

Note: Asterisks indicate statistical significance for A S relative to A L M A.

Note: The best performance is displayed in red. Asterisks indicate the statistical significance (for AS only) of the improvement in ALMA’s performance relative to the considered algorithms. Only the best hyperparameterization is retained for ALMA.

Table 8.

Mean performance scores on Greek (sorted by AS per model class).

Table 8 Long description

The table contains six columns: Model Class, Model, Hyperparameters, A S, Precision, and Recall.

* A L M A: GreBERTa-tuned with Thresh=1 and Ratio=0.7. A S is 95.77, Precision is 95.69, and Recall is 95.91. These values are highlighted as the best performance.

* SimAlign:

- X L M-RoBERTa-tuned: A S 94.74 asterisk, Precision 94.67, Recall 94.88.

- GreBERTa: A S 94.62 asterisk, Precision 94.58, Recall 94.73.

- GreBERTa-tuned: A S 94.56 asterisk, Precision 94.46, Recall 94.74.

- mBERT: A S 93.71 triple asterisk, Precision 93.47, Recall 94.02.

- mBERT-tuned: A S 91.78 triple asterisk, Precision 90.48, Recall 93.27.

* P O S: A S 86.95 triple asterisk, Precision 84.18, Recall 90.11.

* CollateX:

- LevDist = False: A S 84.46 triple asterisk, Precision 81.15, Recall 88.29.

- LevDist = True: A S 84.46 triple asterisk, Precision 81.15, Recall 88.29.

Note: Asterisks indicate statistical significance of A L M A's improvement over other algorithms.

Note: Asterisks indicate the statistical significance of the improvement in ALMA’s performance relative to the considered algorithms. Best performance is displayed in red. Only the best parameterization is retained for ALMA.

These two tables show that ALMA significantly outperforms all tested approaches, including the best-performing version of SimAlign. ALMA improves AS by 1.78 (p-value = 0.001) and 1.03 (p-value = 0.03) in Latin and Greek, respectively. We thus see relative gains of 1.93 percent (Latin) and 1.09 percent (Greek) for ALMA compared to SimAlign, using fine-tuned XLM-RoBERTa for both Greek and Latin. The recall–precision trade-off is balanced across all the algorithms considered. Alignment with Greek manuscripts yields slightly higher scores for all the algorithms considered, due to lower token variation in the Greek manuscript dataset. This is particularly striking for CollateX’s string-similarity algorithm, which shows a performance increase of more than 24 percent for Greek compared to Latin. Despite this improvement in Greek, CollateX’s algorithm, although widely used, performs significantly worse than neural methods. Compared with CollateX, SimAlign (in its best hyperparameterization) achieves 36 percent and 12 percent improvements in AS for Latin and Greek, respectively, whereas ALMA achieves 39 percent and 13 percent improvements. Whether or not the Levenshtein distance is used for strict matching does not affect the results, because morphological or synonymic differences are too significant to be captured by a character-string comparison. The same observation holds for the POS algorithm, which underperforms compared to neural approaches. These results highlight the strength of neural approaches compared to traditional ones for alignment, as they effectively handle differences in word order, morphological variation, and synonymic substitutions encountered in scribal editing. This superior performance is verified even for models that have not been fine-tuned and have been used off the shelf, including non-standard Greek without diacritics. For ALMA, performance increases in all cases when fine-tuning is applied. The best models for SimAlign are not the specialized language models, but rather the XLM-RoBERTa model for both languages, specifically tuned on the considered corpus. SimAlign thus exhibits the opposite of ALMA, whose performance is greatly improved by using pre-trained models specific to the aligned languages. ALMA’s architecture thus seems better suited to exploiting the semantic knowledge encoded by language-specific pretraining, and the integration of linguistic components appears to reinforce the signal provided by embeddings.

Statistical distribution of AS

The statistical distribution of alignment quality is shown in Figure 5.

Figure 5.

Statistical distribution of the AS at the verse level for the best-performing hyperparameterization of each algorithm.

Figure 5 Long description

The Y-axis represents a numerical scale from 30 to 100 with horizontal grid lines every 10 units. The X-axis is labeled Language with two categories: Latin and Greek. A legend on the right identifies four algorithms: Collatex (blue), P O S (orange), ALMA (green), and SimAlign (red).

In the Latin category, the box plots show an upward trend in performance from left to right. Collatex has the lowest median around 70 with a wide range. P O S follows with a median near 78. ALMA and SimAlign show significantly higher performance, with ALMA's median at 100 and SimAlign's median near 95, both featuring several outliers below their whiskers.

In the Greek category, the overall performance is higher and more compressed. Collatex has a median near 87. P O S has a median near 89. ALMA maintains a median of 100 with a very tight interquartile range and several outliers. SimAlign has a median near 98. Across both languages, ALMA and SimAlign consistently outperform Collatex and P O S.

This figure shows that median performance leads to the same ranking of algorithms as average comparisons. The lower performance observed for the Latin witnesses is consistent with their greater degree of token variation, with differences between Latin manuscripts leading to a general decline in performance compared to Greek. In particular, we note that ALMA achieves a median performance of 100 percent on both datasets, whereas SimAlign achieves 94.33 percent (Latin) and 97.82 percent (Greek) in its best configurations. This value indicates that ALMA achieves perfect alignment on more than half of the verses considered, unlike its competitors, and improves by 6.01 percent (Latin) and 2.23 percent (Greek) in median over the state of the art in terms of relative improvement. Regarding the variance across the algorithms, as measured by the interquartile ranges, ALMA shows the smallest interquartile range in both datasets. The Latin dataset, with the greatest differences between tokens, yields a higher interquartile range, especially for Dekker’s algorithm, whose performance drops approximately to 25 percent for some alignments.

Case studies: SimAlign and ALMA

To illustrate the performance differences between SimAlign and ALMA, we analyze three alignments grouped into two case studies in which ALMA is more efficient. Error analysis of ALMA is provided in the “Error analysis and main methodological limitations” section to highlight some of the limits of our approach and possible research directions for improvement.

Case 1 – Uniqueness filter

One significant difference between SimAlign and ALMA is the application of the uniqueness filter for matches based on word similarity available in ALMA. An example of the efficiency of this filter is on the alignment of the readings of John 5:35 in respectively manuscripts Palatinus and Corbeiensis Secundus:

This alignment presents three significant difficulties: (1) the variation in tense between the perfect (fuit) and the imperfect (erat); (2) the transformation between the infinitive passive “to be exulted” (exultari) and the infinitive active “to exult” (exultare); and (3) the reinforcement of lucens by the addition of lucet. The results given by ALMA and SimAlign are shown in Table 9.

Table 9.

Token alignments detected by ALMA and SimAlign on case 1 (Latin).

Table 9 Long description

The table consists of three columns: Algorithm, Matches, and Unmatches.

* The first row for the A L M A algorithm lists 14 matches including: ille double-arrow ille, fuit double-arrow erat, lucerna double-arrow lucerna, ardens double-arrow ardens, et double-arrow et, lucens double-arrow lucens, uos double-arrow uos, autem double-arrow autem, uoluistis double-arrow uoluistis, ad double-arrow ad, oram double-arrow horam, exultari double-arrow exultare, in double-arrow in, lumine double-arrow luce, and eius double-arrow eius. The Unmatches column for A L M A contains: dash double-arrow lucet.

* The second row for the SimAlign algorithm lists 15 matches including: ille double-arrow ille, fuit double-arrow erat, lucerna double-arrow lucerna, ardens double-arrow ardens, et double-arrow et, lucens double-arrow lucet (highlighted in red), lucens double-arrow lucens, uos double-arrow uos, autem double-arrow autem, uoluistis double-arrow uoluistis, exultari double-arrow exultare, ad double-arrow ad, oram double-arrow horam, in double-arrow in, lumine double-arrow lucet, lumine double-arrow luce, and eius double-arrow eius. The Unmatches column for SimAlign is listed as none.

We can thus see the ability of both algorithms to handle the first two difficulties, making them superior to approaches based on sequence similarity: the algorithms were able to handle the grammatical transformations from perfect/imperfect and between active/passive. However, SimAlign, which is based solely on semantic proximity, overmatches lucens with lucet, which come from the same semantic field. ALMA avoids this pitfall by applying a uniqueness filter, which selects the word with the most semantic correspondence: even though lucet and lucens have the same lemma, the exact match will always return a more substantial similarity in terms of embedding and will therefore be chosen by ALMA.

Another case study showing similar behavior is the alignment of two manuscript readings of the verse of John 1:13, respectively, in codices Bezae and Vaticanus:

These two verses present the following difficulties: (1) the addition of the subject οἱ (they) in the Vaticanus; (2) the addition of οὐδὲ ϑελήματος ἀνδρός (nor from the will of man) in the Bezae, near to οὐδὲ ἐκ ϑελήματος σάρκος (nor of the will of the flesh), presenting the challenge for the algorithms of not overmatching by matching together the two identical οὐδὲ and ϑελήματος; (3) the spelling variant between ἐγεννήϑησαν and ἐγενήϑησαν, presenting a single ν in the Vaticanus; and (4) the spelling variant between ἐμάτων and αἱμάτων. The results of the two matches are in Table 10.

Table 10.

Token alignments detected by ALMA and SimAlign on case 1 (Greek).

Table 10 Long description

The table consists of three columns: Algorithm, Matches, and Unmatches.

* The first row for the ALMA algorithm shows eleven matches where Greek tokens are linked by double-headed arrows to identical or near-identical tokens, such as ouk to ouk, ex to ex, and ematōn to aimatōn. The unmatches column for ALMA lists four instances where a token is linked to a dash or vice versa, including the tokens thelēmatos, andros, oude, and oi.

* The second row for the SimAlign algorithm shows a similar set of matches, but includes a specific sequence highlighted in red text: oude linked to oude, thelēmatos linked to aimatōn, and thelēmatos linked to thelēmatos. The unmatches column for SimAlign lists two instances: andros linked to a dash and a dash linked to oi.

While both algorithms successfully handled the addition of the subject οι, the orthographic variation of ἐγεννήϑησαν and of αἱμάτων, the SimAlign algorithm performs an overmatch by considering the doublet οὐδὲ ϑελήματος as a match with the other οὐδὲ ϑελήματος, as well as matching αἱμάτων with ϑελήματος. ALMA performs the alignment without error: the additional expression οὐδὲ ϑελήματος has a weaker semantic proximity to the other οὐδὲ ϑελήματος and is therefore filtered by ALMA’s uniqueness filter, while SimAlign once again performs an overmatch.

Case 2 – POS matcher

Another feature of ALMA compared to SimAlign is the grammatical filter, which enables matching tokens with the same POS and grammatical functions, even if they have different lemmas, provided their semantic similarity exceeds the predefined threshold. SimAlign lacks this feature and relies solely on contextual proximity to determine matches. An example of such a phenomenon can be found in the alignment of the two verses from John 9:5 (Palatinus vs. Corbeiensis Secundus):

This alignment presents three difficulties: (1) the variant between dum and cum, which must be matched together; (2) the use of synonyms for seculum and mundus; and (3) the addition of the demonstratives hoc and huius, which should be left unmatched as additions.Footnote ⁶ The results of the alignments are available in Table 11.

Table 11.

Token alignments detected by ALMA and SimAlign on case 2.

Table 11 Long description

The table consists of three columns: Algorithm, Matches, and Unmatches.

* Row 1: A L M A algorithm.

- Matches: dum double-headed arrow cum, sum double-headed arrow sum, in double-headed arrow in, seculo double-headed arrow mundo, lux double-headed arrow lux, sum double-headed arrow sum, seculi double-headed arrow mundi.

- Unmatches: dash double-headed arrow hoc, dash double-headed arrow huius.

* Row 2: SimAlign algorithm.

- Matches: dum double-headed arrow cum, sum double-headed arrow sum, in double-headed arrow in, seculo double-headed arrow hoc (highlighted in red), seculo double-headed arrow mundo, lux double-headed arrow lux, sum double-headed arrow sum, seculi double-headed arrow huius (highlighted in red), seculi double-headed arrow mundi.

- Unmatches: none.

We observe an overmatch in SimAlign when mapping demonstratives to their referents, as seculo $\leftrightarrow $ hoc and seculi $\leftrightarrow $ huius match together. At the same time, ALMA presents the expected perfect match, thanks to part-of-speech matching: Since hoc and huius have no common lemma between the target sentence and the source sentence, and have no token with the Part-of-Speech DET in the source sentence, they remain unaligned, which is the expected behavior.

Error analysis

Quantitative analysis of errors

Figure 6 shows the statistical distribution of errors made by ALMA for all aligned verses. We can see the strong results of 100 percent median success, with an interquartile range of [ $ Q_1$ =88.88, $Q_3$ =100] for Latin and [ $Q_1$ =93.33, $\!\! Q3$ =100] for Greek, showing ALMA’s good overall performance on the datasets considered, with more than half of the verses perfectly aligned in both languages considered in this study. The performance gap between Greek and Latin can be explained, as with the median scores, by the greater freedom in token choice in the Latin corpus compared to the Greek one.

Figure 6.

Statistical distribution of AS for ALMA with the best-performing configuration.

Figure 6 Long description

A box plot with a horizontal x-axis labeled Language and a vertical y-axis ranging from 60 to 100.

On the left, the Latin category features a blue box. The top of the box and the median line are both at 100. The bottom of the box is at approximately 89. A vertical whisker extends downward to a horizontal cap at approximately 73. Below this cap, a series of eight circular outliers are plotted vertically between 72 and 57.

On the right, the Greek category features an orange box. The top of the box is at 100, and the median line is slightly lower, around 99. The bottom of the box is at approximately 94. A vertical whisker extends downward to a horizontal cap at approximately 84. Below this cap, a series of ten circular outliers are plotted vertically between 83 and 60.

Despite these good performances, this figure highlights several verses with degraded performance, as we observe 84 at or below the first-quartile threshold of $Q_1=88.88$ for Latin, and 75 at or below the quartile $Q_1=93.33$ for Greek. The statistical characteristics of these lower-performing subsets are presented in Table 12, and practical cases from among them are qualitatively analyzed in the “Qualitative analysis” section.

Table 12.

Characteristics of lower-quartile subset (at or below $Q_1$ ) for Latin and Greek and of the complete datasets.

Table 12 Long description

The table consists of five columns: Metric, Lower-scoring subset (Lat), Full dataset (Lat), Lower-scoring subset (Gr), and Full dataset (Gr).

* Total number: Lower-scoring subset (Lat) is 84; Full dataset (Lat) is 285; Lower-scoring subset (Gr) is 75; Full dataset (Gr) is 285.

* Average A S: Lower-scoring subset (Lat) is 81.27 with three asterisks; Full dataset (Lat) is 94.12; Lower-scoring subset (Gr) is 85.29 with three asterisks; Full dataset (Gr) is 95.77.

* Reference verse length (characters): Lower-scoring subset (Lat) is 93.19; Full dataset (Lat) is 92.31; Lower-scoring subset (Gr) is 98.56; Full dataset (Gr) is 95.93.

* Difference in verses length (characters): Lower-scoring subset (Lat) is minus 1.74; Full dataset (Lat) is minus 2.92; Lower-scoring subset (Gr) is 1.47; Full dataset (Gr) is 1.15.

* Number of differing tokens: Lower-scoring subset (Lat) is 5.23 with three asterisks; Full dataset (Lat) is 3.79; Lower-scoring subset (Gr) is 2.29 with two asterisks; Full dataset (Gr) is 1.75.

The study of the characteristics of outliers shows that the lower-scoring subset contains a greater average number of differing tokens. This pattern suggests an association between token variation and lower alignment performance: outliers thus have a higher value of different tokens between the reference verse and the compared verse than in the data considered as a whole. Figure 7 illustrates this trend, showing a decrease in AS as the number of different tokens between the reference verse and the compared verse increases. This trend is particularly evident for Latin, where we see a very sharp decline in AS for verses where the number of different tokens between the reference and the target exceeds 9.

Figure 7.

Evolution of the AS for ALMA with the best-performing configuration as a function of differing token count.

Figure 7 Long description

The X-axis is labeled Total Differing Pairs in Reference, ranging from 0 to 16. The Y-axis represents a percentage scale from 65 to 100.

Two data series are plotted:

* l a t (Latin): Represented by a solid blue line with circular markers. It begins at approximately 98 percent at X equals 0 and shows a gradual downward trend to 91 percent at X equals 8. It then drops sharply to 64 percent at X equals 12 before a slight recovery to 68 percent at X equals 16. A light blue shaded region indicates variance, which widens significantly between X equals 8 and 10.

* g r c (Greek): Represented by a dashed orange line with X markers. It starts at 98 percent and follows a similar downward trajectory as Latin until X equals 6, where it dips to 90 percent. It then rises back to 96 percent at X equals 8. An orange shaded region shows high variance between X equals 4 and 7, forming a wide diamond shape.

The background features a light gray grid.

However, as shown in Table 12, the difference in length between the reference and the target has little impact on the AS, indicating that ALMA can effectively capture additions that lead to length differences thanks to its uniqueness filter. The adverse effects of a high rate of differing tokens are therefore not related to additions, but rather to word substitutions.

Qualitative analysis

We conclude this analysis of errors by studying two case studies, presenting typical examples of errors observed in ALMA.

Case 1 – Syntactic variants: One of ALMA’s common errors involves not aligning several words together, due to an overly strict uniqueness pruning filter that does not account for changes in syntactic structure. This is the case, for example, when aligning John 15:18 in Latin (Corbeiensis Secundus vs. Palatinus):

1. si seculum uos odis scitote quoniam me primo odiit

If the world hates you, know it hated me first [scribal error, odis translated as odit].

2. si mundus uos odit scitote quia me priorem odio habuit

If the world hates you, know it had hate for me first.

The challenge of this alignment is to (1) detect the synonymic change between seculum and mundus and (2) detect the syntactic structure change between a simple verbal construction (to hate me) and a periphrastic expression (to have hate for me), which requires matching the overall semantic unit rather than a word-for-word alignment. The result of the matching by ALMA is presented in Table 13.

Table 13.

Token alignments detected by ALMA.

Table 13 Long description

The table consists of three columns: Algorithm, Matches, and Unmatches.

* The Algorithm column contains a single entry: ALMA.

* The Matches column lists nine pairs of Latin tokens connected by double-headed horizontal arrows: si to si, seculum to mundus, uos to uos, odis to odit, scitote to scitote, quoniam to quia, me to me, primo to priorem, and odiit to habuit.

* The Unmatches column contains a single entry in red text: a hyphen followed by a double-headed horizontal arrow pointing to the word odio.

We see that ALMA correctly detects the synonymous change between seculum and mundus, but fails to detect the variation in syntactic construction between odiit and odio habuit, as ALMA is unable to detect the grammatical differences between the two constructions. The model’s token-level embedding approach struggles to recognize these syntactically distinct yet semantically equivalent structures, and the uniqueness filter prunes them too strictly. SimAlign, however, with its more lenient semantic-similarity requirements due to its more iterative matching using the itermax algorithm, produces the correct matching.

This phenomenon, which is common in the Latin dataset (another example can be the construction of potuit cognoscere [could have known]/sciuit [knew]), explains several of the errors encountered during the evaluation process, and further work on ALMA will aim to improve this aspect of the algorithm.

Case 2 – Complex synonymy substitutions: Another problematic case, again involving the non-detection of synonymous multi-token groups, is found in the alignment in the Greek text of John 4:33 in Codices Bezae and Vaticanus:

(1) ἔλεγον δὲ ἐν ἑαυτοῖς οἱ μαϑηταί μή τις ἤνεγκε αὐτῷ φαγεῖν

The disciples were asking among themselves: has someone brought him to eat?

(2) ἔλεγον οὖν οἱ μαϑηταὶ πρὸς ἀλλήλους μή τις ἤνεγκεν αὐτῷ φαγεῖν

The disciples were asking between themselves: has someone brought him to eat?

The difficulty in matching this phrase is the synonym used to render the expression “among themselves,” with Bezae employing the construction ἐν ἑαυτοῖς and Vaticanus the construction πρὸς ἀλλήλους, which should be matched together. The results of ALMA are displayed in Table 14.

Table 14.

Token alignments detected by ALMA.

Table 14 Long description

The table contains three columns.

1. Algorithm: The row identifies the algorithm as A L M A.

2. Matches: This column lists ten pairs of Greek words connected by a double-headed horizontal arrow. The pairs are:

- ελεγον and ελεγον

- δε and ουν

- οι and οι

- μαϑηται and μαϑϑηται

- εαυτοις and αλληλους

- μη and μη

- τις and τις

- ηνεγκε and ηνεγκεν

- αυτω and αυτω

- φαγειν and φαγειν

3. Unmatches: This column contains two entries in red text separated by a comma.

- The first entry shows the Greek word εν followed by a double-headed horizontal arrow pointing to a dash.

- The second entry shows a dash followed by a double-headed horizontal arrow pointing to the Greek word προς.

We see from these results that ALMA fails to align “ἐν” with “πρός” because the model relies primarily on direct lexical distributional similarities, without sufficient ability to recognize semantic equivalence between distinct prepositions in contextually equivalent constructions. Even though they share the same POS, the similarity filter prunes them too aggressively, leading to a missed match: the chosen approach, therefore, fails to detect the same discursive function of two different prepositions, which, depending on the context, may have synonymous or antonymous meanings (“ἐν” and “πρός”). A possible approach to mitigating this issue is to train the embedding models on aligned parallel corpora.

Current methodological limits of our approach

Resource requirements

Another limitation of the ALMA algorithm that we can highlight is its higher computational resource requirements, both because of the fine-tuning phase on the input dataset, and when executing the algorithm itself, due to the six different layers that the input texts must pass through before obtaining an alignment, some layers being particularly resource-intensive, such as parsing all the words available in the input verses. ALMA’s strength in combining the advantages of existing alignment algorithms, however, automatically entails additional costs in terms of resources and execution time.

Table 15 compares the average time of each algorithm for one alignment of 10 Latin tokens. The computations were averaged over 100 trial runs.

Table 15.

Execution time per algorithm with the best hyperparameters per algorithm class for aligning 10 tokens.

Table 15 Long description

The table consists of two columns: Algorithm and Average time in m s.

* Dekker: 0.4 m s.

* P O S: 65.9 m s.

* SimAlign: 82.3 m s.

* A L M A: 215.7 m s.

We can therefore see in this table the substantial increase in execution time when using ALMA, which combines several approaches, rather than a single approach: ALMA results in approximately 539 times the execution time of CollateX’s implementation of Dekker’s algorithm. However, this significant increase in execution time must be put into perspective in relation to its application. With 215.7 ms of alignment time required, it takes 1 minute 45 seconds and 1 minute 49 seconds, respectively, to align all the Latin and Greek tokens we considered in this study. We thus consider this a reasonable waiting time, given how long the alignment process can take when performed by hand and how faster algorithms like CollateX lead to poorer performance.

Even when putting in perspective the higher computational costs with the more modest gains compared to the SimAlign approach, as the absolute improvement over the best SimAlign configuration remains moderate (+1.78 AS for Latin and +1.03 for Greek), the additional time resource requirements of ALMA are justified for use-cases where alignment quality is paramount, as in editorial work, as it reaches a median AS of 100 percent on both datasets. For such precise and time-consuming tasks as witness alignment, even marginal gains in performance can save the philologist days of post-correction work. ALMA is thus best understood as a tool targeting fine-grained philological accuracy rather than large-scale, speed-critical alignment that can tolerate a noisier output.