Plain language summary
This article discusses large language model (LLM)-assisted methods for cleaning a large dataset about handwritten copies of English-language poetry created between 1475 and 1700. It shows that these methods were not only helpful for their intended purpose of cleaning the data but also provided new insights into the poems themselves, particularly when the LLMs struggled. Specifically, the LLM-assisted methods struggled where early modern poetics were at odds with the modern prose on which general models are trained. Fundamental early modern practices, such as imitation and translation, and unique poems posed particular challenges for the computational methods used. The essay concludes with a reflection on concerns shared by early modern studies and literary theory as well as linguistics and natural language processing.
The structure meant is a structure of meanings, evaluations, and interpretations; and the principle of unity which informs it seems to be one of balancing and harmonizing connotations, attitudes, and meanings
$\ldots $
. The unity is not a unity of the sort to be achieved by the reduction and simplification appropriate to an algebraic formula.
(Brooks Reference Brooks1968, 159)
Introduction
William Caxton introduced printing with moveable type to England in 1476, but this technological innovation did not slow the circulation of poetry in manuscript. Instead, the two systems of literary transmission worked competed with and informed one another for at least 200 more years (and, of course, print still has not fully superseded handwriting). Publishing in manuscript offered early modern literary authors somewhat greater control than print over their works’ initial circulation, but texts could – and often did – escape their intended readership to reach a wider audience (McCarthy Reference McCarthy2020, 12–14; Smith Reference Smith2014, 1–2). Traditionally, early modern manuscript verse is said to have been circulated in four main contexts: the universities, the court, the Inns of Court, and gentry and aristocratic households (Marotti Reference Marotti1995, 30–44). In many cases, these contexts intersected, as when young men moved from the universities to the Inns of Court or returned to their family households (McCarthy Reference McCarthy2018, 664); in others, it is unclear how texts traveled from one setting to another, as when a poem by John Donne usually associated with the Inns reached a young woman in West Midlands (Burke Reference Burke and Knoppers2009, 60; see also Marotti Reference Marotti2014). Furthermore, there was a significant degree of overlap between print and manuscript audiences (Love, 1998). Increasingly, scholars recognize that, as Steven W. May and Arthur F. Marotti put it, “Most scribal culture
$\ldots $
must have been produced outside the centers manuscript scholars have tended to credit with virtually all of it” (May and Marotti Reference May and Marotti2014, 7). When these burgeoning groups of literary readers got their hands on new poems, there was no telling what they might do with them.
One thing early modern readers particularly liked to do was to transcribe poems into small, personalized collections that scholars now call manuscript miscellanies (Eckhardt and Smith Reference Eckhardt, Smith, Eckhardt and Smith2014, 2). Compilers of manuscript miscellanies did not just feel free to alter the poems they copied. Imitating, adapting, and reusing other people’s ideas was a large part of the point. This variability is precisely what makes manuscript circulation so interesting for literary historians – and so confounding for even state-of-the-art computational methods. The text, the form, and the material presentation of a given poem were all liable to change in the course of its circulation in manuscript.
The European Research Council-funded project “STEMMA: Systems of Transmitting Early Modern Manuscript Verse, 1475 to 1700” computationally models the manuscript circulation of early modern poetry to gain macro-level insights into the overlapping contexts in which it was written, read, and copied. The team was given permission to incorporate six of the most comprehensive existing datasets about early modern English manuscript culture into a bespoke relational database developed to support advanced analyses. Reusing this data offered the practical advantage of giving the team a lot of data to work with without having to conduct extensive in-person cataloguing: together, these resources include nearly one million records related to the circulation of English verse. However, these underlying datasets differ in formats and metadata schemes as well as in their intellectual scopes and emphases, and reconciling multiple databases is inevitably a complex undertaking. The team has therefore also developed new computational methods to address the specific challenges presented by this dataset.
This essay argues that early modern manuscript poetry presents specific formal and historical challenges to large language models (LLMs) and other transformer-based models, which are pretrained on very different material.Footnote 1 Problems lie at the intersections of the cultural values, prosodic features, and material practices that are inseparable from literary form in the early modern period (McCarthy Reference McCarthy2020). Consequently, computational methods that were effective for a significant amount of STEMMA data often fell flat on smaller subsets of it. After briefly introducing STEMMA’s data reconciliation methods, I present three case studies showing how early modern poetry resists methods based on more general LLMs: imitative lyrics, Psalm translations, and unique short poems. These case studies, though far from exhaustive, point up areas where models trained on modern prose find themselves caught between meaning, form, and history. These challenges, in turn, shed new light on old questions still of interest in literary studies, including, perhaps surprisingly, how LLMs’ current limitations might give us new purchase on the traditional practice of close reading as well as longstanding questions about the nature of poetry itself.
Methods and related work
The STEMMA dataset
To facilitate STEMMA’s ultimate goal of modeling the circulation of manuscript poetry as a large-scale network, the project began by combining and reconciling five of the most authoritative sources of relevant data: the Catalogue of English Literary Manuscripts (CELM); the Folger Union First-Line Index of English Verse; the master list of poems from DigitalDonne; Joshua Eckhardt’s Index of Selected English Poetry Manuscripts, 1590–1660; and 170 previously unpublished spreadsheets created for the project “RECIRC: The Reception and Circulation of Early Modern Women’s Writing, 1550–1700.” Data from a sixth source, the Perdita Project, could not be imported programmatically and will therefore be entered into the STEMMA database manually. Together, these resources included nearly one million records related to the circulation of English verse, including metadata records describing people, documents, texts, and literary works.
Although each of these resources sheds light on an aspect of early modern manuscript culture, they were created by different people with different research objectives in mind. CELM and the Folger Union First-Line Index were developed as finding aids – that is, as guides to assist researchers in locating poems in physical documents. Digital Donne and the Index of Selected English Poetry Manuscripts, 1590–1660, were intermediate work products prepared to support print publications. Perdita and RECIRC were also created in the course of large-scale, collaborative grant-funded projects focused on recovering women’s compilation practices, and the reception of women’s writing. Consequently, each dataset’s scope, principles of inclusion, and transcription practices varied greatly. There was also a significant degree of duplication. Finally, because none of these projects were developed with computational modeling in mind, no consideration was given to making them computationally tractable at the time of their creation. In many cases, authorities had not been used or created, dates were entered as strings (often inconsistently or with errors), and textual data were missing, incomplete, or ambiguous. Crucially, though, none of these sources includes full text or images.
A Readme file, including a description of the dataset and sample records, has been deposited in the project’s Zenodo community: https://doi.org/10.5281/zenodo.18247735. All relevant code is in a GitHub repository (currently set to private) and will be released in due course.
Data reconciliation, curation, and cleaning
The collation of this material is not the primary aim of this project but rather preliminary to advanced literary and quantitative analyses. Nevertheless, the overlaps (sometimes substantial) between our inherited datasets meant that there was a large proportion of data that needed to be merged, teased apart, or otherwise disambiguated to be interoperable. Unfortunately, there is not yet an existing authority list that would support this work – indeed, the datasets themselves are often treated as authorities, although they were not designed that way. This presented a technical challenge because off-the-shelf reconciliation tools standardize data, either by editing or resolving to a pre-existing list of authorities, rather than preserving the small details where literary historical evidence usually inheres. Therefore, in an effort to balance precision with efficiency, the team experimented with a series of staged LLM-assisted reconciliation pipelines to create unique identifiers without requiring the alteration of existing data. These pipelines are described in detail in a recent article in the Journal of Open Humanities Data (see McCarthy Reference McCarthy2026). The pipelines were useful in addressing about 70 percent of the imported material. However, a remainder of about 30 percent resisted even our more sophisticated computational efforts. Systematic benchmarking is beyond the scope of the current project but is a priority for follow-on funding applications.
Source data record from the Folger Union First-Line Index for the copy of John Donne’s “The Storm” in BL Stowe MS 962, ff. 55v–56v.
Figure 1 Long description
Fourteen structured data fields are shown, each containing bibliographic metadata (e.g. unique ID, author, title, date, and shelfmark) for the copy of John Donne’s “The Storm” in BL Stowe MS 962, ff. 55v–56v.
To support the work of reconciling and deduplicating relevant data, creating authorities, and assigning unique identifiers, the STEMMA team worked with contractors from Ember Ltd. to import this data into a single PostgreSQL database. Although STEMMA aims to produce Linked Open Data by the conclusion of the project, the PI and developers agreed that working initially in a relational database would be easier for a team made up primarily of specialists in literature and manuscript studies with a mixed level of digital humanities expertise. Preparing inherited data related to Manuscript Collections, Manuscripts, People, and Repositories was fairly straightforward through manual curation in spreadsheets and simpler computational methods like regular expressions.Footnote 2 However, at the time of import, STEMMA had approximately 198,000 Manuscript Items, or records of transcription, in total. Moreover, there was not yet an authority that would allow us to identify duplicates or determine how many Works were represented – and because of the variability between data sources, there was no straightforward way to match records. It was important to preserve old spelling and other potentially meaningful scribal evidence wherever possible for use in later work packages, which ruled out modernization and string searching.
To complete this work, STEMMA developed a series of staged LLM-assisted reconciliation pipelines. We began with min-wise independent permutations locality sensitive hashing (LSH or minhashing), progressed to vector embeddings for semantic similarity search, and finally trained a lightweight agentic model to offer recommendations based on several imperfect textual signals in conjunction with expert data retrieved from our dataset. Vector embeddings were created first with OpenAI’s text-embedding-3-small model and later with Enrique Manjavacas and Lauren Fonteyn’s MacBERTh, a domain-adapted model for early modern English (see Manjavacas and Fonteyn Reference Manjavacas and Fonteyn2022; Manjavacas Arevalo and Fonteyn Reference Manjavacas Arevalo, Fonteyn, Hämäläinen, Alnajjar, Partanen and Rueter2021;). These progressively more sophisticated machine learning techniques initially helped the team make rapid progress in deduplicating Items and identifying Works. However, there was a resistant remainder of this material that could not be addressed using these methods, and further computational work seemed to offer diminishing returns for our efforts.
At minimum, an imported Manuscript Item record includes a reference to a Manuscript as well as a first line (see Figure 1).
Most Manuscript Item records also include last lines, and some include second and penultimate lines. Crucially, none of the sources offer full text (except sometimes incidentally for poems of fewer than four lines), although there is a work package that will focus on a restricted corpus of manuscripts. STEMMA’s data model also allows us to preserve and record additional data not present in all sources, including authors, misattributions, dates, number of lines, reference numbers, and free-text notes (see Figure 2). Additional fields such as “All lines cleaned” and “Lang” were algorithmically generated to support data cleaning. Dates display as strings but are linked to a separate “Date Ranges” table as nine integer and binary entities to support later computation.
A merged record showing all available data about the same transcription.
Figure 2 Long description
The merged record includes significantly more data than any one source record, stored in 28 fields. Hyperlinks enable users to navigate to records for the relevant Work, Manuscript, and Author.
Each pipeline stage used slightly different textual signals, which impacted results. For the LSH process, we had parameters for each line individually as well for titles and headings. The vector embeddings approach allowed the team to choose one of two scopes: “line,” which compared vector representations of all available lines, and “title-heading-and-lines,” which included additional editorial and/or scribal data. The final model incorporated the two sets of embeddings, data retrieved from the database (e.g., dates and names), and calculated values for lengths (in words and characters), phonetic overlap, rarity, and proximity to create a probability estimate, P(same_work), for each candidate pair. These probabilities were split into three bands to facilitate automated decision-making about auto-linking, generating new works, and flagging for manual review. Initially, only the review band was passed through an LLM agent that retrieved candidate matches, suggested a decision, and offered a rationale created using gpt-4.1-2025-04-14. However, after noticing a small number of mismatches between reranker and LLM validation results, the team decided not to allow the agent to alter the data directly and instead reviewed all of the LLM agent’s outputs.
Results and discussion
I have shown elsewhere that the staged pipeline approach was very useful for making rapid progress on a large portion of STEMMA’s inherited data. In combination with manual cleaning, the research team reduced the original 195,000 Manuscript Items to a current total of 89,279, and they have identified (at present) 48,107 Works. These numbers will undoubtedly continue to fluctuate over the course of the project, but even these interim figures provide a useful indication of the scope and extent of the cleaning work.
Terminal application showing intermingled transcriptions of “The Passionate Shepherd to His Love” and “The Bait.”
Figure 3 Long description
This is a screenshot of a terminal application, featuring black text on a white background. The applications shows results of a fuzzy search and allows users to perform automated merging.
Nevertheless, the team’s experiments also offered useful insights into the limitations of using LLM-assisted methods, especially those applying general models, to work with specific instances of early modern poetry. Below, I focus on a series of three case studies where specific features of early modern poetics resisted STEMMA’s staged machine learning pipelines. These may point up opportunities for future computational research, but they also reveal where early modern poetic language differs empirically from the kinds of material on which current models were trained.
LSH and poetic imitations
Imitation is fundamental to early modern poetics. Whereas post-Romantic poetics emphasize personal experience and aesthetic originality, sixteenth- and seventeenth-century English educational and artistic practices privileged reliance on earlier models and authorities (Dolven Reference Dolven, Chaouli, Lietz, Müller-Tamm and Schleusener2021, 46–7; Woudhuysen Reference Woudhuysen1996, 162). G. W. Pigman writes, “From Petrarch’s sonnets to Milton’s epics a major characteristic of Renaissance literature is the imitation of earlier texts, and the Renaissance contains a vast and perplexing array of writings on the theory and practice of imitation” (Pigman Reference Pigman1980, 1). Schoolboys were trained to master, memorize, and then spontaneously compose verses aloud in the style of Ovid and Virgil (Riggs Reference Riggs2004, 53). Though they differed in their specific recommendations, classical rhetorics and vernacular poetic treatises alike praised imitation as a key means to poetic creativity. This imperative, in the social context of manuscript verse circulation, encouraged copyists to adapt their copy texts as co-creative acts, big and small (Marotti Reference Marotti1995, 9–10).
The prevalence of imitation in early modern poetry presented a particular challenge for the LSH method. As a probabilistic method for locating nearest neighbors through small, overlapping substrings called “shingles” (Jafari et al. Reference Jafari, Maurya, Nagarkar, Islam and Crushev2021, 1–2, 10–8; Marçais et al. Reference Marçais, DeBlasio, Pandey and Kingsford2019, 127), LSH could not help but struggle with well-executed poetic imitations.
“The Passionate Shepherd to His Love,” now commonly attributed to Christopher Marlowe, and John Donne’s pastiche “The Bait” were indistinguishable to our earliest process (see Figure 3). Readers unfamiliar with these poems may find it helpful to see modernized texts of the first, second, penultimate, and last lines side by side:
Come live with me and be my love,
And we will all the pleasures prove,
$\ldots $
If these delights thy mind may move,
Then live with me, and be my love.
(Marlowe Reference Marlowen.d., emphasis mine)
Come live with me, and be my love,
And we will some new pleasures prove
$\ldots $
That fish, that is not catch’d thereby,
Alas, is wiser far than I.
(Donne Reference Donnen.d., emphasis mine)
One can readily see that Manuscript Items that included penultimate and last lines would be easy to disambiguate using this method. However, those that only had a first line – again, the only piece of data STEMMA is certain to have for a given Manuscript Item – would be indistinguishable. Even those with a second line would be difficult to differentiate in the absence of the concluding lines, as these vary in only two words. This problem is further compounded by the high level of scribal variation that marks texts of the “The Passionate Shepherd” (see Woods Reference Woods1970, 27–9).
Thankfully, these poems are both canonical enough to be well documented, but they are indicative of the unique challenges posed by early modern poems. At present, STEMMA records nine copies of “The Passionate Shepherd” and 45 of “The Bait.” Additionally, the database notes 12 copies of Sir Walter Ralegh’s “The Nymph’s Reply to the Shepherd,” and “The Passionate Shepherd” continued to be imitated and parodied by numerous other authors in the sixteenth and seventeenth centuries (Dolven Reference Dolven, Chaouli, Lietz, Müller-Tamm and Schleusener2021, 45; McCarthy Reference McCarthy2020, 86–6; Woods Reference Woods1970, 30). However, the first lines of these other poems differ enough from “The Passionate Shepherd” that the LSH approach could easily disambiguate them.
Vector embeddings, transcriptions, and translations
The next phase of our work moved from the probabilistic LSH method to vector embeddings, using cosine similarity clustering at a 0.15 distance threshold. This method excelled with works that had a number of small variants. For instance, Donne’s most-transcribed poem, “Break of Day,” is currently recorded 71 times in the STEMMA database, where it is represented by 62 unique strings (see also McCarthy Reference McCarthy2018, 677–84):
’Tis trew, tis day; what though it bee?
’Tis true ’tis break of day, what though it be?
’Tis true ’tis day, what though it be.
’Tis true, ’tis Day; What though itt bee?
’Tis true, ’tis daye what though it bee,
I true t’is day, what though it be?
In truth its day /wt though it bee
It’s true, tis day, what though it bee?
Loue, it is day: what if it bee
T’is true ’tis day what if itt bee,
T’is true t’is day what though it bee
T’is true t’is day what though itt bee
t’is true t’is day wt though it bee
T’is true t’is day; what if it bee?
T’is true tis Day, what thoughe it bee,
T’is true, t’is daie, what though itt bee
T’is true, t’is daie; what thoughe it bee
T’is true, t’is daye; what though it bee
T’is tyme, t’is day, what though itt be
Tis thou kills day wt if it bee
Tis time tis day, what though it bee?
Tis time, t’is day, what though itt be?
Tis trew tis day which though it be
Tis trew, ’tis day wt though it be?
Tis true ’tis day what though it be
Tis true ’tis day what thought it bee
Tis true ’tis day, what though it be
Tis true ’tis day, what though it be?
Tis true t’is day what though it be
Tis true t’is Day, what thoughe it bee?
Tis true this day what if it bee?
Tis true this day, what though it bee
Tis true tis daie what though it bee
Tis true tis day what if it be
Tis true tis day what though it bee
Tis true tis day what though it bee
Tis true tis day what though it bee,
Tis true tis day what though it bee?
Tis true tis day, what though at bee
Tis true tis day, what though it be,
Tis true tis day, what though it bee
Tis true tis day, what though it bee?
Tis true tis day, wt though it be?
Tis true tis daye what though yt bee
Tis true, ’tis day, what though it be
Tis true, ’tis day, what though it be?
Tis true, ’tis day, what though itt bee?
Tis true, ’tis day; what though it be
Tis true, ’tis day; what though it be?
Tis true, tis day and if it bee
Tis true, tis day what though it be?
Tis true, tis day, what if itt bee
Tis true, tis day, what though it be?
Tis true, tis day, what though itt bee
Tis true, tis day, what though yt bee
Tis True, tis day; what though it bee
Tis true, tis day; what though it bee?
Tis true, tis day. What though it bee?
Tis true, tis daye, what though it bee,
Tis true; tis day what though it be?
Tis true; tis day, what though it bee
Tis trwe, tis day what though it be?
Many of these variants are small differences in spelling and punctuation or slight changes (“if” vs. “though,” “t’ is true” vs. “in truth”). Others, like “Loue, it is day: what if it bee” alter the meter (in this case replacing the opening iamb with a trochee) but mostly preserve the sense. But others are so radically different that even a human reader might question whether they were meant to represent the same work, such as “Tis thou kills day wt if it bee.” This final variant appears in British Library Sloane MS 542, where “Break of Day” is conflated with Donne’s “Stay, O Sweet, and do not rise” and headed “at her rising.”
Notably, MacBERTh seemed to outperform the OpenAI model; although the models’ performance on STEMMA’s data have not yet been systematically benchmarked, the team observed that it was, as one would expect, more effective in the face of orthographical variation. We also found that the “line” scope provided better results; while “title-heading-and-lines” might seem to offer more information, much of this was noise. Copyists tended to eschew titles in favor of numbers for kinds of sequential works, like sonnets and epigrams, and miscellany compilers creating collections of short texts for their own use sometimes use vague, attenuated, or generic headings like “Another” (see also Marotti Reference Marotti1995, esp. 327–30). Differences across datasets also complicate matters, as the practice on RECIRC was to supply a first line where a title was missing (marked with a binary flag in the original data source), whereas other data sources had their own practices, such as supplying standard editorial titles or short forms.
The vector embedding approach was not effective on translations, another important mode of early modern literary production. The problem was particularly acute in relation to the Psalms; as Hannibal Hamlin has observed, “no biblical book was translated more often or more widely in the subsequent two centuries [after the Protestant Reformation],” and “[v]irtually every author of the period
$\ldots $
translated, paraphrased, or alluded to the Psalms in their major works” (Hamlin Reference Hamlin2004, 1). Perhaps we should have anticipated this difficulty, but as a team of literary researchers learning about LLMs as we worked, we did not – and our database developer, though very knowledgeable about LLMs, underestimated the popularity of the Psalms among early modern manuscript compilers. At present, 2,501 Manuscript Items contain the word “Psalm” somewhere in their metadata, and each iteration of similarity search attempted to reunite translations that had been disambiguated. For STEMMA’s purposes, it is less important to gather every translation of a particular Psalm than it is to know how many copies of a specific translation exist. In other words, it is the translation itself that is the Work. As A. C. Bradley explains,
Hence in true poetry it is, in strictness, impossible to express the meaning in any but its own words, or to change the words without changing the meaning. A translation of such poetry is not really the old meaning in a fresh dress; it is a new product, something like the poem, though, if one chooses to say so, more like it in the aspect of meaning than in the aspect of form. (Bradley Reference Bradley1961, 19)
While the vector embeddings did exactly what they were meant to do – that is, draw together semantically similar items – this was a rather perverse outcome for the numerous Psalm translations. Because of the complex nature of the texts and their literary, historical, and religious importance, the Psalms are an area where the STEMMA team will need to do a significant amount of manual data curation.
Agentic system and isolates
The first two approaches helped the team associate approximately 54 percent of imported Manuscript Items to Works, leaving a little less than half unassociated. As a manuscript scholar, I had a sense that there would be a long tail of rare or unique Items, but given the levels of variation we found in the first two phases, it was crucial to ensure that Items that should be linked in a single Work had not been missed. For these difficult cases, STEMMA developed a lightweight agentic model, “PRISM: Pipeline for Retrieval Integrated Similarity Mapping,” to find candidate matches for Manuscript Items that remained unassociated with a Work. PRISM combined several textual signals and presented them alongside an algorithmic recommendation and explanatory text (see McCarthy Reference McCarthy2026, 9–10).
The agentic model helped the team match an additional 14 percent of Manuscript Items to Works, but the matches it proposed and rejected provided interesting ways to understand formal differences between the material at the heart of STEMMA and the data on which the models we used were trained. This was even more fascinating because the rationales provided by the explanatory text often bordered on the comical. Reasoning output from LLMs is known to be unreliable and occasionally even misleading for human readers:
CoT reasoning frequently diverges from models’ actual computations of deciding the final answers: small manipulations sway decisions with the CoT merely rationalizing rather than reporting true causes, models silently correct mistakes without reflecting this in their reasoning, and shortcuts are exploited while presenting a reasoning façade. (Barez et al. Reference Barez, Tung-Yu, Arcuschin, Lan, Wang, Siegel, Collignon, Neo, Lee, Paren, Bibi, Trager, Fornasiere, Yan, Elazar and Bengio2025, 6)
Moreover, it can be tempting to anthropomorphize models and attribute thought processes like those we experience as humans (Kambhampati et al. Reference Kambhampati, Valmeekam, Bhambri, Palod, Saldyt, Stechly, Samineni, Kalwar and Biswas2026). But while LLM-generated explanations can call attention to human-understandable features of input that may have impacted the final recommendation, they do not accurately represent the pipeline’s “thinking” (see also Levy, Elyoseph, and Goldberg Reference Levy, Elyoseph and Goldberg2025). In one particularly vexing example, PRISM attempted to match an epitaph “On Sir Thomas Lowell” beginning “All ye nuns in Holywell” to 280 different Items. The poem is known to survive only in a single seventeenth-century copy in Yale Osborn MS fb143, where it is surrounded by numerous other epitaphs.
“On Sir Tho: Lowell” in Osborn MS fb143 (Epitaphs collected 1694), James Marshall and Marie-Louise Osborn collection, Beinecke Rare Book and Manuscript Library, Yale University, p. 22.
Figure 4 Long description
The image shows a double-columned manuscript page that includes several items transcribed in a single early modern hand. One item is struck through but still legible. The poem "On Sir Tho: Lowell" appears in the lower-right quadrant.
The agentic model correctly dismissed these potential matches 279 times and flagged one as uncertain. STEMMA has the poem’s first and last lines via the Folger Union First-Line Index; I have transcribed the two intervening lines from the Beinecke’s digital images (see Figure 4):
On Sir Tho: Lowell
All the Nuns in Holywell
Pray for the Soul of Sir Tho. Lowell
He dy’d the 25 of May at Enfeild, Anno 1524.
As is likely evident, this is not a formally complex poem. Arguably, its most interesting feature is its irregular meter: the first line is seven syllables long and starts with a stressed foot, making it “headless iambic tetrameter” (or acephalous iambic tetrameter), the second line has five iambs, and the final line could be split into two tetrameter lines between “May” and “at,” though it has not been transcribed that way. But PRISM did not have access to any metrics for meter. The poem says little of substance about its subject, a former speaker of the House of Commons, and his date of death precedes the transcription of the poem by more than a century. As Table 1 shows, the explanation texts provide limited insight.
Examples of explanation texts for high-, medium-, and low-confidence matches.
Table 1 Long description
The table consists of three columns and three rows of data.
Row 1:
- Explanation: The unmatched incipit and explicit bear no wording overlap or topical similarity with the matched item. The subject, phrasing, and tone differ (moralizing/abstract reflection versus a local epitaph/prayer).
- Decision: no_match.
- Confidence: High.
Row 2:
- Explanation: Superficial similarity in the vocative opening but strong substantive differences. The matched item is an epitaph for Sir Thomas Lowell, whereas the unmatched item addresses a different audience and is attributed to Sir James Langham.
- Decision: no_match.
- Confidence: Medium.
Row 3:
- Explanation: Both items concern a Thomas, but the unmatched incipit does not match the matched item's incipit or explicit. The unmatched text reads like a devotional couplet invoking a saint rather than Sir Thomas Lowell.
- Decision: Uncertain.
- Confidence: Low.
The uncertain and medium-confidence rejections in Table 1 give some indication of why the system tried to compare these items, although reasons like “They both mention someone named Thomas” may seem silly to a human reader. But why did the first, high-confidence recommendation happen at all?
Further investigation reveals that texts represented by only one or two authentic witnesses in the STEMMA database, like “On Sir Tho: Lowell,” tended to attract an overwhelming number of candidates. The system works as designed, but the lack of strong matches nevertheless means that, for each of these rare items, the reranker must return a relatively weak match, and the LLM is forced to provide an ex post facto justification for this strongest-possible weak match. Five additional poems attracted at least 100 proposed pairings, with 94.2 percent of these matches marked as “no_match” or “uncertain” before human review. These include verses on a range of themes: “No strength of flesh is able to withstand,” “Others can statues cast in breathing brass,” “An aged man which spectacles did use,” “When Philomela with her strains,” and “What mortal man may live but hurt.” Selected explanation texts illustrate their limitations (again with emphasis added):
The unmatched item’s incipit and explicit both foreground transience of life and youth – the same central theme found across the candidate items (e.g. lines about the hour-glass, frailty, death). The date (c. 17th century) also fits the group (c. 1656). There is no direct contradiction (no strongly different subject or phrasing) but the unmatched item lacks any explicit reference to Robert Whetcombe, the King’s School, or a shared heading/signature found in the matched set, so attribution is plausible but not certain.
The unmatched item fits the set: same genre (short numbered epigrams), identical dating (c.1630), and stylistic/incipit pattern – many matched items begin with interrogative openings such as “What is
$\ldots $
” or “Why
$\ldots $
” The heading/numbering (“Epigram 93”) sits sensibly among matched headings (e.g. Epigram 92 and Epigram 98). No strong textual conflicts are present, so it likely belongs to the same work.
The unmatched item is attributed and signed to Robert Herrick and its heading explicitly references “Herrick’s poems,” which aligns with the candidate group (all short Herrick epigrams). There are no strong textual conflicts in incipits/explicits: absence of matching first/last lines is not surprising for different short pieces within the same corpus. The only mild concerns are the lack of any shared opening/closing lines and a slight date discrepancy (unmatched c.1657 vs matched c.1630). These do not outweigh the authorial and genre alignment, so a tentative match with medium confidence is warranted.
Again, it is likely that these justification texts are downstream rationalizations of dubious matches dismissed further up the pipeline. Few human readers would accept such tenuous justifications – domain knowledge is certainly an asset, but most literate non-experts would likely recognize that the “lack of any shared opening/closing lines” would generally indicate a weak or false match. PRISM’s performance thus anecdotally chimes with the recent KRISTEVA benchmark evaluation of LLMs’ literary interpretive reasoning, which showed that “The best performing models trail behind their human counterparts on 10 out of 11 [close reading] tasks” (Sui et al. Reference Sui, Rodriguez, Laban, Che, Nabende, Shutova and Pilehvar2025, 7). Even at the level of metadata, we need not worry that AI will replace human readers quite yet, though it may play a role in supporting them (Zhi et al. Reference Zhi, Long, So and Lee2026, 16).
Conclusions and future work
What began, for STEMMA, as an attempt to solve a practical problem – how to reconcile metadata about early modern poetry without destroying historical and literary evidence – has gradually drawn the project into a conversation about the intersection of form and content that spans disciplines and millennia. Philosophical, rhetorical, and poetic treatises since Plato have oscillated between whether poetry was defined by its form, content, or some combination thereof. The New Critics, following the modernists, seem to have concluded that what defined poetry was the fundamental inseparability of “form” and “content,” or “structure” and “meaning” (see, inter alia, Bradley Reference Bradley1961; Brooks Reference Brooks1968; Eliot Reference Eliot1964). That is, if a poem could be paraphrased effectively, it was not a poem at all.
It might seem futile, then, to attempt to abstract poems, or parts of poems, even further by representing them as 256-dimensional vectors. While embeddings do not numerically signify “meaning” per se, they encode distributional semantic information about how words are used within their training data and allow us to use linear algebraic calculations to understand their relationships; this is true for both encoder–decoder models (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013; see also Pennington, Socher, and Manning Reference Pennington, Socher, Manning, Moschitti, Pang and Daelemans2014) and the transformer-based approaches used in STEMMA. A.C. Bradley quipped, “People say, for instance, ‘steed’ and ‘horse’ have the same meaning; and in bad poetry they have, but not in poetry that is poetry
$\ldots $
” (Bradley Reference Bradley1961, 20). If one were to create vector embeddings for these words, one could quantify and visualize the distance between them, perhaps including other equine terms for comparison. However, if one used a contemporary general model, this approach would be unlikely to account for genre, figurative use, changes in use over time, or other qualitative distinctions.
It is striking, too, that language one associates with literary criticism is finding its way into scholarship about LLMs. Emily M. Bender and Alexander Koller compare LLM output to an eavesdropping octopus’s attempts to communicate with a human, concluding that neither the octopus nor a chatbot can do so because “a system exposed only to form in its training cannot in principle learn meaning” (Bender and Koller Reference Bender, Koller, Jurafsky, Chai, Schluter and Tetreault2020, 5186). For Bender and Koller, the system does not construct meaning so much as imitate what its training data does. But to what extent is this different than Christopher Marlowe’s classmates transcribing, memorizing, reciting, and then finally aping classical models (Riggs Reference Riggs2004, 53–4)? Arguably, the Elizabethan schoolboys also sought to rework existing forms, and reading a large selection of early modern poems shows that some, but far from all, would go on to reach transcendent heights of communication or creativity. To be clear, I do not want to suggest that models are conscious or communicating in the way humans do. Rather, I would argue that not all human utterances demonstrate the kind of unity between subjectivity, expression, and/or meaning that post-Romantic poetics might lead us to idealize.
Even the foundational paper on transformer architecture, “Attention is All You Need” (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), echoes (probably coincidentally) scholarship on poetics. The authors introduced the important concept of “self-attention,” which “relat[es] different positions of a single sequence in order to compute a representation of the sequence” (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017, 2); in other words, self-attention facilitates encoding relationships between linguistic parts and wholes, even as these representations are not direct or isolable.Footnote 3 Similarly, philosopher Peter Lamarque’s definition of poetry drives at a comparably ineffable sense of what poetry is and does:
Whatever a poem might look (or sound) like, if we take it as a poem we bring to it a distinctive kind of attention as broadly determined by the poetic tradition. We assume, for example, that the surface language itself is salient, that its physical textures, sound, rhythm, metre, repetitions, rhymes, are not merely incidental but integral to our attention, to be appreciated in their own right, not just as vehicles for but as identifying conditions of the content conveyed. (Lamarque Reference Lamarque2017, 64 emphasis mine).
What he describes as the “physical textures” of poetry are among the chief confounding elements in STEMMA’s computational efforts to date. LLMs, largely trained on modern prose, cannot (yet) grasp the full range of poetic elements that humans can. Poetic language often defies even human expectations for the next most likely token, much less statistical ones. At the same time, our computational approach allowed for retrieval and semantic linking across a dataset too large for most human users to recall quickly. In literary studies, as in other domains, combining machine learning models with human interpretation allows us to make the most of the relative strengths of each (Kerrigan, Smyth, and Steyvers Reference Kerrigan, Smyth and Steyvers2021).
STEMMA’s pipelines facilitated the efficient cleaning of a large proportion of the inherited data, but they also cast new light on the ways early modern poetic practice radically differs from the modern prose on which general models are trained. This defamiliarizing lens offers a new way to address longstanding questions about the relationship between poetic form and poetic meaning as well as the ways readers, then and now, understand it. Joining these insights with other computational methods, like network analysis, and literary techniques like paleography, philology, and close reading, promises an even fuller view of literary history. But perhaps most importantly, putting LLMs into conversation with complex data sets reveals how much computational research and traditional humanities research have in common – and how much they stand to learn from one another.
Data availability statement
An open version of the full dataset that supports this study will be deposited when it is fully cleaned and ready for analysis (likely summer 2026) and updated throughout the remainder of the funding period. A Readme file, including sample records from the source datasets, has been deposited on Zenodo at https://doi.org/10.5281/zenodo.18247735. Code is on GitHub and will be released at or around the same time as the data.
Author contributions
Conceptualization: E.A.M.; Data curation: E.A.M., C.B., D.C., K.D., M.K., M.R., and L.V.; Funding acquisition: E.A.M.; Investigation: E.A.M.; Methodology: E.A.M.; Project administration: E.A.M.; Resources: E.A.M.; Software: E.A.M. and J.P.; Supervision: E.A.M.; Validation: E.A.M., C.B., M.K., and L.V.; Writing – original draft: E.A.M.; Writing – review and editing: E.A.M.
Funding statement
Funded by the European Union (ERC Grant Agreement No. 101088497). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Competing interests
The author declares none.
Ethical standards
The research meets all ethical guidelines, including adherence to the legal requirements of the study country.
Open access
Rapid Responses
No Rapid Responses have been published for this article.