1 Background on LLMs and data augmentation

LLMs are large-scale neural network models built on the Transformer architecture (Vaswani et al. Reference Vaswani2017). The Transformer’s main innovation is the attention mechanism, which allows the model to build contextual embeddings for each word based on all words in a sequence.Footnote ⁴ As a consequence, the word “bank” in “they robbed the bank” and “they sat on the river bank” has very different numerical representations, leading to much greater contextual understanding.Footnote ⁵ Since LLMs are pretrained on very large text corpora, they have extensive knowledge of natural language in context and can be applied to specific tasks with generally strong levels of performance.

Two Transformer families dominate current text-as-data research, but they handle text and labeled data in different ways. Encoder-only models such as BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019) and RoBERTa (Liu et al. Reference Liu2019) take a sequence of text as input and produce a contextual representation that can be passed to a classification head. Researchers fine-tune these models on a labeled training set, which adjusts the model’s weightsFootnote ⁶ so that the final representation reliably maps onto specific categories of interest. Encoders therefore learn category boundaries directly from labeled examples, whose quantity and quality determine how well the model performs.Footnote ⁷ A relevant point for this article is that performance on a category is tied to how often the category appears during fine-tuning,Footnote ⁸ which means that a category with very few labeled instances effectively disappears from the parts of training where most of the learning happens.

Decoder-only models such as the GPT family work differently (Brown et al. Reference Brown2020). They are trained to predict the next word in a sequence based on all the words that came before. For classification, these models are not usually fine-tuned on labeled data but are used with a prompt that describes the task and includes a small set of labeled examples.Footnote ⁹ The model produces the category label as part of its generated output (see Gilardi et al. Reference Gilardi, Alizadeh and Kubli2023; Timoneda and Vallejo Vera Reference Timoneda and Vallejo Vera2026). The labeled examples in the prompt act as anchors that tell the model what each category looks like in practice. This style of use, often called few-shot prompting, places very different demands on labeled data: labeled examples are needed for the prompt itself, but not for fine-tuning. It is important, however, for model evaluation to compare the model’s output against expert-coded labels, both to estimate performance reliably and to correct for measurement error in downstream analyses (Egami et al. Reference Egami, Hinck, Stewart and Wei2023). Lastly, note that encoder–decoder models like BART (Lewis et al. Reference Lewis2020) and T5 (Raffel et al. Reference Raffel2020) combine elements of both model types, but they are used less often in classification tasks and thus not considered here.

The distinction between encoders and decoders matters because sparse categories generate problems for each. For encoder models, a category with too few examples does not appear often enough during training for the model to learn a reliable boundary. For decoder models, too few examples mean that evaluation against expert-coded labels becomes unstable. There are not sufficient labeled instances against which to assess performance. In both cases, the result is worse performance on the rare category, which also affects overall model performance. This is the problem that data augmentation methods are designed to address.

Indeed, several existing data augmentation approaches attempt to increase the number of examples for small-N categories, but many induce new problems in practice. Paraphrasing methods, including back-translation, usually change the wording only slightly, making synthetic texts ultimately too similar to the original (Beddiar, Jahan, and Oussalah Reference Beddiar, Jahan and Oussalah2021; Huang et al. Reference Huang, Iyer, Hsu, Kumar, Chang and Galstyan2023).Footnote ¹⁰ This can make classifier performance artificially strong, as the model learns the repeated textual patterns in the training data and always classifies similar texts correctly. A related problem emerges with more sophisticated strategies, such as SSMBA, which uses masking and reconstruction steps to create new text from the existing examples (Ng, Cho, and Ghassemi Reference Ng, Cho and Ghassemi2020).Footnote ¹¹ However, reconstructed texts with SSMBA replace words but do not alter the structure of the sentence, which again yields artificially high-performance metrics. Generative LLMs, on the other hand, provide more flexibility due to the probabilistic nature of their outputs (see Ornstein et al. Reference Ornstein, Blasingame and Truscott2025; Walker and Timoneda Reference Walker and Timoneda2025). They are good at expressing the same idea in different ways and can retain the category-defining content of the text using different structures and vocabulary. Synthetic imputation offers a clear guided approach to generative LLM prompting that yields the right balance to retain category-defining content while increasing lexical and structural variation in synthetic samples, which is crucial for optimal classifier performance and model evaluation.

2 Why sparse categories challenge encoder-only and decoder-only LLMs

LLMs require a critical number of expert-coded observations per category for optimal model training and evaluation in annotation tasks (Egami et al. Reference Egami, Hinck, Stewart and Wei2023; Timoneda and Vallejo Vera Reference Timoneda and Vallejo Vera2025a). This is true for both encoder-only models (e.g., BERT and RoBERTa), which rely on sufficient labeled data during training and validation, and for generative LLMs, whose evaluation still depends on having enough clear and well-defined examples for each category (see Egami et al. Reference Egami, Hinck, Stewart and Wei2023). For tasks using encoder-only models, there should generally be multiple instances of each category in the trainer batches—small subsets of the data that are passed through the neural network during training. The most common batch sizes are 16 or 32 observations, and each batch should contain at least a few observations of all categories (Liu et al. Reference Liu2019). With smaller batch sizes or with a larger number of categories in the data, it may often be the case that some categories are not represented in all the batches if data are scarce (Zhou, Lin, and Ren Reference Zhou, Lin and Ren2021).Footnote ¹²

Sparse categories also affect the evaluation of a decoder’s performance. This area has become salient in LLM research in political science in recent times, especially as it relates to the problem of replication (Barrie, Palmer, and Spirling Reference Barrie, Palmer and Spirling2024). First, researchers need sufficient samples of a given category to properly evaluate performance and correct downstream biases (Egami et al. Reference Egami, Hinck, Stewart and Wei2023). Second, if a rare category is underrepresented in the expert-coded sample and during annotation, the model will drift toward more abundant categories, assigning the more common label even if the underlying text belongs to a low-N category (Albattah and Khan Reference Albattah and Khan2025). This dynamic introduces measurement error that differs from the encoder-only case (Gao et al. Reference Gao2026). Therefore, even if generative LLMs reduce the need for expert-coded examples for fine-tuning, they still require them for model evaluation.

Sparsity is common in political science applications. Some categories are rare because the underlying events are uncommon (Timoneda Reference Timoneda2021), and in others, researchers are too resource-constrained to be able to produce a large balanced sample. It is thus not unusual for a category to contain only a few dozen labeled examples, which is insufficient for training and evaluation (see Grimmer and Stewart Reference Grimmer and Stewart2013; Halterman Reference Halterman2025). An approximate threshold of around 50 examples is useful in practice, not as a hard cutoff but as a general operational guideline.Footnote ¹³ The reasons are different for encoder-only and generative LLMs. First, the batching problem already described means that the model receives little direct information about a sparse category during gradient updates. In most text-classification settings, performance curves also show the steepest gains between 20 and 50 examples, consistent with natural scaling laws. In fact, Pecher, Srba, and Bielikova (Reference Pecher, Srba and Bielikova2025) find that 100 examples are often enough to achieve satisfactory LLM classifier performance, and my own testing finds that 50 observations yield adequate results as well.Footnote ¹⁴ Second is the evaluation problem described earlier for generative LLMs, as having fewer than 50 human-labeled examples makes model evaluation unreliable.

3 The synthetic imputation approach

The core idea behind synthetic imputation is to use generative LLMs to expand sparse categories while keeping the process grounded in the category-defining content of the original texts. The approach rests on the idea that a handful of high-quality human-coded examples capture the key features of a category. Those examples give the model enough information to (1) keep the new text consistent with the category-defining content in the training examples and (2) vary how those texts are written so the classifier is not just memorizing repeated patterns. The synthetic imputation approach thus solves two problems at once. It generates new observations that expand the effective sample size of the rare category, but without shifting the category’s underlying conceptual boundaries or generating texts that are overly similar to one another. The resulting synthetic texts preserve the meaning, tone and defining structure of the category, but introduce sufficient variation for the model to learn without artificially inflating performance.Footnote ¹⁵ It is, therefore, a data augmentation technique especially crafted to address the sparse-category problem in annotation tasks with encoder-only or generative Transformer LLMs. The logic for the synthetic imputation approach is similar to multiple imputation: if sufficiently informative data exist, one should be able to generate imputations that approximate the true but unknown value of missing observations. The advent of generative LLMs makes it possible to impute training data in supervised text classification. These models can condition on a small set of curated examples and generate new instances that approximate the underlying distribution of the category without the researcher having to specify in advance how the text should be edited, swapped or rewritten.

The synthetic imputation approach connects to long-standing work on data augmentation in computer science, where researchers have developed various approaches in data augmentation for text data (see Halterman Reference Halterman2025). One approach has been rules-based algorithms, where tokens are swapped, inserted and deleted at random to generate a similar but different text sample from one original text (Wei and Zou Reference Wei and Zou2019; Wei et al. Reference Wei, Huang, Xu and Vosoughi2021). A second approach is example interpolation, where tokens from multiple samples are mixed to generate a synthetic example (Guo Reference Guo2020). A third and more complex approach uses deep learning models to augment data. Nie et al. (Reference Nie, Tian, Wan, Song and Dai2020) use pretrained embeddings to generate synthetic texts based on semantic neighbors in a sentence, while Ng et al. (Reference Ng, Cho and Ghassemi2020) use autoencoders to deconstruct and reconstruct sentences using BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019). These approaches improve the quantity of annotated text but offer limited gains in conceptual fidelity and lexical variability. Generative LLMs can now address these limitations.

The synthetic imputation approach has four steps. The first is to identify at least 50 text instances for each category as a practical guideline, paying special attention to underrepresented categories in the data. The goal is to ensure that the sparse category has at least 50 meaningful examples for the model to identify its defining features (the dataset need not be balanced, as the other categories can vary in size above 50). The second step is to use a generative model (here, GPT-5) to produce synthetic texts that resemble the original examples in category-defining content but differ in their writing. The model is prompted using a general instruction plus five examples drawn at random from the subset of hand-coded texts for each new synthetic text. By drawing a fresh set of five examples each time, the model avoids focusing only on a single text, and the resulting samples show more natural variation.Footnote ¹⁶

Third, the researcher uses a nearest-neighbor similarity test to remove near-duplicates and synthetic texts too similar to the originals.Footnote ¹⁷ For each synthetic text, one calculates the cosine similarity between its sentence embedding and the embeddings of five human-coded texts used by the model as examples.Footnote ¹⁸ If the maximum similarity exceeds a fixed threshold set ex-ante (and applied uniformly across applications), the synthetic text is discarded and regenerated using a fresh draw of examples. Appendix E of the Supplementary Material describes this process in detail and provides an audit table with examples using synthetic imputation and SSMBA.Footnote ¹⁹ The researcher then checks the text generation process manually, sampling a subset of texts and ensuring that each one fits the category-defining content and tone of the category without redundancy. While less concerning than with alternative approaches, generative models can sometimes drift stylistically or stretch a category’s content boundaries what the human-coded data supports.

The fourth and final step is to either fine-tune the encoder-only classifier (BERT or RoBERTa) on the expanded dataset or to apply and evaluate the generative LLM classifier on the full data. For encoder-only models, the augmented sample (original plus synthetic texts) increases the effective sample size of the sparse categories, making them appear more frequently in the training batches. For generative LLMs, the researcher compares LLM-labeled data with the expanded expert-coded hold-out data to evaluate out-of-sample model performance.

Throughout this procedure, I keep the generation of synthetic texts separate from model evaluation. I generate synthetic examples only from human-coded texts in the training data, and I do not use validation or test observations as inputs to the generation process. In the encoder-only applications using RoBERTa, I generate synthetic texts within each training fold and add them only to that fold, so information from held-out data never enters model training. I also avoid evaluating synthetic texts against the specific human-coded examples from which they were generated.

Step 2 of the procedure leaves open the practical question of how many synthetic texts should be generated. In the applications above, I impute up to the full number of true texts in the target category (151 for nostalgia and 218 for speeches) because the goal is to validate performance against a known full-data benchmark. Applied researchers, however, will not usually have that reference point. The recommendation, derived from the logic of the encoder-only classifier and then checked against the decoder, is to augment to approximately 200 total texts for sparse categories. For RoBERTa, with a typical batch size of 16, a category should appear in training batches with at least 2 or 3 examples per batch, on average. This ensures that gradient updates reflect the category consistently. In most applied datasets, that requirement translates into approximately 150–200 texts in the sparse category. That range is large enough to populate batches reliably without pushing the synthetic-to-true ratio too high.

The encoder logic thus provides a principled mechanical anchor for the imputation target, but the decoder results in the empirical section point to the same range. Per-category F1 for GPT-4o also stabilizes around 150–200 texts, even though the underlying mechanism is different. For generative LLMs, the issue is not batch composition but evaluation stability. This convergence across the encoder and decoder settings is useful because it suggests that the recommended target is not an artifact of one particular model class. Researchers could, in principle, continue augmenting beyond that range and obtain additional marginal gains. But doing so would also increase the ratio of synthetic to true texts and, with it, the risk that the model begins to learn patterns associated with the style of the synthetic data rather than the category-defining content of the texts. That failure mode is visible in the SSMBA results.

The synthetic imputation approach is intended for three broad types of applied settings. The first, and likely the most common, is a dataset that is otherwise adequate but contains one or two sparse categories that lag behind the rest and weaken both training and evaluation. The second is a multi-category task in which several categories are simultaneously sparse. In a six- or ten-category problem, even a moderately sized dataset will often fail to provide 150–200 observations per category. Synthetic imputation allows the researcher to bring each sparse category to a usable sample size without hand-coding the entire corpus. The third is a project that would otherwise be infeasible because there are too few examples of the category of interest, or one in which sparse categories would need to be collapsed into broader and less precise ones, thereby introducing a different form of measurement bias. The method can still be applied below roughly 50 true texts, but doing so requires explicit recognition of the overfitting that is likely to arise.

Synthetic imputation is also complementary to Egami et al.’s (Reference Egami, Hinck, Stewart and Wei2023) design-based supervised learning (DSL), which provides a principled way to correct downstream estimators when classifiers introduce measurement error. The two approaches operate at different stages of the measurement pipeline. DSL corrects downstream estimates after a classifier has been fit, while synthetic imputation augments the labeled data so the classifier can be trained and evaluated reliably in the first place. In settings with severe class imbalance, where there are too few hand-coded positives to fit a classifier well and too few held-out positives to estimate its performance with precision, the bottleneck arises before DSL’s correction step becomes feasible. The two approaches are therefore complementary, and combining them in a single workflow is a natural direction for future applied work. Appendix A of the Supplementary Material provides further discussion.

3.1 Prompt design

Effective prompt design is essential for generating high-quality synthetic texts. When constructing the GPT-5 prompt, researchers should clearly instruct the model to produce texts that differ from the examples it is shown, both in how they are written and in the details they include (Gao Reference Gao2023; Hatakeyama-Sato et al. Reference Hatakeyama-Sato, Yamane, Igarashi, Nabae and Hayakawa2023; White et al. Reference White2023). The prompt itself consists of two parts: a general system instruction and the set of five human-coded examples drawn at random from the category-specific sample for each iteration. This structure follows existing evidence that few-shot prompting tends to yield stronger performance than zero-shot prompting for models such as GPT-5 (Gilardi et al. Reference Gilardi, Alizadeh and Kubli2023; Ornstein et al. Reference Ornstein, Blasingame and Truscott2025; Walker and Timoneda Reference Walker and Timoneda2025). The five examples are re-sampled for every new synthetic text. Each prompt therefore exposes the model to a slightly different slice of the human-coded data, which helps produce a broader range of outputs and reduces the likelihood that the model will repeat the same formulations across iterations.Footnote ²⁰ Because the examples are drawn from the original data, each generated text remains grounded in category-defining content, even as the wording and structure vary across outputs. Figure 1 displays a flowchart describing the synthetic imputation approach from prompt creation to synthetic text output, using the nostalgia dataset as an example.

Figure 1

Flowchart for an example of synthetic text generation using the synthetic imputation approach (notes: each example is included in the prompt in place of the “{ex_}”; mean_length represents the average length of the five examples, which helps guide the LLM to produce synthetic texts of appropriate length).

Figure 1 Long description

The flowchart consists of seven vertically stacked boxes.

* The top box is labeled Prompt and contains instructions to generate a sentence in English based on examples, maintaining a nostalgic tone but changing content, names, and topics. It includes placeholders for mean length and five examples labeled ex 1 through ex 5.

* The second box, Example 1, contains a sentence about the Sami Parliament's control and participation in decisions affecting Sami culture and industries.

* The third box, Example 2, contains a sentence stating that social democracy can report good results of 20th century government policy.

* The fourth box, Example 3, contains a sentence about the 1945 Labour government establishing the N H S as a central institution of fairness.

* The fifth box, Example 4, contains a sentence about cultural democracy based on the highest possible participation of society in creating cultural values.

* The sixth box, Example 5, contains a sentence about social science disciplines focusing on the culture and history of the Czech Republic.

* The final box at the bottom is labeled Final synthetic text generated by G P T 5 based on the prompt and the five examples. It contains the output: We will safeguard the village post offices and local libraries, preserving the civic spirit of the 1950s that connected communities through shared public service.

Several practical considerations shape the prompt’s effectiveness, such as generation temperature, length guidance, avoidance of formulaic phrasing and the principle that the substantive content of each synthetic text should come from the rotating examples, not the category-naming language in the prompt. Appendix C of the Supplementary Material discusses these design choices in detail and explains how the prompt is structured to minimize sensitivity to wording.

4 Applications

The first task uses data on political nostalgia from Müller and Proksch (Reference Müller and Proksch2024). The authors collected sentences from different party manifestos in 24 European countries. Four human annotators coded the data into either nostalgic or not nostalgic, and the authors aggregated these into a binary variable where a sentence is classified as nostalgic if at least three coders agree. The final dataset contains 1,200 annotated sentences, with 151 nostalgic and 1,049 not nostalgic texts. This dataset represents a binary application with relatively short texts, which is a text structure common in political science applications that use sentence-level and Twitter data.

The second application uses data from the Global Populism Database (GPD) by Hawkins et al. (Reference Hawkins, Aguilar, Silva, Jenne, Kocijan and Kaltwasser2019).Footnote ²¹ The project built a large dataset of populist speeches by world leaders, with a total of 843 speeches by 190 leaders from a total of 58 countries. The original speeches are in 35 different languages and have been translated into English.Footnote ²² They range from 139 to 1,468 words long, with a mean length of 414.75 and a median length of 410 words. From the GPD data, I will use one human-labeled variable for analysis: the speech type. The measure has four categories: campaign, ribboncutting, famous and international, our category of interest, where the audience is foreign and the speech is given outside the speaker’s country of origin. The focus on international speeches stems from the fact that it is a highly contextual category with very specific textual patterns. Appendix G of the Supplementary Material discusses two methodological challenges with this dataset, encoder-only model token limits and per-category sample size, and how I address them in the analysis below.

The two applications cover the structural dimensions in common text-as-data applications in terms of unit of analysis (manifesto sentence vs. full speech), text length (mean 20.8 vs. 414.7 words), class structure (a heavily imbalanced binary task vs. four roughly balanced categories) and language pipeline (English-only vs. translations from 35 source languages). Most importantly, they differ in the substantive nature of the category itself. Nostalgia is tone-defined: what makes a sentence nostalgic is its affective framing of the past, which can be expressed through varied phrasings and therefore yields a relatively diffuse boundary. International speeches, by contrast, are context-defined: the category is signaled by concrete cues like the audience and setting, producing a tighter signal even though the texts are much longer. The contrast allows me to evaluate whether the approach generalizes across substantively different task structures. Table 1 summarizes these characteristics across both applications.

Table 1

Dataset characteristics across the two applications

Table 1 Long description

The table compares two applications across ten characteristics.

1. Source: Nostalgia uses Müller and Proksch 2024; International speeches uses Hawkins et al. 2019.

2. Unit of analysis: Manifesto sentence for Nostalgia; Full speech for International speeches.

3. Total dataset N: 1,200 for Nostalgia; 843 for International speeches.

4. Target category N: Nostalgic 151; International 218.

5. Categories: 2 (nostalgic or not) for Nostalgia; 4 (international, campaign, ribbon-cut., famous) for International speeches.

6. Class balance: Heavily imbalanced at 12.6 percent positive for Nostalgia; Roughly balanced between 22.7 percent and 26.1 percent for International speeches.

7. Text length in words: Nostalgia has a mean of 20.8 and range of 3 to 174; International speeches has a mean of 414.7 and range of 139 to 1,468.

8. Language pipeline: English covering 24 E U countries for Nostalgia; English translated from 35 source languages for International speeches.

9. Category structure: Tone-defined and diffuse for Nostalgia; Context-defined with tighter cues for International speeches.

10. Unique tests: Nostalgia tests short texts, severe imbalance, and conceptually diffuse categories; International speeches tests long texts, multi-category structures, and contextual cues.

Lastly, Appendix D of the Supplementary Material reports a third validation test using sentiment analysis data from SemEval 2016 (Mohammad et al. Reference Mohammad, Kiritchenko, Sobhani, Zhu and Cherry2016), which provides a simpler and more standardized setting for the approach.

4.1 Sampling strategy

To apply the synthetic imputation procedure, I draw random samples for each dataset. For the political nostalgia data, I draw three samples from the 151 nostalgic sentences, containing 50, 75 and 100 true texts. I then generate 101, 76 and 51 synthetic texts so that each augmented set returns to the full size of 151. All samples use fixed random seeds for reproducibility. For the international speeches, I use the same approach and draw samples of 50, 75, 100 and 150, generating 150, 125, 100 and 50 synthetic speeches, respectively. Each augmented set has 200 observations, which is close to the original 218 and keeps the subsets comparable.

Figure 2 illustrates this pipeline. Both methods share an identical starting point and evaluation procedure. They differ only in sample construction: synthetic imputation augments the N real positives with $(151 - N)$ LLM-generated synthetic positives, while the baseline uses only the N real positives. Each sample is then evaluated with both RoBERTa-large (10 $\times $ 10-fold cross-validation) and GPT-4o as a few-shot annotator. Performance is measured against expert-coded labels using the F1-score for the target category. The contrast between the two tracks isolates the effect of synthetic data augmentation.

Figure 2

Pipeline for the synthetic imputation approach (top track) and the class-weighted baseline (bottom track). Both pipelines share an identical starting point (a labeled dataset and a sampled subset of N real positives) and identical evaluation methods (RoBERTa-large fine-tuning and GPT-4o few-shot annotation). They differ only in the construction of the training/evaluation sample: synthetic imputation augments the N real positives with $151 - N$ LLM-generated synthetic positives, while the baseline uses only the N real positives. Numbers shown reflect the nostalgia application; the same procedure applies to the international-speeches application with corresponding sample sizes.

Figure 2 Long description

The flowchart begins on the left with a shared starting point: Original labeled dataset (e.g., 1,200 sentences; 151 positives), which flows into Sample N real positives (N = 50, 75, 100).

The process then splits into two tracks:

1. Top Track (Synthetic Imputation):

- G P T-5 with 5-example rotating prompt: generate (151 minus N) synthetic positives.

- Augmented sample: N real + (151 minus N) synthetic positives (+ all negatives).

- This flows into two evaluation methods: Fine-tune RoBERTa-large (class-weighted loss, 10 times 10-fold C V) and G P T-4o annotation (10 stratified in-context examples).

2. Bottom Track (Class-Weighted Baseline):

- No synthetic generation.

- Sparse sample: N real positives (+ all negatives).

- This flows into the same two evaluation methods: Fine-tune RoBERTa-large and G P T-4o annotation.

Both tracks converge at the final node on the right: Mean F 1 vs. annotated labels (+ corpus prediction for downstream).

A legend at the bottom indicates that light blue boxes represent shared steps, medium blue boxes represent the synthetic imputation track, and dark blue boxes represent the class-weighted baseline track.

5 Analysis and results

For each application, I run encoder-only and generative LLM annotators (1) on a sparse subset of the data, (2) on that same subset after adding the synthetic texts, and (3) on the full dataset. The model trained on the complete data provides the benchmark for comparison. SSMBA will serve as the state-of-the-art benchmark. I report results separately for encoder-only and generative LLMs because the two model families incorporate labeled data differently: RoBERTa during fine-tuning and GPT-4o in examples and evaluation. I evaluate RoBERTa with repeated cross-validation and GPT-4o against expert-coded test labels.

5.1 Encoder-only models (RoBERTa)

For each application, I fine-tune RoBERTa-large classifiers on sparse original samples, their synthetic imputation counterparts and the full dataset.Footnote ²³ All models are trained using 10-times repeated 10-fold cross-validation to obtain stable performance estimates (Timoneda and Vallejo Vera Reference Timoneda and Vallejo Vera2025a).Footnote ²⁴ Figure 3 reports mean $F1$ -scores for the relevant category in each application.

Figure 3

Model performance across sparse subsets with synthetic imputation, baseline and SSMBA augmentations. Shaded bands show 95% confidence intervals. $F1$ -score averages based on 10 $\times $ 10 CV runs.

Figure 3 Long description

A two-panel line graph. Both panels share an x-axis titled Increasing Amount of True Data and a y-axis for F 1 score.

Panel a, titled Nostalgia results, has a y-axis from 0.5 to 1.0 and x-axis values of 50, 75, 100, and 151.

* S S M B A Benchmark (red line) starts at approximately 0.98 and shows a downward curve to 0.82.

* Synthetic Imputation 151-x (blue line) starts at 0.85 and slightly declines to 0.82.

* True Performance (dotted red line) is a horizontal constant at 0.822.

* No Synthetic Data Baseline (grey line) starts at 0.47 and shows a steep logarithmic increase to meet the other lines at 0.82.

Panel b, titled International speeches results, has a y-axis from 0.6 to 1.0 and x-axis values of 50, 75, 100, 150, and 218.

* S S M B A Benchmark (red line) starts at 0.97 and gradually curves down to 0.90.

* Synthetic Imputation 218-x (blue line) starts at 0.93 and converges to 0.90.

* True Performance (dotted red line) is a horizontal constant at 0.895.

* No Synthetic Data Baseline (grey line) starts at 0.56 and increases rapidly to 0.90.

In both panels, shaded bands around the lines represent 95 percent confidence intervals, which narrow as the amount of true data increases toward the right.

Figure 3a shows the results for the nostalgia data. First, as the number of true category-specific texts decreases, the baseline models without synthetic data (gray line) decline sharply. With only 50 examples, the model drops to an $F1$ -score of 0.471, a steep decline from the 0.822 average score in the model with all 151 true labeled texts. The synthetic imputation models (blue line) reverse this pattern. With 50 true texts, the augmented model reaches 0.851—slightly above the full-data benchmark of 0.822. This indicates a small and predictable overfitting effect from the model picking up on lexical and grammatical patterns in the synthetic data. The key point in the figure is that once we reach 75 true texts, the model augmented with synthetic imputation lands close to the full-data benchmark (0.840 vs. 0.822). With 100 texts, the difference is negligible. SSMBA performs worse, with F1-scores well above the benchmark at every sample size and reaching 0.982 when trained on only 50 true texts. This is a clear sign that the augmented texts are too close to the originals and induce artificially high-performance scores.

Figure 3b reports the results for the international speeches data. As with the nostalgia application, the baseline models without synthetic data (gray line) deteriorate quickly as the amount of true data shrinks. With only 50 speeches, the model falls to an $F1$ -score of 0.561, a large drop from the full-data benchmark of 0.895. The synthetic imputation models (blue line), on the other hand, remain much more robust to both overfitting and performance decline. With 150 true speeches, the augmented model’s $F1$ -score is 0.893, effectively the same as the full-data score. Using 100 true texts also produces the same $F1$ -score as the benchmark. More importantly, at 75 texts, the model edges up very slightly to 0.903, a small difference that falls well within the variability of the cross-validation runs. With only 50 true speeches, the model climbs to 0.923, but as in the nostalgia results, this reflects a small and predictable overfitting effect from the model learning recurring patterns across the synthetic texts.

Taken together, the two RoBERTa applications point to the same pattern. With roughly 75 true texts, the model using augmented data from the synthetic imputation approach reaches full performance, with $F1$ -scores equivalent to the model trained on the full set of observations. Even at 50 texts, it does far better than the baseline without any augmented data. The overfitting at this point is very limited and follows a highly predictable pattern of minor inflation that can be corrected. As reported in detail in Appendix B of the Supplementary Material, however, overfitting becomes more pronounced below 50 true texts: at $n=25,$ the F1 reaches 0.903, and at $n=10,$ it climbs to 0.944, well above the full-data benchmark of 0.822.

5.2 Generative LLMs

For each application, I evaluate a generative LLM model on the sparse original samples, their synthetic imputation counterparts and the full dataset. I use a short few-shot classification prompt that states the task and lists the possible categories to label. The prompt provides ten examples with human labels drawn at random from the data using stratified sampling. The same prompt structure is used across all experiments.Footnote ²⁵ Model performance is measured by comparing the LLM’s classifications to the expert-coded labels from the original datasets. Figure 4 reports the $F1$ -scores for both the nostalgia and international speeches categories.

Figure 4

Model performance across sparse subsets with synthetic imputation and baseline results. Shaded bands show 95% confidence intervals from nonparametric bootstrap resampling of GPT-4o predictions against the full set of annotated labels.

Figure 4 Long description

A two-panel line graph. Both panels share an x-axis titled Increasing Amount of True Data and a y-axis representing F 1-score.

Panel a, titled Nostalgia results, shows the y-axis from 0.55 to 0.85. A horizontal red dotted line representing True Performance is constant at 0.800. The Synthetic Imputation 151-x line, in blue with a light blue shaded confidence interval, starts near 0.82 at x=50 and remains nearly flat, slightly decreasing to 0.80 at the full sample of 151. The No Synthetic Data Baseline line, in solid red with a light red shaded confidence interval, starts low at approximately 0.60 and follows a logarithmic growth curve, meeting the other lines at 0.80 at the full sample.

Panel b, titled International speeches results, shows the y-axis from 0.5 to 0.9. The True Performance red dotted line is constant at 0.859. The blue Synthetic Imputation 218-x line starts at 0.86, peaks slightly near x=100, and ends at 0.859 at the full sample of 218. The red Baseline line starts at approximately 0.60 and increases steadily in a convex curve until it converges with the blue line and dotted line at 0.859 at the full sample. In both panels, the shaded confidence intervals for the baseline are much wider at low x-values and narrow as they approach the full sample size.

Each condition in Figure 4 is designed to simulate a realistic applied setting at the stated sample size. In the baseline condition (red line), GPT-4o annotates a dataset in which the target category contains only the indicated number of true expert-coded texts and no synthetic texts. In the synthetic imputation condition (blue line), those same true texts are supplemented with synthetic examples until the category reaches the full sample size. In both conditions, F1 is calculated against the full set of annotated labels—namely, the expert-coded labels in the baseline condition and the expert-coded plus synthetic labels in the augmented condition. Measured performance thus reflects the annotation distribution a researcher would face after applying synthetic imputation. The contrast between conditions therefore isolates the effect of synthetic supplementation on the annotation distribution, that is, on the texts GPT-4o is asked to label, rather than on the prompt itself.

Figure 4a reports the results for the nostalgia dataset using GPT-4o as an annotator. As with the encoder-only models, performance declines quickly when the model is asked to annotate the sparse datasets without any synthetic imputation. With only 50 true nostalgic texts, the model’s $F1$ -score drops substantially relative to the benchmark model using the full set of 151 human-coded sentences. Once synthetic texts are added, however, the model’s performance moves sharply toward the full-data benchmark. The augmented model with 50 true texts and 101 synthetic ones performs much closer to the true performance for the model than the baseline, and the confidence intervals show that there are no significant overfitting concerns. The same result applies to datasets with 75 and 100 original observations and the rest synthetic ones, as their performance is essentially indistinguishable from the full-data model.

Figure 4b reports the results for the international speeches dataset using GPT-4o as an annotator. As with the nostalgia application, performance declines quickly when the model is asked to annotate the sparse datasets without any synthetic imputation. With only 50 true international speeches, the model’s $F1$ -score falls well below the benchmark of 0.859 and remains far from it even at 75 and 100 true texts. After adding the synthetic texts, however, performance is much closer to the model using the full data. With 50 true texts, the augmented model reaches an average $F1$ -score of 0.861, essentially matching the full-data benchmark of 0.859 even at very low levels of N. The same result holds for the datasets with 75, 100 and 150 original observations, as their augmented models cluster close to the benchmark and are essentially indistinguishable from full-data model performance. The overall pattern mirrors the RoBERTa results. Once the model has access to a modest number of true speeches, the synthetic imputation approach provides enough informative variation for GPT-4o to approximate full-sample performance without drifting away from the category-defining content of the text. Appendix G of the Supplementary Material discusses the generalizability of these results to other classification tasks.