2.1 Three paradigms of image representation

Computer vision approaches to image representation have evolved profoundly over recent decades. For our purposes, three paradigm shifts are relevant. Each offers conceptual and performance advances that support distinct paths to modeling visual topics. Continuous embeddings are important in these developments: the more we move toward complex semantic layers, the more we need to draw on continuous embeddings.

2.2 Previous approaches to visual topic modeling

Pioneering work to tackle political image data with topic models has largely focused on the first two paradigms, remaining in the domain of graphical and literal visual topics. These approaches fall into two tracks.

2.3 Toward a vSTM with continuous embeddings

We contend that it is desirable for visual topic modeling in the context of political analysis to draw on approaches that can offer a spectrum of possible emphases (i.e., can capture graphical and literal, but also connotative similarity) and have the capacity to handle complex semantics in which multiple semantic layers of visual meaning come into play—that is, approaches based on continuous embeddings. We also hold that it is important to retain the flexibility of mixed membership modeling in semantically complex contexts.

There are two potential obstacles to moving in this direction. The first centers on interpretability. A critique of continuous embeddings is their opacity when used as model inputs compared to more straightforward discrete features (Scholz et al. Reference Scholz, Weidmann, Steinert-Threlkeld, Keremoğlu and Goldlücke2024; Torres Reference Torres2024). However, the balance is reversed when interpreting output-side topics. In the case of semantically complex images, a representation that is too simple risks producing topics that are less coherent and meaningful, and consequently difficult to interpret. We acknowledge that embracing continuous embeddings shifts the onus of interpretability to robust post hoc interpretation. As reported below, the vSTM therefore provides a number of outputs that can be used in this manner. We also provide evidence for the claim that the vSTM with continuous embeddings provides comparatively more interpretable and meaningful topics.

The other potential obstacle is that incorporating continuous embeddings requires an alternative formulation of the statistical underpinnings of the topic model. In the next section, we therefore provide such a formulation, extending a mixed-membership model that enables us to think of image embeddings as a weighted sum of topics and topic proportions. Our contribution is two fold. First, we introduce a hierarchical prior for topic proportions, linking covariates to the prevalence of topics through a generalized linear model, analogous to how STM extends LDA and CTM (Blei and Lafferty Reference Blei and Lafferty2006; Blei, Ng, and Jordan Reference Blei, Ng and Jordan2003). Second, we focus on scalability, implementing and demonstrating how the model can be estimated using stochastic variational inference (Hoffman et al. Reference Hoffman, Blei, Wang and Paisley2013; Kucukelbir et al. Reference Kucukelbir, Tran, Ranganath, Gelman and Blei2017). This makes it feasible to explore several topic models in large image datasets, while also facilitating extensions and modifications of the model.

3.1 Statistical model

Figure 1

Plate diagram of the generative model of vSTM. $\boldsymbol {\beta }_k$ is a topic, $\boldsymbol {\theta }_i$ is the vector of topic proportions for image i, $\boldsymbol {\Gamma }$ is the matrix of topic prevalence coefficients, $\boldsymbol {z}_i$ is the image embedding and $\boldsymbol {x}_i$ are image-level covariates

.

Figure 1 Long description

The diagram shows the dependencies in the visual Structural Topic Model. Image-level covariates, x_i, and topic-prevalence coefficients, Γ, influence image-specific topic proportions, θ_i. Topic proportions and topic vectors, β_k, in turn determine the image embedding, z_i. Variables x_i, θ_i, and z_i are repeated for images i ε N, while β_k is repeated for topics k=1, …, k.

Formally, our data consist of N image embeddings $\boldsymbol {z}_i \in \mathbb {R}^D$ and image-level covariates $\boldsymbol {x}_i \in \mathbb {R}^P$ , assumed to influence the prevalence of visual topics. We model each image embedding $\boldsymbol {z}_i$ using a normal distribution

(1) $$ \begin{align} \boldsymbol{z}_i \sim \text{normal}\left(\sum^K_{k=1}\theta_{i,k} \boldsymbol{\beta}_k, \boldsymbol{\Sigma}_z\right), \end{align} $$

where K is the number of topics, $\boldsymbol {\theta }_i$ is a vector of topic proportions, such that $\theta _{i,k} \geq 0$ and $\sum _k \theta _{i,k} = 1$ , and $\boldsymbol {\beta } = \left [\boldsymbol {\beta }_1, \dots , \boldsymbol {\beta }_K\right ]^T$ is a matrix of topics, where (Figure 1) $\boldsymbol {\beta }_k \in \mathbb {R}^D$ . The image-specific topic proportions thus define how much each topic contributes to $\boldsymbol {z}_i$ . A specific topic $\boldsymbol {\beta }_k$ is a point in the embedding space. If $\boldsymbol {z}_i$ is close to $\boldsymbol {\beta }_k$ , then the corresponding topic proportion $\theta _{i,k}$ will also tend to be “large.” $\boldsymbol {\Sigma }_z$ is the covariance matrix.

Topic proportions are sampled from a logistic-normal distribution, whose mean depends on the covariates $\boldsymbol {x}_{i} \in \mathbb {R}^P$ and a set of topic prevalence coefficients $\boldsymbol {\Gamma } \in {\mathbb {R}^{P \times (K-1)}}$

(2) $$ \begin{align} \begin{aligned} \boldsymbol{\eta}_{i} & \sim \text{normal}(\boldsymbol{x}_{i} \boldsymbol{\Gamma}, \boldsymbol{\Sigma}_{\theta}) \\ \boldsymbol{\theta}_i & = f(\boldsymbol{\eta}_i). \end{aligned} \end{align} $$

This reflects the idea that images with similar covariates are also more likely to use the same topics. To ensure statistical identifiability, $\boldsymbol {\eta }_i$ is sampled as a $(K-1)$ -dimensional Gaussian and mapped to the K-dimensional simplex via a transformation function f, rather than using an overparameterized K-dimensional softmax. This projection resolves the translation invariance of the standard softmax by anchoring the latent space. Our software defaults to the isometric log-ratio (ILR) transformation for this mapping, although other options like appending a fixed reference value (e.g., zero) to $\boldsymbol {\eta }_i$ achieve the same identifiability. Unlike the Dirichlet distribution, this construction allows correlations between topics to be modeled explicitly (Aitchison and Shen Reference Aitchison and Shen1980; Blei and Lafferty Reference Blei and Lafferty2006). We additionally implement a sparsemax transformation (Martins and Astudillo Reference Martins and Astudillo2016) as an alternative to the ILR link, which can generate sparser topic proportions.

We also place priors on $\boldsymbol {\beta }_k, \boldsymbol {\Sigma }_z, \boldsymbol {\Sigma }_\theta $ and $\boldsymbol {\Gamma }$ . We use student-t distributions for

$$\begin{align*}\gamma_{p,k} \sim \text{student-t}(\nu_\gamma, 0, \sigma_\gamma)\end{align*}$$

and $\beta _{k,d} \sim \text {student-t}(\nu _\beta , 0, \sigma _{\beta })$ . Similar to Gaussian mixture models, multiple parameterizations are possible for the likelihood covariance $\boldsymbol {\Sigma }_z$ , in this case, reflecting different assumptions about the error structure of the embeddings (see also Gruhl and Erosheva Reference Gruhl and Erosheva2014; Heller, Williamson, and Ghahramani Reference Heller, Williamson and Ghahramani2008). A simple covariance (e.g., $\boldsymbol {\Sigma }_z=\sigma _z\boldsymbol {I}$ ) with fixed $\sigma _z$ can work well in practice. Crucially, the choice must be considered in light of the specific geometry of the embeddings. In our analysis in Section 5.1, we found L2-normalizing the embeddings prior to fitting the model, and calibrating the variance to the hypersphere by scaling the standard deviation by $1/\sqrt {D}$ (since the expected magnitude of any single dimension on a unit hypersphere decreases as dimensionality D increases) to be robust and work well across different embeddings.

Finally, we decompose the topic proportion covariance $\boldsymbol {\Sigma }_{\theta }$ into a correlation matrix $\boldsymbol {\Omega }_{\theta }$ and standard deviations $\boldsymbol {\sigma }_{\theta }$ , such that $\boldsymbol {\Sigma }_{\theta } = \text {diag}(\boldsymbol {\sigma }_{\theta }) \boldsymbol {\Omega }_{\theta } \text {diag}(\boldsymbol {\sigma }_{\theta })$ . We use the Lewandowski–Kurowicka–Joe (LKJ) prior on $\boldsymbol {\Omega }_{\theta } \sim \text {LKJ}(\eta _{\theta })$ , where $\eta _{\theta }>0$ controls the degree of correlation between the visual topics. A value of $\eta _{\theta } = 1$ corresponds to a uniform distribution over correlation matrices, while values greater than one concentrate more mass around the diagonal (Lewandowski, Kurowicka, and Joe Reference Lewandowski, Kurowicka and Joe2009). More specific details on the priors and hyperparameters used in Sections 4 and 5 are deferred to the Supplementary Material.

3.2 Inference

We implement model inference to be flexible and scalable. Specifically, vSTM utilizes the automatic differentiation and parallelization capabilities of the Python library JAX (Bradbury et al. Reference Bradbury2018). This enables the user of our software to defer heavy computations to a GPU (if available), and to leverage existing implementations of state-of-the-art inference algorithms implemented in libraries like BlackJAX and NumPyro (Cabezas et al. Reference Cabezas2024; Phan, Pradhan, and Jankowiak Reference Phan, Pradhan and Jankowiak2019). We make this easy by defining the model’s log joint density $p(\boldsymbol {\theta }_i, \boldsymbol {\beta }, \boldsymbol {\Gamma }, \boldsymbol {\Omega }_\theta , \boldsymbol {\sigma }_\theta , \boldsymbol {\sigma }_z \vert \boldsymbol {z}_i, \boldsymbol {x}_i)$ and gradient $\nabla \log p(\boldsymbol {\theta }_i, \boldsymbol {\beta }, \boldsymbol {\Gamma }, \boldsymbol {\Omega }_\theta , \boldsymbol {\sigma }_\theta , \boldsymbol {\sigma }_z \vert \boldsymbol {z}_i, \boldsymbol {x}_i)$ , conditioned on the data, in terms of parameters in an unconstrained space. Necessary reparameterizations and Jacobian adjustments (due to a change of variable) for constrained parameters (e.g., topic proportions $\boldsymbol {\theta }_i$ ) are handled internally. This is similar to the approach in probabilistic programming languages like Stan (Stan Development Team 2024) and is what enables compatibility with a wide range of general-purpose optimization and sampling methods that are designed for unconstrained spaces.

In this article, we use mean-field variational inference (MF-VI) throughout. We describe the details of our implementation and how we make it scalable to large data in Section D of the Supplementary Material. Since VI is known to underestimate the uncertainty of the posterior distribution, but can provide good point estimates of the posterior mean (Wang and Blei Reference Wang and Blei2019), our focus is on the latter. These quantities are often of primary interest, so we argue that VI offers a practical trade-off that is often reasonable in topic modeling contexts where the researcher is likely to explore many models. Furthermore, recent work has shown that making accurate statistical and causal inferences using latent quantities inferred from unstructured data, requires additional considerations at a more fundamental level than the choice of a particular inference method (Egami et al. Reference Egami, Fong, Grimmer, Roberts and Stewart2022, Reference Egami, Hinck, Stewart and Wei2023).

3.3 Covariates and topic prevalence

A key output of the model is the relation between the covariates and topic prevalence. Since the image-level covariates $\boldsymbol {x}_i$ map to topic proportions $\boldsymbol {\theta }_i$ nonlinearly, we characterize the relationship more interpretably via the marginal distribution of topic proportions

(3) $$ \begin{align} \mathbb{E}[\boldsymbol{\theta}_i \vert \boldsymbol{x}_i] = \mathbb{E}\left[ f(\boldsymbol{\eta}_i)\right], \end{align} $$

where $\boldsymbol {\eta }_i \sim \text {normal}(\boldsymbol {x}_i\boldsymbol {\widehat {\Gamma }}, \boldsymbol {\widehat {\Sigma }}_\theta )$ , f is the function mapping $\boldsymbol {\eta }_i$ to the simplex and $\boldsymbol {\widehat {\Gamma }}$ and $\boldsymbol {\widehat {\Sigma }}_\theta $ are point estimates of the prevalence coefficients and covariance matrix of topic proportions.

In practice, the researcher can use Equation (3) to predict the value of topic proportions at some specific covariate values of interest, use predictions at different values to make comparisons or compute an average prediction over a range of values. We show a concrete example in Section 4.

3.4 Interpreting the topic-element distribution

The vSTM can in principle be combined with many types of image representations, some of which are more easily interpretable, while others contain elements that do not individually carry a clear meaning. As discussed in Section 2.1, less interpretable representations can be desirable for capturing complex semantics. In both cases, we need a procedure to summarize the topic distribution $\boldsymbol {\beta }_k$ .

When the input is already based on interpretable elements, the interpretation is straightforward, as we can simply look at the elements that are the most characteristic for the images associated with each topic.

When image embeddings are difficult to interpret directly, it is still possible to obtain interpretable summaries of the topics post hoc using an auxiliary representation of the images. Let $\tilde {\boldsymbol {z}}_i$ be an auxiliary representation of $\boldsymbol {z}_i$ , with interpretable elements. The probability of any particular element appearing in topic k is then

(4) $$ \begin{align} \tilde{\beta}_{kd} = \frac{\sum^N_{i=1} \hat{\theta}_{ik} \tilde{z}_{id}}{\sum^D_{d=1}\sum^N_{i=1}\hat{\theta}_{ik} \tilde{z}_{id}} \ , \end{align} $$

where $\hat {\boldsymbol {\theta }}_i$ are the topic proportions obtained from the vSTM. As $\tilde {\boldsymbol {\beta }}_k$ is now a distribution over interpretable elements, it can be analyzed directly or be used for calculating other quantities, such as frequent and exclusive (FREX) keywords (Bischof and Airoldi Reference Bischof and Airoldi2012).

We view this approach as complementary to the necessary step of inspecting representative images, regardless of the embedding type. Sections 4 and 5 exemplify the trickier case when embeddings are less interpretable. In both cases, we use CLIP to produce the input embeddings ( $\boldsymbol {z}_i$ ) and an LLM to generate text descriptions, which are encoded into an image-term matrix ( $\boldsymbol {\tilde {z}}_i$ ). We note that this approach, in principle, can be used for any type of (input or auxiliary) interpretable elements, for example, object detection labels (Scholz et al. Reference Scholz, Weidmann, Steinert-Threlkeld, Keremoğlu and Goldlücke2024) or visual words (Torres Reference Torres2024).

4.1 Data and setup

We focus on Twitter/X communication around United Nations Climate Change Conferences (COP) 21–27. The data consist of $154{,}177$ unique images posted with the #COP2X hashtag by the most active users in the hashtag stream ( $N=1{,}707$ ), defined by the number of posts. We use the COP year, stakeholder type, and stance as our image-level covariates, assumed to influence the prevalence of visual topics. Each account was manually coded according to stakeholder type based on profile description, following a coding scheme developed from Mooseder et al. (Reference Mooseder, Brantner, Zamith and Pfeffer2023). We also manually coded the account stance toward progressive climate action (for/against/unclear), based on profile description and recent (multi-modal) post content.

4.1.1 Step 1: Choosing the embedding model

A first step in applying the vSTM is choosing which embeddings to use. Many factors influence how the meaning of images is encoded.

Table 1 gives an overview of example embeddings, following our discussion in Section 2. If research goals require connotative and complex topics, we recommend a “CLIP-style” embedding. These embeddings are easy and fast to generate and can represent semantic layers that extend beyond literal depiction. For more tailored representations, using an LLM as in Arminio et al. (Reference Arminio, Magnani, Piqueras, Rossi and Segerberg2026) is also a good option. Pretrained embeddings from models trained under a supervised objective (e.g., VGG16 on ImageNet) should be considered primarily if the training data and learning task closely resemble the analysis data. If the goals of the research call for graphically and/or literally defined topics, self-supervised models (e.g., DINO) are likely to offer good off-the-shelf performance. We provide initial evidence of these claims in Section B of the Supplementary Material, by evaluating the coherence of topics resulting from different embeddings, in terms of denotation (what is depicted) and connotation (the implied meaning), under different embeddings.

Table 1

Image embedding options

.

Table 1 Long Description

The table has three columns: paradigm, example model, and embedding semantics. Supervised learning is represented by pretrained VGG16 and described as having bespoke semantics tied to categories in the pretraining data. Self-supervised learning is represented by DINO and captures graphical and literal semantics. Weakly supervised learning is represented by CLIP and captures graphical, literal, and connotative semantics. Generative AI is represented by Gemma 3 and offers tailored semantics that can be steered using a prompt.

Note: The table shows example models that can be used to extract embeddings. 1. By bespoke, we mean that the semantics are tied to categories in pretraining data (e.g., ImageNet). 2. Tailored refers to the possibility to steer the focus of the embedding using a prompt.

In this section, we use CLIP’s image encoder as an example to show how the model is run step by step.Footnote ¹

4.1.2 Step 2: Selecting the number of topics

The next step is to select the number of topics. To this end, two sets of diagnostics can provide practical guidance in narrowing down a range or selection of models to consider for further inspection.

The first concerns the predictive performance of the model. In this regard, we use five-fold cross-validation to compute perplexity, which is the exponential of the negative log likelihood on held-out data, normalized by the number of held-out observations and embedding dimensions. Lower perplexity indicates that the model, relatively speaking, is better at predicting unseen data. Since topic proportions are local to each image, we fit new ones for the held-out set, while keeping global parameters fixed.

In addition, coherence, and exclusivity scores (details in Section A.3 of the Supplementary Material) can serve as a surrogate for semantic validity—how internally coherent and yet distinctive the topics are (Quinn et al. Reference Quinn, Monroe, Colaresi, Crespin and Radev2010). Figure 2 shows the diagnostics for several models, varying the number of topics between $10$ and $150$ . The left panel shows perplexity, averaged over five held-out sets. It initially decreases sharply, before leveling off around $K=50$ , and later increases again. The right panel reports coherence and exclusivity averaged over all topics. Low exclusivity suggests that many topics are similar to each other, while low coherence indicates that the images within a topic are more heterogeneous.

Figure 2

Diagnostics for model selection. The left panel shows perplexity, averaged over five held-out sets. The right panel shows coherence and exclusivity for each model, averaged over all topics

.

Figure 2 Long description

The left panel plots perplexity against the number of topics from 10 to 150. Perplexity decreases rapidly at first, reaches its lowest region at roughly 60–80 topics, and then increases slightly. The right panel plots exclusivity against coherence, with each point labelled by the number of topics. Models with fewer topics generally have greater exclusivity and lower coherence, while models with more topics generally have higher coherence and lower exclusivity. The model with 45 topics is highlighted as a compromise between these diagnostics.

In this example, we select $K=45$ as a model that offers a balance between coherence, exclusivity and perplexity. We emphasize that this decision is ultimately made in a specific research context, where priorities can differ. The figure should in this sense not be read prescriptively, but rather as a tool to navigate how the model behaves as the resolution (broad vs. fine-grained topics) increases or decreases.

4.1.4 Step 4: Interpreting and labeling topics

Step 4 is to interpret and label topics. We maintain that it is important to include an image-native approach to interpreting visual topics. In contrast to reading representative documents, inspecting a large number of images is fast and intuitive. In order to label topics, we recommend visualizing images that are representative for each topic (see also Torres Reference Torres2024; Zhang and Peng Reference Zhang and Peng2022). One can consider either the top N images with the highest topic proportion $\theta _{ik}$ for each k, or a random sample of images with high proportions. The former provides a sense of what prototypical images look like; the latter gives a better sense of the variety of images included, and if the meaning is still coherent in less pronounced cases.

As a complement, the approach in Section 3.4 can be used to generate an interpretable representation of topics post hoc, using both images and text. Here, we use the Gemma 3 model (Gemma Team et al. 2025) to generate a description for each image, using a simple open-ended prompt.Footnote ² A part-of-speech tagger is then used to extract relevant word classes (nouns, verbs, and adjectives) from the descriptions, which are transformed into an image-term matrix $\tilde {\boldsymbol {Z}} \in \mathbb {N}^{N \times V}$ , where V is the number of unique terms. We then compute the topic-term distribution $\tilde {\boldsymbol {\beta }_k}$ using Equation (4) and use this to derive FREX scores (Bischof and Airoldi Reference Bischof and Airoldi2012).

We assign topic labels by inspecting 50 top images and a random sample of 200 images for each topic. We also look at the top 25 FREX nouns, verbs and adjectives for each topic. Figure 3 shows an example of what this looks like in practice, for a selection of topics. In this case, we find that topics are generally coherent but some are more difficult to label than others, mainly due to being similar to other topics. We include an overview of all topics and broader macro-topic categories in Table 1 in the Supplementary Material.

Figure 3

Top six images from a selection of topics. The top FREX nouns, verbs, and adjectives of a topic are shown above the images and are identified automatically using a part-of-speech tagger on the LLM image descriptions

.

Figure 3 Long description

Six horizontal rows each show six representative images with FREX nouns, verbs, and adjectives. Climate activism contains demonstrations, crowds, and climate-protest signs. Nature contains lakes, forests, mountains, turquoise water, and streams. Political commentary contains cartoons, memes, and satirical illustrations. Political leaders contains close-up or podium photographs of political figures. Causes and consequences contains industrial facilities, smokestacks, smoke or steam, and fire. Event flyers contains text-heavy promotional graphics for climate-related webinars and events.

4.1.5 Step 5: Analyzing covariates and topic prevalence

While steps 1–4 are generally applicable, the use of further parameters for analysis depends on the research question. This said, a key output of the model is $\boldsymbol {\Gamma }$ , which characterizes how image-level covariates influence the prevalence of topics.

Figure 4 exemplifies this in the context of a question that concerns the online discourse of actors for and against progressive climate action. It shows how, in our case, topics vary by actor type, stance, and COP event for a selection of topics, computed using the marginal distribution of topic proportions (Equation 3). Top-producing private individual accounts are likely to post images of “climate activism,” but diverge in their rendition: “For” accounts post undramatic images of turnout in street demonstrations; “against” accounts post images that follow a delegitimizing “protest paradigm” frame (Chan and Lee Reference Chan and Lee1984) by highlighting police confrontations, disruption (e.g., blocking traffic) or spectacle (e.g., activists dressed in costumes). As a group, top-producing politicians, business actors, and scientists post a different set of topics, such as witnessing conference scenes (e.g., “climate panels”). Journalistic actors document summit scenes, political elites, and protests. Their visual coverage of summit scenes (e.g., “event spaces”) decreases, correlating with the pandemic and increasing constraints on media access (Hayes and O’Neill Reference Hayes and O’Neill2025).

Figure 4

Predicted topic proportions by actor type and stance, across each COP for a selection of topics. Topics with a “ $+$ ” are merged into one

.

Figure 4 Long description

Four panels plot predicted topic proportions across COP21–COP27. Blue lines represent actors supporting progressive climate action and orange lines actors against it; marker shapes indicate political, private, journalistic, advocacy, business, and scientific actors. Climate activism varies mostly between about 1 and 10 percent. Political commentary is substantially higher for some actors against climate action, reaching close to 20 percent. Event spaces and venues are generally more prevalent before COP26 and decline sharply at COP26. Climate panels and presentations increase substantially around COP23–COP25, drop at COP26, and rise again at COP27.

Note that our focus here is to show the model in application, and that this figure should be interpreted with caution. Recall that the data consist of unique images posted by highly productive accounts, not the most popular content. There were only a handful of accounts with a stance “against” climate action in this data.

5.1 Semantic validity of topics

We compare the semantic validity of topics produced by vSTM to discrete approaches with STM. We use a dataset of almost 12,000 images related to climate communication from Zhang and Peng (Reference Zhang and Peng2022), for which Arminio et al. (Reference Arminio, Magnani, Piqueras, Rossi and Segerberg2026) produced expert annotations for 500 pairs of images that were annotated for their denotative and connotative similarity on a three-point scale (0 = unrelated, 1 = somewhat related and 2 = closely related). To quantify semantic validity, we use the connotative annotations to adopt a measure for clustering quality from Grimmer and King (Reference Grimmer and King2011). We assign each image to its dominant topic and calculate a quality score Q as the average expert-rated similarity for same-topic pairs, minus the average expert-rated similarity for different-topic pairs. The maximum possible score is 2, the minimum is $-$ 2, and a score of 0 suggests that the model performs no better than random assignment.

We fit several configurations of vSTM and STM to the same set of images. For robustness, we test vSTM with two embeddings: CLIP and GPT4o-generated image descriptions, embedded using a sentence transformer.Footnote ³ For STM, we use BoVW, the same GPT4o descriptions, but represented in a document-feature matrix (DFM), and object detection labels produced by the YOLOE model (Wang et al. Reference Wang, Liu, Chen, Lin, Han and Ding2025). For BoVW, we must set the number of visual words that form the vocabulary. We test several sizes (500, 1,000, 2,500, 5,000, 7,500, and 10,000) and report here the one producing the best results, which is 2,500 visual words. For the object-based STM, we use YOLOE as it has a comparably large vocabulary ( $1,000+$ objects). We represent the images by the number of times an object appears in the image, akin to a DFM. A threshold on the confidence of the predictions must be set to determine which of the detected objects should be kept. This is a trade-off between accuracy and coverage. We use $0.1$ , as we find that a lower threshold generally leads to better results on the evaluation, even as the labels are noisier. A higher threshold quickly leads to many images without a single label, which would force us to remove them from the analysis.Footnote ⁴

We vary the number of topics between $10$ and $100$ to avoid the risk of bias in favor of any particular approach. For each configuration, we fit the model 10 times. We use short-run LDA to initialize STM (Roberts, Stewart, and Tingley Reference Roberts, Stewart and Tingley2019) and draw the initial parameters from a uniform distribution for vSTM.

Figure 5 reports the average quality score Q with a $95\%$ confidence interval. We find that more complex representations (CLIP+vSTM, GPT4o+vSTM, and GPT4o+STM) significantly outperform BoVW+STM and YOLOE+STM across all K. The results indicate that vSTM performs marginally better than STM with GPT4o representations for most K. This aligns with our expectation that converting a rich textual description into a simpler DFM leads to loss of information, which is avoided by vSTM’s use of transformer embeddings. While on average scoring lower than the GPT4o configurations, CLIP+vSTM performs remarkably well compared to BoVW+STM and YOLOE+STM. This is important. CLIP embeddings are available off the shelf, are generally much faster to obtain and do not require designing a prompt. Moreover, the GPT4o prompt was in this case explicitly tuned to summarize the connotative meaning of the images, whereas CLIP’s embeddings are general-purpose. Our main takeaway is that modern, complex image representations are crucial for good performance in visual topic modeling. The continuous vSTM framework is uniquely positioned to fully leverage the strength of these representations.

Figure 5

Quality score for different discrete and continuous configurations using STM and vSTM, respectively

.

Figure 5 Long description

The graph plots quality score Q against the number of topics from 10 to 100 for GPT4o+STM, YOLOE+STM, BoVW+STM, GPT4o+vSTM, and CLIP+vSTM. GPT4o+vSTM, GPT4o+STM, and CLIP+vSTM have substantially higher scores than YOLOE+STM and BoVW+STM across the topic range. GPT4o+vSTM reaches the highest scores at larger numbers of topics. Shaded regions show 95 percent confidence intervals.

5.2 Qualitative inspection of vSTM and bag of visual words + STM topics

We test vSTM’s capacity to generate topics that are coherent, interpretable, and accurate in capturing concepts of substantive interest in comparison to BoVW + STM. Therefore, we replicate Torres’s (Reference Torres2024) analysis of the extent to which media ideology affects coverage of a “caravan” of migrants attempting to enter the United States on foot. An STM is fitted to 688 images represented using a BoVW and identifies a topic of interest: the extent to which dense crowds are present in an image. The “Dense crowd” topic is assumed to be a frame that casts the size of the migrant group as a threat.

We applied vSTM using the same number of topics ( $15$ ) as in the original study and identified those portraying large crowds and migrants. We again use CLIP to generate the image embeddings and Gemma 3 to produce image descriptions and generate topic keywords in the same way as in Section 4. Figures 6 and 7, respectively, show the three vSTM topics that portray migrants and the “Dense crowd” topic from Torres (Reference Torres2024). As the method for generating keywords is agnostic to how the topics were generated, we use the most representative images and top FREX nouns, verbs, and adjectives to label the topics in both cases.

Figure 6

Top 18 most representative images for three migrant topics. The top FREX keywords are computed using the method in Section 3.4

.

Figure 6 Long description

Three grids each contain 18 representative photographs. “Migrant caravan, distant view” shows very large groups travelling along roads and bridges, often photographed from above or at a distance. “Migrant caravan, close-up” shows smaller groups and individuals at street level, walking, waiting, travelling beside vehicles, and carrying belongings. “Children and circumstances” emphasizes children, families, resting or waiting, temporary conditions, limited belongings, and difficult circumstances. Each topic is accompanied by FREX nouns, verbs, and adjectives.

Figure 7

The top 18 most representative images for the “Dense crowd” topic from Torres (Reference Torres2024). The top FREX keywords are computed using the method in Section 3.4

.

Figure 7 Long description

The grid contains 18 representative photographs from the “Dense crowd” topic. Images include tightly packed crowds viewed from above, street gatherings, demonstrations with signs, rallies, seated audiences, public ceremonies, and groups of uniformed soldiers. Some images are dominated by dense crowds, while others contain smaller groups. FREX nouns, verbs, and adjectives are displayed above the grid.

The results show that the vSTM migrant topics are comparatively more coherent with respect to the semantic meaning, both in terms of how similar representative images are, and the interpretability of keywords. The topic “Migrant caravan (distant view)” consistently shows images of the caravan: people walking in a dense crowd, often taken from above or from a distance. The keywords contain words, such as caravan, dense, and migrant. Meanwhile, the “Dense crowd” topic discovered by the BoVW+STM approach is more ambiguous. Among the top images, we find pictures of protests, Trump rallies and soldiers. While the first two include dense crowds, they are arguably less relevant for the particular frame under study.

The upshot is that while the “Dense crowd” topic may better capture how much of a picture is filled with people, it is less likely to adequately capture what is actually portrayed. This suggests that relying on low-level visual features can produce topics that are well defined according to specific graphical or literal criteria but fail to capture meaning in a broader sense. In contrast, by relying on a more powerful representation (here CLIP), vSTM can produce semantically valid topics that are easier to interpret. The other topics in Figure 6 are similarly meaningful yet distinct in portraying migrant groups. For example, the topic we label “Children and circumstances” emphasizes difficult circumstances and vulnerable individuals, especially mothers with children, rather than the size of the caravan.