2.1 Answer-set programming

Answer-Set Programming (ASP) (Brewka et al. Reference Brewka, Eiter and Truszczynski2011; Lifschitz Reference Lifschitz2019) is a well-known approach to declarative problem solving, in which solutions to a problem are described by programs that consist of sets of logical rules. Efficient ASP solvers for evaluating programs are available such as clingo Gebser et al. (Reference Gebser, Kaminski, Kaufmann and Schaub2019), dlv Leone et al. (Reference Leone, Pfeifer, Faber, Eiter, Gottlob, Perri and Scarcello2006), or wasp Alviano et al. (Reference Alviano, Dodaro, Faber, Leone and Ricca2013).

For our concerns, an ASP program is a finite set $P$ of rules $r$ of the form

(1) \begin{equation} a \mathrel {\mathrm{:\!\!{-}}}\ b_1, \ldots ,\ b_n,\ not\ c_1, \ldots ,\ \mathit{not}\ c_n \quad m,n \geq 0,\, \end{equation}

where $a$ (the head), all $b_i$ , and all $c_j$ (the body) are atoms in a first-order predicate language, and $\mathit{not}$ stands for negation as failure (weak negation). We allow that $a$ may be missing (viewed as falsity); then $r$ acts as a constraint. Intuitively, the rule means that whenever all $b_i$ are true and none of the $c_j$ can be shown to be true, then $a$ must be true. A fact is a rule with empty body (in which case $\mathrel {\mathrm{:\!\!{-}}}$ is omitted). Facts represent knowledge that is unconditionally true.

The semantics of a ground (variable-free) ASP program is given by answer sets, which are Herbrand models that satisfy a stability condition (Gelfond and Lifschitz Reference Gelfond and Lifschitz1988). A Herbrand interpretation of $P$ is a set $I$ of ground atoms in the language of $P$ (intuitively, the atoms that are true); it is a model of $P$ if for each rule $r\,{\in }\,P$ either

1. (i) $a \in I$ or

2. (ii) $\{b_1,\ldots ,b_n\}\not \subseteq I$ or

3. (iii) $I\cap \{c_1,\ldots ,c_n\}\neq \emptyset$ ; that is $I$ satisfies $r$ viewed as implication in classical logic.

Furthermore, $I$ is an answer set of $P$ (Faber et al. Reference Faber, Leone and Pfeifer2004)Footnote ² if $I$ is a $\subseteq$ -minimal model of the program

\begin{equation*} P^I = \{ r \in P \mid I \mbox{ satisfies neither } (ii) \mbox{ nor } (iii) \}\,. \end{equation*}

Intuitively, $I$ must result by applying the rules $r$ whose bodies “fire” w.r.t. $I$ starting from facts. The semantics of programs with variables is defined in terms of their groundings (uniform replacement of variables in rules with all possible ground terms).

ASP provides further constructs that extend its modeling capabilities: choice rules allow for selecting a subset of atoms under cardinality bounds. They are of the form

\begin{equation*} l \ \{ a_1,\ldots ,a_k \} \ u \;\mathrel {\mathrm{:\!\!{-}}}\ \texttt {body} \end{equation*}

where $l,u$ are integers specifying the minimum and maximum number of atoms $a_i$ that may be true if the body holds. This supports controlled nondeterminism and is useful for selection problems.

Aggregates enable reasoning over sets of terms, for instance, to impose conditions on their cardinality or sums. Here is an example of a rule that involves an aggregate expression for counting:

\begin{equation*} p(X) \;\mathrel {\mathrm{:\!\!{-}}}\ \#\texttt {count} \{ Y : q(X,Y) \} \gt N. \end{equation*}

This example states that $p(X)$ holds if the number of $Y$ such that $q(X,Y)$ holds exceeds $N$ . ASP supports various aggregates, including #sum, #min, and #max.

While a hard constraint is a constraint that must be satisfied by any feasible solution to the ASP program, a weak constraints (soft constraint) can be violated, but violating it incurs a penalty in the objective function. A weak constraint has the following form:

\begin{equation*} :\ \!\!\sim \texttt {body}.\; [w@l] \end{equation*}

where $w$ is the penalty cost and $l$ the priority level. ASP solvers seek answer sets that minimize the total penalty incurred by the weak constraints whose body is satisfied. Ties are broken using the priority level. For more details on ASP, we refer to (Brewka et al. Reference Brewka, Eiter and Truszczynski2011; Lifschitz Reference Lifschitz2019; Calimeri et al. Reference Calimeri, Faber, Gebser, Ianni, Kaminski, Krennwallner, Leone, Maratea, Ricca and Schaub2020).

2.2 Modular VQA with answer-set programming

We consider three VQA datasets as application area for our rule distillation approach: GQA (Hudson and Manning Reference Hudson and Manning2019), CLEVR (Johnson et al. Reference Johnson, Hariharan, van der Maaten, Fei-Fei, Zitnick and Girshick2017), and $\mbox{CLEGR}^V$ (Bauer et al. 2023, 2024). Example instances of these datasets are shown in Figure 1. They involve realistic scenes in GQA, synthetic scenes in CLEVR, and images of labeled graphs in $\mbox{CLEGR}^V$ .

Fig. 1.

Example scenes and questions from GQA (a), CLEVR (b), and $\mbox{CLEGR}^V$ (c).

For our distillation process, we adopt the logical representations and encodings of neurosymbolic systems that use ASP for reasoning, and instantiate a generic modular neurosymbolic architecture, shown in Figure 2 that has the following core components:

• • Vision module: Neural networks that process the visual scene to produce a symbolic representation of the scene, called the scene graph. It contains a structured representation of each object in the image, with their corresponding attributes and relationships.

• • Language module: Neural networks that process the natural language question to produce a symbolic representation of the question, called a functional program. This is a tree-like structure, where every node defines a primitive function, and edges between the nodes represent the flow of information from the leaves to the root.

• • Reasoning module: A component that uses an ASP solver to derive the answer. First the scene graph and functional program are translated into sets of ASP facts. Details of these representations depend on the dataset, but they have in common that each primitive function of the functional program is represented by a single ASP fact in the question representation. We use an ASP program, which we also refer to as ASP theory, to encode how to derive an answer based on scene and question encodings. An ASP solver computes then an answer set from which the answer to the question is extracted.

Fig. 2.

Generic Neurosymbolic VQA pipeline with three main components: (i) vision module, (ii) language module and (iii) reasoning module. Our distillation method operates on the symbolic representations of scenes and questions, ASP theory, as well as the ground-truth answers to the questions.

The ASP theory is a set of rules that – in contrast to the question and scene encoding – does not change from question to question. The theory is thus general and encodes the semantics of the primitive functions in a uniform way. Using ASP as the symbolic formalism for reasoning provides us not only with a mature ecosystem of tooling and solvers but, and notably, allows one to capture the uncertainty in the predictions of the image-processing component.

The collection of primitive functions the ASP theory implements depends on the task. For GQA, the focus is on reasoning on a complex scene graph, CLEVR focuses on processing compositional questions, and $\mbox{CLEGR}^V$ involves multi-modal reasoning tasks on labeled graphs, for example deciding reachability between nodes with certain labels. Also, the details of how the vision and language module are realized varies from one application to another. We next review some of these details and the characteristics of the datasets used. For each dataset, we present examples of a scene and question as ASP facts, and the complete ASP theory in Appendix A.

2.2.1 The GQA dataset

We use the state-of-the-art GQA dataset, which has been widely adopted recently (Amizadeh et al. Reference Amizadeh, Palangi, Polozov, Huang and Koishida2020; Surís et al. Reference Surís, Menon and Vondrick2023; Liang et al. Reference Liang, Niu, Reganti, Thattai and Tur2020; Li et al. Reference Li, Li, Savarese and Hoi2023). It contains over 22 M open and binary questions that are complex in structure, involve a wide variety of reasoning skills, and have a large number ( $1\,878$ ) of possible answers.

The questions cover more than $100\,000$ images from the Visual Genome dataset (Krishna et al. Reference Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, Bernstein and Fei-Fei2017) that present real-world scenes with a wide variety of object classes, attributes, and relations. GQA provides both a functional representation of each natural-language question and a Visual Genome scene graph for every image, which allows us to construct ASP programs that always infer the correct answer whenever there is perfect language and visual information. We later will erase parts of these programs to simulate incomplete correct programs and ask an LLM to extend them accordingly.

We adopt the representations and theory of a recent system for GQA (Hadl Reference Hadl2023; Eiter et al. Reference Eiter, Hadl, Higuera, Lange, Oetsch, Scheuvens and Strötgen2025) that adheres to the general architecture described above.

For the vision module, the system uses “question-driven” partial scene-graph extraction, where only information is extracted from the scene that is relevant for answering the question at hand. It does so by relying on the foundation models CLIP (Radford et al. Reference Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger and Sutskever2021) and OWL-ViT (Minderer et al. Reference Minderer, Gritsenko, Stone, Neumann, Weissenborn, Dosovitskiy, Mahendran, Arnab, Dehghani, Shen, Wang, Zhai, Kipf and Houlsby2022) for processing the visual scene. They do not need any training and thus solve the task in a zero-shot manner. A snippet of ASP facts from an ASP scene representation is:

These facts describe parts of a scene that contains an object whose most specific class is “baseball_player” (cf. Figure 1).

Questions are translated from their functional program into ASP facts. For example, the question “Is the umpire to the right or to the left of the standing person that is wearing a helmet?” may be rendered as follows:

The numbers encode steps of an evaluation plan for deriving the answer, where the first argument is the current step, followed by numerical input steps providing input and/or additional arguments. All of our ASP question encodings adopt variations of this representation. In comparison with the original functional program, each of these facts represents a primitive function, while numbers, in them represent states, denoting the flow of computation from state to state, serving virtually as edges in a graph where the nodes are the primitives.

The ASP theory encodes the semantics of the primitive functions that can appear in the question encoding. The whole theory has 60 rules; some rules are:

In these rules, TI and TO are variables representing input/output step references, ID represents an object id, and C an object class. These rules define how the state of candidate objects is updated based on the scene, selection criteria, and attribute filters. For example, the select rule propagates objects from a previous step TI to the current step TO if they have the specified class C.

The entire theory is solved alongside both scene and question encodings to produce an answer.

2.2.2 The CLEVR dataset

The second dataset we consider is CLEVR (Johnson et al. Reference Johnson, Hariharan, van der Maaten, Fei-Fei, Zitnick and Girshick2017), which uses synthetic scenes but challenges VQA systems with long compositional questions. The ASP-based VQA system used for CLEVR has been extended by an explanation component (Eiter et al.2022, 2023). The dataset contains synthetically generated images with different objects. The latter vary in their shape, color, size, and material. CLEVR questions require, for example identifying objects, counting, filtering for attributes, comparing attributes, and spatial reasoning. They are formulated in natural language, but, as for GQA, a functional representation is also provided that can be directly parsed into ASP facts.

The architecture of the system for CLEVR from where we take the ASP theory is similar to the one for solving GQA. However, the scenes are less complex as geometric objects are clearly distinguishable, using the popular object-detection framework YOLOv5Footnote ³ to identify all objects in the image, their attributes, and bounding-box information.

A snippet of ASP facts from a scene representation, question representation, and theory are:

These facts describe objects and their different attributes, such as ID, center of the bounding box $X,Y$ positions, material, etc. The question “Is there a small green cube in the scene?” is rendered as follows:

In a similar fashion to GQA, the numbers encode evaluation steps. The ASP theory consists of 72 rules; some of the rules are:

Here, the unique rule propagates exactly one object from step T to T + 1, enforced by a constraint that eliminates any answer set where multiple objects are present at step T. The rules follow essentially the same pattern as those for GQA, evaluating some predicate in a state and then passing this information to the following state.

2.2.3 The $\mbox{CLEGR}^V$ dataset

The recently introduced $\mbox{CLEGR}^V$ dataset (Bauer et al. 2023, 2024) consists of images of graphs that resemble transit networks; the task is to answer questions concerning such graphs. An illustration of a graph and a question that requires to find the shortest path between two stations and count the number of stations that are on that path is shown in Figure 1.

We adopt the ASP theory of the modular neurosymbolic architecture from the same publication. The vision module is multi-modal in the sense that it deals with graph and text information: it combines optical graph recognition (Auer et al. Reference Auer, Bachmaier, Brandenburg, Gleißner and Reislhuber2013) for graph parsing, a tool that parses the graph image into an abstract representation that contains nodes and edges, and a pretrained optical character recognition neural network (Sabu and Das Reference Sabu and Das2018) to parse the image into text labels that are associated with the closest node. The language module parses the natural language questions using either regular expressions or LLMs. Finally, the ASP theory encodes the semantics of the question as a logic program.

A snippet of ASP facts from a scene representation, question representation and theory are:

These facts describe the different nodes, edges and the “metro lines” the nodes and edges belong to, for example the node “Thraonk” is in the brown line. The question “How many stations are in the shortest path between Thraonk and McCiourly?” is encoded as follows:

Again, the structure resembles the same evaluation procedure of GQA and CLEVR. This theory consists of 56 rules; some of the rules are:

The first rule identifies the station pair (N, N’) for which a shortest path is requested, specified by the question encoding using the shortestPath predicate. The second rule uses a choice rule to non-deterministically select directed edges that may form a path between the stations in the pair, allowing at most one direction of the edge. The next two rules define the transitive closure over the selected edges, determining which stations are reachable from each other via the current selection. The constraint ensures that the chosen edges indeed connect the given station pair, eliminating answer sets that fail to do so. The penultimate rule computes the number C of selected edges used for each station pair using the #count aggregate. Finally, the #minimize statement ensures that among all valid edge selections, the one with the smallest number of edges is preferred.

We note that the initial ASP theory is completely manually constructed. In our experiments, we simulate the extension process by removing rules from a “perfect” theory, that is one that always calculates the ground truth answer, to create an incomplete version. This means that we only use theories we know are correct; using faulty initial theories is not in the scope of this article but is an interesting possible direction that involves the remediation of an ASP program.

3.1 Rule distillation algorithm

We present our rule distillation method in Algorithm 1. Assume we want to extend an initial ASP theory $T$ and we have a distillation suite of new VQA examples at our disposal. Each example is a triple $(s,q,a)$ consisting of a scene description $s$ as ASP facts, a question description $q$ as ASP facts, and the expected answer $a$ as an ASP fact.

Algorithm 1

Rule distillation from LLMs.

Algorithm 1 Long description

The code snippet outlines a process for rule distillation from large language models (LLMs). It starts with an initial ASP theory T and a distillation suite of new VQA examples. Each example is a triple consisting of a scene description s as ASP facts, a question description q as ASP facts, and the expected answer a as an ASP fact. The algorithm iterates over each example in the distillation suite, checking if the ASP solver with the current theory and question equals the expected answer. If it does, it continues; otherwise, it generates rules using different prompts and checks their syntax and semantics. If the rules pass these checks and regression tests, they are added to the theory.

The algorithm iterates over all examples (line 5); for each example, it calls an ASP solver to check whether the expected answer can be derived from the theory accompanied with the scene and question facts. If so, it skips to the next example (line 8); otherwise it tries to update the theory.

We apply different prompting strategies – more details follow below – to obtain a list of candidates for extending the theory. This involves one strategy in which we ask for several variants of a solution in one prompt, as well as two different chain-of-thought prompts. Typically, the quality of the resulting rules varies and they are not always correct. By synergetic use of the prompting strategies, we thus create a pool of options from which we will subsequently aim to pick one that works (lines 10–12).

For each candidate rule (line 14), we use an ASP solver to deduce the answer for the current example:

• – If we get a syntax error (line 15), we pass the error message to the LLM and prompt it to revise the rules (line 16); we call this step syntactic mending.

• – If we get an answer set from the ASP solver, we check whether it matches the expected answer (line 18). If not, we pass the answer to the LLM and ask it to update the rules (line 19); we call this step semantic mending.

• – We then we do regression testing, that is testing whether a new rule appended to the theory gives us correct answers for all previously used examples (line 21). This assures consistency and is in line with an online setting where future examples cannot be assumed to be known in advance. Only after passing this step, we accept the suggested theory extension (line 22), add the example to the regression list (line 23) and break to the next example (line 24).

The models used tend to generally follow the instruction to return ASP rules, yet we observed that mistakes and hallucinations broadly fall into three categories: (i) syntactic, such as unsafe rules; (ii) semantic, where the generated rules fail to produce the expected answer for the current example; and (iii) regressive, where a rule may answer a specific question correctly but introduces a semantic conflict with previously distilled rules. These errors are systematically handled by our algorithm, using the ASP solver to ensure only syntactically valid and globally consistent rules are integrated into the theory. We illustrate the three error types using the target primitive verify_attr(TO, TI, A, V), which checks whether an object at state $\texttt {TI}$ possesses an attribute $\texttt {A}$ with value $\texttt {V}$ , and if so, state $\texttt {TO}$ is assigned “yes.”

• • (i) Syntactic errors: The model generates invalid ASP syntax, often hallucinating mathematical notation (e.g. $\forall$ , $\in$ ) symbols that are not supported by the solver.

  

• • (ii) Semantic errors: The rule is syntactically valid but fails to produce the correct answer for the current example. In the case below, the model incorrectly requires the object to be “burnt” to satisfy the verification.

  

• • (iii) Regressive errors: The model generates a rule that works for the current example but conflicts with a previously distilled rule.

  

  Now the program derives yes (and no) when we present the uncooked red lobster, provoking a regression failure.

3.2 Prompting

We prompt the LLM in Algorithm 1 for different purposes: either to generate new rules or to revise rules if a syntax or semantics test failed. For generating rules, we use different prompting strategies, but all extend the same preprompt that sets the stage. The prompts are rather general and only need small adjustments if different VQA tasks are considered. Although it is well known that details of how prompts are formulated can have a great impact on the results, we are striving for clarity and readability instead of performance optimization. The complete prompts are presented in Appendix B.

3.2.2 Multi-prompting

We use the strategy that instructs the LLM to produce not only one solution for the task, but to suggest several alternative solutions; we call this multi-prompting. We apply this strategy in line 10 of Algorithm 1. The advantage is that we can possibly obtain several solution candidates that we can evaluate using a single call to the LLM. This increases the chances of finding a correct solutions without the need for several potentially expensive LLM calls. We use the following prompt:

The parameter N is set to 5 by default.

3.2.3 Chain-of-thought

Chain-of-Thought (CoT) prompting is a technique in which an LLM is instructed to break down complex tasks into a series of intermediate steps (Wei et al. Reference Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le and Zhou2022). This often helps the model to produce a more accurate and coherent response. We adopted this idea for generating declarative rules, where we instruct the model to break down a problem solution as a set of if-then conditions and to translate them in a subsequent step into ASP rules. We use two versions of this idea in Algorithm 1: a simple, rather straight-forward version (line 11), and a more intricate one that asks for more detailed explanations (line 12). The simple version is as follows:

The complex version of chain-of-thought prompting, which is more verbose and structured than the previous, is as follows:

3.2.4 Prompts for rule mending

If the suggested rules do not produce the expected results, we prompt the LLM to correct them. We leverage feedback from the ASP solver to accomplish this. In particular, if a syntax test fails, we provide the LLM with the error message from the solver and a request to suggest a corrected version. Here is the prompt to achieve this:

If the syntax is correct but we do not get the expected answer, we pass the answer from the ASP solver to the LLM and explain that we expected the correct answer instead. The prompt looks as follows:

The semantic mending also includes the expected answer and the inferred answer, if any.

4.1 Evaluation platform

The evaluation platform is a workstation with an Intel Core i7-12700K CPU, 32 GB of RAM, and an NVIDIA GeForce RTX 3080 Ti GPU with 12 GB of video memory. The ASP solver we use is clingo Gebser et al. (Reference Gebser, Kaminski, Kaufmann and Schaub2019), version 5.8.0, from the Potassco solver suite,Footnote ⁵ an state-of-the-art system. All experiments were run 5 times with a timeout of 600 s. For reproducibility, we logged all our parameters, random seeds, input prompts, and resulting rules.

4.2 Instances

We create two sets of examples for each dataset, a distillation suite for the distilling process of Algorithm1, and a testing suite to benchmark the distillation results. The suites were created aiming at having proportionally balanced examples of all primitive functions of each dataset.

For GQA, we used about $45k$ instances from its “balanced training set” and then divided this set into a distillation and test suites of about $35k$ and $10k$ instances, respectively. For CLEVR, the splits are taken from its training set and contain $35k$ examples for distillation and $15k$ for testing. For $\mbox{CLEGR}^V$ , they also belong to the training set and contain $2k$ examples for distillation and $1k$ for testing.

Distillation sample. Our preliminary experiments showed that the rules generated during distillation tend to be general, often covering multiple examples at once when a rule is distilled. To control resource usage and demonstrate the efficiency of our approach with limited data, we do not use the complete distillation suite at every step, but instead sample a small, fixed number of examples ( $N = 10$ ) for each distillation attempt. While using larger sample sizes may improve performance by exposing more patterns, it is more costly and can also lead to deterioration due to increased prompt length and the resulting strain on the LLM’s reasoning capabilities. We have chosen this value empirically after our preliminary experiments, as it was large enough to produce rules without prohibitive cost. On the other hand, for evaluation always the full test suite is used.

4.3 LLM selection

The landscape of LLMs is rapidly evolving, with new models and variants emerging frequently, making it infeasible to cover all available options in our evaluation. We therefore focus on a representative subset of competitive and accessible models.

We ran experiments with several LLMs, both locally hosted and online, API-based ones. Small models that can be easily self-hosted showed unfortunately very poor performance for the considered rule distillation task. We tried in particular GPT4ALLFootnote ⁶ “wizardlm-13b” and Zephyr 7b $\beta$ ,Footnote ⁷ which is a fine-tuned version of the Mistral 7B model developed by the Hugging Face H4 team.

Ultimately, we resort to larger API-based models. Our selection includes GPT-4oFootnote ⁸ from OpenAI, which is the company’s most advanced model, notable for its multi-modal capabilities (text, vision, and audio) and high alignment performance, accessible via a paid API service. In addition, we include three open-source models: DeepSeek,Footnote ⁹ a flagship Chinese-English model designed for long-context reasoning and GPT-level performance; Mistral Large,Footnote ¹⁰ optimized for fast inference and general-purpose usage under an open-weight license (code not public but model weights are); Meta’s LLaMA 3 70B-Instruct,Footnote ¹¹ which offers strong instruction-following ability and high-quality outputs across tasks; and last, Gemini-3 Pro,Footnote ¹² which exhibits a new “high reasoning” mode that in practice translates to an embedded preprompt that makes the LLM “think” before answering. In terms of scale, the models range from LLaMA 3 (70 billion parameters), Mistral Large ( $\approx 123$ billion) DeepSeek-V2 (236 billion), and Gemini-3 Pro (estimated 1 trillion), up to GPT-4o, with estimated 1.8 trillion parameters.

The temperature parameter of an LLM regulates the randomness of its output: lower values (e.g. near 0) make the model more deterministic by favoring high-probability tokens, while higher values introduce greater variability. We set a temperature of 0 for the LLMs which theoretically assures deterministic behavior. Yet in practice, there are multiple sources of non-determinism when working with LLMs, such as different tokenizers Ali et al. (Reference Ali, Fromm, Thellmann, Rutmann, Lübbering, Leveling, Klug, Ebert, Doll, Buschhoff, Jain, Weber, Jurkschat, Abdelwahab, John, Suarez, Ostendorff, Weinbach, Sifa, Kesselheim and Flores-Herr2024), floating point errors, and versions of systems under the same name which can evolve in the background.

4.4 Theory extension simulation

For each VQA dataset, we start with the complete ASP theory $T$ , and remove all rules that mention a selected predicate $P$ that occurs in some question representation. Intuitively, $P$ represents the ASP implementation of some primitive function, as described in Section 2, which can be encoded in the theory using one or more rules. We take a representative selection from the complete set of predicates and use samples as described in Subsection 4.2 from the distillation suite, where each example selected must mention $P$ . We then iterate over the examples using Algorithm1 to distill rules that can effectively calculate the expected answer.

The results for the datasets GQA, CLEVR, and $\mbox{CLEGR}^V$ are shown in Tables 1, 2 and 3, respectively. The tables report, for each model and predicate in the dataset, the average number of examples used ( $n$ ) and the average accuracy (%) achieved over the test suite, along with its standard deviation; the best results are in bold. The column labeled $\mathrm{T}\,{\setminus }\,P$ displays the accuracy of the initial ASP encoding with predicate $P$ removed, which serves as a baseline. The drop in accuracy depends on the number of questions affected and the role of predicate $P$ ; for example select is used in almost every question of GQA as removing it drops the accuracy to 9.60%. The other columns illustrate the performance of the evaluated LLMs; “—” indicates that there is no improvement, meaning, no new rules where produced.

Table 1.

Restoring the initial theory $\mathrm{T}$ of GQA after removing all rules mentioning predicate $P$

Table 1 Long description

A table comparing the accuracy of different models on various predicates. The table has 10 rows and 6 columns. The columns are labeled Predicate, GPT-4o, DeepSeek, Mistral, LLaMA3, and Gemini-3. Each row corresponds to a different predicate and lists the accuracy percentages and standard deviations for each model. Row 1: query, 97.44 ± 2.48, 98.01 ± 2.27, 98.86 ± 0.08, 86.62 ± 21.9, 98.92 ± 0.00. Row 2: exist, 100.0 ± 0.00, 99.98 ± 0.01, 99.38 ± 1.06, 89.58 ± 6.01, 100.0 ± 0.00. Row 3: or, 100.0 ± 0.00, 100.0 ± 0.00, 100.0 ± 0.00, 94.27 ± 1.67, 100.0 ± 0.00. Row 4: filter, 97.24 ± 3.38, 98.62 ± 2.76, 100.0 ± 0.00, 100.0 ± 0.00, 100.0 ± 0.00. Row 5: choose_attr, 99.83 ± 0.00, 99.79 ± 0.06, 98.44 ± 2.76, 99.74 ± 0.07, 99.83 ± 0.00. Row 6: verify_rel, 98.94 ± 1.23, 100.0 ± 0.00, 94.70 ± 2.52, 96.44 ± 3.00, 100.0 ± 0.00. Row 7: select, 100.0 ± 0.00, 99.77 ± 0.46, 99.53 ± 0.38, 32.20 ± 39.1, 100.0 ± 0.00. Row 8: negate, 99.42 ± 0.70, 100.0 ± 0.00, 98.98 ± 0.57, 99.04 ± 0.59, 100.0 ± 0.00. Row 9: relate, 98.13 ± 2.69, 98.76 ± 1.52, 99.11 ± 1.54, 76.74 ± 12.3, 100.0 ± 0.00. Row 10: two_different, 100.0 ± 0.00, 100.0 ± 0.00, 100.0 ± 0.00, 99.55 ± 0.45, 100.0 ± 0.00. Row 11: two_same, 100.0 ± 0.00, 100.0 ± 0.00, 100.0 ± 0.00, 98.76 ± 0.00, 100.0 ± 0.00.

Table 2.

Restoring the initial theory $\mathrm{T}$ of CLEVR after removing all rules mentioning predicate $P$

Table 2 Long description

Panel A: Table comparing the performance of different models on various predicates for the GQA dataset. The table lists predicates in the first column and the performance metrics for each model in subsequent columns. The models include GPT-4o, DeepSeek, Mistral, LLaMA3, and Gemini-3. Each cell contains the average accuracy percentage along with the standard deviation. The best results are highlighted in bold. Panel B: Table comparing the performance of different models on various predicates for the CLEVR dataset. The structure is similar to Panel A, with predicates listed in the first column and performance metrics for each model in subsequent columns. Panel C: Table comparing the performance of different models on various predicates for the CLEGR^V dataset. The structure is similar to Panels A and B, with predicates listed in the first column and performance metrics for each model in subsequent columns.

Table 3.

Restoring the initial theory $\mathrm{T}$ of $\mbox{CLEGR}^V$ after removing all rules mentioning predicate $P$

Table 3 Long description

A table comparing the accuracy of different models on various predicates. The table has 12 rows and 6 columns. The columns are labeled Predicate, GPT-4o, DeepSeek, Mistral, LLaMA3, and Gemini-3. Each row corresponds to a different predicate, and the values represent the average accuracy percentage along with the standard deviation. Row 1: cycle, 99.19 ± 0.00, 88.75 ± 0.32, 87.44 ± 2.31, 99.78 ± 0.00. Row 2: shortest_path, 91.73 ± 6.07, 85.69 ± 1.93, 85.10 ± 1.16, 100.0 ± 0.00. Row 3: paths, 94.56 ± 2.98, 80.74 ± 7.45, 95.56 ± 2.12. Row 4: count_nodes, 91.91 ± 8.52, 86.50 ± 0.38, 100.0 ± 0.00. Row 5: adjacent, 100.0 ± 0.00, 78.90 ± 7.30, 88.99 ± 0.33, 81.29 ± 1.96, 86.11 ± 23.3. Row 6: adjacent_to, 100.0 ± 0.00, 74.74 ± 11.8, 88.59 ± 0.57, 100.0 ± 0.00. Row 7: same_line, 98.76 ± 2.34, 64.35 ± 22.7, 87.14 ± 0.20, 100.0 ± 0.00. Row 8: common, 98.60 ± 0.53, 96.70 ± 3.70, 98.22 ± 0.50. Row 9: line_names, 100.0 ± 0.00, 87.62 ± 4.22, 100.0 ± 0.00. Row 10: line_count, 100.0 ± 0.00, 92.11 ± 8.68, 100.0 ± 0.00. Row 11: exist, 95.38 ± 6.17, 93.80 ± 3.10, 100.0 ± 0.00.

A general overview per dataset allows one to see that for both GQA and CLEVR, LLMs were highly successful when distilling rules for them. On the other hand, $\mbox{CLEGR}^V$ appears to be more challenging, as accuracies are in general inferior to the ones in other datasets, and some models could not produce an answer in some cases. This can be explained by the difficulty of the predicates involved in $\mbox{CLEGR}^V$ , such as cycle and shortest path, in contrast to the focus on spatial and comparison relationships from GQA and CLEVR, which are more straightforward to implement. The predicates in $\mbox{CLEGR}^V$ are usually larger in terms of number of rules used to encode them but are also more complex, as they involve recursion, while none of the rules for GQA nor CLEVR are recursive.

Analyzing the results of every model reveals consistent patterns across datasets. GPT-4o remained the most reliable model overall, achieving near or perfect accuracy with low deviation, across most predicates and datasets. Notably on CLEVR, it was able to recover or produce semantically equivalent rules to the ones removed. DeepSeek showed strong performance as well, often trailing GPT-4o on GQA and CLEVR, but noticeably worse in $\mbox{CLEGR}^V$ .

Mistral-Large demonstrated competitive results as well, being on par with DeepSeek on GQA and CLEVR (with the exception of the unique predicate in CLEVR where no rules were produced). Yet it followed the trend of deteriorating results for $\mbox{CLEGR}^V$ , now some distillation attempts on particular predicates without improvement.

Last, LLama3-70b’s performance was acceptable for GQA, producing rules for almost every predicate, but with high variance and a low score on select, where the accuracy was only 32.20%. For CLEVR, and in particular $\mbox{CLEGR}^V$ , the results were noticeable worse; no new rules were produced for the distillation of many predicates.

Gemini-3 demonstrated exceptional performance across all datasets, consistently achieving near-perfect accuracy (Tables 1–3). Notably, it achieved 100% accuracy on several complex graph-based predicates in $\mbox{CLEGR}^V$ , where other high-end models showed slight performance drops. We attribute this to the model’s high-reasoning mode, which allows an LLM to (i) explore multiple reasoning paths, (ii) using “verification loops” to ensure that their reasoning is correct, and (iii) allocates a significant share of time to this preparatory reasoning step. In our setting, these characteristics are translated into a model that is more precise and robust than the rest.

The findings highlight that the distillation process used few examples and proved considerably successful for GPT-4o, DeepSeek, Mistral-Large and Gemini-3 on GQA and CLEVR, whereas Mistral-Large and LLaMA3-70b struggled on $\mbox{CLEGR}^V$ where complex graph reasoning tasks posed greater difficulty.

Table 4.

Average number of examples used by model and dataset

Table 4 Long description

A table with four rows and five columns compares the average number of examples used by different models across various datasets. The columns are labeled with the names of the models: GPT-4o, DeepSeek, Mistral, LLaMA3, and Gemini-3. The rows are labeled with the names of the datasets: GQA, CLEVR, and CLEGRV. Each cell contains a value representing the average number of examples used, along with a standard deviation. Row 1: GQA, GPT-4o, 4.04 ± 1.09; DeepSeek, 2.16 ± 0.32; Mistral, 4.02 ± 1.50; LLaMA3, 4.03 ± 0.67; Gemini-3, 2.79 ± 0.10. Row 2: CLEVR, GPT-4o, 3.22 ± 0.50; DeepSeek, 2.34 ± 0.16; Mistral, 3.90 ± 1.29; LLaMA3, 6.48 ± 1.83; Gemini-3, 2.58 ± 0.32. Row 3: CLEGRV, GPT-4o, 4.02 ± 1.51; DeepSeek, 2.40 ± 0.32; Mistral, 5.80 ± 0.75; LLaMA3, 5.02 ± 0.24; Gemini-3, 1.73 ± 0.53.

Regarding the number of examples used, it always happened that fewer than all available $N=10$ examples per distillation attempt were used. This is because the LLM may have already produced a set of rules general enough such that it can cover all future examples, or a set of rules was produced that the LLM could not improve upon. In general, the amount was remarkably low, using 3–8 examples to produce a set of rules. While we set an upper limit of $N=10$ examples for our experiments, in practice, typically we use 1–4 examples per distillation attempt as shown in Table 4. The LLM often reaches a “fixpoint” during the process: once it produces a rule general enough to cover the representative logic of a question type, presenting further examples does not result in additional rules. Gemini-3 was highly efficient, often requiring the fewest number of examples (averaging to only 1.73 for $\mbox{CLEGR}^V$ ) to converge on a correct set of rules.

Table 5.

Average and standard deviation token usage per dataset and model

Table 5 Long description

A table titled ‘Average and standard deviation token usage per dataset and model’ presents data for three datasets: GQA, CLEVR, and CLEGRV. The table includes four models: GPT-4o, DeepSeek, Mistral-Large, LLaMA3-70B, and Gemini-3. The table has three main sections, each corresponding to a dataset, with columns for input (In) and output (Out) token usage. Each cell contains the average token usage and its standard deviation. Row 1: GPT-4o, GQA In: 11259 ± 762, GQA Out: 359 ± 353, CLEVR In: 6862 ± 743, CLEVR Out: 356 ± 42, CLEGRV In: 3615 ± 336, CLEGRV Out: 1931 ± 206. Row 2: DeepSeek, GQA In: 8096 ± 360, GQA Out: 319 ± 24, CLEVR In: 3092 ± 445, CLEVR Out: 793 ± 130, CLEGRV In: 3321 ± 357, CLEGRV Out: 325 ± 458. Row 3: Mistral-Large, GQA In: 13591 ± 1494, GQA Out: 788 ± 635, CLEVR In: 6551 ± 532, CLEVR Out: 475 ± 540, CLEGRV In: 12604 ± 1188, CLEGRV Out: 1331 ± 164. Row 4: LLaMA3-70B, GQA In: 33565 ± 2494, GQA Out: 2587 ± 2048, CLEVR In: 25935 ± 3007, CLEVR Out: 2942 ± 3708, CLEGRV In: 39716 ± 2574, CLEGRV Out: 5166 ± 4701. Row 5: Gemini-3, GQA In: 8050 ± 205, GQA Out: 128 ± 40, CLEVR In: 3892 ± 311, CLEVR Out: 82 ± 36, CLEGRV In: 4585 ± 278, CLEGRV Out: 368 ± 214.

For practitioners, the efficiency and operational cost of the distillation process are primarily determined by the token consumption of the LLMs. While input costs are largely a function of the preprompt size and the number of examples, output costs vary significantly based on the model’s verbosity and its ability to adhere to the constraint of returning only ASP rules. Table 5 provides a breakdown of the average input and output tokens per distillation attempt on 100 distillation attempts for each dataset using a random predicate. We observe that the results align with the number of examples needed per distillation attempt shown in Table 4, as models like DeepSeek and Gemini-3 use fewer tokens than the competitors. This is because their higher precision reduces the number of failed distillation attempts and repeated API calls triggered by the feedback loop. In contrast, other models like LLama often incur higher costs through multiple retries and exhibit high variance.

4.5 Ablation experiments

Our distillation algorithm consists of a series of constituents including syntax mending, semantic mending, multi-prompting, two types of CoT and regression testing. We produce ablation studies focusing on three major elements of our prompting approach, which we refer to as distillation strategies:

• • No mending: When removing both syntactic and semantic mending,

• • No CoT: When removing both simple and complex chain of thought prompts, and

• • No multiprompt: When removing multi-prompting from the prompt.

Table 6 reports the average accuracy of each model on each dataset with and without ablation. The best accuracy in each row is shown in bold; the best model without ablation for each dataset is additionally underlined. When a model failed to generate valid rules for a predicate, the accuracy was replaced with that of the theory $\mathrm{T} \setminus P$ as baseline.

Table 6.

Average accuracy by model and ablation condition. For each dataset, best configuration per model is in bold, best model under full configuration is underscored

Table 6 Long description

The table presents the average accuracy of different models across three datasets: GQA, CLEVR, and CLEGRV. It includes six models: GPT-4o, DeepSeek, Mistral-Large, LLama3-70b, and Gemini-3. The table is divided into four columns: Full, No mending, No CoT, and No multi-prompt. Each row represents a model’s performance under different conditions. The best configuration per model for each dataset is highlighted in bold, and the best model under the full configuration for each dataset is underscored. The datasets and models are listed as follows: GQA with models GPT-4o, DeepSeek, Mistral-Large, LLama3-70b, and Gemini-3; CLEVR with models GPT-4o, DeepSeek, Mistral-Large, LLama3-70b, and Gemini-3; CLEGRV with models GPT-4o, DeepSeek, Mistral-Large, LLama3-70b, and Gemini-3. The values in the table represent the average accuracy percentages along with their standard deviations. For example, under the GQA dataset, GPT-4o has an accuracy of 99.18 ± 1.09 under the Full condition, while Gemini-3 has an accuracy of 99.89 ± 0.32 under the same condition. Similar detailed accuracy values are provided for other models and conditions across all datasets.

Across datasets, GPT-4o and DeepSeek achieve consistently strong results. GPT-4o in particular benefits from the full prompting pipeline, especially on GQA and $\mbox{CLEGR}^V$ : disabling any strategy in these settings leads to clear performance drops of up to 10%. DeepSeek shows similar behavior, though in CLEVR, disabling CoT slightly improves performance, indicating that simpler prompting can sometimes improve performance even on strong models. In $\mbox{CLEGR}^V$ , the impact of these strategies is generally less pronounced. Mistral-Large and LLama3-70b, which trail behind on GQA and $\mbox{CLEGR}^V$ , perform comparably to larger models on CLEVR, especially when some strategies are deactivated. In these cases, reduced prompting appears to ease the burden on smaller models. For instance, LLama3-70b improves by 5%–10% on several CLEVR predicates when mending is disabled. The same models exhibit high variance and inconsistent behavior when distilling rules. LLama3-70b frequently fails to distill rules for harder tasks in $\mbox{CLEGR}^V$ , regardless of strategy, and Mistral-Large sometimes fails completely for certain GQA or $\mbox{CLEGR}^V$ predicates if strategies like multi-prompting are disabled. The ablation study for Gemini-3 reveals a remarkable stability compared to other models. While most models suffered significantly when mending or multi-prompting were disabled, Gemini-3 maintained an average accuracy of over 93% even without mending. Unlike Mistral or LLaMA3, Gemini-3 was not hindered by complex prompting strategies, showing that its reasoning depth is sufficient to handle the full pipeline without being overburdened.

A more thorough analysis of the experiments is given in Appendix C.

The efficacy of distillation strategies is not uniform but rather highly correlated with the intrinsic reasoning capabilities of the underlying model. We observe three distinct performance archetypes: (i) Overburdened Models (Mistral-Large, LLaMA3-70b), where the computational or context overhead of complex strategies like CoT and multi-prompting often outweighs their benefits, leading to better performance when these modules are deactivated; (ii) Balanced Models (GPT-4o, DeepSeek), which effectively leverage these strategies to bridge reasoning gaps, showing clear performance gains when all distillation components are active; and (iii) Saturated Models (Gemini-3), which possess such high baseline capabilities that they render some strategies redundant.

4.6 Composing new ASP predicates

We explore furthermore the synthesis of new ASP predicates by composing existing operations from a fixed ASP theory $T$ , specifically on the GQA dataset. The primitives (e.g. select, has relation, filter, verify_rel) provide the foundational semantics for reasoning over visual scenes. Our goal is to test whether language models can define new predicates compositionally from those in $T$ , in particular those with higher arity and those that use different ASP constructs.

We showcase how to introduce a new predicate, in_between, which is a spatial predicate that relates three objects, and evaluate whether LLMs can provide its semantics using only the existing predicates in $T$ .

The process unfolds in several steps:

1. 1. Manual extension of the theory: We begin by adding rules to handle a new predicate $P$ to $T$ , such that it implements the desired predicate using predicates previously defined. For example, the predicate in_between can be implemented as:

   

2. 2. Sampling modifiable examples: We identify examples where the structure of the scene graph allows for a transformation into a query that involves the new predicate. These examples are modifiable into ones that speak about the desired predicate. For example, for the in_between predicate, one searches for scenes with three or more objects.

3. 3. Transforming the examples: Each selected example is transformed to

   • • a new ASP question using the new predicate (e.g. in_between),

   • • an updated answer, and

   • • an optionally modified scene if additional attributes or relations are needed.

4. 4. Dataset splitting: We create distillation and testing suites.

5. 5. Rule distillation: To obtain new rules, we use Algorithm 1, testing whether the LLM can produce $P$ by composing existing operations from $T$ using the generated examples as guidance.

This method enables us to use a VQA dataset, in this case GQA, to probe the capabilities of LLMs to generate new predicates from existing ones. Besides in_between, we construct additional predicates to further evaluate predicate composition (their ASP encodings are given in Appendix D):

• • For connected, we select two objects from the scene and ensure the presence of a path between them through one or more intermediate objects, that is we synthesize relational chains by adding has_rel(A, connects, B) relations (e.g. object A is next to object B, and B is to the left of C) to form paths of length greater than two. The query then tests whether such indirect connections exist.

• • For isolated, we identify an object and systematically remove all of its relationships from the scene graph, making it structurally disconnected. The associated query asks whether the object is isolated.

• • For count_class, we leave the scene unchanged and simply construct a query that counts the number of objects belonging to a given class. We note counting is not part of the original GQA primitives.

We use new distillation and test suites following the procedure described before, with $500$ examples each.

For each new predicate, we report the average number $n$ of examples used, accuracy, and standard deviation, as shown in Table 7.

Table 7.

Distillation for producing new predicates removing all rules mentioning predicate $P$ . Best average accuracy per predicate in bold

Table 7 Long description

A table comparing the performance of different models on various predicates. The table has 5 columns and 5 rows, including the header row. The columns are labeled as follows: P, G P T-4 o, DeepSeek, Mistral-Large, L Llama3-70b, and Gemini-3. The rows are labeled as follows: in_between, connected, isolated, and count_class. Each cell contains a value representing the average accuracy and standard deviation for each model on each predicate. The values are as follows: Row 1: in_between, G P T-4 o: 97.39 ± 2.82, DeepSeek: 99.23 ± 1.31, Mistral-Large: 93.46 ± 3.26, L Llama3-70b: 92.82 ± 3.20, Gemini-3: 98.85 ± 1.15. Row 2: connected, G P T-4 o: 100.0 ± 0.0, DeepSeek: 99.84 ± 0.32, Mistral-Large: 99.92 ± 0.16, L Llama3-70b: 84.20 ± 22.4, Gemini-3: 100.0 ± 0.0. Row 3: isolated, G P T-4 o: 100.0 ± 0.0, DeepSeek: 100.0 ± 0.0, Mistral-Large: 100.0 ± 0.0, L Llama3-70b: 74.09 ± 26.3, Gemini-3: 100.0 ± 0.0. Row 4: count_class, G P T-4 o: 100.0 ± 0.0, DeepSeek: 100.0 ± 0.0, Mistral-Large: 100.0 ± 0.0, Gemini-3: 100.0 ± 0.0.

GPT-4o can successfully provide the intended semantics for all novel predicates connected, isolated, and count_class with perfect accuracy, the exception being in_between, which achieves ca. $97\%$ .

Deepseek and Mistral both match GPT-4o on isolated and count_class. The former even surpasses GPT-4o on in_between, but at the cost of a lower than perfect score on connected. Mistral presents lower scores than DeepSeek on both in_between and connected. LLama3-70b produced the worst results, in particular no rules were produced for count_class. Last, Gemini-3 positions itself again as the best performant model overall, achieving perfect accuracy on most of the predicates.

This experiment demonstrates the potential of LLMs to extend symbolic theories compositionally, a critical capability for scenarios where rules must be dynamically generated or adapted to new domains.

4.7 Assessment of research questions

We now turn to our research questions R1–R4:

1. (R1) Given a VQA task and correctly parsed examples of scenes, questions, and answers into ASP, can our approach extend the ASP theory to deal with questions that require operations/steps not yet encoded?

   LLMs can effectively recover missing ASP rules, with strong performance on GQA and CLEVR. Although performance drops on $\mbox{CLEGR}^V$ , key primitive operations were still captured, promoting the viability of rule distillation across domains.

2. (R2) Which LLMs are suitable for our method?

   Only large-scale models like GPT-4o, DeepSeek and Gemini-3 consistently succeed across tasks. Mistral-Large performs moderately well, while LLama3-70b struggles, showing that both scale and instruction-following capabilities are crucial for robust rule distillation.

3. (R3) How do the individual prompting strategies contribute to the overall performance?

   Distillation strategies (mending, CoT, multi-prompting) substantially improve performance for top models, but can hinder weaker ones. GPT-4o and DeepSeek depend on them for complex cases, while Mistral and LLama3-70b often benefit from simpler prompting.

4. (R4) How do LLMs fare when prompted to compose new predicates from already existing ones?

GPT-4o, DeepSeek, and Gemini-3 can compose new ASP predicates using primitives from the base theory, even for recursive and aggregation-based rules. Mistral succeeds on simpler constructs, while LLama3-70b remains unreliable for compositional synthesis.

Regarding scalability with respect to the number of rules of the initial theory, we observe that the distillation algorithm operates locally by processing one primitive predicate at a time. Consequently, the total size of the theory has low impact on the effectiveness of it. In fact, larger initial theories may improve performance by providing the LLM with richer context to prevent reasoning drift, whereas extremely sparse theories risk insufficient context. In terms of reasoning capabilities, our experiments across GQA, CLEVR, and $\mbox{CLEGR}^V$ demonstrate robustness in spatial, equality, counting, and recursive graph-related tasks, covering a vast landscape of ASP constructs. A future direction is to extend this approach to temporal datasets, leveraging ASP’s native temporal extensionsFootnote ¹³ to handle time-dependent predicates within the VQA encoding.

In summary, our results show that LLMs can effectively recover missing ASP rules in the context of VQA and extend symbolic theories with new logical rules. Success depends heavily on both dataset complexity and model capabilities: simpler tasks in GQA and CLEVR are addressed with high accuracy, while structurally demanding domains like $\mbox{CLEGR}^V$ , which involve recursion and aggregation, require larger models and robust prompting strategies.

5.1 Rule redundancy heuristic

We present our redundancy elimination procedure in Algorithm2. Assume we have the results of a distillation process for a predicate $P$ : a set $R$ of new ASP rules produced by the LLM, the initial theory $T \setminus P$ , and a pruning sample drawn from the distillation suite.

Since rules generated later in Algorithm 1 tend to be more general due to regression testing, as new distilled rules must capture past examples as well, we attempt to eliminate rules from first to last. The algorithm iterates over all rules $r_i \in R$ in the order they were generated (line 7).

For each rule, it temporarily removes $r_i$ to form a reduced set $Q$ (line 8), and calls evaluate to compare the behavior of extending $T$ with both sets on the pruning sample (line 9). If the outputs are identical, the rule is marked as redundant (line 10). After all candidates have been checked, the redundant rules are removed from $R$ (line 12).

Algorithm 2

Rule redundancy elimination.

Algorithm 2 Long description

An algorithm for eliminating redundant rules in a set of ASP rules. The algorithm starts with an initial theory TP, a list of new ASP rules R produced by the LLM, and a pruning sample of training scenes, questions, and answers. It initializes an empty list of redundant rules. For each rule ri in R, it sets Q to R minus the rule ri. It then evaluates the theory TP plus R against the pruning sample and compares it to the evaluation of TP plus Q against the pruning sample. If the evaluations are the same, the rule ri is added to the list of redundant rules. Finally, the algorithm sets R to R minus the set of redundant rules.

We use a sample from the examples for distillation also for pruning. If a rule can be removed without affecting correctness on this sample, it is marked as redundant and discarded.

Figure 3 visualizes the outcomes of applying Algorithm2 on the new rules generated in the main experiments (Section 4.4) for GQA, CLEVR, and $\mbox{CLEGR}^V$ ; values in tabular form are presented in Appendix E. For each predicate, the bars indicate the average number of rules pruned across the sets of new rules produced by the LLM. We use a sample of 1000 examples from the corresponding distillation suite of each predicate.

Across all datasets, pruning reveals meaningful differences in how LLMs approach rule generation. GPT-4o consistently produces more extensive rule sets, often leading to high pruning counts – such as 8.8 rules for unique in CLEVR – suggesting a thorough but sometimes redundant exploration of possible solutions. In contrast, Mistral-Large and LLama3-70b tend to yield more compact outputs, with near-zero pruning for several predicates, such as relate or select in GQA. Gemini-3 exhibited the lowest pruning rates. In many cases, it produced zero redundant rules. This indicates that the model “high” reasoning mode allows to target the exact logic required by the VQA primitive as an ASP rule.

Fig. 3.

Average number of rules pruned for each predicate across different models on the GQA, CLEVR, and $\mbox{CLEGR}^V$ datasets.

Low pruning, combined with high accuracy and a low number of examples, indicates that the LLM has produced general and useful rules. In contrast, high pruning with many examples suggests exploratory behavior, where the LLM initially generates a broader set of rules that are gradually refined through the distillation process. However, when high pruning occurs despite few examples being used, it may indicate that the LLM is generating superfluous rules – statements that do not affect the outcome.

GQA and CLEVR present the highest pruning rates overall, as their simpler and more compositional predicates encourage LLMs to generate varied candidate rules. In contrast, $\mbox{CLEGR}^V$ shows consistently lower pruning counts across models – not due to more efficient generation, but because its increased complexity makes rule synthesis harder in the first place, often resulting in fewer or no rules to prune. The pruned rules – alongside observed syntactic or semantic errors – offer possibly valuable data that can be potentially incorporated as prompt feedback or leveraged as training data to fine-tune models toward more accurate rule generation.

6.1 Answer-set programming and LLMs

Recently, LLMs have been explored in the context of ASP. Ishay et al. (Reference Ishay, Yang and Lee2023) observed that LLM reasoning capabilities are shallow, but they can serve as a highly effective semantic parser to transform input into ASP representations. These are then used to solve logical puzzles. Furthermore, Bauer et al. (Reference Bauer, Eiter, Ruiz and Oetsch2024) recently proposed to use LLMs to parse questions into ASP facts in the context of VQA for images of graphs.

Rajasekharan et al. (Reference Rajasekharan, Zeng, Padalkar and Gupta2023) showed that the combination of LLMs with ASP in their STAR framework produces good results for natural language tasks that require qualitative reasoning, mathematical reasoning, and goal-directed conversation. Santana et al. (Reference Santana, Kareem and Ricca2024) proposed a two-step system for translating natural language to ASP via a controlled natural language intermediate form, demonstrating accurate rule generation on graph-based tasks. LLASP (Coppolillo et al. Reference Coppolillo, Calimeri, Manco, Perri and Ricca2024) is an approach that involves fine-tuning LLMs specifically for ASP, showing that model adaptation can significantly improve rule generation accuracy in logic programming tasks.

Our work is different as we do not focus on semantic parsing but on the more challenging task of knowledge distillation, where we aim for a system that can produce sound logical rules capturing knowledge about a particular domain.

Recent work by Zhu et al. (Reference Zhu, Xue, Chen, Zhou, Tang, Schuurmans and Dai2023) explores the use of LLMs to learn rules from arithmetic and kinship relationships, yet the rules they learn do not contain variables and their semantics is informal.

6.2 Statistical-relational learning

Similar to our approach, methods from SRL (Raedt and Kersting Reference Raedt and Kersting2017), in particular from ILP (Muggleton Reference Muggleton1991; Muggleton and Raedt Reference Muggleton and Raedt1994; Cropper et al. Reference Cropper, Dumancic, Evans and Muggleton2022), aim to produce rules from example data and a background theory. SRL has seen great advances in terms of scalability by, for example applying gradient-based boosting (Gutmann and Kersting Reference Gutmann and Kersting2006), and systems like ILASP (Law et al. Reference Law, Russo and Broda2020b) and FastLAS (Law et al. Reference Law, Russo, Bertino, Broda and Lobo2020a) provide means for inductively learning expressive ASP programs. Kareem et al. (Reference Kareem, Gallagher, Borroto, Ricca and Russo2024) combine LLMs with ILASP to induce commonsense rules from answer sets for robust QA, requiring only few examples.

SRL adopts a statistical and probabilistic perspective that is fundamentally data-driven. In contrast, our approach does not perform learning. While we also make use of a dataset, its role is to guide a knowledge distillation process by conditioning an LLM, rather than to serve as training data. Notably, only a few examples are sufficient for this purpose.

Traditional ILP uses a search-based approach where the solutions produced are correct and minimal under some criteria. A key aspect of many ILP systems are mode declarations that define the syntactic form of allowed rules to restrict the search space of possible programs. They are given in a formal language and tacitly assume an intuition about the form of the solution. The distillation approach does not require this; we only elicit knowledge that is already present in the LLM, and the information in the prompt that instructs what rules we want is informal and in natural language. When prompting LLMs, rules are general by command – while optimality is not enforced, it may happen implicitly. In very recent work, Yang et al. (Reference Yang, Wu and Yue2025) propose to use LLM agents to generate the language bias itself, enabling automated hypothesis generation in ILP.

6.3 Modular neurosymbolic VQA

There are several VQA systems that feature a modular architecture that combines subsymbolic with symbolic components (Yi et al. Reference Yi, Wu, Gan, Torralba, Kohli and Tenenbaum2018; Mao et al. Reference Mao, Gan, Kohli, Tenenbaum and Wu2019; Amizadeh et al. Reference Amizadeh, Palangi, Polozov, Huang and Koishida2020; Eiter et al. Reference Eiter, Higuera, Oetsch and Pritz2022; Surís et al. Reference Surís, Menon and Vondrick2023; Johnston et al. Reference Johnston, Nogueira and Swingler2023).

Specifically, Yi et al. (Reference Yi, Wu, Gan, Torralba, Kohli and Tenenbaum2018) used a pipeline to extract a scene graph (a list of all objects detected in the image with their attributes and positions) from the image. They then translated the provided question into a structured representation of the reasoning steps, called functional program, and executed this program on the structural scene representation to obtain an answer. The authors showed excellent results on the popular CLEVR dataset (Johnson et al. Reference Johnson, Hariharan, van der Maaten, Fei-Fei, Zitnick and Girshick2017). This approach has been advanced with logic-based reasoning processes, for example by Differentiable First-Order Logic ( $\nabla$ -FOL) (Amizadeh et al. Reference Amizadeh, Palangi, Polozov, Huang and Koishida2020), or by ASP (Eiter et al. Reference Eiter, Higuera, Oetsch and Pritz2022). These reasoning processes can consider not just the most probable scene-graph prediction, but rather the entire vector of probabilities as output by the object detection and attribute/relation classifier networks that form the visual perception component of the VQA pipeline.

While our focus is on VQA, related developments exist in textual QA. For instance, Nguyen et al. (Reference Nguyen, Pham, Vuong, Trinh and Nguyen2025) use LLMs in combination with ASP to detect misleading content by generating logic-based explanations. Their system highlights parts of a text responsible for incorrect or hallucinated answers, illustrating the potential of symbolic reasoning for enhancing transparency of QA pipelines.

Foundational models including Vision-Language Models (VLMs), such as BLIP-2 (Li et al. Reference Li, Li, Savarese and Hoi2023) and SimVLM (Wang et al. Reference Wang, Yu, Yu, Dai, Tsvetkov and Cao2022) have become sufficiently strong through their pre-training regimes to generalize well to multiple different datasets. Approaches that use these VLMs as components are, for example ViperGPT (Surís et al. Reference Surís, Menon and Vondrick2023), CodeVQA (Subramanian et al. Reference Subramanian, Narasimhan, Khangaonkar, Yang, Nagrani, Schmid, Zeng, Darrell and Klein2023), and PnP-VQA (Tiong et al. Reference Tiong, Li, Li, Savarese and Hoi2022).

Regarding systems for neurosymbolic learning, it should be mentioned that there are logic-based systems that employ the semantic loss (Xu et al. Reference Xu, Zhang, Friedman, Liang and den Broeck2018) to use the information produced by the reasoning module to improve the learning tasks of the neural networks involved (Yang et al. Reference Yang, Ishay and Lee2020; Manhaeve et al. Reference Manhaeve, Dumancic, Kimmig, Demeester and Raedt2021).