2.1 FrameNet

Frame Semantics (Fillmore Reference Fillmore1985) seeks to describe the meaning of a sentence as it is actually understood by characterizing the background knowledge necessary to understand the sentence. Background knowledge is represented in the form of frames, conceptual structures modelling prototypical situations. From a linguistic perspective, a frame is a semantic class containing predicates that can evoke the described situation. These target words or expressions are called frame evoking elements (FEE). Table 1 shows the frame Statement, which describes a specific type of a communication situation and is evoked by verbs such as acknowledge or admit, and by nouns such as affirmation.

Table 1.

Example frame from the FrameNet database

Each frame comes with its own set of semantic roles called frame elements. These are the participants and propositions in the abstract situation described. In the case of Statement, they are the Speaker and Addressee of the statement, the Message conveyed and its Topic. Roles are local to individual frames, thus avoiding the commitment to a small set of universal roles, whose specification has turned out to be unfeasible in the past.

The Berkeley FrameNet project (Baker et al. Reference Baker, Fillmore and Lowe1998) has been developing a frame semantic lexicon for the core vocabulary of English since 1997. The current FrameNet release contains about 800 frames and 10,000 lexical units. Part of FrameNet is also a corpus of annotated example sentences from the British National Corpus, currently containing 135,000 sentences. As FrameNet is still an ongoing project, the coverage of the resource is not complete. One of the goals of our study is to assess if coverage is already sufficient for RTE.

In FrameNet, asymmetric frame relations can relate two frames (called Superframe and Subframe), forming a complex hierarchy which situates frames in a semantic space. A relation also provides a partial or complete role mapping between the two linked frames. The most important frame relations are Inheritance and Subframe, complemented by the Uses relation. Inheritance is the strongest relation: anything that is true in the semantics of the Superframe, must also be true for the Subframe (e.g. Killing – Execution). Subframe indicates that the Subframe describes a subpart of a more complex situation described in the Superframe (e.g. Criminal-Process – Sentencing). Using is a loose relation in which a part of the situation evoked by the Subframe refers to the Superframe. In addition to these IS, A-like relations, FrameNet provides four additional relations: Causative_of, Inchoative_of, Precedes and Perspective_on. Frame relations can be exploited in RTE to infer complex entailments when text and hypothesis represent different but related situations (e.g. Killing – Death).

In recent years, a number of projects are investigating the use of Berkeley FrameNet frames for languages other than English, such as German ((Boas Reference Boas2005) and the SALSA project (Burchardt et al. Reference Burchardt, Erk, Frank, Kowalski, Pado and Pinkal2006)), Spanish (Subirats and Petruck Reference Subirats and Petruck2003), Italian (De Cao et al. Reference De Cao, Croce, Pennacchiotti and Basili2008) and Japanese (Ohara et al. Reference Ohara, Fujii, Ohori, Suzuki, Saito and Ishizaki2004).

2.2 The role of frame semantics in RTE

The annotation of predicate-argument structure in general, and of FrameNet frames in particular, is interesting for its intermediate position between syntax and ‘deep’ compositional semantics. Frame semantics is not concerned with problems of deep semantic analysis such as modality, negation or scope ambiguity; instead, it focuses on structured meaning information on the level of aboutness (‘who did what to whom’). This shallow level of semantic analysis is well suited for RTE, which in its original definition aims at discovering entailment relations looking at the shallow textual level, disregarding deep semantic axiomatizations (Dagan et al. Reference Dagan, Glickman, Magnini, Quiñonero-Candela, Dagan, Magnini and D'Alché-Buc2006).

Several studies (e.g. Bar-Haim, Szpektor and Glickman Reference Bar-Haim, Szpektor and Glickman2005; Vanderwende and Dolan Reference Vanderwende and Dolan2005; Burchardt and Frank Reference Burchardt and Frank2006; Litkowski Reference Litkowski2006; Clark et al. Reference Clark, Harrison, Thompson, Murray, Hobbs and Fellbaum2007) indicate that the level of granularity offered by FrameNet is relevant for modelling many phenomena which occur in the current textual entailment corpora. For example, Bar-Haim et al. (Reference Bar-Haim, Szpektor and Glickman2005) show that paraphrases at the predicate level constitute the most relevant inference mechanism for detecting textual entailment: they contribute (either fully or partially) in 31% of the RTE-2 positive dataset.Footnote ¹ These numbers are comparable to those obtained in the RTE-2 annotating RTE (ARTE) annotation (Garoufi Reference Garoufi2007, see Section 4.1), which demonstrates that around 20% of the inferences in the positive examples of the RTE-2 corpus involve nominalization or argument variations, which can be (fully or partially) solved by using knowledge at the frame level. Similar numbers are reported for the RTE-1 dataset by (Vanderwende and Dolan Reference Vanderwende and Dolan2005): authors demonstrate that the use of syntactic alternations and a general thesaurus is sufficient to solve 49% of entailment cases in RTE-1.

We further analysed the ARTE corpus to estimate how many of the positive examples could be solved by using PropBank instead of FrameNet. Since PropBank can capture only lemma-specific syntactic alternation (i.e. those involving the same predicate), not surprisingly it turned out that it is potentially useful in only 2% of positive examples, far below the 20% contribution offered by FrameNet. This clearly indicates that the higher degree of generalization offered by FrameNet is more promising for inference oriented tasks with respect to PropBank.

The main FrameNet contribution for RTE is the generalization across near meaning-preserving transformations such as alternation in voice and word class or in lexicalization. For example, all of the following sentences are analysed in a uniform way, as instances of the frame Commerce_goods-transfer with BuyerBMW and GoodsRover.

1. (3) BMW bought Rover from British Aerospace.

2. (4) Rover was bought by BMW, which financed [. . .] the new Range Rover.

3. (5) BMW's purchase of Rover for $1.2 billion was a good move.

4. (6) BMW, which acquired Rover in 1994, is now dismantling the company.

These types of meaning-preserving transformations defined inside a single frame can be leveraged in RTE, as in the following positive example:

1. (7) T: [Virtual reality]_Means is used to train [surgeons, pilots, astronauts, police officers, first-responders and soldiers]_Student. (frame: Education_teaching)

2. (8) H: [Soldiers]_Student are trained using [virtual reality]_Means.(frame: Education_teaching)

FrameNet can also contribute to RTE by suggesting not straightforward, inferential relations via the frame hierarchy. As an example, consider the positive pair (9)/(10) from the RTE-2 test set corpus, where entailment can only be established by linking ‘arrested’ and ‘accused’:

1. (9) T: Wyniemko, now 54 and living in Rochester Hills, was arrested and tried in 1994 for a rape in Clinton Township.

2. (10) H: Wyniemko was accused of rape.

Here, the word ‘accused’ evokes the frame Notification_of_charges, while the word ‘arrested’ evokes Arrest. As can be seen in Figure 1, a Notification_of_charges situation typically presupposes (i.e. entails) an Arrest. This piece of knowledge can be successfully used to derive the correct value of entailment.

Fig. 1.

Making use of frame relations.

Although the above examples illustrate the intuitive appeal of using FrameNet frames to model textual entailment, it remains to be seen if its use is feasible in actual systems for RTE. This will be the focus of the next section.

3.1 Automatic frame semantic analysis

Automatic role semantic analysis (or shallow semantic parsing) aims at labelling texts with predicate-argument information derived from predicate repositories such as FrameNet and PropBank. In the literature, many approaches have been proposed for solving this task, starting from Gildea and Jurafsky (Reference Gildea and Jurafsky2002), and up to the shared tasks at CoNLL (Carreras and Marquez Reference Carreras and Marquez2005) and Senseval-3 (Mihalcea and Edmonds Reference Mihalcea and Edmonds2004). Most approaches adopt machine learning techniques, where manually annotated examples are used to train a classifier based on lexical and syntactic features.

Automatic frame semantic analysis is the more specific task of labelling unrestricted text with FrameNet frame and role information. It is typically treated as two separate subtasks: frame assignment, i.e. decide which word should be labelled with which frame label; and role assignment, i.e. decide which constituents in the text realize which roles of the evoked frame. In frame assignment, the target FEE is given, and systems only have to find the correct frame evoked by the FEE, a task similar to Word Sense Disambiguation. In role assignment, systems have both to identify the candidate role fillers (role recognition), and to assign the correct role to each candidate (role labelling).

In some of the experimental studies described in this paper, we adopt Shalmaneser (Erk and Pado Reference Erk and Pado2006), a freely available state-of-the-art system for shallow semantic parsing based on machine learning techniques and pretrained on the FrameNet corpus. Shalmaneser offers a complete ‘tool-box’ architecture for frame and role assignment, with preprocessing modules to elaborate input from the Collins and the Minipar syntactic parsers. Shalmaneser offers high performance, with an accuracy of 0.93 on frame assignment, and F-scores of 0.85 and 0.78 respectively on role recognition and labelling (Padó Reference Padó2007) if evaluated on the FrameNet sample corpus. Note, however, that the degree of lexical ambiguity is artificially low in this corpus because of the incompleteness of the FrameNet lexicon, so the baseline is already relatively high. More realistic figures can be observed if we consider the German data annotated by the SALSA project (Erk and Pado Reference Erk and Pado2006), namely F-score 0.60 and 0.67 respectively on role recognition and labelling.

Shalmaneser can be boosted with ‘Detour to FrameNet’ (Burchardt, Erk and Frank Reference Burchardt, Erk, Frank, Fisseni, Schmitz, Schröder and Wagner2005a), a rule-based frame assignment system, which addresses gaps in FrameNet's coverage by using WordNet to infer correct frame assignment for unknown FEE.

Shalmaneser and Detour are used in combination in the SALSA RTE system (see next section), to support the recognition of FrameNet based entailments.

3.2 Predicate-argument structure in RTE systems

Several systems for computing textual entailment explicitly integrate predicate-argument structure and frame knowledge. The general approach adopted by these system is to first use shallow semantic parsers to automatically annotate (T, H) pairs with either FrameNet or PropBank semantic classes and roles. Then, they use statistics on class and role matchings across text and hypothesis, to build a feature space or to elaborate decision rules.

The best performing RTE system to date, proposed by (Hickl and Bensley Reference Hickl and Bensley2007) at the RTE-3 challenge, partially uses PropBank knowledge. In detail, the system performs a first step in which it extracts the most relevant textual fragments of the text (called commitment). Then, it uses a shallow semantic parser to annotate both the commitment and the hypothesis. Finally, several features are extracted from the commitment and the hypothesis; among these, PropBank features indicate if aligned entities in the commitment and the hypothesis are assigned the same role labels. Unfortunately, authors do not indicate what is the contribution of the PropBank features in achieving the high performance.

PropBank is also used to some extent in (Adams et al. Reference Adams, Nicolae, Nicolae and Harabagiu2007), but authors demonstrate that the performance obtained by the system is lower than those of a simpler algorithm using only extended lexical overlap techniques. Finally, in (Raina et al. Reference Raina, Haghighi, Cox, Finkel, Michels, Toutanova, Bill, Marie-Catherine, Christopher and Andrew2005) PropBank role information is used to augment syntactic dependencies. Yet, in both systems, the role of PropBank in the overall entailment prediction is not clear.

FrameNet, on the other hand, has been used for RTE so far only in the SALSA system (Burchardt and Frank Reference Burchardt and Frank2006) participating at the RTE-2 and RTE-3 challenges. The system is centred around the frame semantic analysis provided by the Shalmaneser and the Detour systems introduced above. It performs three main steps: (a) a linguistic analysis in terms of lexical functional grammar (LFG) f-structures and a FrameNet projection assigning frames and role to constituents; (b) the computation of a ‘match graph’ encoding the ‘semantic overlap’ between text and hypothesis and (c) a machine learning decision step, building the feature space by using matching information at the syntactic and frame level, derived from the match graph.

The system approximates textual entailment in terms of the ‘semantic overlap’ between text and hypothesis. It compares their LFG f-structures with frame-semantic and ontological projection by determining compatible matching nodes and edges. Nodes match if they are labelled with identical frames or predicates, or if the nodes are semantically related on the basis of WordNet or FrameNet frame relations. A node in the hypothesis may match multiple nodes from the text and vice versa. Edge matching is restricted to edges that connect matching nodes, or nodes taking identical atomic values. The SALSA system scored in the RTE competitions in the middle ranges, obtaining 0.59 and 0.62 accuracy respectively at the RTE-2 and RTE-3 competitions.

As reported in Burchardt et al. (Reference Burchardt, Reiter, Thater and Frank2007), the SALSA system using FrameNet-based features does not achieve significantly higher performance than a simple lexical overlap system. This evidence, strengthened by the above mentioned results obtained in Adams et al. (Reference Adams, Nicolae, Nicolae and Harabagiu2007), indicates that predicate-argument structures and frame knowledge have not yet been implemented in a very successful way in RTE systems. In the next section we will investigate more in depth possible causes of this problem.

3.3 Practical issues in using frame semantics in RTE

Examples discussed in Section 2.2 clearly indicate that an analysis at the level of frame semantics, and of predicate-argument structure in general, is a promising semantic contribution to the challenge of modelling textual entailment. Yet, as the previous section has shown, only few RTE systems have tried to leverage this type of knowledge in practice. Also, there is no assessed experimental evidence proving that the use of FrameNet or PropBank actually leads to a relevant increase in performance. For example, the SALSA system's frame semantic component did not significantly outperform the lexical baseline on the RTE-2 corpus, as one would expect, given that around 20% of the entailment pairs involve variation in predicate argument structure (as reported in Section 2.2).

This ‘implementation gap’ between the intuitive appeal of frame semantics and its practical use in textual inference, has so far discouraged a wide use of FrameNet in real settings. It is then critical to understand which are the main reasons of this gap. Different factors can influence the applicability of frame semantics to tasks such as RTE.

1. (i) Quality of automatic frame semantic analysis. Systems using predicate-argument knowledge strongly rely on semantic parsers producing the annotation. One reason for low overall performance could be that the quality of the annotation produced by the parsers is not good enough to significantly support RTE inferences.

2. (ii) Knowledge modelling. Even if appropriate annotations are provided, it can also be the case that this information is not optimally accessed and modelled by current RTE systems, either as a feature space or as a set of rules. For example, the SALSA system has been proven to be too permissive when computing the compatibility of text and hypothesis in its learning model.

3. (iii) Resource coverage. A third possible factor is that the coverage of the FrameNet lexicon is not good enough.

The goal of the following sections is to explore the above mentioned factors in depth, by studying the impact of frame semantics on the task of RTE, and estimating its potential for related tasks. To this end, we built FATE, a manual frame annotation of the RTE-2 challenge test set, on top of an automatic syntactic analysis (Section 4). Such annotation gives a relevant contribution in solving the problem, by providing the following evidence. First, it enables the evaluation of the impact of the FrameNet resource coverage, as we explicitly ask the annotators to indicate frame information evoked in the entailment pairs that are not covered by the FrameNet resource (Section 5.1). Second, the manual annotation indicates how relevant the quality of automatic semantic analysis is in using FrameNet. This can be achieved by directly computing the accuracy of a state of the art semantic parser (in our case Shalmaneser) over the manual gold standard (Section 5.2). Finally, it can be used to assess the complexity of frame semantics knowledge modelling in real applications. Indeed, the corpus is a valid resource to manually study to what extent FrameNet normalizations can be practically modelled and leveraged by an RTE system, by discriminating positive and negative examples (Section 5.3).

4.1 Annotation of the RTE-2 test set

The annotation has been carried out on the test set used in the second RTE challenge (Bar-Haim et al. Reference Bar-Haim, Dagan, Dolan, Ferro, Giampiccolo, Magnini and Szpektor2006). The corpus consists of 800 text-hypothesis pairs, evenly split into 400 positive and 400 negative pairs. Frames are annotated on top of a syntactic structure produced by the Collins parser (Collins Reference Collins1999). FEEs can be individual words or constituents, and the role fillers of a frame refer to syntactic constituents. In addition to FrameNet frames, two special ‘pseudo-frames’ are used in the annotation: a so-called unknown frame is used in cases where a FEE evokes a situation not represented in the FrameNet lexicon; an Anaphora frame is used to relate anaphoric expressions to their referents, keeping the annotation local, in the sense that the role fillers of a frame must be within the maximal projection of the FEE's constituent.Footnote ³ In rare cases, existing FrameNet frames are extended with unknown roles. The annotation of support verb and copula constructions follows the FrameNet annotation guidelines, i.e. the noun rather than the verb is analysed as a FEE. We found four sentences in the dataset involving a metaphoric expression, for which we annotated the figurative rather than the literal meaning. The annotated corpus is available at the SALSA project page.Footnote ⁴

Relevant FEEs. One important property of the FATE annotation is that only relevant FEEs are annotated, i.e. words that evoke frames which are somehow relevant to the overall situation(s) described in the text at hand. Indeed, textual entailment inferences are mainly supported by properties and descriptions of relevant facts. To illustrate the idea, consider the following example:

1. (11) T: Authorities in Brazil say that more than 200 people are being heldhostage in a prison in the country's remote, Amazonian jungle state of Rondonia.

2. (12) H: Authorities in Brazil hold 200 people as hostage.

A complete annotation would select all words displayed in boldface as FEEs. In the FATE annotation, we only annotate the underlined relevant FEEs (but words considered not relevant as FEE are still available as arguments of frames). Indeed, these are the only words which evoke frames describing the overall situations in H and T: ‘hostage’ evokes the Kidnapping frame, ‘say’ evokes Statement.

While the notion of a relevant FEE clearly relies on intuition, we achieved a very high inter-annotator agreement in a pilot annotation for a small, randomly selected set of 15 sentences from the RTE-2 test set. Two experienced annotators have been asked to annotate all and only those words which they consider relevant, achieving an average agreement of 87% if individual annotations are compared to the set of commonly annotated FEEs. Given this pilot study, we elaborated some operational characterization of relevance, which has been used to guide the main annotation. In particular, only words which evoke frames that have at least one role instantiated in the text are candidates for counting as relevant.

Relevant spans. For many (positive) pairs in the RTE datasets only parts of the texts contribute to the inferential process that allows the hypothesis to be derived from the text. A typical example is shown below, where only the sections in bold face are really important:

1. (13) T: His wife Strida won a seat in parliament after forging an alliance with the main antiSyrian coalition in the recent election.

2. (14) H: Strida elected to parliament.

We annotated only the specific sections within the text/hypothesis pairs that contain relevant material for the task of textual entailment recognition. The annotators were provided with a markup of the these sections, referred to as relevant spans. Relevant spans are derived automatically on the basis of a separate alignment annotation (see below). Note that we compute relevant spans only for positive pairs. For negative pairs we perform full annotation, as no (reliable) alignment that could be used to compute them is available.

Alignment corpora. The basic intuition behind the span annotation is that if a text entails a hypothesis, it is usually possible to ‘embed’ the hypothesis into the text, i.e. to map the words in the hypothesis to corresponding words in the text in some systematic and consistent way. For datasets that have been annotated with such lexical alignments, it is straightforward to automatically compute relevant spans by selecting the smallest section of the text that contains all words used in the alignment links.

We are aware of two publicly available annotations that provide annotations for such mappings: The ARTE-Annotation (Garoufi Reference Garoufi2007), and an annotation provided by the Microsoft Research Natural Language Processing Group (Brockett Reference Brockett2007). While these two annotations differ in some aspects,Footnote ⁵ they produce similar results for the task of automatically computing relevant spans: if we pairwise compare the spans for the ARTE annotation to the spans for the Brockett annotation using the Jaccard coefficient (considering spans as sets of words), we obtain an average value of 0.84.

In a pilot study, we checked if the relevant spans derived for these annotations indeed capture the relevant information of the text. To this end, we gave 20 positive pairsFootnote ⁶ from the RTE2 test set to two annotators, who were asked to mark the relevant FEEs. We then checked whether the relevant FEEs are part of the span computed on the basis of the two alignment annotations. Results show that all spans for the ARTE-annotation contain the most important FEEs, while the spans for the Brockett annotation don't contain it in three cases. This study thus indicates that relevant spans can be computed reliably. Furthermore, it indicates that the ARTE annotation is slightly better suited for computing relevant spans.

For our annotation, we decided to use the ARTE annotation to compute relevant spans.

4.2 Annotation process and statistics

The annotation has been carried out by an experienced annotator, initially trained and calibrated on a pilot dataset. Texts and hypotheses have been annotated independently of each other to avoid influences of the T-annotation while the annotator annotates H, and vice versa.

The whole annotation was carried out in 230 h: 90 h for the positive examples, 140 h for the negatives. In average, it took 13 min to annotate a positive pair, and 21 min to annotate a negative. As positive and negative examples have in average the same number of tokens, these statistics clearly show the contribution of the span-annotation on speeding up the process. In particular, the span-annotation focused the annotation of the positive texts on only 46% of tokens, i.e. more than a half of the positive texts are outside the spans and left unannotated.

Due to resource and time constraints, we could not afford a full double annotation, but we computed inter- and intra-annotator agreements for subsets of the corpus, in order to verify consistency and correctness.

Inter-annotator agreement. 80 pairs (10%) of the dataset have been annotated by a second annotator. For this subcorpus, we computed the agreement on frame recognition, frame labelling, role identification. Frame recognition agreement is the percentage of plain words (verbs, nouns, adjectives and adverbs) that have been commonly annotated as either being or not being a FEE. We obtained an agreement of 93%, corresponding to a 0.60 Kappa level (moderate agreement). Frame labelling agreement is the percentage of commonly assigned frames, among those evoked by all FEEs recognized by both annotators. Annotators agreed on 87% of cases, corresponding to a Kappa level of 0.86 (almost perfect agreement). Role identification agreement is the percentage of phrases that have been commonly annotated (same phrase and same role label) by the annotators.Footnote ⁷ We obtained an agreement of 97%, corresponding to a 0.88 Kappa level (almost perfect agreement). Role agreement is fairly in line with results obtained in other complex annotation efforts, as in the PropBank annotation, where annotators achieved 0.91 Kappa on the combined task of role recognition and labelling (see Palmer et al. Reference Palmer, Gildea and Kingsbury2005).

These results indicate that the overall annotation is reliable. In particular, agreement results on frame recognition show that our notion of relevant FEE is effective. Disagreement cases mostly occur in long and complex sentences, which describe more than one situation. In such cases, it is less easy for annotators to discern which situations are relevant (i.e. a FEE is annotated) and which are not. Also, once the FEE has been selected, the tasks of finding the correct frame and the correct roles is fairly easy and unambiguous. The sporadic cases of disagreement on frames usually involve the choice of different but highly similar frames (e.g. Risky_situation versus Run_risk) or an unknown frame used by one annotator instead of the correct one present in the FrameNet hierarchy. Cases of disagreements on roles are generally due by one annotator missing a role.

Intra-annotator agreement. The RTE-2 test set contains repeated sentences across the pairs, which allow an intra-annotator agreement test to be performed. We estimated the agreement over the positive corpus. In all, we counted 109 repetitions. Over this set we computed a FEE agreement of 97%, a frame agreement of 98% and a role agreement of 88%, proving a reliable level of consistency of the overall annotation.

5.1 FrameNet coverage

The FATE corpus contains 4488 annotated frames and 9512 annotated roles, with an average of 5.6 frames per pair, and 2.1 roles per frame. Table 2 reports the most frequent frames occurring in the corpus, and their number of occurrences. This list gives a general idea about the semantic domains characterizing the RTE-2 corpus, mostly referring to killing, disaster and competition events.

Table 2.

Most frequent annotated frames in the RTE-2 test set

The annotation contains 373 Unknown frame instances, accounting for only the 8% of the total frame instances. Unknown roles are 1% of the total roles. This means that FrameNet coverage for the RTE corpus is surprisingly good. These numbers differ from figures reported, for example, for SALSA's German corpus annotation (Burchardt et al. Reference Burchardt, Erk, Frank, Kowalski, Pado and Pinkal2006), where one third of the verb occurrences could not be annotated with available FrameNet frames (largely due to the incompleteness of the frame inventory, not to cross-lingual differences). One possible reason for the discrepancy may be that only relevant frames have been annotated in FATE. Also, the annotators of FATE were allowed to annotate frames that looked appropriate in a rather flexible way, while the SALSA annotation for German followed a stricter annotation guideline. All in all, the 8% coverage lack of FrameNet is unlikely to be an important factor for the implementation gap discussed in Section 3.3.

5.2 Quality of automatic frame semantic analysis

In this section, we report an experiment carried out to assess the accuracy of systems for automatic frame semantic analysis on the RTE corpus. We compared the output of the Shalmaneser system (Erk and Pado Reference Erk and Pado2006) against the FATE corpus as a gold standard.

5.2.2 Analysis of Results

Table 3 shows the results for the different tasks. Results on frame match indicate that the systems have some difficulties in assigning exactly the gold standard frame (Frame match strict). Yet, the large improvement from strict match to relaxed match (e.g., Precision +13% and Recall +15% for SHA), shows that the systems often assign a frame which is similar to the gold standard frame, and candidate to be correct. Filler match results are relatively good, showing that in most cases, a correctly hypothesized role is filled with the correct syntactic constituent.

Table 3.

Systems performance on frame, role and filler match over the gold standard

Results for frame, role, and filler match are lower than results of the same shallow semantic parser for English reported in (Erk and Pado Reference Erk and Pado2006). This is probably due to two facts. First, Shalmaneser has been trained and tested on different kinds of corpora; sentences in the RTE dataset are typically longer and less ‘prototypical’ if compared to the FrameNet sample corpus Shalmaneser has been trained on. Second, the FATE annotation allows a liberal annotation style. The latter aspect can be addressed by the use of the Detour system, improving Recall at the cost of lower Precision.

Overall, the comparison of the gold standard annotation with automatic frame and role assignments shows that the performance of Shalmaneser is comparably low and needs to be improved, either by implementing ad-hoc heuristics, or by training semantic parsers over RTE-specific corpora, such as our RTE-2 test set gold standard.

5.3 Knowledge modelling

In this section we examine knowledge modelling as a possible cause of the implementation gap. The most straightforward means to do that would be to run a FrameNet-based RTE algorithm, such as the SALSA system, on our gold standard corpus. Low performance would most likely indicate a lack in knowledge modelling. Yet, as the gold standard has only the size of a test set, we cannot expect significant results from rerunning a full system on this small dataset (especially in a machine learning setting). Indeed, an exploratory run of the SALSA system on four-fold cross validation on the gold standard, while obtaining slightly better performance than those on the Shalmaneser output, showed a too large variance among folds to draw reliable conclusions.

As an alternative means to study the impact of knowledge modelling, we present an experiment based on a simple algorithm which extracts frame-based statistical information from the positive and the negative examples of the annotated corpus. This information aims at capturing the overlap of frame structures between text and hypothesis in an entailment pair. In that way, we reproduce at the frame semantic level a classical ‘embedding’ approach to knowledge modelling, adopted by many existing RTE systems, including SALSA. The underlying assumption of the approach is that the more of the hypothesis can be ‘embedded’ into the text, the more probable it is that entailment holds. This approach has been adopted at different levels of analysis, such as lexical, syntactic, or semantic (e.g. Bos and Markert Reference Bos and Markert2006; Kouylekov and Magnini Reference Kouylekov, Magnini, Magnini and Dagan2006). In our case, embedding is at the frame semantic level, and consists in finding how many frames, roles and role fillers of the hypothesis are also in the text.

5.3.2 Analysis of results

Results for the simple algorithm are displayed in Tables 4–6. Results for frame overlap (Table 4) show that indeed the gold standard offers a notable discriminative power between positive and negative examples, around 14%. Compared to the baseline lexical overlap, frame overlap shows a slightly but nonsignificantly higher discriminative power (+10.5% versus +14% difference). To get a clearer picture, we produced two rankings of the pairs in the dataset, one according to frame overlap, and another according to lexical overlap. Then, we computed the best accuracy, i.e. the accuracy in dividing positive and negative examples, at the most discriminative point of the ranking. Results obtained by the frame and the lexical overlap rankings are nonsignificantly different, showing respectively a best accuracy of 0.57 and 0.59.Footnote ⁹ This further indicates that frame and lexical overlap have comparable discriminative power. The question is then if the two overlap notions capture the same information, i.e. if and to what extent frame overlap introduces new informative and discriminative frame semantic knowledge with respect to lexical overlap. To answer this question, we computed the Spearman rank correlation coefficient ρ between the two rankings. The coefficient is used to find out the level of correlation of the two, thus showing if lexical and frame overlap are modelling the same information or not. We obtain ρ = 0.346, indicating a moderate correlation. From this we can conclude that frame overlap partially uses different information (at the frame semantic level) for discriminating between positive and negative examples, with respect to the lexical overlap.

Table 4.

Average frame overlap over positive and negative pairs in the gold standard and in Shalmaneser output; and average lexical overlap baseline

Table 5.

Average role overlap over positive and negative pairs in the gold standard and in Shalmaneser output

Table 6.

Average role fillers overlap over positive and negative pairs in the gold standard and in Shalmaneser output

Results using hierarchy match show that including part of the FrameNet hierarchy does not strengthen the discriminative power. An explanation is that we used the hierarchy in a limited way, focusing on a limited type of basic paths connecting frames.

The overall analysis of frame overlap indicates that the discriminative power is not high, as accuracy results show. This result is not surprising, as one would expect that both positive and negative pairs by and large talk about the same situation. Also, correlation results show that frame overlap and lexical overlap model slightly independent information, that could be exploited successfully together in an RTE system.

Results for role overlap, reported in in Table 5, suggest similar conclusions. The number of matching roles is notably but not significantly lower in the negative cases (8.7% difference). Similarly to frame overlap, this indicates that the type of participants (e.g. Speaker, Leader) talked about in text and hypothesis are largely the same.

So far, results indicate that overlap at the frame and role level is not highly relevant for discriminating positive and negative example, suggesting that text and hypothesis mostly deal with the same facts, situations and participants. In contrast, one would expect opposite results for filler overlap. Indeed, in theory the most significant indication as to whether entailment holds should be observed on the level of filler match and mismatch. Yet, results for filler mismatch, reported in Table 6, do not confirm this intuition, as the discriminative power for strict match is low (2.9% difference). Using the shallow Levenshtein measure improves the results, indicating that further elaborating the comparison of role fillers is an urgent and promising challenge for future research, as suggested in the ablation study reported in (Snow, Vanderwende and Menezes Reference Snow, Vanderwende and Menezes2006).

Regarding Shalmaneser, results show that frame overlap computed using the parser's analysis does not have a discriminative power higher than the baseline (+10% versus +10.5% difference). The output of Shalmaneser is noisy, thus impeding the rise of relevant distinctions at the frame level between positive and negative examples. We computed the correlation between the gold standard overlap ranking and the Shalmanser overlap ranking, obtaining a good, but not optimal, level of correlation (ρ = 0.502). This confirms that Shalmaneser adds significant noise, but still captures information at the frame level which is not represented at the lexical level (correlation with the lexical overlap is indeed lower, ρ = 0.331).

Similar results are obtained by the Shalmaneser data on role and filler match, revealing an informational gap existing between the information produced by a gold standard annotation and an automatic parser.

Overall, the goal of the experiment was to use a basic frame-based model, as close as possible to the embedding approach currently adopted by many RTE system. Results show that this way of modelling frame semantics does not make optimal use of the information contained in the dataset. Strong evidence to discriminate between negative and positive examples cannot be derived in this straightforward manner. What is needed are more complex models which go beyond frame and role overlap, and that make use of more sophisticated evidence coming from the frame analysis.