2 The adjudication task

2.1 Definition and motivation

The adjudication task can be defined as follows: let A = {a ₁, a ₂, . . ., a _n} be a set of annotators and let σ be a sentence that each annotator in A tagged with a sense from a reference inventory (e.g. WordNet). Given a word w ∈ σ, we define the set of annotations provided for w by the annotators as S _A = {s ₁, s ₂, . . ., s _m} ⊆ Senses(w), where Senses(w) is the set of senses of w in the reference inventory and m ≤ n.

If |S _A| > 1, i.e. if at least one annotator disagreed on which sense to associate with w, an adjudication step is required. The final judgment is typically made by an adjudicator who selects for word w a sense s ∈ Senses(w) over the others. Notice that s is a word sense for w in the sense inventory, but is not necessarily in S _A, although it is likely to be. Also note that the annotators in A can be either human or automatic, depending upon the purpose of the exercise:

• • Manual annotation is usually performed in the creation of experimental data sets (e.g. SemCor, the Senseval data sets, etc.): the possible divergencies of opinion between human annotators need to be smoothed away so as to produce high-quality data sets and guarantee that the experiments will provide meaningful results;

• • Automatic annotation is gaining more and more interest with the advent of the so-called Semantic Web (Berners-Lee Reference Berners-Lee1999), as users are not willing to sense tag and semantically index their web pages. However, compared to human annotators, automated systems tend to diverge in their choices more often and at a coarser level of granularity. As a result, adjudicating these disagreements is a laborious and time-consuming task.

As an example, suppose that human annotators manually tagged the following sentence (we subscript the sense annotations beside each word):Footnote ¹

1. (a) She loves_#1 the smell_#1,2 of basil_#1 leaves_#1

The uncertainty on the appropriate choice for the noun smell is reflected by the ambiguity of its WordNet sense inventory, reported in Table 1. This is a case where even a good judge might not be able to choose between the first two senses of the word, even knowing the hypernyms of both (as reported in the table). However, a closer look at the lexicon structure would reveal that the appropriate sense of smell is #1, as shown in Figure 1. In fact, the second sense of smell expresses one of the abstract properties typical of human beings (vision, audition, touch, taste, smell). This information (the taxonomical structure, but also other important textual, lexical, and semantic information) is not immediately available to the annotator. As a result, inconsistent choices can be made.

Table 1.

The WordNet sense inventory of the noun smell

Fig. 1.

An excerpt of the WordNet taxonomy including smell#1 and smell#2.

3 Lexical chains and semantic interconnections

Lexical chains (Morris and Hirst Reference Morris and Hirst1991) are sequences of words w ₁, . . ., w _n in a text that represent the same topic, i.e. such that w _i is related to w _i+1 by a lexico-semantic relation (e.g. hypernymy, meronymy, etc.).

Lexical chains have been applied to the analysis of discourse cohesion (Morris and Hirst Reference Morris and Hirst1991), text summarization (Barzilay and Elhadad Reference Barzilay and Elhadad1997), the correction of malapropisms (Hirst and St-Onge Reference Hirst, St-Onge and Fellbaum1998), WSD (Galley and McKeown Reference Galley and McKeown2003), etc. This idea was developed in further approaches to WSD based on lexico-semantic heuristics (Rigau, Atserias, and Agirre Reference Rigau, Atserias and Agirre1997; Harabagiu, Miller, and Moldovan Reference Harabagiu, Miller and Moldovan1999) and link analysis (Mihalcea, Tarau, and Figa Reference Mihalcea, Tarau and Figa2004; Véronis Reference Véronis2004; Agirre, Martínez, de Lacalle, and Soroa Reference Agirre, Martínez, de Lacalle and Soroa2006).

Recently, a knowledge-intensive, untrained algorithm for WSD, called Structural Semantic Interconnections Footnote ² (SSI) (Navigli and Velardi Reference Navigli and Velardi2005), has been shown to provide interesting insights into the choice of word senses by producing structural justifications in terms of semantic graphs.

SSI exploits an extensive lexical knowledge base, built upon the WordNet lexicon and enriched with collocation information representing semantic relatedness between sense pairs. Collocations are acquired from existing resources [like the Oxford Collocations (Lea Reference Lea2002), the Longman Language Activator (Longman Reference Longman2003), and collocation web sites]. Each collocation is mapped to the WordNet sense inventory in a semiautomatic manner and transformed into a relatedness edge [for further details, the interested reader can refer to Navigli (Reference Navigli2005)]. Notice that, at present, the lexical knowledge base does not include any information from the SemCor semantically annotated corpus (Miller, Leacock, Tengi and Bunker Reference Miller, Leacock, Tengi and Bunker1993).Footnote ³

Given a word context W = {w ₁, . . ., w _k}, SSI builds a graph G = (V, E) such that Senses(w _i) (i.e. V includes a vertex for each sense of a word in W) and (s, s′) ∈ E if there is at least one semantic interconnection between senses s and s′ in the lexical knowledge base. A semantic interconnection pattern is a relevant sequence of edges selected according to a manually created context-free grammar, i.e. a path connecting a pair of word senses, possibly including a number of intermediate concepts. The grammar consists of a small number of rules, inspired by the notion of lexical chains. The full context-free grammar encoding semantic interconnection patterns for the WordNet lexicon is reported in Table 2. For further details, the reader can refer to (Navigli and Velardi Reference Navigli and Velardi2005).

Table 2.

The full context-free grammar for the recognition of semantic interconnections

The rules in the table are divided into three groups: pattern rules, basic nonterminals (identifying basic sequences of hypernymy, meronymy, and relatedness), and additional relations (most of which connect different parts of speech, like nominalization, pertainymy, etc.). The grammar is general enough to be applied to any sufficiently structured computational lexicon, where only the last group should be replaced by the set of lexicon-specific nonterminals.

The purpose of the grammar is twofold: first, to prune out a large number of unwanted lexical chains (e.g. universe#1 natural object#1 object#1 commodity#1 merchandise#1); second, to ensure that the chains contain a maximum number of edges of a certain kind (e.g. the grammar does not allow any sequence of three relatedness edges, like in job#1 money#1 coin#1 metal#1). Both aspects aim at avoiding undesired shifts of meaning.

Even though the grammar produces an infinite number of edge sequences, we limit the recognition to stringsFootnote ⁴ of length ≤ 5. While it is true that this reduces the expressivity of the grammar to a finite state automaton (or, equivalently, a regular grammar), yet we want to maintain a general, clear, and compact formalism to express edge sequences.

We report good and bad examples of semantic interconnections in Table 3. In the second column, we mark each interconnection in the table with a check mark (✓) if the interconnection connects two concepts which are really semantically related (we mark it with ×, otherwise).

Table 3.

Good and bad examples of semantic interconnections

SSI performs disambiguation in an iterative fashion, by maintaining a set of senses as a semantic context. Initially, = V (the entire set of senses of words in the word context W). At each step, for each word w ∈ W, and for each sense s ∈ Senses(w), the algorithm calculates a score of the degree of connectivity between sense s of w and the other senses in :

where IC(s, s') is the set of interconnections between senses s and s'. The contribution of a single interconnection i is given by the reciprocal of its length (1/length(i)), calculated as the number of edges connecting its ends. The overall degree of connectivity is then normalized by the number of contributing interconnections. The highest ranking sense is chosen, the other senses of w are removed from the semantic context (i.e. ), and the word w, disambiguated as a result of the current iteration, is removed from W. During the next iteration, will be used as an additional semantic context for the remaining words in W, i.e. for the words yet to be disambiguated. The algorithm terminates when either or there is no sense such that its score exceeds a fixed threshold. The threshold was experimentally set to 0.2.

4 Adjudication with structural semantic interconnections

In this section, we illustrate the use of structural semantic interconnections for the automatic adjudication of the most appropriate word sense in case of disagreement.

Given a sentence σ and a set of words with disagreement W ⊆ σ, according to Section 2.2, we apply SSI to W by taking into account for disambiguation only the senses selected by the annotators, and using as a fixed context the agreed senses chosen by the annotators for the words in σ \ W.

Recall the annotated sentence from section 2.1:

1. (a) She loves_#1 the smell_#1,2 of basil_#1 leaves_#1

This sentence is a real case from an annotation experiment we conducted in a previous work. We initialize the word context W = {smell#n} and the semantic context (the first set is the fixed semantic context of agreed senses, while the second set contains the disagreed word senses). Only the first two senses of smell (i.e. the disagreed senses) are taken into account for the calculation of the Score _SSI, leading to:

The choice of the first sense of smell as a solution to this disagreement is structurally supported by a number of semantic interconnections according to the grammar in Table 2. Figure 2(a) shows some interconnections identified by the algorithm. In contrast, sense 2 is not related to other senses in through any semantic interconnection, as illustrated in Figure 2(b).

Fig. 2.

(a) Some semantic interconnections supporting the choice of sense #1 of smell in sentence (a). (b) The choice of smell#2 is not supported by any semantic interconnection.

As a second example, consider the WordNet definition of motorcycle:

1. (b) Motorcycle: a motor vehicle with two wheels and a strong frame

In the Senseval-3 Gloss WSD task (Litkowski Reference Litkowski2004), the human annotators assigned the first sense to the word frame (a structure supporting or containing something), unintentionally neglecting that the dictionary encodes a specific sense of frame concerning the structure of objects (e.g. vehicles, buildings). In fact, according to WordNet, a chassis#3 is a kind of frame#6 (the internal supporting structure that gives an artifact its shape), and is also part of a motor vehicle#1. Figures 3(a) and (b) illustrate the graphs resulting from the sense choices of frame#6 and #1, respectively.

Fig. 3.

(a) Some semantic interconnections supporting the choice of sense #6 of frame in sentence (b). (b) The choice of frame#1 has a weaker structural support within the SSI lexical knowledge base.

Semantic interconnections reflect the fine granularity of the inventory, as they are expressions of the lexical knowledge base from which they are extracted. In fact, the choice of frame#1 still produces relevant semantic interconnections, as illustrated in Figure 3(b), but the overall ranking of this sense selection, i.e. the degree of overall connectivity of the resulting graph, is smaller than that obtained for frame#6:

These two real-world cases make it evident that semantic interconnections can point at inconsistent, though acceptable, choices made by human annotators due, among others, to the fine granularity of the sense inventory and to regular polysemy, i.e. the recurring and predictable sense alternations certain classes of words are subject to.

We recognize that subtle distinctions, like those encoded in WordNet, are rarely useful in any NLP application, but as a matter of fact WordNet is at the moment the de facto standard within the research community, as no other computational lexicon of that size and complexity is freely available.

5 Evaluation

In this section, we present an evaluation of the effectiveness of semantic interconnections applied to the adjudication of disagreed word senses. We assessed the method for both manual (Section 5.1) and automatic annotations (Section 5.2).

5.1 Evaluating the adjudication of manual annotations

As mentioned above, the adjudication of disagreements resulting from manual annotations is a critical task when we want to guarantee the high quality of a data set. To assess the quality of the adjudications made by SSI on manual sense annotations, we performed two different experiments on SemCor (Miller et al. Reference Miller, Leacock, Tengi and Bunker1993), a corpus of more than 200,000 content words manually tagged with WordNet word senses:

• • A large-scale simulation of disagreements between two annotators (Section 5.1.1);

• • An experiment on a smaller scale concerning the adjudication of real disagreements between annotators of a portion of the SemCor corpus (Section 5.1.2).

5.1.1 Adjudicating simulated disagreements

As a first experiment, we simulated in vitro a disagreement between two annotators in which an annotator provides an appropriate sense and the other selects a different sense at various semantic levels. To this end, we needed a way to distinguish meanings at different levels of granularity. For each word of interest, we used the Oxford Dictionary of English (ODE) (Soanes and Stevenson Reference Soanes and Stevenson2003) to manually produce a hierarchical version of the WordNet sense inventory of that word.

The ODEFootnote ⁵ provides a multilevel structure of senses, distinguishing between homonymy (i.e. completely distinct senses, like race as a competition and race as a taxonomic group) and polysemy (e.g. race as a channel and as a current). Each polysemous sense is further divided into microdistinctions (e.g. a division of humankind versus a group of people with a common ancestor). Table 4 shows the lexical entries of the noun race (we represent an ODE sense of a word w as w#p#h.i, where p is its part of speech and i denotes the i-th polysemous entry of the h-th homonym of w).

Table 4.

The sense inventory of race#n in the ODE*

* definitions are abridged, bullets (•) indicate a subsense in the ODE, arrows (→) indicate hypernymy.

For each word of interest, the manual mapping of WordNet senses to ODE polysemous entries induces a hierarchical structure on the former. For example, consider the sense inventories provided by the two dictionaries for the noun race (Tables 4 and 5). As a result of the mapping, a semantic correlation between the WordNet senses is identified at different levels of granularity, as shown in Figure 4.

Table 5.

The sense inventory of race#n in WordNet*

* arrows (→) indicate hypernymy.

Fig. 4.

A hierarchical version of the WordNet sense inventory for noun race.

We randomly selected a set W of 207 polysemous words from WordNet (64 nouns, 82 verbs, 61 adjectives) resulting in 1,488 different senses overall (7.18 senses per word on average) that we manually mapped to the appropriate ODE senses.Footnote ⁶ Notice that mapping fine-grained to coarse-grained word senses is much easier than any semantic annotation or one-to-one mapping task. This intuition is also substantiated by a quantitative assessment: 548 WordNet senses of 60 words were mapped to ODE entries by two annotators, with a pairwise agreement of 92.7% (κ agreement: 0.854).

This mapping technique was also successfully used in the organization of the Semeval-2007 coarse-grained English all-words WSD task (Navigli, Litkowski, and Hargraves Reference Navigli, Litkowski and Hargraves2007). An expert lexicographer estimated that an amount of time between 20 and 30 s is enough for mapping a fine-grained sense (e.g. from WordNet) to a coarser sense (e.g. in ODE) with the aid of an appropriate interface. Accordingly, the creation of our mapping took an overall amount of 11 man-hours.

Next, we selected all those sentences in SemCor which included at least one occurrence of words from W (overall, 183 of the 207 words in W occur at least once and are sense-tagged in SemCor). Formally, we considered all the sentences σ = w ₁w ₂ . . . w _n annotated in SemCor with the senses , j ∈ {1, 2, . . ., n}), such that σ contains at least one word w _i ∈ W. Then, for each sentence we simulated a disagreement on the word w _i by randomly selecting a different sense for the word w _i at three different levels:

1. (1) microdistinctions: is in the same lexical entry as (e.g. race#n#1 and race#n#3, both mapped to the ODE entry race#n#1.1) – we expect that most of the human disagreements fall into this category (as also confirmed by the experiments in Section 5.1.2);

2. (2) polysemy: is in a different polysemous entry from that of (e.g. race#n#1 and race#n#4, mapped respectively to the ODE polysemous entries race#n#1.1 and race#n#1.2);

3. (3) homonymy: is a homonym of (e.g. race#n#1 and race#n#2, mapped respectively to the ODE homonyms race#n#1.1 and race#n#2.1).

We further experimented on a random choice of the disagreed sense:

1. (4) random disagreements: we built a fourth data set in which sense of word w _i is chosen randomly from .

In Table 6 we report the size of the four test sets, i.e. the number of disagreements by part of speech (and overall) and the total number of sentences involved, ranging from 4,633 sentences for random disagreements (i.e. the full set of sentences which contain at least one word from our initial set W of 207 words) to 633 for homonymy (i.e. the set of sentences which include at least a homonym). This difference is due to the low number of homonyms compared to polysemous entries. Notice that the number of disagreements per sentence is a function of the initial set of words W whose senses were clustered according to ODE. As a consequence, the resulting ratio (ranging from 1.05 to 1.2 disagreements per sentence depending upon the class of granularity) can differ significantly from that of a real data set of human disagreements. This aspect is further discussed in Section 5.1.2. Finally, the average number of context (i.e. agreed) words per sentence ranges between ten and twelve words depending on the data set.

Table 6.

Size and composition of the four disagreement test sets

We applied SSI to the annotated sentences (as discussed in Sections 2.2 and 4) and evaluated the performance of the approach in suggesting the appropriate choice for the words with disagreement. We assessed precision (the number of correct suggestions over the overall number of suggestions from SSI) and recall (the number of correct suggestions over the total number of words to be adjudicated), and we calculated the F1-score, a harmonic mean of the two measures (). The results are reported in Table 7.

Table 7.

Performance of automatic adjudication at different levels of disagreement (sentences from SemCor)

As expected, the accuracy of automatic adjudication increases as the “semantic distance” of the disagreed senses grows. In fact, while we get an F1-score of 65.72% in the adjudication of microdistinctions, this figure rises to 80.7% when we deal with homonyms. This confirms the intuitive idea that closer senses are inherently difficult to disambiguate.

Moreover, it is interesting to note that there is a 6.5% increase in precision from microdistinctions to polysemy and a 3.1% increase from polysemy to homonymy. As expected, we observe the largest increase in recall from polysemy to homonymy (+15.22%). Completely distinct senses of the same word convey indeed very different semantics and it is less likely to find semantic interconnections supporting the wrong sense.

We observe that the performance on randomly selected disagreements is close to that obtained for polysemous sense distinctions (indeed, the difference is not statistically significant according to a χ² test). We will see in Section 5.2 that random disagreements apparently simulate the average level of difficulty encountered by an automatic system.

We preferred to calculate precision and recall distinctly, rather than accuracy, as the latter does not provide any hint about the quality of the structural suggestions produced by SSI: a good, justifiable adjudication is better than a couple of mediocre answers.

We studied the performance by part of speech for the most difficult case, i.e. disagreements on microdistinctions (similar trends were observed for the other classes of granularity). We report the figures in Table 8. The most interesting result can be observed on noun performance: here SSI proves to be really effective, by correctly adjudicating almost three-fourth of the disagreements. The drop in recall for verbs and, even more, for adjectives is mostly due to the lack of structural information. However, precision keeps high in both cases, which allows for high-quality adjudication and leaves the remaining uncertain cases to a more careful, manual inspection.

Table 8.

Performance by part of speech of automatic adjudication of subtle disagreements (sentences from SemCor)

We also calculated the accuracy of the most frequent sense (MFS) heuristic for the four levels of granularity: 89.54% (microdistinctions), 93.85% (polysemy), 98.95% (homonymy), and 92.17% (random). The MFS baseline selects the disagreed sense which is most frequent in the SemCor annotated corpus. Consequently, it cannot be considered as a baseline, as it employs the test corpus to determine its sense choices. However, it can be used to establish an upper bound for this adjudication task.

Finally, we studied how the degree of connectivity of each word instance affects the performance of SSI on the adjudication task. Specifically, we extrapolated the trend on the microdistinction test set, that we report in Figure 5 (a similar trend was observed for the other levels of granularity). On the x-axis we have the out-degree of the correct sense of each word instance in the test set. The out-degree counts the number of WordNet relation edges and relatedness edges (cf. Section 3) outgoing from the word sense at hand. Each interval in the figure includes a minimum of 300 word instances from the test set, with approximately two-thirds of the word instances distributed in the range 0–13.

Fig. 5.

The SSI performance on the microdistinction test set by degree of connectivity. Each interval includes a minimum of 300 word instances from the test set.

The graph shows that the higher the degree of connectivity in the computational lexicon, the better is the overall performance of structural semantic interconnections (both in terms of precision and recall). When we have more than eight to thirteen relation edges that enable the connectivity from a specific sense to other senses in the lexicon, the performance grows noticeably. The trend shown in the graph corroborates the apparently trivial observation that the availability of large knowledge bases (even (semi)automatically acquired) tends to improve the disambiguation performance (Cuadros and Rigau Reference Cuadros and German2006). Moreover, it shows that the improvement can be really significant, with an increase of several tenths of points in terms of both precision and recall.

5.1.2 Adjudicating real disagreements

Performing experiments on a data set of simulated disagreements has the advantage of providing results on a large scale. However, experimenting on a data set of real, human disagreements would make the results more meaningful. Moreover, it would allow us to understand the level of disagreement attained by sense taggers according to the different classes of granularity that we introduced in the previous section.

Unfortunately, only few data sets of human disagreements are available to the research community. Among these, we cite the outcome of the Open Mind Word Expert (OMWE) project (Chklovski and Mihalcea Reference Chklovski and Mihalcea2002). However, this annotation effort does not fit our problem, as it was created on a lexical sample basis, whereas we expect annotators to tag all words in a text. Another interesting data set of disagreements is an outcome of the MultiSemCor project (Pianta, Bentivogli, and Girardi Reference Pianta, Bentivogli and Girardi2002; Bentivogli, Forner, and Pianta Reference Bentivogli, Forner and Pianta2004), aiming at the creation of a parallel English/Italian version of the SemCor corpus. This data set includes about a hundred disagreements; as a result, its sample size does not allow to perform a statistically significant evaluation.

Given the lack of all-words disagreement data sets, we asked two annotators to manually sense-tag a portion of the SemCor corpus with WordNet senses (we adopted version 2.0). The data set included an overall number of 2,927 words (285 sentences, 2,274 polysemous words) from three randomly chosen documents on different topics, specifically, football (document code: br-a13), biography (br-e22), and mathematics (br-j19).

We measured a pairwise interannotator agreement of 81.1% (i.e. the annotators – expert lexicographers – disagreed on 553 words). This figure is only slightly higher than the inter-tagger agreement ratio reported by Fellbaum et al. (Reference Fellbaum, Joachim, Shari and Fellbaum1998) on a subset of SemCor (ranging between 72.8% and 79.9% under different conditions), corroborating the difficulty of annotating with fine-grained word senses. However, in their experiments several factors lowered the agreement: the use of several annotators (with a comparison of the agreement among nonexperts and between experts and nonexperts), an older sense inventory (several sense entries in WordNet now report SemCor sentences as examples, which make it easier for an annotator to select the appropriate meaning), etc. We do not provide here a figure of κ agreement (Cohen Reference Cohen1960) as for most words we have a very low number of occurrences. As a result, the κ agreement is below 0.60 (i.e. between moderate and poor agreement) for over 90% of the words.

After an adjudication step, performed by a human arbiter, we discovered that the 553 disagreements included 38 incorrect sense choices due to inattention (i.e. unvoluntary choices) of either of the two annotators, and 22 words tagged with a different part of speech than that assigned in SemCor. Moreover, the arbiter found out that 92 of the remaining disagreements had to be resolved in contrast with the corresponding senses chosen in the SemCor data set. We can say that most of these cases are due to imprecise choices in SemCor (we report some in Table 9).

Table 9.

Some of the inexact sense tags in the SemCor corpus

Our aim is not to criticize SemCor, which is an invaluable resource for WSD and related fields. Most of these imprecisions indeed may be due to several factors, such as the mappings between different versions of WordNet, typos, and agreements by chance of the original annotators.

In order to study the distribution of the disagreements according to the three classes of sense granularity introduced in Section 5.1.1, and similarly to what we did in that section, we mapped all disagreed senses (770 distinct senses overall) to the corresponding ODE senses.

The application of SSI to solve the 553 disagreements in our data set led to the figures in Table 10. We report the overall performance of SSI together with its performance on the three classes of sense granularity. We registered only one case of homonymous disagreement, due to a tagging mistake (a low figure was expected indeed). We were not able to determine the granularity of five disagreements because of missing entries in ODE.

Table 10.

Performance of automatic adjudication on real human disagreements at different levels of disagreement

First, we note that the overall performance (68.77% F1) gets very close to the MFS baseline (69.98% F1). As in the previous experiment, we remark that the MFS baseline employs sense frequencies from the SemCor annotated texts of which our test set is a part. Secondly, the overall performance is comprised between the results of microdistinctions and the polysemous case of our simulated experiment (cf. Table 7). This result was expected, as the disagreements concerned these two classes of granularity.

Regarding the performance on distinct parts of speech, reported in Table 11, we observe an increase in recall for verbs and adjectives compared to the figures from our simulated experiment on microdistinctions (cf. Table 8). However, all the differences in precision from Tables 8 and 11 calculated distinctly for each part of speech are not statistically significant according to χ² tests.Footnote ⁷

Table 11.

Performance of automatic adjudication on real human disagreements by part of speech

We remark here that the results of this experiment on real human disagreements are not entirely comparable with those from our previous experiment on simulated disagreements. Firstly, here we are forcibly working on a smaller number of disagreed senses. Unfortunately, as we mentioned above, no medium-scale data set of disagreements from all-words annotation efforts is available, so we believe our data set constitutes an interesting effort in this direction. Secondly, the simulated experiment was easier than this experiment on human disagreements, as it presupposed a disagreement per sentence ranging from 1.05 to 1.2 (depending upon the class of granularity). In contrast, as shown in Table 12, we can find between one and six disagreements per sentence in our data set of real annotations, with an average of 1.94 disagreements per sentence, thus making automatic adjudication a harder task, as its difficulty depends on the quality and number of agreed senses. We did not determine the performance by number of disagreements per sentence because of the small size of each sample.

Table 12.

Distribution of real human disagreements over a subset of SemCor (285 sentences)

5.2 Evaluating the adjudication of automatic annotations

For assessing semantic interconnections applied to the adjudication of disagreements between automatic systems, we chose the Senseval-3 corpus for the English all-words task (Snyder and Palmer Reference Snyder and Palmer2004). The task required WSD systems to provide a sense choice for a total of 2,041 content words in a set of 301 sentences from the fiction, news story, and editorial domains.

For our experiments, we focused on the outcome of the three best-ranking systems: GAMBL (Decadt, Hoste, Daelemans, and van den Bosch Reference Decadt, Hoste, Daelemans and Antal2004), SenseLearner (Mihalcea and Faruque Reference Mihalcea and Faruque2004), and Koc University (Yuret Reference Yuret2004). The application of SSI to the disagreement set led to the figures in Table 13, where we compare our results with:

• • the chance baseline, calculated as the sum of the uniform probabilities of correctly solving each disagreement divided by the total number of disagreements;

• • the most frequent sense baseline, i.e. the choice of the most frequent or predominant sense in SemCor, selected among those output by the three systems;

• • a majority voting combination strategy, i.e. the sense output by a majority of the three systems is chosen – in case of full disagreement, a random choice is made;

• • the best-performing Senseval-3 system, i.e. GAMBL;

• • the oracle performance: if we had an oracle adjudicating the appropriate answer for each word instance, its accuracy would be 76.9% (this percentage represents the number of words for which at least one correct answer was provided by any of the three systems). This figure constitutes an upper bound for our task, as it would be impossible for any automatic adjudication algorithm to assign a greater number of correct answers.

Table 13.

Results on the Senseval-3 all words task

In Table 14 we report the oracle accuracy based on the kind of agreement (three-way agreement, two-way agreement, or total disagreement). This is given by the percentage of word occurrences for which at least a correct answer is given by any of the three systems. Clearly, the oracle is more accurate when the three systems agree (78.1%), while its accuracy decreases to 67.3% when they fully disagree, indicating an inherent condition of difficulty.

Table 14.

The oracle performance by kind of agreement between the three best-ranking systems at Senseval-3

Notice that the chance and MFS heuristic, as well as the majority voting strategy, have been applied to the adjudication of disagreed senses, while the performance of the best-performing system is from the original Senseval-3 task. It could be argued that, in the majority voting strategy, in case of tie the sense of the best system (namely, GAMBL) could be selected. We tested this option and obtained a result which is comparable (65.1%) with the random backoff strategy. This result can be probably attributed to the lack of information in GAMBL for assigning the appropriate sense to these challenging instances. We also experimented with a higher number of systems, and found out that the improvement in accuracy was negligible (+0.5% with five best-ranking systems), compared to the costly requirement of a higher number of state-of-the-art systems to combine.

We could not apply other kinds of untrained combination strategies, such as rank-based or probability mixture, as the confidence of state-of-the-art systems on each word sense was not available. However, experimentally, we do not expect a big increase in performance (see, e.g. Brody, Navigli, and Lapata (Reference Brody, Navigli and Lapata2006), where the performance increase is lower than 1% with a rank-based ensemble compared to majority voting, both applied to disagreed nouns from the entire SemCor).

SSI applied to automatic adjudication performs better than the best-performing Senseval system, with a difference in precision and recall of 3.5% and 3.1%, respectively. Both differences are statistically significant (χ² test, p < 0.05).

An interesting remark is that the F1 performance of SSI is not dissimilar from that obtained in the previous section with the random selection of disagreed senses. This result is plausible given that automated systems can make very good choices as well as provide inexplicable answers (especially when they disagree).

Finally, Table 15 reports the precision of SSI on nouns, adjectives, and verbs. The good performance on nouns is not surprising, as SSI relies on the lexicon structure, which is more interconnected for nouns. In contrast, verbs suffer from the lower degree of connectivity with other parts of speech, especially because of short sentences in the test set such as “that's what the man said”, “I'm just numb”, etc. The good performance on adjectives is given by their low degree of polysemy.

Table 15.

Results on the Senseval-3 all words task by part of speech

6 Related work

It is a matter of fact that achieving a high quality for sense annotations is a very hard task (see, e.g., Hanks (Reference Hanks2000) and Véronis (Reference Véronis2001)) studies on the interannotator agreement report figures between 70% and 80% when WordNet is adopted as a sense inventory (Fellbaum et al. Reference Fellbaum, Joachim, Shari and Fellbaum1998; Edmonds and Kilgarriff, Reference Edmonds and Adam2002; Snyder and Polmer Reference Snyder and Palmer2004). An even worse figure of 67% is reported when sense choices are made by unknown contributors on the Web (Chklovski and Mihalcea Reference Chklovski and Rada2003). Sense distinctions are rarely incontestable, and this is one of the reasons why we think it is important to refer to the lexicon structure as a justification for sense choices, as our method does.

A big dispute is even whether word senses exist at all (Hanks Reference Hanks2000; Kilgarri. Reference Kilgarriff1997). Discretizing sense distinctions is difficult and can lead to different equally adequate choices. In contrast to the widespread enumerative approach, a generative approach has been proposed (Pustejovsky Reference Pustejovsky1995) which consists of generating word senses based on the combination of a set of features called qualia roles. The latter approach overcomes the granularity problem by adopting a continuous representation of senses, but presents other issues: it is unclear how to compare different system outputs in an objective manner when applied to a gold standard data set or how to consistently provide an unequivocal semantic annotation of text.

If we believe in word senses, the use of coarse-grained sense distinctions (or, alternatively, different levels of granularity) seems to be a promising approach to overcome the problem of low interannotator agreement. Ng, Lım and Foo (Reference Ng, Lim and Foo1999) show that, when a coarse-grained sense inventory is adopted, a consistent increase of the interannotator agreement is observed. Numerous manual and automated approaches to sense clustering have been proposed, ranging from syntactic and semantic criteria for grouping senses (Palmer Reference Palmer2000; Palmer, Dang, and Fellbaum Reference Palmer, Dang and Fellbaum2007) to the use of heuristics (Peters, Peters, and Vossen Reference Peters, Peters and Vossen1998), dictionary mappings (Dolan Reference Dolan1994; Navigli Reference Navigli2006c), and confusion matrices (Chklovski and Mihalcea Reference Chklovski and Rada2003; Agirre and de Lacalle Reference Agirre and de Lacalle2003).

Recently, Hovy, Marcus, Palmer, Ramshaw, and Weischedel (Reference Hovy, Marcus, Palmer, Ramshaw and Weischedel2006) presented the OntoNotes project. The proposed approach aims at creating sense groupings: senses are iteratively partitioned until an interannotator agreement of 90% is reached in a sense annotation task. As a result the method does not need a further evaluation of its outcome, as it achieves its main objective, i.e. guaranteeing a high interannotator agreement. However, this approach – although faster than a classic manual sense creation procedure – still requires a great amount of work to produce a coarse set of senses on a large scale. Finally, an approach for clustering word senses based on lexico-syntactic features has been proposed which achieves state-of-the-art performance in WSD (Kohomban and Lee Reference Kohomban and Lee2007).

Unfortunately, none of these approaches led to the release of an acknowledged de facto standard. As a result, until the WordNet sense inventory is not fully revised or a new sense inventory is widely adopted, enumerative approaches to coarse-grained WSD will remain unfeasible on a large scale. Until then, the most reasonable solution is to support annotators and, at a later stage, adjudicators in the difficult task of making the most appropriate choice from a discrete list of senses.

An important step in the direction of supporting sense annotations is the OMWE, a project for the acquisition of annotations with the aid of a web interface that presents the sense inventory of a word to be disambiguated in context (Chklovski and Mihalcea Reference Chklovski and Mihalcea2002). The main limitation of such an interface is that it is very difficult to provide an annotator, especially an unskilled one, with enough information to choose between subtle sense distinctions. In contrast, the method presented in this paper, even though applied to adjudication rather than sense annotation, has the advantage of supporting sense choices which are or seem coherent with respect to the adopted lexicon. A further difference is given by the fact that the OMWE expects a large number of contributions for the same item to be annotated reliably, whereas our approach can be applied to items annotated by any number of humans or systems. In our experiments we focused on two- or three-way disagreements; however, the very same approach can be employed when more annotators are involved at virtually no additional cost.

Other structural approaches, either based on semantic distances (e.g., Jiang and Conrath, Reference Jiang and Conrath1997; Leacock, Chodorow, and Miller Reference Leacock, Chodorow and Miller1998) or graphs (e.g., Mihalcea et al., Reference Mihalcea, Tarau and Figa2004), could be employed to support the adjudication task. Unfortunately, knowledge-based methods are likely to perform poorly if they do not rely on rich lexical knowledge bases. This assertion is also supported by the findings discussed in the last part of Section 5.1.1. Also, most of these approaches cannot produce justifications for their choices in terms of semantic graphs as SSI does.

We stress that the adjudication task presented in this paper is only apparently easy: while it is true that the task works on a restricted number of senses for disagreed words, these are usually the hardest cases (i.e. those which make WSD an “AI-hard” task), simply due to the fact that humans or state-of-the-art systems disagreed on them.

As remarked in Section 2.2, ensemble approaches, i.e. combinations of WSD methods, are mostly related to the adjudication task. However, they differ in many aspects. Firstly, they do not exploit the agreed senses as an aid for solving disagreements. Secondly, they are applied to the outcome of automatic systems (often requiring a confidence degree to be output for each sense choice), in contrast to our approach which was applied to disagreements resulting from both manual and automatic annotation efforts. Thirdly, some ensemble approaches are supervised and require a training phase either for the disambiguation algorithms (first-order classifiers) or for the combination scheme (second-order classifier). This third aspect makes ensembles involving supervision unsuitable for the adjudication of disagreements resulting from the annotation of both unrestricted and domain-oriented texts.

Florian, Cucerzan, Schafer, and Yarowsky (Reference Florian, Cucerzan, Schafer and Yarowsky2002) propose several kinds of combination approaches, based on voting, probability mixture, etc. Experiments are performed on the Senseval-2 lexical sample task: this kind of task is different from the one proposed in the present paper (an all-words task indeed), as it targets a single word per sentence (and neither requires nor exploits the annotation of the other words with senses). Klein, Toutanova, Ilhan, Kamvar, and Manning (Reference Klein, Toutanova, Ilhan, Kamvar and Manning2002) present ensemble approaches based on majority voting, weighted voting, and maximum entropy. The latter performs best (obtaining an improvement of 1% over the best single classifier), but requires a training phase on the second-order maximum entropy classifier. Experiments are again performed on a lexical sample basis, which makes a comparison with our approach not feasible.

Among the unsupervised approaches, purely relying on first- and second-order unsupervised classifiers, Stevenson and Wilks (Reference Stevenson and Wilks2001) present a method for combining different disambiguation techniques, such as selectional preferences, subject codes, simulated annealing, etc. However, their evaluation is performed on a portion of the SemCor corpus, where each original sense tag has been mapped to an entry of the LDOCE dictionary. As a result, the authors obtain a performance around 90% and beyond 94% for fine- and coarse-grained disambiguation, respectively, which is difficult to compare to other systems, including ours.

Finally, we cite a study by Brody, Navigli and Lapata (Reference Brody, Navigli and Lapata2006), which concerns ensembles of unsupervised disambiguation algorithms. One of the combination approaches involves the use of SSI, the algorithm adopted in the present paper, as an arbiter of disagreed senses. Notice however that in that work the authors focus on the adjudication of disagreed nouns. As a result, verbs, adjectives, and adverbs are left ambiguous and do not provide a strong contribution to the structural disambiguation of disagreed nominal senses. The other ensembles presented in the paper (majority voting, probability mixture, rank-based) would not benefit from agreed senses of other parts of speech, and this is the reason for the lower performance of SSI in that specific setting. In that work, SSI on the adjudication of disagreed nouns in SemCor obtains 56.3% versus 58.1% accuracy obtained by the best ensemble method, namely the rank-based approach (−1.8%). In contrast, in this paper we exploit the full power of sense-tagging agreements, thus obtaining results which overcome the state of the art in all-words WSD by around 3.3% (cf. Section 5.2).

In previous works (Navigli Reference Navigli2006a, Reference Naviglib), we presented preliminary experiments on the manual and automatic adjudication of sense annotations. However, those experiments were inconclusive: on the manual side, we simulated disagreements on a much smaller scale without even distinguishing among classes of granularity as we did in this paper; on the automatic side, we performed experiments only on a part of the Senseval-3 test set, whereas in this paper we applied SSI to the full set of disagreements. All the experiments in the present paper are novel and provide interesting insights into the contribution of structural information to the resolution of disagreements.