5.1 Corpus

The EPEC corpus (the Reference Corpus for the Processing of Basque) aims to function as a “reference” corpus for the development and improvement of several NLP tools for Basque (Aduriz et al. Reference Aduriz, Aranzabe, Arriola, Atutxa, Díaz de Ilarraza, Ezeiza, Gojenola, Oronoz, Soroa and Urizar2006). It is a 300,000-word sample collection of news published in Euskaldunon Egunkaria, a Basque-language newspaper. This corpus has been manually tagged at different levels (morphology, syntax, phrases, etc.). Recently, mentions and coreference chains have also been tagged by two expert linguists in a subpart of EPEC corpus. The estimation of the inter-annotator agreement, including the chance-based factor for the task of tagging mentions has not been researched yet according to Artstein and Poesio (Reference Artstein and Poesio2008). In Kopeć and Ogrodniczuk (Reference Kopeć, Ogrodniczuk, Sobecki, Boonjing and Chittayasothorn2014) only the observed agreement is presented, since it is difficult to estimate the probability of a random tagging of a mention. The same procedure is used in Ohta et al. (2012). One annotation is considered as gold and the other as system. We evaluated the annotation consistency in mention detection in the same way, which was scored using the Strict Matching protocol, explained in Section 5.2. The F-measure obtained for agreement was 94.07%.

The mention detector has been developed and tested using a subpart of the EPEC corpus consisting of 46,383 words that correspond to 12,792 mentions. The relation between the number of words and the number of mentions is about 27%, meaning that approximately every four words a mention is detected. Similar statistics regarding the proportion of words/mentions can be found in Màrquez, Recasens, and Sapena (Reference Màrquez, Recasens and Sapena2013), where English, Spanish, and Catalan values are presented.

5.2 Scoring protocols

The most used measures to evaluate mention detection are precision, recall, and F-measure. To calculate these, mentions extracted by the mention detector and the set of manually tagged mentions are compared. Typically, two matching methods have been used to make the comparison between mentions.

The first one, known as Lenient Matching or Partial Matching, considers mentions to be correct if their span is within the span of the gold mention and contains the head word (Kummerfeld et al. Reference Kummerfeld, Bansal, Burkett and Klein2011). This protocol was used in Message Understanding Conferences (MUC-6 1995; MUC-7 1998). Slightly different measure was used in SemEval-2010 Task 1 (Recasens et al. Reference Recasens, Màrquez, Sapena, Martí, Taulé, Hoste, Poesio and Versley2010) to score the mention detection task. Mention were rewarded with one point if their boundaries coincide with those of gold mentions and the mentions that whose boundaries are within the gold mention including its head with 0.5 points. And in ACE programs (NIST 2008) only system mentions that overlap more than 30% in terms of character span with the gold mention and which have the same entity type were credited.

The second scoring protocol is clearly a stricter variation, because it only considers correct mentions that are exactly the same as the gold mentions. This matching method is known as Strict Matching and has been used in conferences such as CoNLL-2011 Shared Task (Pradhan et al. Reference Pradhan, Ramshaw, Marcus, Palmer, Weischedel and Xue2011).

We present our mention detector scores using both Lenient Matching and Strict Matching.

5.3 Nominal chunks as mentions

For mention detection, as a starting point we decided to use a rule-based approach using a generic NLP tool, namely a chunker (Aduriz and Díaz de Ilarraza Reference Aduriz, Díaz de Ilarraza and Oyharçabal2003), an analyser that identifies verbal and nominal chunks based on rule-based grammars. This tool is integrated in a pipeline for Basque processing where other modules are also applied: (i) A morphological analyser that performs word segmentation and PoS tagging (Alegria et al. Reference Alegria, Artola, Sarasola and Urkia1996), (ii) A lemmatiser that also disambiguates the PoS and the syntactic function (Alegria et al. Reference Alegria, Aranzabe, Ezeiza, Ezeiza, Urizar, Watson and Wood2002), (iii) A multi-word item identifier that determines which groups of two or more words are to be considered multi-word expressions (Alegria et al. Reference Alegria, Ansa, Artola, Ezeiza, Gojenola and Urizar2004), (iv) A named-entity recogniser that identifies and classifies named entities (person, organization, location) in the text (Alegria et al. Reference Alegria, Ezeiza, Fernandez and Urizar2003).

The chunker uses Constraint Grammar (CG) formalism (Karlsson et al. Reference Karlsson, Voutilainen, Heikkilä and Anttila1995) and obtains a precision value of 81.08%, a recall of 81.09% and F-measure of 81.08% in chunking.

The nominal-chunks can be considered the structures that are most similar to mentions, so we consider them and discard the verbal chunks for mention detection. The obtained results can be seen in Table 3. We obtained an F-measure of 64.28% using the Strict Matching protocol and 71.12% using Lenient Matching. These scores were considered as baseline.

Table 3.

Mention detection using nominal chunks. B=Baseline

Regarding the scores obtained using only a chunker for mention detection, we conclude that mention detection itself is a very challenging task since expressions can have complex syntactic and semantic structures. Preprocessing with generic NLP tools, while helpful, did not always succeed in identifying mention boundaries correctly. The same fact is pointed out by Nguyen, Kim and Tsujii (Reference Nguyen, Kim and Tsujii2008), who suggest that using a base noun phrase chunker is insufficient for mention detection in the bio-domain, and that therefore some adaptations are needed. In less resourced languages such as Basque, a processing of the output obtained by generic NLP tools is needed to obtain better results in mention detection. In contrast, in richer resource languages where generic NLP tools such as parsers obtain good results, these tools can be used for accurate mention detection with little or no processing.

The example in Figure 1 illustrates the differences between the chunker output and mentions. In this example, we deal with the sentence Mikelek erosi zituen etxeak. . . “The houses that Mikel bought. . .”. The first row shows that Mikel “Mikel” and etxeak “houses” are the nominal chunks obtained by the chunker, which are automatically considered mentions in this first approach. Mikel is a mention itself but etxeak cannot be considered a mention, because it is only the head of a larger mention Mikelek erosi zituen etxeak “The houses that Mikel bought”. It is clear that some processing of the output generated by a chunker is needed to correctly identify mentions.

Fig. 1.

Wrongly identified mention. NP =Noun Phrase, BVP = Begin Verbal Phrase, EVP = End Verbal Phase.

5.4 Mention detection with finite state transducers

As explained in Section 5.3, using a preprocessing tool such a chunker does not really succeed in mention identification. Some mention structures are more complex than nominal chunks and mentions exceed the chunk boundaries. Thus, to identify even the complex type of mentions, we decided to combine the chunker with a clause tagger, that is, an analyser that identifies clauses, combining rule-based-grammars and machine learning techniques (Arrieta Reference Arrieta2010).

We combined the tags provided by these tools and created a mention detector system, defining a set of hand-crafted rules that have been compiled into Finite State Transducers (FST).

The FSTs are able to detect complex structures that should be identified as mentions and that chunkers do not recognise. In reference to the classification of mentions presented in Section 3, the FSTs detect verbal nouns, NPs as part of complex postpositions, NPs containing subordinate clauses, ellipses, coordination cases, and location adverbs.

To better understand how the FSTs work, an example of a mention composed by an NP containing a relative clause is presented, and then a rule that identifies these structures and marks them as mentions is defined.

Suppose that the mention detector receives as input the sentence Armada britainiarrak Ipar Irlandan dituen bi kuartel eraitsi dituzte.

Figure 2 shows the above sentence with information obtained during the preprocessing step and the mentions that the system proposes.

Fig. 2.

The input and the output of the mention detector. BNP = Begin-NP, ENP = End-NP, BVP = Begin-VP, EVP = End-VP, VREL = Relative Verb, CB = Clause Boundary.

Using the preprocessing information, the rule in Figure 3 identifies the verb containing a relative suffix (dituen “that has”), which is tagged with VREL. Next, the right side boundary is established in the NP or coordinated NPs that follow the relative verb (bi kuartel Two military barracks). Finally, the left side boundary of the mention is established at the closest clause boundary ({CB) to the left (Armada army). Following these steps, the system proposes the correct mention (Armada britainiarrak Ipar Irlandan dituen bi kuartel Two military barracks that the British army has in Northern Ireland) and tags the whole structure to show where the mention begins (<MENTION>) and ends (</MENTION>).

Fig. 3.

A simplified rule to recognise NPs containing subordinate clauses. NP = Noun Phrase, CB = Clause Boundary, W=Word, RV = Relative Verb, RM = Relative Mention.

Further discussion about the behaviour of the FSTs can be found in Soraluze et al. (Reference Soraluze, Arregi, Arregi, Ceberio and Díaz de Ilarraza2012).

The use of Finite State Technology enables the processing of large datasets at a high processing speed and with low memory usage. Using foma Footnote ¹ (Hulden Reference Hulden2009), an open source platform for finite-state automata and transducers, we defined 12 FSTs, composed of 34 hand-crafted rules.

We compared the results of the mention detector thus obtained with a baseline, established by considering all the nominal chunks that the chunker outputs to be mentions. The results are presented in Table 4.

Table 4.

Baseline and mention detection scores. B=Baseline, BMD=Basque Mention Detector

Our mention detector outperforms the Strict Matching protocol F-measure baseline by 9.08 percentage points, and that of Lenient Matching by 8.56 percentage points. The improvements obtained by our system are significant.Footnote ²

With the aim of comparing our mention detector system with standard systems, we carried out experiments with the Stanford Mention Detector (SMD) presented in Lee et al. (Reference Lee, Chang, Peirsman, Chambers, Surdeanu and Jurafsky2013) and with BART (Versley et al. Reference Versley, Ponzetto, Poesio, Eidelman, Jern, Smith, Yang and Moschitti2008) for mention detection. Using a high-recall algorithm, SMD identifies nominal and pronominal mentions from all NPs, pronouns, and named entity mentions, and then non-mentions are filtered out. BART uses a built-in mention extraction module, computing boundaries heuristically from the output of a parser. It creates a list of candidate mentions by merging basic NP chunks with named entities.

The tokens and PoS tags provided to both mention detectors were analysed by Basque Linguistic processors, nevertheless, the PoS tags were mapped to equivalent Penn PoS tags (Marcus, Marcinkiewicz and Santorini Reference Marcus, Marcinkiewicz and Santorini1993). The parse trees used by the two systems were created using the Stanford PCFG parser (Klein and Manning Reference Klein and Manning2003).

Table 5 presents the scores obtained by the three compared systems. SMD, BART Mention Detector (BRTMD) and Basque Mention Detector (BMD).

Table 5.

Comparison of different mention detectors. SMD=Stanford Mention Detector, BRTMD= BART Mention Detector, BMD=Basque Mention Detector

Compared to the SMD, our mention detector outperforms Strict Matching protocol F-measure by 47.53 and that of Lenient Matching by 22.63 percentage points. BMD also improves the scores that BRTMD obtains by 35.01 when Strict Matching protocol is used, and by 17.29 using Lenient Matching. The great difference in F-measures of Stanford and BRTMD compared with those obtained by our mention detector clearly shows that language specific characteristics (agglutinative, free-word order, etc. for Basque) should be taken into account when mention detection is performed.

6.1 Error types

To help identify the different types of errors, we carried out a qualitative evaluation on a subcorpus of 1,904 mentions that were used to develop the system.

We classify the error types made in the mention detection step in eight different categories. The error categorization idea is based on (Kummerfeld and Klein 2013) where errors of a coreference system are classified. We have adapted this categorization to classify errors produced in mention detection. To better understand this, for each category we present an example. The first item in each example is the incorrect case (×) and the second the respective correct one (√). The categories are the following:

1. 1. Missing mention (MM): The mention in gold standard has not been identified.

   × Guk bere garaian apustu handia egin genuen Sa Pinto fitxatu genuenean, jokalaria egoera berezi batean aurkitzen baitzen.

   √ Guk bere garaian apustu handia egin genuen Sa Pinto fitxatu genuenean, [jokalaria] egoera berezi batean aurkitzen baitzen.

   • 

   “At the time we hired Sa Pinto we took a great gamble because [the player] was in a special situation.”

2. 2. Extra mention (EM): The system proposes a mention that is not present in the gold standard.

   × [Bestetik], Zestoako Udalak mozio bat onartu zuen aho batez AEKren kontrako prozesua salatzeko.

   √ Bestetik, Zestoako Udalak mozio bat onartu zuen aho batez AEKren kontrako prozesua salatzeko.

   • 

   “Moreover, Zestoa City Council accepted a motion unanimously to condemn the trial against AEK.”

3. 3. Span errors (SE): The system mention does not match exactly with the true or gold mention. Two different span error cases can happen.

   1. 3.1 Head identified (HI): The head of the mention to be identified is inside the span proposed by the system.

      1. 3.1.1 Missing text (MT): The system mention span misses one or more words that should be considered part of the gold mention.

         × [Jiri Vanek] txekiarrak irabazi egin zion atzo.

         √ [Jiri Vanek txekiarrak] irabazi egin zion atzo.

         • 

         “[Czech Jiri Van] beat him/her yesterday.”

      2. 3.1.2 Extra text (ET): The system mention span has one or more words that are not part of the gold mention.

         × [Luis Uranga harrituta] azaldu da Portugalgo klubaren jokaerarekin.

         √ [Luis Uranga] harrituta azaldu da Portugalgo klubaren jokaerarekin.

         “[Luis Uranga] is surprised with the attitude of the club from Portugal.”

         • 

         “[Luis Uranga] is surprised with the attitude of the club from Portugal.”

   2. 3.2 Head not identified (HNI): The head of the mention to be identified is not inside the system mention span.

      × Europako Kontseiluetan herrialde handiek indar [gehiago] izango dute.

      √ Europako Kontseiluetan herrialde handiek [indar gehiago] izango dute.

      • 

      “The great nations will have [more strength] in European councils.”

4. 4. Divided mention (DM): Mention to be identified is divided in two or more mentions.

   × Eztandak ez du inor zauritu, eta kalte material txikiak eragin ditu, Hammersmith zubian, [Londres] [mendebaldean].

   √ Eztandak ez du inor zauritu, eta kalte material txikiak eragin ditu, Hammersmith zubian, [Londres mendebaldean].

   • 

   “The explosion in Hammersmith Bridge, [west London], has not hurt anyone and material damage has been small.”

5. 5. Conflated mentions (CM): Two or more mentions are identified as they were only one.

   × Gogotsu ariko da [Euskaltel Alpeetako Klasikoan] eta Dauphine Liberen ariko dira hurrena.

   √ Gogotsu ariko da [Euskaltel] [Alpeetako Klasikoan] eta Dauphine Liberen ariko dira hurrena.

   • 

   “[Euskaltel] will participate enthusiastically in the Classique des Alpes and then they will take part in Dauphine Libere.”

6. 6. Combination of errors (CN): Two or more errors explained above can be identified in the mention proposed by the system. The following examples illustrate two cases of combined errors:

• • Divided Mention + Missing Text

  × [Siriako Atzerri ministro Farouk] Al [Xaraak] ukatu egin du Israelek Xebaako landetxeak okupatzea onartzen duela.

  √ [Siriako Atzerri ministro Farouk Al Xaraak] ukatu egin du Israelek Xebaako landetxeak okupatzea onartzen duela.

  • 

  “[The Syrian Foreign Minister Farouk al Sharaa] has denied that Israel accepts to occupy the farms of Shebaa.”

• • Conflated Mentions + Extra text

  × [Clintonen irudiko, Alderdi Errepublikarraren egitasmoa] ez da egokia europarren interesetarako.

  √ [Clintonen] irudiko, [Alderdi Errepublikarraren egitasmoa] ez da egokia europarren interesetarako.

  • 

  “Clinton believes the project of the Republican Party is not suitable for the interest of Europeans.”

Table 6 summarises the examples explained above.

Table 6.

Examples of error types

Table 7 shows the number of each error type when automatic mention detection is carried out and the percentages. The percentages are calculated taking into account the total number of correct mentions (1,904) in the subcorpus used for the evaluation.

Table 7.

Error types automatic mentions

As can be observed, the majority of errors are Missing Mentions (11.50%) and Extra Mentions (7.46%). The last row shows that the errors are 32.10% of the total correct mentions.

Each error type differently affects the precision and recall values. The differences also depend on the scoring protocol that has been used for scoring. For instance, the conflated mentions error type provokes one precision error and k (the number of mentions in which they have been conflated) recall errors when Strict Matching protocol is used. To better understand this, we can observe the following example, which has previously been presented.

In this example, the system conflates two mentions in one ([Euskaltel Alpeetako Klasikoan]), so k = 2. This mention is considered incorrect when Strict Matching is applied as it does not exactly match with any mention in the gold, and also when Lenient Matching is applied. Although it has the head of the correct mention inside its span, it exceeds the span of the gold mention. Consequently, it is considered a precision error. Regarding recall, as k is two in this case, two recall errors are counted because [Euskaltel] and [Alpeetako Klasikoan] gold mentions are not present in the system output.

Table 8 summarises how error types affect precision and recall when different scoring protocols are used. In the case of combination of errors, the effect in precision and recall depends on the nature of the combined errors. The same happens when the scoring protocol is Lenient Matching.

Table 8.

Effects of error types in P and R

6.2 Error causes

To help identify the error causes, we carried out a qualitative evaluation on a corpus of 1,070 mentions, a subcorpus of the 1,904 mentions that were used to analyse the error types.

We divided the errors encountered in nine categories. For each category, we present some examples. The first item in each example is the incorrect case (×) and the second the respective correct one (√).

1. 1. Erroneous morphosyntactic analysis (EMA): POS tags and/or syntactic functions are incorrectly disambiguated by the lemmatiser. The same string of letters can be a noun in some contexts and an adjective in others; this error occurs when a non-noun instance is tagged as a noun and, consequently, incorrectly regarded as a chunk. In addition, some complex postpositional are not identified correctly. In the following example, arte is tagged as a noun (arte, n.: art) and, consequently, considered a nominal chunk, although in this context it is a part of a complex postposition that has not been identified, meaning “until” which is a preposition in English.

   × Futbolean azken minutura [arte] edozer gauza gerta daiteke.

   √ Futbolean azken minutura arte edozer gauza gerta daiteke.

   • 

   “In football, anything can happen until the last minute.”

2. 2. Incorrect chunk tag (ICT): The chunker does not provide correct chunk tags.

   × Estatu Batuetako presidente Bill [Clintonek] antimisilen inguruan Pentagonoa atontzen ari den sistema babesteko eskatu zion atzo Europar Batasunari Lisboan.

   √ [Estatu Batuetako presidente Bill Clintonek] antimisilen inguruan Pentagonoa atontzen ari den sistema babesteko eskatu zion atzo Europar Batasunari Lisboan.

   • 

   “Yesterday in Lisbon, [United States president Bill Clinton] asked the European Community to support the anti-missile system that the Pentagon is preparing.”

3. 3. Missing chunk tag (MCT): The chunker does not tag a chunk, or it only tags the beginning or the end of the chunk. Not opened or not closed tags can occur when CG formalism (Karlsson et al. Reference Karlsson, Voutilainen, Heikkilä and Anttila1995) is used.

   × [Luis Uranga harrituta azaldu da Portugalgo klubaren jokaerarekin.

   √ [Luis Uranga] harrituta azaldu da Portugalgo klubaren jokaerarekin.

   • 

   “[Luis Uranga] is surprised at the attitude of the Portuguese club.”

   The error could be partially solved in the case when the chunker missed the ending tag of a chunk. In this case, we applied a heuristic that is explained in Section 6.4.

4. 4. Incorrect clause tag (ICLT): The left clause boundary proposed by the clause tagger is not the same as the gold mention left boundary, so the mention is incorrect.

   × Behin betiko hitzarmen bat sinatu behar dute bi aldeek, [horrela Oslon hasitako prozesua] bukatzeko.

   √ Behin betiko hitzarmen bat sinatu behar dute bi aldeek, horrela [Oslon hasitako prozesua] bukatzeko.

   • 

   “The two parts are going to sign a permanent agreement so that [the process that began in Oslo] will be brought to a close.”

5. 5. Predication (PRE): Two words are joined in one chunk even though there is a predication relation between them. In coreference resolution it is common to join two mentions in one cluster when they have a predicational relation. For this reason, it is necessary to split the chunk into two parts when there is a predication. However, in some cases it is difficult to predict the predication relation and the chunk is not split, causing a mention detection error.

   × [Bush buru] duen Alderdi Errepublikarraren proiektua okerragoa da.

   √ [Bush] [buru] duen Alderdi Errepublikarraren proiektua okerragoa da.

   • 

   “The project of the Republican party whose [leader] is [Bush] is worse”.

   The opposite case also occurs, i.e., a chunk is split when it should not be, causing a mention detection error.

   × [Madariagaren] [ametsa] da aspalditik.

   √ [Madariagaren ametsa] da aspalditik.

   • 

   “It has long been [Madariaga’s dream].”

   The error is caused by a rule that does not divide a mention when it is presented in a predication relation or divides it when it should not. The rule can be adapted to better identify whether the mentions should be divided.

6. 6. Coordination (COR): In coordination cases, the entire coordinate structure is considered a mention. However, in some cases the submentions to the left and right of the coordinating conjunction do not need to be marked. The detector should not split the coordinate structure when the submentions do not have to be included, otherwise we get incorrect mentions in the output.

   × [[Lintxamendu] eta [jazarpenerako erabili nahi den aitzakia]] da.

   √ [Lintxamendu eta jazarpenerako erabili nahi den aitzakia] da.

   • 

   “It is [an excuse to use for lynching and persecution].”

   The same happens when the whole coordinate structure is not joined.

   × [Bost urteko kontratua] eta [sekulako dirutza] lortu zituen.

   √ [[Bost urteko kontratua] eta [sekulako dirutza]] lortu zituen.

   • 

   “He/she obtained [[a five-year contract] and [lots of money]].”

   In addition, sometimes coordinate structures are enumerations. In enumeration cases, apart from the entire coordinate mention, submentions divided by the comma must also be obtained as mentions. When these enumerations are not divided correctly mentions are missed.

   × [Van der Linden, Malafosse, Carboneau, Carrat, Venditti, Arbizu, Lamaison eta Smith] dira jokalariak.

   √ [[Van der Linden], [Malafosse], [Carboneau], [Carrat], [Venditti], [Arbizu], [Lamaison] eta [Smith]] dira jokalariak.

   • 

   “The players are [[Van der Linden], [Malafosse], [Carboneau], [Carrat], [Venditti], [Arbizu], [Lamaison], and [Smith]].”

   Adapting the rules that decide whether a coordinate structure needs to be divided into submentions and joined into bigger structures would necessarily improve mention detection scores. In addition, specific rules to treat enumerations would also improve mention detection.

7. 7. Comparatives (COMP): In comparatives structures, the related two elements are not joined and, consequently, the adverb (baino than) is considered out of the mention.

   × [Bilerak] baino [oihartzun handiagoa] lortu zuen atentatuak.

   √ [Bilerak baino oihartzun handiagoa] lortu zuen atentatuak.

   • 

   “The attack received [greater coverage than the meeting].”

   A specific rule in the mention detector to treat comparative structures would solve the problem.

8. 8. Referentiality (RF): Some referential chunks are discarded as they do not have nouns inside them.

   × Italiarrak onartezintzat hartu du erabakia.

   √ [Italiarrak] onartezintzat hartu du erabakia.

   • 

   “[The Italian] considers the decision unacceptable.”

   On the contrary, non-referential chunks that have nouns inside are considered as mentions, for example, linking words.

   × [Era berean], lurralde suntsitua berreraikitzeko nazioarteko laguntza eskatu zuen.

   √ Era berean, lurralde suntsitua berreraikitzeko nazioarteko laguntza eskatu zuen.

   • 

   “In the same way, he/she called for international help to rebuild the destroyed territory.”

9. 9. Miscellaneous (MISC): Errors with varied causes that cannot be classified into the categories discussed earlier. Some examples include:

   • • Parenthetical: The chunks are divided by a parenthetical structure.

     × [David Trimble] ([UUP]) [lehen ministroak] adierazpenak egin zituen.

     √ [David Trimble ([UUP]) lehen ministroak] adierazpenak egin zituen.

     • 

     “[Prime Minister David Trimble ([UUP])] made statements.”

   • • Typing error: Some texts have typing errors, missing spaces between words, unreadable characters. . . When processing these texts with automatic tools the typing errors are not always solved and cause a mention detection error.

     × [1995eanhartu] zuen parte.

     √ [1995ean] hartu zuen parte.

     • 

     “He/she participated in [1995].”

     Solving these problems is complicated, and they would be better corrected in the preprocessing step.

   • • Extra Words in Boundaries: The chunks contain words that fall outside the mention boundaries.

     × Kolpistek elkarrizketak hautsi zituzten [erabaki horren ondorioz].

     √ Kolpistek elkarrizketak hautsi zituzten [erabaki horren] ondorioz.

     • 

     “The leaders of the coup broke off the talks because of that decision.”

     The error of mentions with extra words can be solved by defining a set of words that appear at the boundaries of mentions even though they are not part of the mention. This way such words can be removed from within the mention boundaries.

   • • Unjoined chunks: Even though the chunks are correct they do not match the mentions exactly. In most cases, the chunks should be joined to create a correct mention.

     × Gentzelek [atzo] [goizean] hitz egin zuen.

     √ Gentzelek [atzo goizean] hitz egin zuen.

     • 

     “Gentzel talked [yesterday morning].”

     The error of unjoined chunks that should form a mention can be solved by identifying the exact cases in which this phenomenon occurs.

   Although specific rules to treat miscellaneous (MISC) cases would likely improve mention detection, the phenomenon is a broad one and the improvement may not be substantial.

We observed that the errors explained above can be classified in three main categories, some are caused by preprocessing tools (EMA, ICT, MST), some others are because of using the output of general-purpose tools that must be adapted to our mentions specification (ICLT, PRE, COR, CONP), while the rest are caused by inaccurate mention detection rules (RF). The MISC errors fall in all the categories.

Table 9 shows the percentages of the main error causes resulting from the mention detection step.

Table 9.

Main causes of errors

As Table 9 shows, the automatic preprocessing tools are the main source of errors: 27.93% are caused by the chunker providing incorrect chunk tags (ICTs), and 18.92% by it missing the chunk tag. The morphosyntactic analysis that the mention detector receives as input also causes a number of errors in mention detection, accounting for 14.11% of the errors. Another considerable cause of error is that the left clause boundaries identified by the clause tagger do not match the left boundaries of gold mentions; this accounts for 7.51% of the errors.

Given the above, we can conclude that the input that our mention detector receives has a substantial impact on mention detection errors. This is also pointed out by Uryupina and Moschitti (Reference Uryupina and Moschitti2013), who suggest that better performance can be obtained if a robust preprocessing is achieved. The performance of the mention detector is highly dependent on the preprocessing tools, and errors caused by these tools are difficult to tackle. We can, however, attempt to address other types of errors.

We found that the greatest improvements can be achieved by treating the errors classified as COR and RF, which account for 10.51% and 9.61% of errors, respectively.

6.3 Distribution of errors among mention types

Apart from identifying the main causes of errors, observing how these errors are distributed among mention types is essential to implementing improvements effectively. Figure 4 shows this distribution in terms of percentages of the total number of errors.

Fig. 4.

Distribution of error causes into mention types. Automatic processing.

As Figure 4 shows, ICT error cause is mainly divided into two mention types, proper nouns (10.81%) and common noun phrases (15.62%). A similar pattern can be found in the Missing Chunk Tag (MCT) errors. They cause 18.92% of the total errors and mainly affect common noun phrases (9.91%) and proper nouns (4.50%). Incorrect Clause Tag (ICLT) errors, in turn, are mainly errors in NPs containing subordinate clauses (6.91%). Other considerable errors are errors caused by Erroneous Morphosyntactic Analysis (EMA) (9.01%), Referentiality (RF) (9.01%), and MISC (6.31%) in Common Noun Phrases.

In the next Section, we explain how we improved our mention detection to overcome the problems identified by the error analysis.

6.4 Improvements in mention detection

To attempt to resolve some of the errors presented in Table 9, we implemented the following five adjustments and applied them in the mention detection process:

1. 1. To solve the errors caused by the chunker only tagging the beginning of the chunk and not the end, we applied a heuristic that closes the nominal chunk when a verbal chunk starts. In other words, if a nominal chunk is not closed when a verbal one starts, we close it just before the verbal chunk. With this improvement, we tried to solve some of the MCT errors (third row in Table 9). In example (a), the nominal chunk borroka armatua “armed struggle” is opened properly but the close tag is missed. Since a verbal chunk itzultzeko “to return” starts after the word armatua “armed”, we close the nominal chunk, obtaining a correct nominal chunk as in example (b).

   1. (a) IRA [borroka armatura itzultzeko asmotan da.

      • 

      “The IRA intends to return to [armed struggle.”

   2. (b) IRA [borroka armatura] itzultzeko asmotan da.

      • 

      “The IRA intends to return to [armed struggle].”

2. 2. We observed in the error analysis that some mentions in a predication relation are divided when they should not be, or not divided when they should be. We adjusted the rule in the FSTs that was responsible for deciding whether a mention should be divided or not, depending on whether it was in a predication relation. With this improvement, we aimed to solve PRE errors (fifth row in Table 9). In the following examples, (a) is a case when a mention should be divided and (b) an example where the mention should not be divided.

   1. (a) [Ibarretxe] [lehendakaria] da.

      • 

      “[Ibarretxe] is the [president].”

   2. (b) [Bushen proposamena] da.

      • 

      “It is [Bush’s proposal].”

3. 3. We observed that chunks composed only of determiners were considered mentions. However, not all of these chunks are mentions. For instance, examples such as biok “we both” in (a) should be considered mentions, but examples such as bestetik “on the other hand” (b) should be discarded. We defined a new rule that discards chunks of undefined determiners because they are rarely referential. With this improvement, we aimed to detect some mentions that were missed and quantified as RF error (eighth row in Table 9).

   1. (a) [Biok] egin genuen lan hura.

      • 

      “[We both] did the work.”

   2. (b) Bestetik, orain arte jasotako elkartasuna eskertu zuen.

      • 

      “On the other hand, he expressed his thanks for the support he had received so far.”

4. 4. Chunks composed of only adjectives were all discarded. Upon closer examination, however, we observed that some of them could in fact be referential, as clarified by the examples below. In example (a), handia “big” is not referential so it must not be considered as a mention, but in example (b) the same word, handia, is referential and so is a mention. Therefore, we adjusted the rule that discarded chunks composed of nothing but adjectives so that it only discarded undefined adjectives and adjectives that were in a predication relation, as in example (a). With this improvement, our aim was to detect mentions that were missed and quantified as RF error (eighth row in Table 9).

   1. (a) Etxea oso handia da.

      • 

      “The house is very big.”

   2. (b) [Handia] garestiagoa da.

      • 

      “[The big one] is more expensive.”

5. 5. Whether chunks to the left and the right of the coordinating conjunction are considered a single chunk or separate chunks is determined by the nature of the chunk, in particular whether the element(s) in the chunk are declined. In example (a), both Pello and Ane are declined (-ri to) and are therefore considered two separate chunks, but in (b) Pello is not declined (-Ø) and therefore the elements are considered a single chunk. We defined new rules that identify coordinate structures that are not joined but that should be considered mentions, e.g. (a). With this improvement, our goal was to detect coordinate structures that were missed, thereby decreasing COR error (sixth row in Table 9).

   1. (a) [Pellori] eta [Aneri] oparia eman zioten.

      • 

      “They gave the present to [Pello] and [Ane].”

   2. (b) [Pello eta Aneri] oparia eman zioten.

      • 

      “They gave the present to [Pello] and [Ane].”

After implementing these five improvements, we obtained a gain of 1.21% in the F-measure using Strict Matching and of 0.89% with Lenient Matching. These gains are significant.Footnote ³ The results can be seen in Table 10, and show that the improvements have succeeded in resolving some errors.

Table 10.

Improved scores in mention detection. BMD = Basque Mention Detector, BIMD = Basque Improved Mention Detector

Table 11 shows the main error types with improved mention detection (BIMD) and without it (BMD). In absolute values, 61 errors have been resolved, 611 without improvements and 548 after applying them. The biggest improvement has been achieved in Missing Mention error type, 44 mentions that before were not identified are now in the system output and are correct mentions. In percentage values, an improvement of 3.3% is obtained, 28.80% being errors of total mentions.

Table 11.

Comparison of main error types when improvements in mention detection are applied