3.1 Current measures: strong and weak points

This section reviews the main advantages and drawbacks of the principal coreference evaluation measures. The main difference resides in the way they conceptualize how a coreference set within a text is defined: either in terms of links, i.e. the pairwise links between mentions (MUC, Pairwise F1, Rand), or in terms of classes or clusters, i.e. the entities (B³, CEAF, ACE-value, mutual information). Although the two approaches are equivalent in that knowing the links allows building the coreference classes, and knowing the classes allows inferring the links, differences in instantiation design produce a range of evaluation metrics that vary to such an extent that still today there is no widely agreed upon standard. Table 1 shows how the different system outputs in Figure 3 (borrowed from Luo (Reference Luo2005)) are scored by the various scoring algorithms presented next.

Table 1.

Comparison of evaluation metrics on the examples in Figure 3

Fig. 3.

Example entity partitions (from Luo (Reference Luo2005)).

MUC (Vilain et al. Reference Vilain, Burger, Aberdeen, Connolly and Hirschman1995). This is the oldest and most widely used measure, defined as part of the MUC-6 and MUC-7 evaluation tasks on coreference resolution. It relies on the notion that the minimum number of links needed to specify either GOLD or SYS is the total number of mentions minus the number of entities. The MUC measure computes the number of all coreference links common between GOLD and SYS. To obtain recall (R), this number is divided by the minimum number of links required to specify GOLD. To obtain precision (P), it is divided by the minimum number of links required to specify SYS.

As observed by Bagga and Baldwin (Reference Bagga and Baldwin1998) and Luo (Reference Luo2005), the MUC metric is severely flawed for two main reasons. First, it is indulgent, as it is based on the minimal number of missing and wrong links, which often results in counterintuitive results. Classifying one mention into a wrong entity counts as one P and one R error, while completely merging two entities counts as a single R error, although this is further away from the real answer. As a result, the MUC score is too lenient with systems that produce overmerged entities (entity sets containing many referring expressions), as shown by system responses (b) and (c) in Table 1. If all mentions in each document of the MUC test setsFootnote ² are linked into one single entity, the MUC metric gives a score higher than any published system (Finkel and Manning Reference Finkel and Manning2008). Second, given that it only takes into account coreference links, the addition of singletons to SYS does not make any difference. It is only when a singleton mention is misclassified in a multimention entity that there is decrease in the MUC score. This is why the entry for system response (d) in Table 1 is empty.

B ³ (Bagga and Baldwin Reference Bagga and Baldwin1998). To penalize clustering too many mentions in the same entity, this metric computes R and P for each mention, including singletons. The total number of intersecting mentions between the GOLD and the SYS entities is computed and divided by the total number of mentions in the GOLD entity to obtain R, or in the SYS entity to obtain P. The average over the individual mention scores gives the final scores.

Although B³ addresses the shortcomings of MUC, it presents a drawback in that scores squeeze up too high due to singletons: when many singletons are present, scores rapidly approach 100 per cent. This leaves little numerical room for comparing systems, and forces one to consider differences in the second and third decimal places when scores are high (while such differences are meaninglessly small in lower ranges). It is not possible to observe this in Table 1, as the truth in Figure 3 does not contain any singleton. However, it turns out that singletons are the largest group in real texts (see Table 2): about 86 per cent of the entities if the entire set of mentions is considered, like in the AnCora corpora; 61 per cent of the entities in the ACE corpora, where the coreference annotation is restricted to seven semantic types (person, organization, geo-political entity, location, facility, vehicle, and weapon). A side effect is that B³ scores are inflated, obscuring the intuitively appropriate level of accuracy of a system in terms of coreference links.

Table 2.

Distribution of mentions into entities in two corpora: the English ACE-2004 (Doddington et al. Reference Doddington, Mitchell, Przybocki, Ramshaw, Strassel and Weischedel2004) and the Spanish AnCora-Es (Recasens and Martí Reference Recasens and Martí2010)

CEAF (Luo Reference Luo2005). Luo (Reference Luo2005) considers that B³ can give counterintuitive results due to the fact that an entity can be used more than once when aligning the entities in GOLD and SYS. In Figure 3, B³–R is 100 per cent for system response (c) even though the true set of entities has not been found; conversely, B³–P is 100 per cent for system response (d) even though not all the SYS entities are correct. Thus, he proposes CEAF, which finds the best one-to-one mapping between the entities in GOLD and SYS, i.e. each SYS entity is aligned with at most one GOLD entity, and the best alignment is the one maximizing the similarity. Depending upon the similarity function, Luo (Reference Luo2005) distinguishes between the mention-based CEAF and the entity-based CEAF, but we will focus on the former as it is the most widely used. It employs Luo's (Reference Luo2005) φ₃ similarity function. When true mentions are used, R and P scores are the same. They correspond to the number of common mentions between every two aligned entities divided by the total number of mentions.

CEAF, however, suffers from the singleton problem just as B³ does. This accounts for the fact that the B³ and CEAF scores are usually higher than the MUC on corpora where singletons are annotated (e.g. ACE, AnCora), because a great percentage of the score is simply due to the resolution of singletons. In addition, CEAF's entity alignment might cause a correct coreference link to be ignored if that entity finds no alignment in GOLD (Denis and Baldridge Reference Denis and Baldridge2009). Finally, all entities are weighted equally, irrespective of the number of mentions they contain (Stoyanov et al. Reference Stoyanov, Gilbert, Cardie and Riloff2009), so that creating a wrong entity composed of two small entities is penalized to the same degree as creating a wrong entity composed of a small and a large entity.

ACE-value (Doddington et al. Reference Doddington, Mitchell, Przybocki, Ramshaw, Strassel and Weischedel2004). The ACE-value, the official metric in the ACE program, is very task-specific, and not really useful for the general coreference problem that is not limited to a set of specific semantic types. A score is computed by subtracting a normalized cost from 1. The normalized cost corresponds to the sum of errors produced by unmapped and missing mentions/entities as well as wrong mentions/entities,Footnote ³ normalized against the cost of a system that does not output any entity. Each error has an associated cost that depends on the type of the ACE-entity and on the kind of mention, but these costs have changed between successive evaluations. The ACE-value is hard to interpret (Luo Reference Luo2005): a system with 90 per cent does not mean that 90 per cent of system entities or mentions are correct, but that the cost of the system, relative to the one producing no entity, is 10 per cent.

Pairwise F1. Also known as positive-link-identification F-score. If reported, this metric is always included in addition to MUC, B³ and/or CEAF, as it is meant to give some further insight not provided by the other metrics (Choi and Cardie Reference Choi and Cardie2007; Poon and Domingos Reference Poon and Domingos2008; Haghighi and Klein Reference Haghighi and Klein2009). Pairwise F1 simply computes P, R, and F over all pairs of coreferent mentions. As noted by Haghighi and Klein (Reference Haghighi and Klein2009), merging or separating entities is over-penalized quadratically in the number of mentions. Besides, it ignores the correct identification of singletons.

Mutual information, H (Popescu-Belis Reference Popescu-Belis2000). The H measure draws on information theory to evaluate coreference resolution. GOLD and SYS are seen as the two ends of the communication channel, GOLD being the sender or speaker, and SYS being the receiver or the hearer. The coreference information of GOLD and SYS correspond to the entropy of GOLD and SYS, respectively. Then the GOLD and SYS partitions are compared on the basis of the mutual coreference information. R is obtained by subtracting the conditioned entropy of GOLD given SYS (loss of information) from the entropy of GOLD. P is obtained by subtracting the conditioned entropy of SYS given GOLD (irrelevant information gains) from the SYS entropy. Both values are then normalized. This measure has been hardly used for reporting results of real systems, and it emerges from the results reported by Popescu-Belis (Reference Popescu-Belis2000) that H is not superior to the other existing measures. Popescu-Belis concludes that each metric, by focusing on different aspects of the data, provides a different perspective on the quality of the system answer.

Rand index (Rand Reference Rand1971). The Rand index is a general clustering evaluation metric that measures the similarity between two clusterings (i.e. partitions) by considering how each pair of data points is assigned in each clustering. Stated in coreference terms, the Rand index equals the number of mention pairs that are either placed in an entity or assigned to separate entities in both GOLD and SYS, normalized by the total number of mention pairs in each partition. The motivations behind this measure are three (where we replace ‘point’ by ‘mention’, ‘cluster’ by ‘entity’, and ‘clustering’ by ‘entity partition’): (1) every mention is unequivocally assigned to a specific entity; (2) entities are defined just as much by those points which they do not contain as by those mentions which they do contain; and (3) all mentions are of equal importance in the determination of the entity partition.

The only use of the Rand index for coreference resolution appears in Finkel and Manning (Reference Finkel and Manning2008). Although Rand has the potential to capture well the coreference problem, it is not useful if applied as originally defined due to the significant imbalance between the number of coreferent mentions and the number of singletons (Table 2). The extremely high number of mention pairs that are found in different entities in GOLD and SYS explains the high figures obtained by all systems reported in Finkel and Manning (Reference Finkel and Manning2008), and by system response (d) in Table 1. Hence, the low discriminatory power of Rand. The BLANC measure that we introduce in Section 4 implements Rand in a way suited to the coreference problem.

It is often hard for researchers working on coreference resolution to make sense of the state of the art. Compare, for example, the scores shown in Table 3 that correspond to various systemsFootnote ⁴ and two baselines: (1) all singletons (i.e. no coreference link is created, but each mention is considered to be a separate entity), and (2) one entity (i.e. all document mentions are clustered into one single entity). The only measure for which we have the results of all systems is MUC, but this is the one with the largest number of drawbacks, as evidenced by the one-entity baseline's high score. It is clear that the measures do not produce the same ranking of the systems, other than the fact that they all rank Luo et al. (Reference Luo, Ittycheriah, Jing, Kambhatla and Roukos2004) and Luo and Zitouni (Reference Luo and Zitouni2005) as the best systems for each data set. This sort of discrepancy makes it impossible in the long term to conduct research on this question: which measure should one trust, and why?

Table 3.

Performance of state-of-the-art coreference systems on ACE

Apart from the pros and cons of each measure, the difficulty in comparing the performance of different coreference resolution systems is compounded by other factors, such as the use of true or system mentions and the use of different test sets (Stoyanov et al. Reference Stoyanov, Gilbert, Cardie and Riloff2009). Some systems in Table 3 are not directly comparable since testing on a different set of mentions or on a different data set is likely to affect scoring. Ng (Reference Ng2009) did not use true but system mentions, and Luo and Zitouni (Reference Luo and Zitouni2005) had access to the entire ACE-2004 formal test sets, while the remaining systems, due to licensing restrictions, were evaluated on only a portion of the ACE-2004 training set.

3.2 Desiderata for a coreference evaluation measure

Coreference is a type of clustering task, but it is special as each item in a cluster bears the same relationship, referential identity, with all other items in the same cluster, plus the fact that a large number of clusters are singletons. Thus, only two of the four formal constraints for clustering evaluation metrics pointed out by Amigó et al. (Reference Amigó, Gonzalo, Artiles and Verdejo2009) apply to coreference. Amigó et al.'s (Reference Amigó, Gonzalo, Artiles and Verdejo2009) formal constraints include: (1) cluster homogeneity, i.e. clusters should not mix items belonging to different categories; (2) cluster completeness, i.e. items belonging to the same category should be grouped in the same cluster; (3) rag bag, i.e. it is preferable to have clean clusters plus a cluster with miscellaneous items over having clusters with a dominant category plus additional noise; and (4) cluster size versus quantity, i.e. a small error in a large cluster is preferable to a large number of small errors in small clusters.

While the first two constraints undoubtedly hold for coreference resolution, the last two do not necessarily. What makes coreference resolution special with respect to other clustering tasks is the propagation of relations within an entity caused by the transitive property of coreference. That is to say, unlike regular clustering, where assigning a new item to a cluster is a mere question of classifying that item into a specific category, in coreference resolution assigning a new mention to an entity implies that the mention is coreferent with all other mentions that have been assigned to that same entity. Thus, the larger an entity is, the more coreferent links will be asserted for each new mention that is added.

To illustrate: To us, given the GOLD in Figure 4, the output produced by system S2 is not better than that produced by system S1, as it would follow from constraint (3). In fact, if the rag-bag entity contained more singletons, including an additional wrong singleton would make S2 even worse than S1. Similarly, in Figure 5, S2 is not better than S1, as suggested by constraint (4).

Fig. 4.

An example not satisfying constraint (3): The output S2 with a rag-bag cluster is equally preferable to S1.

Fig. 5.

An example not satisfying constraint (4): The output S2 with a small error in a large cluster is equally preferable to S1.

Amigó et al. (Reference Amigó, Gonzalo, Artiles and Verdejo2009) show that whereas B³ satisfies all four constraints, measures based on counting pairs, such as the Rand index, satisfy only constraints (1) and (2). This is the reason why Rand is a good starting point for developing the BLANC measure for coreference resolution in Section 4. As described in Section 3.1, the three most important points that remain unsolved by the current coreference metrics are as follows:

1. (1) Singletons. Since including a mention in the wrong chain hurts P, a correct decision to NOT link a mention should be rewarded as well. Rewarding correctly identified singletons, however, needs to be moderate, leaving enough margin for the analysis of correctly identified multimention entities.

2. (2) Boundary cases. Special attention needs to be paid to the behavior of the evaluation measure when a system outputs (1) all singletons, or (2) one entity (i.e. all mentions are linked).

3. (3) Number of mentions. The longer the entity chain, the more coreferent mentions it contains, each mention inheriting the information predicated of the other mentions. Thus, a correct large entity should be rewarded more than a correct small entity, and a wrong large entity should be penalized more than a wrong small entity.

We suggest that a good coreference evaluation measure should conform to the following desiderata:

1. (1) Range from 0 for poor performance to 1 for perfect performance.

2. (2) Be monotonic: Solutions that are obviously better should obtain higher scores.

3. (3) Reward P more than R: Stating that two mentions are coreferent when they are not is more harmful than missing a correct coreference link.Footnote ⁵ Hence, the score should move closer to 1 as

   • • More correct coreference links are found,

   • • more correct singletons are found,

   • • fewer wrong coreference links are made.

4. (4) Provide sufficiently fine scoring granularity to allow detailed discrimination between systems across the whole range [0, 1].

5. (5) As nearly as possible, maintain the same degree of scoring granularity throughout the whole range [0, 1].

4.1 Implementing the Rand index for coreference evaluation

From what has been said in Section 3, the Rand index seems to be especially adequate for evaluating coreference since it allows us to measure ‘non-coreference’ as well as coreference links. This makes it possible to correctly handle singletons as well as to reward correct coreference chains commensurately with their length.Footnote ⁶ The interesting property of implementing Rand for coreference is that the sum of all coreference and non-coreference links together is constant for a given set of N mentions, namely the triangular number N(N − 1)/2. By interpreting a system's output as linking each mention to all other mentions as either coreferent or non-coreferent, we can observe the relative distributions within this constant total of coreference and non-coreference links against the gold standard.

The Rand index (1) uses N ₀₀ (i.e. the number of mention pairs that are in the same entity in both GOLD and SYS) and N ₁₁ (i.e. the number of mention pairs that are in different entities in both GOLD and SYS) as agreement indicators between the two partitions GOLD and SYS. The value of Rand lies between 0 and 1, with 0 indicating that the two partitions do not agree on any pair of mentions and 1 indicating that the partitions are identical.

(1)

BLANC borrows the ‘bilateral’ nature of Rand to take into consideration both coreference links (N ₀₀) and non-coreference links (N ₁₁), but modifies it such that every decision of coreferentiality is assigned equal importance. Thus, BLANC models coreference resolution better by addressing the significant imbalance between the number of coreferent mentions and singletons observed in the real data. Further, whereas class-based metrics need to address the fact that GOLD and SYS might not contain the same number of entities, and the MUC metric focuses on comparing a possibly unequal number of coreference links, BLANC is grounded in the fact that the total number of links remains constant across GOLD and SYS.

4.1.1 Coreference and non-coreference links

BLANC is best explained considering the following two kinds of decisions:

1. (1) The coreference decisions (made by the coreference system)

   1. (a) A coreference link (c) holds between every two mentions that corefer.

   2. (b) A non-coreference link (n) holds between every two mentions that do not corefer.

2. (2) The correctness decisions (made by the evaluator)

   1. (a) A right link (r) has the same value (coreference or non-coreference) in GOLD and SYS (i.e. when the system is correct).

   2. (b) A wrong link (w) does not have the same value (coreference or non-coreference) in GOLD and SYS (i.e. when the system is wrong).

Table 4 shows the 2 × 2 confusion matrix obtained by contrasting the system's coreference decisions against the gold standard decisions. All cells outside the diagonal contain errors of one class being mistaken for the other. BLANC resembles Pairwise F1 as far as coreference links are concerned, but it adds the additional dimension of non-coreference links.

Table 4.

The BLANC confusion matrix

Let N be the total number of mentions in a document d, and let L be the total number of mention pairs (i.e. pairwise links) in d, thereby including both coreference and non-coreference links, then

The total number of links in the SYS partition of d is the sum of the four possible types of links, and it equals L:

where rc are the number of right coreference links, wc are the number of wrong coreference links, rn are the number of right non-coreference links, and wn are the number of wrong non-coreference links.

The confusion matrix for the example in Figure 1 is shown in Table 5. As the text has fourteen mentions, the total number of links is ninety-one. The system correctly identifies two coreference links (m ₅–m ₁₂, m ₇–m ₉), and wrongly another three coreference links (m ₄–m ₆, m ₇–m ₁₄, m ₉–m ₁₄). Every right coreference link that is missed by the system necessarily produces a wrong non-coreference link (m ₅–m ₁₄, m ₁₂–m ₁₄). The rest are eighty-four right non-coreference links. The confusion matrix shows the balance between coreference and non-coreference links with respect to the gold partition.

Table 5.

The BLANC confusion matrix for the example in Figure 1

The singleton problem pointed out in Section 3 becomes evident in Table 5: The number of non-coreference links is much higher than the number of coreference links. The class imbalance problem of coreference resolution causes that if the Rand index is applied as originally defined by Rand (Reference Rand1971), it concentrates in a small interval near 1 with hardly any discriminatory power. A chance-corrected Rand index has been proposed (Hubert and Arabie Reference Hubert and Arabie1985), but it is of no use for the coreference problem, given that the computation of expectation only depends on the number of pairs in the same cluster, thus ignoring singletons.

In order to take into account the under-representation of coreference links in the final BLANC score, we compute P, R, and F separately for the two types of links (coreference and non-coreference) and then average them for the final score. The definition of BLANC is shown in Table 6. In BLANC, both coreference and non-coreference links contribute to the final score, but neither more than 50 per cent. BLANC-P and BLANC-R correspond to the average of the two P and R scores, respectively. The final BLANC score corresponds to the average of the two F-scores. Applying the Rand index, the novelty of BLANC resides in putting equal emphasis on coreference and non-coreference links. Table 7 shows the different measures under discussion for the example in Figure 1.

Table 6.

Definition: formula for BLANC

Table 7.

Performance of example in Figure 1

4.1.3 The α parameter

After analyzing several coreferentially annotated corpora, we found that the average text contains between 60 per cent and 80 per cent singletons (depending on the coding scheme). Thus, simply averaging the coreference and non-coreference scores seems to be the best decision. However, given extraordinary cases like the one presented at the end of Section 4.1.2 or for those researchers that consider it to be convenient, we present the weighted version of BLANC:

BLANC_α lets users choose the weights they want to put on coreference and non-coreference links. In the default version of BLANC (Table 6), α = 0.5. Setting α closer to 1 will give a larger weight to coreference links, while setting α closer to 0 will have the opposite effect. For the problematic near-boundary case in which all links but one are non-coreferent in GOLD, evaluating with BLANC_α=0.1 will be much less severe than evaluating with the default BLANC.

4.2 Identification of mentions

An additional drawback that has been pointed out for the class-based metrics like B³ and CEAF is their assumption of working with true mentions, ignoring the problem of evaluating end-to-end systems, where some mentions in SYS might not be correct; i.e. might not be mapped onto any mention in GOLD and vice versa. These are called ‘twinless’ mentions by Stoyanov et al. (Reference Stoyanov, Gilbert, Cardie and Riloff2009). Bengtson and Roth (Reference Bengtson and Roth2008) simply discard twinless mentions, and Rahman and Ng (Reference Rahman and Ng2009) limit to removing only those twinless system mentions that are singletons, as in these cases no penalty should be applied. Recently, Cai and Strube (Reference Cai and Strube2010) have proposed two variants of B³ and CEAF that put twinless gold mentions into SYS as singletons and discard singleton twinless system mentions. In order to calculate P, wrongly resolved twinless system mentions are put into GOLD; to calculate R, only the gold entities are considered.

We agree that proper evaluation of a coreference system should take into account true versus system mentions. However, the mention identification task strictly belongs to syntax as it is closely related to the problem of identifying noun-phrase boundaries, followed by a filtering step in which only referential noun phrases are retained. It is clearly distinct from coreference resolution, whose goal is to link those noun phrases that refer to the same entity. One single metric giving the overall result for the two tasks together is obscure in that it is not informative as to whether a system is very good at identifying coreference links but poor at identifying mention boundaries, or vice versa. Therefore, instead of merging the two tasks, we propose to consider mention identification as its own task and separate its evaluation from that of coreference resolution (Popescu-Belis et al. Reference Popescu-Belis, Rigouste, Salmon-Alt and Romary2004). In brief, a measure for each problem is as follows:

• • Mention identification. This evaluation computes the correctness of the mentions that are being resolved, regardless of the structure of coreference links. Standard P and R are computed to compare the sets of mentions of GOLD and SYS. P is defined as the number of common mentions between GOLD and SYS divided by the number of system mentions; R is defined as the number of common mentions between GOLD and SYS divided by the number of true mentions. Two versions for the matching module are possible:

  • — Strict matching: A system mention is considered to be correctly identified when it exactly matches the corresponding gold mention.

  • — Lenient matching: A system mention is considered to be correctly identified when it matches at least the head of the corresponding gold mention (and does not include any tokens outside the gold mention).Footnote ⁷

• • Correctness of coreference. This evaluation computes the correctness of the coreference links predicted between the mentions shared by GOLD and SYS. The BLANC measure is applied to this set of correctly recognized mentions.

In this way, it might be possible to improve under-performing systems by combining, for instance, the strengths of a system that obtains a high coreference score but a low mention-identification score with the strengths of a system that performs badly in coreference resolution but successfully in the identification of mentions. Similarly, one should not be led to believe that improving the set of coreference features will necessarily result in higher scores, as the system's mention-identification score might reveal that the underlying problem is a poor detection of true mentions.

5.1 Results on artificial data

We take the gold partition in the first row of Table 8 as a working example. It is representative of a real case: it contains seventy mentions, 95 per cent singleton entities, a two-mention entity, a three-mention entity, and a four-mention entity. Each number represents a different mention; parentheses identify entities (i.e. they group mentions that corefer); and multimention entities are highlighted in bold. Table 8 also contains eight sample responses – output by different hypothetical coreference resolution systems – that contain different types of errors. See the decomposition into BLANC's four types of links in Table 9, a quantitative representation of the quality of the systems given in Table 8. The responses are ranked in order of quality, from the most accurate response to the least (response A is better than response B, B is better than C, and so on, according to our intuitionsFootnote ⁸).

Table 8.

Different system responses for a gold standard Gold₁

Table 9.

Decomposition of the system responses in Table 8

System A commits only one P error by linking two non-coreferent mentions; system B looks similar to A but is worse in that a singleton is clustered in a four-mention entity, thus producing not one but four P errors. System C exhibits no P errors but is weak in terms of R, as it fails to identify a three-mention entity. Although system D is clean in terms of R, it suffers from a severe P problem due to the fusion of the three- and four-mention entities in one large entity. System E is worse than the previous responses, as it shows both P and R errors: the four-mention entity is split into two and a singleton is added to both of them. System F worsens the previous output by completely failing to identify the four-mention entity and creating four incorrect two-mention entities. Finally, systems G and H represent the two boundary cases, the former being preferable to the latter, since at least it gets the large number of singletons, while the latter has a serious problem in P.

The performance of these system responses according to different measures is given in Tables 10 and 11. In them, we can see how BLANC addresses the three problems noted in Section 3.2.

1. (1) Singletons. The BLANC score decreases as the response quality decreases. It successfully captures the desired ranking, so does CEAF (although with fewer distinctions, see the ‘number of mentions’ problem below), and so does B³ if we leave aside the boundary responses G and H. BLANC, however, shows a much wider interval (from 97.61 to 49.90 per cent) than CEAF (from 98.57 to 91.43 per cent) and B³ (from 99.28 to 94.99 per cent), thus providing a larger margin of variation, and a finer granularity. The singleton problem is solved by rewarding the total number of correct singletons as much as the total number of correct mentions in multimention entities. Note that the original Rand index makes it impossible to discriminate between systems and it does not even rank them as intuitively expected.

2. (2) Boundary cases. MUC fails to capture the fact that the all-singletons response G is better than the one-entity response H. On the other hand, B³ and CEAF give a score close to 0 per cent for H, yet close to 100 per cent for G. It is counterintuitive that a coreference resolution system that outputs as many entities as mentions – meaning that it is doing nothing – gets such a high score. BLANC successfully handles the boundary responses by setting an upper bound of 50 percent on R.

3. (3) Number of mentions. The fact that MUC and CEAF give the same score to responses A and B shows their failure at distinguishing that the latter is more harmful than the former, as it creates more false coreference links. Namely, the information predicated of mention 1 is extended to mentions 61, 62, 63, and 64, and reciprocally mention 1 gets all the information predicated of mentions 61, 62, 63, and 64. Similarly, CEAF does not distinguish response D from E. In contrast, BLANC can discriminate between these responses because its reward of multimention entities is correlated with the number of coreference links contained in them.

Table 10.

Performance of the systems in Table 8

Table 11.

P and R scores for the systems in Table 8

The constructed example in Table 12 serves to illustrate BLANC's major weakness, which we have discussed at the end of Section 4.1.2. Performance is presented in Table 13. Notice the enormous leap between the BLANC_α=0.5 score for system D and the other three. This is due to the fact that partitions A, B, and C contain no right coreference link, and so BLANC is equal to the correctness of non-coreference links divided by two. The α parameter introduced in Section 4.1.3 is especially adequate for this type of cases. The difference in the scores for D and the rest of systems diminishes when α = 0.2 or α = 0.1 (the two last columns).

Table 12.

Different system responses for a gold standard Gold₂

Table 13.

Performance for the systems in Table 12

This same example, in fact, reveals weaknesses of all the measures. Owing to the fact that the MUC score does not reward correctly identified singletons, it is not able to score the first three responses, thus showing even a larger rise in response D. The B³ and CEAF measures score responses A and D the same, but only the latter succeeds in identifying the only coreference link that exists in the truth – a very relevant fact given that the ultimate goal of a coreference resolution system is not outputting only singletons (as system A does) but solving coreference. Finally, it is puzzling that CEAF considers response B to be appreciably better than response C – they are scored the same by B³ and BLANC. This is a weakness due to CEAF's one-to-one alignment: In B, the three final entities find a counterpart in the gold standard, whereas in C, only one of the two final entities gets mapped.

5.2 Results on real data

In order not to reach conclusions solely derived from constructed toy examples, we run a prototype learning-based coreference resolution system – inspired by Soon et al. (Reference Soon, Ng and Lim2001), Ng and Cardie (Reference Ng and Cardie2002), and Luo et al. (Reference Luo, Ittycheriah, Jing, Kambhatla and Roukos2004) – on 33 documents of the ACE-2004 corpus. A total of five different resolution models are tried to enable a richer analysis and comparison between the different evaluation metrics. The results are presented in Table 14. For a detailed analysis we address the reader to Recasens and Hovy (Reference Recasens and Hovy2010).

Table 14.

Different coreference resolution models run on ACE-2004

The first two are baselines that involve no learning: model A is the all-singletons baseline, and B clusters in the same entity all the mentions that share the same head. In C, D, and E, a pairwise coreference classifier is learnt (i.e. given two mentions, it classifies them as either coreferent or non-coreferent). In C and D, whenever the classifier considers two mentions to be coreferent and one of them has already been clustered in a multimention entity, the new mention is only clustered in that same entity if all pairwise classifications with the other mentions of the entity are also classified as coreferent. The difference between C and D lies in the initial mention pairs that form the basis for the subsequent process: C takes the first mention in textual order that is classified as coreferent with the mention under consideration, while D takes the mention that shows the highest confidence among the previous. E is a simplified version of C that performs no additional pairwise checks.

The best way to judge the quality of each response is to look at the actual data, but space limitations make this impossible. However, we can gain an approximation by looking at Table 15, which shows the number of entities output by each system and how many are singletons, as well as the number of correct and incorrect links of each type. Note that high numbers in the wc column indicate poor P, whereas high numbers in the wn column indicate poor R. Although the trade-off between P and R makes it hard to reach a conclusion as to whether C or D should be ranked first, the low quality of A, and especially E, is an easier conclusion to reach. The head-match baseline achieves high P but low R.

Table 15.

Decomposition of the system responses in Table 14

If we go back to Table 14, we can see that no two measures produce the same ranking of systems. The severe problems behind the MUC score are again manifested: it ranks model E first because it identifies a high number of coreference links, despite containing many incorrect ones. This model produces an output that is not satisfactory because it tends to overmerge. The fact that B³ ranks D and B first indicates its focus on P rather than R. Thus, B³ tends to score best those models that are more conservative and that output a large number of singletons. Finally, CEAF and BLANC agree in ranking C the best. An analysis of the data also supports the idea that strong match achieves the best trade-off between P and R.

Similar problems with the currently used evaluation metrics were also shown by the six systems that participated in the SemEval-2010 Task 1 on ‘Coreference Resolution in Multiple Languages’ (Recasens et al. Reference Recasens, Màrquez, Sapena, Martí, Taulé, Hoste, Poesio and Versley2010), where the BLANC measure was publicly used for the first time. Unlike ACE, mentions were not restricted to any semantic type, and the B³ and CEAF scores for the all-singletons baseline were hard to beat even by the highest performing systems. The BLANC scores, in contrast, tended to stay low regardless of the number of singletons in the corpus. However, it was not possible to draw definite conclusions about the SemEval shared task because each measure ranked the participating systems in a different order.

Finally, in Table 16 we reproduce Table 3 adding the BLANC score for the performance of state-of-the-art systems and the all-singletons and one-entity baselines. We can only include the results for those systems whose output responses were provided to us by the authors. It is worth noting that BLANC is closer to B³ when using the ACE-2 corpus but closer to CEAF when using the ACE-2004 corpus, which is probably due to the different distribution of singletons and multimention entities in each corpus. Knowing the state of the art in terms of BLANC will enable future researchers on coreference resolution to compare their performance against these results.

Table 16.

Performance of state-of-the-art systems on ACE according to BLANC

5.3 Plots

A graph plotting the BLANC slope as the percentage of correct coreference links (rc) increase is depicted in Figure 6, where the slopes of B³, CEAF, and MUC are also plotted. The curve slope for BLANC gradually increases, and stays between the other measures, higher than MUC but lower than B³ and CEAF, which show an almost flat straight line. The ‘pinching’ of scores close to 100 per cent by B³ and CEAF is clearly apparent. A coreference resolution system can obtain very high B³ and CEAF scores (due to the high number of singletons that are present in the gold partition), leaving a too small margin for the evaluation of coreference proper.

Fig. 6.

The BLANC score curve as the number of right coreference links increases.

We illustrate in Figure 7 the dependency of the BLANC score on degrees of coreference and non-coreference. Figure 7 plots the scores for the example in Table 8. The left rear face of the cube – where the right non-coreference (i.e. rn) level is a constant 1 and right coreference (rc) ranges from zero to 1 – displays the BLANC curve from Figure 6. The front face of the cube shows how – for a constant right coreference of 1 – the BLANC score ranges from near zero to 0.5 as right non-coreference ranges from zero to 1. The bend in the surface occurs due to the asymmetry in the number of true coreferences: the smaller the proportion of coreference links to non-coreference links, the sharper the bend and the closer it is to the left face. Systems must achieve correctness of almost all coreference and non-coreference links to approach the steep curve.

Fig. 7.

The BLANC score surface as a function of right coreference and right non-coreference links, for data from Table 8.