2 Citation purpose and polarity

The analysis of citation polarity and purpose can be extremely helpful in evaluating and summarizing the impact of papers. For our study, we first created an annotated dataset. We selected thirty papers from AAN that had different numbers of incoming citations and were consistently cited since they were published. These thirty papers received a total of about 3,500 citations from within AAN (average = 115 citation/paper, Min = 30, and Max =338). These citations come from 1,493 unique papers.

As described above, one of the goals of this project was to study the interplay of citation purpose and polarity. Therefore, we asked our annotators to mark each sentence for both purpose and polarity. The citation polarity labels were positive, negative, and neutral. These labels are defined in a slightly different way than their usual sense. A citation is marked positive if it either explicitly states a strength of the target paper or indicates that the work done in the target paper has been used either by the author or a third-party. It is also marked as positive if it is compared to another paper (possibly by the same authors) and deemed better in some way. A citation is marked negative if it explicitly points to a weakness of the target paper. It is also marked as negative if it is compared to another paper and deemed worse in some way (our criteria for marking negative citations is equivalent to Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006)). A citation is marked as neutral if it is only descriptive.

For annotating citation purpose, we created a taxonomy consisting of six categories based on our study of similar taxonomies proposed in previous work (Spiegel-Rösing Reference Spiegel-Rösing1977; Teufel et al. Reference Teufel, Siddharthan and Tidhar2006). We selected the categories that we believe are more important and useful from a bibliometric point of view, and the ones that can be detected through citation text analysis. We also tried to limit the number of categories by grouping similar categories proposed in previous work under one category. The six categories, their descriptions, and an example for each category are listed in Table 1.

Table 1.

Annotation scheme for citation purpose, motivated by the work of Spiegel-Rösing (Reference Spiegel-Rösing1977) and Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006)

We asked graduate students with good background in NLP to provide two annotations for each citation example. For the first annotation, we asked them to determine the purpose of citing the target reference by choosing from the six purpose categories that we described earlier. For the second annotation, we asked them to determine whether the citation is negative, positive, or neutral. As mentioned above, asking annotators to do both of these allows us to study the interrelationship between citation purpose and polarity.

The complete annotated dataset is available for download at http://clair.si.umich.edu/corpora/citation_sentiment_umich.tar.gz.

To estimate the inter-annotator agreement, we randomly picked 400 sentences and assigned them to an additional annotator who did not originally annotate the sentences. We used the Kappa coefficient (Cohen Reference Cohen1968) to measure the agreement. Since both purpose and polarity annotations are categorical, Kappa coefficient can be directly used. The agreements on the purpose and the polarity classification task were K = 0.61 and K = 0.66 respectively, which indicates substantial agreement on the (Landis and Koch Reference Landis and Koch1977) scale. These additional annotations produced for inter-annotator agreement are not part of the dataset.

2.1 Data analysis

The distribution of the purpose categories in the data was: 14.7 per cent criticism, 8.5 per cent comparison, 17.7 per cent use, 7 per cent substantiation, 5 per cent basis, and 47 per cent other. The distribution of the polarity categories was: 30 per cent positive, 12 per cent negative, and 58 per cent neutral. The amount of negative citations we found in our data is on the higher side, but within the range reported by previous work; (White Reference White2004) reports that the previous literature has found the proportion of negative citations to be between 1 per cent and 14 per cent, with the highest number being reported in Moravcsik and Murugesan (Reference Moravcsik and Murugesan1975).

Next, we study the correlation between the counts of the different purpose and polarity categories as well as total number of citations. This is shown in Table 2. Here we see the distribution of each citation purpose category across different polarity categories. Some of these correlations are expected. Criticizing sentences are usually interpreted as negative in nature, substantiating sentences are interpreted as positive, and neutral in both purpose and polarity correlate well. It is interesting to see that comparison usually results in a neutral citation polarity, indicating the lack of negative results in the scientific community. Similarly, basis and use usually lead to a positive polarity, which agrees with previous work. Finally, citations that are labeled with polarity label negative tend to co-occur exclusively with the purpose labels criticizing and comparison, which provides an experimental validation of the heuristics previously used by Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006) to derive polarity labels automatically from purpose labels.

Table 2.

Distribution of the citations belonging to different citation purpose categories across polarity categories

We also computed the Pearson correlation coefficient between the counts of citations from the different categories that a paper received per year since its publication. We found that, on average, the correlation between positive and negative citations is negative (AVG P = − 0.194) and that the correlation between the count of positive citations and the total number of citations is much higher than the correlation between negative citations and total citations (AVG P = 0.531 for positive versus AVG P = 0.054 for negative).

Similarly, we noticed that there is a higher positive correlation between Use citations and total citations than in the case of both Substantiation and Basis. One possible explanation for this is that publications that present new algorithms, tools, or corpora that are used by the research community become popular with time and thus receive more and more citations. An experiment to confirm this would be to label the citing sentences for the most cited papers in our corpus with citation purpose and measure which purpose categories dominate the citing sentences to these papers. We will do this as part of future work.

2.2 Automatic classification

We now turn to the problem of automatically labeling citation purpose and polarity. We provide a summary of our methods and experiments below, more details can be found in Abu-Jbara, Ezra and Radev (Reference Abu-Jbara, Ezra and Radev2013).

2.2.1 Methodology

We use a supervised approach whereby a classification model is trained on a number of lexical and structural features extracted from a set of labeled citation contexts. Some of the features that we use to train the classifier are listed in Table 3.

Table 3.

Some of the features used for citation purpose and polarity classification

We experimented with several classifiers including: Support Vector Machines (SVM), Logistic Regression (LR), and Naive Bayes. Our initial experimentation showed that SVM outperformed Logistic Regression and Naive Bayes. Therefore, we report the results of ten-fold cross validation using SVM classifier on our data. All the results have been tested for statistical significance using a two-tailed paired t-test.

For polarity classification, due to the high skewness in the data (more than half of the citations are neutral), we use two setups for binary classification. In the first setup, the citation is classified as Polarized (subjective) or neutral (objective). In the second one, subjective citations are classified as positive or negative. We find that this method gives more intuitive results than using a three-way classifier.

2.2.2 Citation purpose classification evaluation results

Table 4 shows the precision, recall, and F1 for each of the six categories. It also shows the overall accuracy and the Macro-F measure.

Table 4.

Summary of Citation Purpose Classification Results (10-fold cross validation, SVM: Linear Kernel, c = 1.0). Substantiating is abbreviated as Subst

The chi-squared evaluation of the features shows that both lexical and structural features are important. It also shows that among lexical features, the ones that are limited to the existence of a direct relation to the target reference (such as closest verb, adjective, adverb, subjective cue, etc.) are the most useful. This can be explained by the fact that restricting the features to having direct dependency relations introduces much less noise than other features (such as Dependency Triplets). Among the structural features, the count of references in the citation context was found to be most useful.

2.2.3 Citation polarity classification evaluation results

Table 5 shows the precision, recall, and F1 values for both the subjectivity classifier (that classifies citation into either subjective or objective) and the polarity classifier (that classifies citation into either positive or negative) along with the accuracy for each classifier. Overall numbers for each of the three categories (positive, negative, and neutral) can be found in Abu-Jbara et al. (Reference Abu-Jbara, Ezra and Radev2013).

Table 5.

Summary of citation polarity classification results (10-fold cross validation, SVM: Linear Kernel, c = 1.0)

The analysis of the features used to train this classifier using chi-squared analysis leads to the same conclusions about the relative importance of the features as described in the previous subsection. However, we noticed that features that are related to subjectivity (Subjectivity Cues, Negation, Speculation) are ranked higher which makes sense in the case of polarity classification.

2.3 Applications

2.3.1 Measuring research dynamics

Figure 2 shows the results of running our purpose classifier on all the citations to Papineni et al. (Reference Papineni, Roukos, Ward and Zhu2002)’s paper about BLEU, an automatic metric for evaluating Machine Translation (MT) systems. The figure shows that this paper receives a high number of Use citations. This makes intuitive sense because this paper describes an evaluation metric that has been widely used in the MT area. This also demonstrates that the citation purpose profiles of research papers can be used to do a more graded citation analysis which allows us to measure not just the amount of impact a paper is having, but also the nature of its impact. The figure also shows that in the recent years, this metric has started to receive some criticizing citations that has resulted in a slight decrease in the number of use citations. Such a temporal analysis of citation purpose and polarity is useful for studying the dynamics of research. It can also be used to detect the emergence or de-emergence of research techniques.

Fig. 2.

Change in the purpose of the citations to Papineni et al. (Reference Papineni, Roukos, Ward and Zhu2002). The values for the categories Substantiating and Basis for this paper are very close together and therefore, their lines are highly overlapping in the graph.

Similarly, Figure 3 shows that Church (Reference Church1988) received significant positive feedback during the 1990s and until early 2000s before it started to receive more negative feedback. This can be explained by the emergence of better statistical models for part-of-speech (POS) tagging (e.g. Conditional Random Fields, CRF; Lafferty, McCallum and Pereira Reference Lafferty, McCallum and Pereira2001) that outperformed Church’s approach. However, as indicated by the neutral citation curve, Church’s work continued to be cited as a classical pioneering research on the POS tagging task, but not as the state-of-the-art approach.

Fig. 3.

Polarity for Church (Reference Church1988).

2.3.2 Faceted summarization

Citation-based summarization is based on the idea of using citing sentences to a target paper to generate a community-driven summary. Previous work has shown that such summaries can be more informative than the abstract of the target paper, because they take into account the response of the community to the target paper after the publication of the paper (Qazvinian and Radev Reference Qazvinian and Radev2008; Qazvinian, Radev and Özgür Reference Qazvinian, Radev and Özgür2010). Existing citation-based summarization methods do not take into account the attitude of citing sentences however, which can result in biased summaries. For example, any shortcomings of a generally positively cited paper are unlikely to appear in a summary, because the summarization algorithms depend on redundancy in the input to determine important facts.

The ability to assign polarities to citing sentences opens up new possibilities for more factored citation-based summarization. Citation-based summaries can be organized around the citation polarity or citation purpose, leading to faceted summaries. Figure 4 illustrates this idea by showing some sample sentences from a possible faceted summary for Magerman (Reference Magerman1995), an early NLP paper about syntactic parsing. This summary was obtained by first classifying citing sentences to the paper into positive, negative, and neutral citations and then summarizing the positive and negative sentences separately. Given the set of input sentences for each category, each sentence was assigned a relevance score using Lexrank (Erkan and Radev Reference Erkan and Radev2004). Lexrank is a network-based content selection algorithm that works by first building a graph of all the document sentences in a cluster. The edges between corresponding nodes represent the cosine similarity between them. Once the network is built, the algorithm computes the salience of sentences in this graph based on their eigenvector centrality in the network. We hypothesize that such faceted summaries might provide a better overview of the strengths and weaknesses of a target paper compared to traditional citation-based summaries.

Fig. 4.

Sample sentences from faceted citation-based summary for the paper Magerman (Reference Magerman1995).

The idea of faceted summaries for scholarly data links traditional scientific summarization with the highly active area of opinion summarization. However, more exploration is needed to create appropriate datasets and evaluation metrics for this task. This is because for evaluating faceted scientific summaries, we must come up with datasets and evaluation metrics that allow us to measure the coverage of facts in the summaries (achieved by traditional summarization evaluation metrics such as pyramid evaluation (Nenkova and Passonneau Reference Nenkova and Passonneau2004)) as well as a balanced presentation of opinion (which is what datasets and metrics for opinion summarization tend to focus on Kim and Zhai (Reference Kim and Zhai2009)). This would require creation of new datasets and evaluation metrics; we are currently pursuing research in this direction.

3 Citation context identification

Citing papers contain some explicit information about the paper being cited. The following example is an excerpt from a paper that contains information about Eisner’s work on bottom-up parsers and the notion of span in parsing:

• Another use of bottom-up is due to Eisner (1996), who introduced the notion of a span.

However, the citation to a paper may not always include explicit information about the cited paper:

• This approach is one of those described in Eisner (1996)

Although this sentence by itself does not provide any information about the cited paper, it suggests that its surrounding sentences describe the proposed approach in Eisner’s paper:

We refer to such sentences that contain information about a specific secondary source but do not explicitly cite it as context sentences.

To build a corpus for studying such context sentences, we picked ten recently published papers from various areas in NLP and annotated them. Table 6 lists these papers together with their authors, publication year, number of references within AAN, and the number of sentences.

Table 6.

Papers chosen from AAN as source papers for the evaluation corpus, together with their authors, publication year, number of references (in AAN) and number of sentences. Papers marked with * are used to calculate annotation inter-judge agreement

Each annotation instance in our setting corresponds to a paper-reference pair, and is a vector in which each dimension corresponds to a sentence and is marked with a C if it explicitly cites the reference, and with a one if it implicitly talks about it. All other sentences are marked with zeros. Table 7 shows a portion of two separate annotation instances of N03-1003 corresponding to two of its references. Our annotation has resulted in 203 annotation instances, each corresponding to one paper–reference pair. The complete annotation dataset is available for download at http://www-personal.umich.edu/~vahed/context-ext.html.

Table 7.

Part of the annotation for N03-1003 with respect to two of its references ‘Lin and Pantel (2001)’ (the first column) ‘Shinyama et al. (2002)’ (the second column). Cs indicate explicit citations, 1s indicate implicit citations and 0s are none

We also asked a neutral annotator to annotate two of our datasets that are marked with * in Table 6. These additional annotations are used for measuring inter-annotator agreement and are not part of the dataset. For each paper–reference pair, the annotator was provided with a vector in which explicit citations were already marked with Cs. The annotation guidelines instructed the annotator to look at each explicit citation sentence, read up to fifteen sentences before and after, and mark context sentences around that sentence with 1s. Next, the twenty-nine annotation instances done by the external annotator were compared with the corresponding annotations that we did, and the Kappa coefficient (κ) was calculated. To calculate κ, we ignored all explicit citations (since they were provided to the external annotator) and used the binary categories (i.e. one for context sentences, and zero otherwise) for all other sentences. Table 8 shows the annotation vector size (i.e. number of sentences), number of annotation instances (i.e. number of references), and average κ for each set. The average κ is above 0.85 in both cases, suggesting that the annotation process has a low degree of subjectivity and can be considered reliable.

Table 8.

Average κ coefficient as inter-judge agreement for annotations of two sets

3.1 Data analysis

First, we look at some statistics about the number of explicit citations a reference receives in a paper. Figure 5 shows a histogram corresponding to this distribution, which is calculated over all the references in the ten papers shown in Table 6.Footnote ¹ It indicates that the majority of references get cited in only one sentence in a scientific article, while the maximum being nine in our collected dataset with only one instance (i.e. there is only one reference that gets cited nine times in a paper). Moreover, the data exhibits a highly positive-skewed distribution. This is illustrated on a log-log scale in Figure 6. This highly skewed distribution indicates that the majority of references get cited only once in a citing paper. The very small number of citing sentences cannot make a full inventory of the contributions of the cited paper, and therefore, extracting explicit citations alone without context sentences may result in information loss about the contributions of the cited paper.

Fig. 5.

Histogram of the number of different citations to all references (regardless of whether in AAN or not) found in the ten papers in our dataset.

Fig. 6.

The distribution observed for the number of different citations on a log-log scale.

Next, we investigate the distance between the context sentences and the closest citations. For each context sentence, we find its distance to the closest context sentence or explicit citation. Formally, we define the gap to be the number of sentences between a context sentence (marked with one) and the closest context sentence or explicit citation (marked with either C or one) to it. For example, the second column of Table 7 shows that there is a gap of size one in the ninth sentence in the set of context and citation sentences about both Shinyama et al. (2002) and Lin and Pantel (2001). Table 9 shows the distribution of gap sizes in the annotated data. This distribution suggests that the majority of context sentences directly occur after or before a citation or another context sentence. However, it shows that gaps between sentences describing a cited paper actually exist, and a proposed method should have the capability to capture them.

Table 9.

The distribution of gaps in the annotated data

3.2 Automatic classification

In this section, we sketch our methodology for automatically identifying the context sentences of a cited paper and summarize the experimental results. For more details, we refer the readers to Qazvinian and Radev (Reference Qazvinian and Radev2010).

3.2.1 Methodology

The first step in building an automated system for identifying context sentences is creating informative features. We can use the structure of the problem to find such features. For example, since we are trying to find context sentences citing a specific target paper, sentences that have a high similarity to the target paper are likely to be citing sentences even if they donot have an explicit citation marker. Additionally, the fact that these sentences are all part of one discourse imposes certain constraints on their distribution in the text. The likelihood of a sentence being a context sentence depends upon how far it is from an explicit citing sentence for the target-cited paper: sentences closer to explicit sentences are more likely to be context sentences than those that are farther away. Thus, there are two classes of features that can help us determine context sentences: features inherent to a sentence itself, and features based on the relationship of a sentence with its neighboring sentences.

Markov Random Fields (MRFs) provide a natural way to model these two kinds of dependencies. An MRF is an undirected probabilistic graphical model where the predicted label of a node depends on both its intrinsic features as well as features based on its association with its neighbors. The intrinsic features that help us determine the label for each sentence can be modeled using the node potential function of the MRF, while the association between sentences can be modeled using the compatibility function.

We use three features in our node potential function. The first feature is based on whether the sentence is an explicit citing sentence. The second feature is based on whether the sentence matches two lexical patterns. The first pattern is a bigram pattern where the first term matches any of ‘this; that; those; these; his; her; their; such; previous’ and the second term matches any of ‘work; approach; system; method; technique; result; example’. The second pattern matches any sentence that starts with ‘this; such’. These patterns provide a heuristic way of identifying coreference between adjacent sentences. In future work, we hope to use standard coreference detection methods to model this more directly, as well as use other forms of discourse dependencies (for example, using explicit and implicit discourse markers from the Penn Discourse Treebank (Prasad et al. Reference Prasad, Dinesh, Lee, Miltsakaki, Robaldo, Joshi and Webber2008)). The third feature is based on the lexical similarity of the sentence with the target cited paper, where the similarity is computed between the sentence and the full text of the cited paper.Footnote ² The node potential function is then computed using an unweighted linear combination of these three features.

The dependencies between different sentences can be modeled in several ways depending on how we structure the dependency graph of the MRF. We can make a sentence depend on all the other sentences in the paper or we could make it depend on only a small window of sentences around it. Figure 7 shows the structure of the two MRFs at either extreme of the local dependency assumption. In Figure 7(a), each sentence depends on only its following and preceding sentences, while in Figure 7(b) each sentence depends on all the others. We refer to the former by MRF₁ and to the latter by MRF_n . Generally, we use MRF_i to denote a MRF in which each sentence is connected to i sentences before and after it. In the MRF framework, the dependency between different nodes is modeled using the compatibility function. We define this compatibility function in such a way that a sentence that is not labeled as a context sentence does not affect the labels of its neighbors in the graph. A sentence that is labeled as a context sentence affects its neighbors based on its lexical similarity with them. More details about the exact formulation of these MRF functions can be found in Qazvinian and Radev (Reference Qazvinian and Radev2010). We experiment with three different structures of MRF: MRF ₁, MRF ₄, and MRF_n .

Fig. 7.

The structure of the MRF constructed based on the dependency of non-adjacent sentences; (a) left, each sentence is independent on all other sentences given its immediate neighbors. (b) right, sentences have dependency relationship with each other regardless of their position.

We compare our MRF-based method with three baselines. The first baseline, B_IR , is an information retrieval-driven method. Given a paper-reference pair, for each explicit citation sentence, B_IR picks its preceding and following sentences if their similarities to that sentence are greater than a cutoff (the median of all such similarities), and repeats this for neighboring sentences of newly marked sentences. Intuitively, B_IR tries to find the best chain (window) around citing sentences.

As the second baseline, we use the hand-crafted discourse-based features used in MRF’s potential function. This baseline, B_DISC , marks as context those sentences that are within a particular distance (four in our experiments) of an explicit citation and match one of the discourse patterns mentioned earlier. After marking all such sentences, B_DISC also marks all sentences between them and the closest explicit citation no farther than four sentences away as context sentences.

Finally, as a third baseline, B_SVM , we train a SVM classifier to label sentences as context/non-context. We use four features to train the SVM model. These four features comprise the three sentence level features used in MRF’s potential function (similarity to reference, explicit citation, matching regular-expression) and a network level feature: distance to the closest explicit citation.

3.2.2 Evaluation results

The evaluation of our methodology means to directly compare the output of our method with the gold standards obtained from the annotated data. Our methodology finds the sentences that cite a reference implicitly. Therefore, the output of the inference method is a vector, υ, of 1’s, and 0’s, whereby a 1 at element i means that sentence i in the source document is a context sentence about the reference while a 0 means an explicit citation or neither. The gold standard for each paper-reference pair, ω, is also a vector of the same format and dimensionality.

Precision, recall, and F _β for this task can be defined as

(1) $$\begin{eqnarray} p = \frac{\upsilon \cdot \omega }{\upsilon \cdot \textbf{1}}; \quad r = \frac{\upsilon \cdot \omega }{\omega \cdot \textbf{1}}; \quad F_\beta = \frac{(1+\beta ^2)p \cdot r}{\beta ^2 p +r} \end{eqnarray}$$

where 1 is a vector of 1’s with the same dimensionality and β is a non-negative real number.

Table 10 shows the results of our experiments. We report F-score with β as three because higher values of β favor recall over precision. For this task, we are more interested in recall (e.g. if the citing sentence has five context sentences, we would want to get all of them, as opposed to getting only one and maximizing precision).

Table 10.

Average F _{β = 3} for similarity-based baseline (B_IR ), discourse-based baseline (B_DISC ), a supervised method (B_SVM ) and three MRF-based methods

The best performing method is MRF₄ . Of the baseline methods, B_IR seems to do the best. This suggests that similarity to the explicit or implicit citing sentences is the strongest cue for finding context sentences. B_DISC does not do well on its own, but does improve the performance when combined with similarity-based features in the MRF-based classifiers.

B_IR does much better than B_SVM as well. In addition to sentence level features, B_SVM has a feature that computes distance to the closest explicit citing sentence, but this feature is non-iterative and does not take into account lexical similarity. B_IR , on the other hand, works iteratively by adding additional sentences similar to the sentences already labeled as context sentences. It is difficult to create a feature that captures this iterative dependency in standard-supervised classifiers such as SVM. However, sequence-based classifiers such as MRFs are designed to model exactly this sort of dependency, and thus, it is not surprising that MRF₁ and MRF₄ do much better than all the other methods. These models are able to effectively capture sentence level features as well as the dependency between adjacent sentences.

Among the MRF-based methods, MRF₄ does better than MRF₁ which shows that sentences depend on more than one sentence on each side. However, the significantly worse performance of MRF_n suggests that dependencies on far away sentences do not improve results and in fact lead to a drop in performance, probably by leading to more false positives where sentences very far away from explicit citing sentences (and hence unlikely to be in their context) might be added as a context sentence due to an arbitrary similarity match.

3.3 Applications

3.3.1 Survey generation

Previous work in scientific summarization has shown the importance of citations in scientific domains and has indicated that citations include survey-worthy information (Teufel Reference Teufel, Shanahan, Qu and Wiebe2006; Elkiss et al. Reference Elkiss, Shen, Fader, Erkan, States and Radev2008; Qazvinian and Radev Reference Qazvinian and Radev2008; Mohammad et al. Reference Mohammad, Dorr, Egan, Hassan, Muthukrishan, Qazvinian, Radev and Zajic2009). Here we show how context sentences add useful information to such summaries in addition to the information present in the explicit citations.

We use the data from Mohammad et al. (Reference Mohammad, Dorr, Egan, Hassan, Muthukrishan, Qazvinian, Radev and Zajic2009) that contains two sets of cited papers and corresponding citing sentences, one on question answering (QA) with ten papers and the other on dependency parsing (DP) with sixteen papers. For comparing different methods, we use pyramid evaluation (Nenkova and Passonneau Reference Nenkova and Passonneau2004) using nuggets extracted manually from the citing sentences to these papers.

A nugget is defined as a specific fact that is found in the input sentences. These are found by using human generated regular expressions representing the facts that can then be matched directly to the text. Each nugget is assigned a weight based on the importance of the nugget, where the importance of a nugget is computed based on its frequency in the input text. Summaries that contain more highly weighted nuggets get better scores in the pyramid evaluation.

In pyramid evaluation, the nuggets are organized in a pyramid of order n. The top tier in this pyramid contains the highest weighted nuggets, the next tier contains the second highest weighted nuggets, and so on. The score assigned to a summary is the ratio of the sum of the weights of the nuggets it contains to the sum of weights of an optimal summary with the same number of nuggets. Pyramid evaluation allows us to capture how a method performs in terms of selecting sentences with the most highly weighted nuggets.

Both sets (QA and DP) contain nuggets extracted by experts from citation sentences. We use these nugget sets, which are provided in the form of regular expressions, to evaluate automatically generated summaries. To perform this experiment, we needed to build a new corpus that includes context sentences. For each citation sentence, we use our automatic context extraction method (described in Section 3.2.1) to extract the context sentences around it. Here, we limit the context size to be four on each side. That is, we attach to a citing sentence any of its four preceding and following sentences that our method marks as context sentences. Therefore, we build a new corpus in which each explicit citation sentence is replaced with the same sentence attached to at most four sentence on each side. After building the context corpus, we use LexRank (Erkan and Radev Reference Erkan and Radev2004) to generate two QA and two DP surveys using the citation sentences only, and the new context corpus explained above.

We limit these surveys to be of a maximum length of 1,000 words. Figure 8 shows a portion of the survey generated from the QA context corpus. This example shows how context sentences add meaningful and survey-worthy information along with citation sentences. Table 11 shows the Pyramid F _{β = 3} score of automatic surveys of QA and DP data. Both surveys are evaluated using nuggets drawn from citation texts. In all evaluation instances, the surveys generated with the context corpora excel at covering nuggets.

Fig. 8.

A portion of the QA survey generated by LexRank using the context information. Bolded sentences are context sentences.

Table 11.

Pyramid F _{β = 3} scores of automatic surveys of QA and DP data. The surveys are evaluated using nuggets drawn from citation texts (CT)

We notice that adding context sentences tends to improve results for the topic of QA much more than DP. In general, the topic of DP seems to be more challenging for citation-based survey generation methods than the topic of QA. In earlier experiments (Qazvinian et al. Reference Qazvinian, Radev, Mohammad, Dorr, Zajic, Whidby and Moon2013), we also found that the average pyramid score obtained by summaries generated by four human evaluators was 0.599 for QA while it was 0.413 for DP. Thus, a simple explanation for the difference in results between the two topics is that the factoid distribution in citing sentences (and hence context sentences) for QA is better compared to DP, leading to more incremental improvement as additional context information is added. For a better understanding of what topics might be more suited for such methods, a more detailed evaluation with a larger set of topics is needed. This will be part of our future work. Based on our understanding of the problem, this might have something to do with the structure of the lexical similarity networks generated for the different topics, which in turn is related to the distribution of different subtopics within these larger topics.

3.3.2 Polarity classification

To study the impact of using citation context in addition to the citing sentence on classification performance, we ran an additional polarity classification experiment. In the first polarity experiment (described in Section 2.2.3), we had only used the explicit citing sentences to extract the features that are used to train the classifiers.

For the second polarity experiment, we experimented with context information annotated by humans. For each of the citing sentences in our polarity experiments, we asked the human annotators to look at a window of four sentences around the explicit citation sentence and label the context sentences that are relevant to the target reference. We then extracted features from this gold context and ran a second round of classification.

Table 12 shows the results of both experiments. The results show that adding citation context improves performance, as shown in the increased F-Scores for each of the three categories (positive, negative, and neutral). Adding context improves precision by 1–3 points for each of the categories. The improvement in recall is unevenly distributed: for positive and neutral categories, recall improves a little, while for the negative category, we see an improvement of about ten points for recall. This supports the intuition about polarized citations that authors start their review of the cited work with an objective (neutral) sentence and then follow it with their criticism if they have any. Thus, taking these context sentences into account allows us to correctly label several negative sentences that were labeled as positive or neutral without this information.

Table 12.

Summary of Citation Polarity Classification Results with context sentences. (ten-fold cross validation, SVM: Linear Kernel, c = 1.0). Results under column wo/ctx are the results with only explicit citing sentences and no context sentences, while the results under column w/ctx are the results with context sentences included

4 Reference scope identification

When a reference appears in a scientific article, it is usually accompanied by a span of text that highlights the important contributions of the cited article. For example, sentence (1) below is a citing sentence that cites a paper by Philip Resnik and describes the problem Resnik addressed in his paper.

(1) Resnik (1999) addressed the issue of language identification for finding Web pages in the languages of interest.

The sentence above contains only one reference, so there is no ambiguity about what reference is being talked about. However, sentences that contain references to multiple papers are very common in scientific writing. For example, sentence (2) below contains three references.

(2) Grefenstette and Nioche (2000) and Jones and Ghani (2000) use the web to generate corpora for languages where electronic resources are scarce, while Resnik (1999) describes a method for mining the web for bilingual texts.

In this sentence, the first fragment describes the contribution of Grefenstette and Nioche (2000) and Jones and Ghani (2000). The second fragment describes the contribution of Resnik (1999).

We use the term Reference Scope to refer to the fragments of a sentence that are relevant to a specific target paper. Two additional examples for reference scope are shown below, where sentences (4) and (5) are labeled for the target references Tetreault and Chodorow (2008), and Cutting et al. (1992) respectively. The underlined words are in the reference scope for the respective target papers.

(4) For example, Tetreault and Chodorow (2008) use a maximum entropy classifier to build a model of correct preposition usage, with 7 million instances in their training set, and Lee and Knutsson (2008) use memory-based learning, with 10 million sentences in their training set.

(5) There are many POS taggers developed using different techniques for many major languages such as transformation-based error-driven learning (Brill, 1995), decision trees (Black et al., 1992), Markov model (Cutting et al. , 1992), maximum entropy methods (Ratnaparkhi, 1996) etc for English.

Our goal is to build an automatic classifier for reference scope detection. For this, we first built a dataset for reference scope from papers in AAN in the following way. In the complete set of citing sentences in AAN, 56 per cent contain two or more references and 44per cent contain one reference only. From this set, we randomly selected 3,500 citing sentences, each containing at least two references. The total number of references in this set of sentences is 19,591. We then asked graduate students with good background in NLP to provide three annotations for each sentence in the data set described above. First, we asked them to determine whether each of the references in the sentence was correctly tagged or not. Second, we asked them to determine, for each reference, whether it is a syntactic constituent in the sentence or not. Third, we asked them to determine and label the scope of the reference in each sentence which was marked as a target reference. We designed a user-friendly tool to collect the annotations from the students. The dataset is available for download at http://clair.si.umich.edu/corpora/refscope_data.tar.gz.

To estimate the inter-annotator agreement, we picked 500 random sentences from our data set and assigned them to an additional annotator who didn’t originally annotate these sentences. We then measured inter-annotator agreement on the reference scope annotation task (not on the first two tasks of identifying correctly tagged references and identifying syntactic references). This is done by labeling each word as either one for inside or zero for outside the reference scope for each of the annotations and then computing the Kappa coefficient between the two annotations. The Kappa coefficient of agreement between the two sets of annotations on the scope identification task was K = 0.61. On Landis and Kochs scale (Landis and Koch Reference Landis and Koch1977), this value indicates substantial agreement. These additional annotations are not part of the dataset.

4.2 Automatic classification

We now provide a sketch of the methods we explored for automatically identifying the scope of a given reference within a citing sentence. More details about the methodology and pre-processing steps are available in Abu Jbara and Radev (Reference Jbara and Radev2012).

4.2.1 Methodology

Our general problem formulation is to label each word in the citing sentence as either belonging to the reference scope of a specific target reference or not. One of the most important clues about whether a word belongs to the reference scope is provided by the positional and syntactic relationship of the word with the target reference anchor. One approach would be to use a set of features for each word and learn a supervised classifier that can label each word as ‘inside’ or ‘outside’ the reference scope based on these features. The set of features that we used to train our classifier are listed in Table 13. This is our simplest approach, which we call the Word Classification approach.

Table 13.

The features used for word classification and sequence labeling

The Word Classification approach does not take into account the fact that the label of a word can depend on the labels of words around it. These relationships can be modeled using a sequence classifier; we use CRF to explore this. In CRF, the predicted label for a word depends on both the features for the word (which are the same as used in Word Classification) as well as the words around it. This second approach is our Sequence Labeling approach.

Both of the above mentioned approaches ignore the fact that words in a sentence are organized in a hierarchical syntactic structure. Thus, it might be useful to model the classification problem at units of higher granularity than words. One way to do this is to use a chunking tool to identify noun groups, verb groups, preposition groups, adjective groups, and adverb groups in the sentence. Each such group (or chunk) forms a segment. If a word does not belong to any chunk, it forms a singleton segment by itself. Labels are assigned by first using the Sequence Labeling approach to label the individual words in the sentence and then assigning to each segment the majority label of the words it contains. This third approach is our Segment Classification approach.

Finally, we also compare our systems with the baseline approach from Abu-Jbara and Radev (Reference Abu-Jbara and Radev2011). In this approach, we first parse the citing sentence using a link grammar parser and extract as reference scope the smallest subtree rooted at a sentence node that contains the target reference anchor. If the text corresponding to this subtree is not grammatical, the second smallest subtree rooted at a sentence node is tried and so on. We call this approach AR-2011.

4.2.2 Evaluation results

Table 14 reports the precision, recall, F1-score, and accuracy for all the methods in ten-fold cross validation. All the metrics were computed at the word level. All the methods outperform the baseline method AR-2011. We also notice that the CRF-based sequence labeling method performs significantly better than the word classification method, corroborating our intuition that the labels of neighboring words are dependent.

Table 14.

Results of scope identification using the different algorithms described in the paper

The segment classification method outperforms both word classification and sequence labeling methods, showing that syntactic structure provides useful information for the task. On the other hand, the baseline method AR-2011 that relies exclusively on syntax-driven heuristics performs worse than all other methods. Thus, the combination of a feature-driven model along with syntax-based heuristics provides the best approach for this task. This suggests that for tasks of this nature, while machine learning methods perform well, good hybrid algorithms that also employ appropriate syntactic heuristics derived from an understanding of the structure of the problem can improve results further.

4.3 Applications

4.3.1 Focused Summarization

Reference scope extraction provides another set of possibilities for improving citation summarization by building more focused summaries complementary to the improvement provided by faceted summaries which are more comprehensive but can still contain irrelevant information due to the presence of multi-reference sentences.

The simplest way in which reference scope extraction can be used to improve citation summaries is by using it as a post-processing step. Thus, while the input sentences for the salience model (e.g. Lexrank) are still full citation sentences as in previous work, once the model produces the set of most salient sentences, they can be run through a reference scope extraction module that can preserve only the sentence segments that are relevant to the target paper being summarized. Figure 9 illustrates how this might benefit summarization. The left column shows the first two sentences from the citation summaries generated for two different papers: Och and Ney (Reference Och and Ney2003) which introduced Giza++ and Yarowsky (Reference Yarowsky1995) which introduced bootstrapping for word sense disambiguation. The right column in the table shows the same sentences after applying reference scope extraction.

Fig. 9.

Top sentences from the summaries obtained for two papers along with their reference scopes. The token < > is used to separate non-adjacent fragments of the same reference scope.

It is easy to see how reference scope-based post-processing can help provide more focus to the summaries. For the summary in Och and Ney (Reference Och and Ney2003), it removes potentially irrelevant details about grow-diag-final-and method from the first sentence and about Moses and SRILM from the second sentence. Similarly, for Yarowsky (Reference Yarowsky1995), it removes fragments about supervised sense disambiguation from the first sentence and details of other tasks using bootstrapping algorithms from the second sentence. It is possible that the user might not want to completely remove these details from sentences, in which case reference scopes can be highlighted while keeping the full citation sentence to emphasize the text fragments relevant to the target paper being summarized.

A second possibility is to use reference scope extraction as a pre-processing step for summarization. In this method, reference scopes are extracted from input citing sentences and the salience model is applied on them. This might lead to better summaries because only the text fragments relevant to the target paper are used by the salience model to assign importance to sentences, leading to an improved signal versus noise ratio. The final summary can consist of only the most salient reference scopes or the full sentences that consist of the most salient reference scopes as scored by the salience model.

Our future work includes creating a dataset of reference scope annotations geared towards summarization so that we can experiment with each of these possibilities. The evaluation strategy for this would be a mixture of automatic evaluation to measure the information content of reference scope-based summaries and human evaluation to measure the readability of such summaries.

4.3.2 Citation sentiment analysis

Reference scope extraction can also help us do more accurate citation-based sentiment analysis. Sentence (1) below shows an examples of this case.

(1) Cohn and Lapata (2008) used the GHKM extraction method (Galley et al. 2004), which is limited to constituent phrases and thus produces a reasonably small set of syntactic rules.

If the target reference is Cohn and Lapata (2008), only the underlined segment should be used for feature extraction. The limitation stated in the second segment of sentence is referring to Galley et al., (2004). Similarly, consider citing sentence (2):

(2) Automatic text summarization approaches have offered reasonably well-performing approximations for identifying important sentences (Lin and Hovy, 2002; Schiffman et al., 2002; Erkan and Radev, 2004; Mihalcea and Tarau, 2004; Daume III and Marcu, 2006) but, not surprisingly, text (re)generation has been a major challenge despite some work on sub-sentential modification (Jing and McKeown, 2000; Knight and Marcu, 2000; Barzilay and McKeown, 2005).

Clearly the sentence expresses two different sentiments towards the first and second group of papers. A sentiment extraction system that uses this sentence for analyzing the impact of a specific paper, say Lin and Hovy (2002), may additionally aggregate attitudes and information about a completely different set of papers. Reference scope extraction can help us use only the segments of a citation sentence relevant to a specific target to determine the citation polarity and purpose for the sentence. This would lead to more accurate assessment of the attitude of the community towards the target paper. In future work, we plan to measure the effect of using reference scopes on citation sentiment analysis.

5 Related work

Our work is related to a large body of research on citations. Studying citation patterns and referencing practices has interested researchers for many years (Hodges Reference Hodges1972; Garfield Sher and Torpie Reference Garfield, Sher and Torpie1984). White (Reference White2004) provides a good survey of the different research directions that study or use citations.

Several research efforts have focused on studying the different purposes for citing a paper (Garfield Reference Garfield1964; Weinstock Reference Weinstock and Kent1971; Chubin and Moitra Reference Chubin and Moitra1975; Moravcsik and Murugesan Reference Moravcsik and Murugesan1975; Bonzi Reference Bonzi1982). Bonzi (Reference Bonzi1982) studied the characteristics of citing and cited works that may aid in determining the relatedness between them. Garfield (Reference Garfield1964) enumerated several reasons why authors cite other publications, including alerting researchers to forthcoming work, paying homage to the leading scholars in the area, and citations which provide pointers to background readings. Weinstock (Reference Weinstock and Kent1971) adopted the same scheme that Garfield proposed in her study of citations.

Spiegel-Rösing (Reference Spiegel-Rösing1977) proposed thirteen categories for citation purpose based on her analysis of the first four volumes of Science Studies. Some of them are: cited source is the specific point of departure for the research question investigated, cited source contains the concepts, definitions, interpretations used, and cited source contains the data used by the citing paper. Nanba and Okumura (Reference Nanba and Okumura1999) came up with a simple schema composed of only three categories: Basis, Comparison, and other Other. They proposed a rule-based method that uses a set of statistically selected cue words to determine the category of a citation. They achieved an F-score of 83per cent on a dataset of fifty citing sentences for these types. The high scores compared to our results on citation purpose classification can be attributed to a simpler task and limited data. It must be noted that their main focus was scientific paper summarization, which justifies the simpler schema. Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006) did a much larger evaluation on 548 citation sentence in their work on citation function classification. They adopted twelve categories from Spiegel–Rosing’s taxonomy and trained an SVM classifier to label each citing sentence with exactly one category. Further, they mapped the twelve categories to four top level categories namely: weakness, contrast (four categories), positive (six categories), and neutral. Their performance in terms of F-score is similar to ours (between 28per cent to 86per cent).

The polarity (or sentiment) of a citation has also been studied previously. Previous work showed that positive and negative citations are common, although negative citations might be expressed indirectly or in an implicit way (Ziman Reference Ziman1968; MacRoberts and MacRoberts Reference MacRoberts and MacRoberts1984; Thompson and Yiyun Reference Thompson and Yiyun1991). Athar (Reference Athar2011) addressed the problem of identifying sentiment in citing sentences. They used a set of structure-based features to train a machine learning classifier using annotated data with 8,736 manually annotated citations. Their SVM-based approach got a macro F-score of 76.4 per cent, which is higher than the macro F-score achieved by our system (62.1 per cent). This difference can be attributed to their use of effective sentence structure-based features.

Citation purpose and relevance has been used for doing scientometric analysis in a number of different fields. Li et al. (Reference Li, Chambers, Ding, Zhang and Meng2014) use citation motivation to study science linkage: a widely used patent bibliometric indicator to measure patent linkage to scientific research based on the frequency of citations to scientific papers within the patent. Liu et al. (Reference Liu, Chen, Ho and Li2014a) use citation relevance-based main path analysis for tracing main paths of legal opinions and show that relevancy information helps main path analysis uncover legal cases of higher importance. Cheang et al. (Reference Cheang, Chu, Li and Lim2014) use citation classification to do a multidimensional evaluation of thirty-nine selected management journals.

Other uses of citation purpose include a study by Bonzi and Snyder (Reference Bonzi and Snyder1991) to understand citation purpose in the context of self-citation in natural sciences. Wan and Liu (Reference Wan and Liu2014b) present a regression method for automatically estimating the strength value of each citation and show the estimated values can achieve good correlation with human-labeled values.

We now look at some work on linguistic analysis of citation text. Nakov, Schwartz and Hearst (Reference Nakov, Schwartz and Hearst2004) proposed the use of citation text as a tool for semantic interpretation of bioscience text and propose several applications. Ding et al. (Reference Ding, Zhang, Chambers, Song, Wang and Zhai2014) introduced the notion of Citation Content Analysis (CCA) and discussed the nature and purposes of CCA along with potential procedures to conduct CCA. Halevi and Moed (Reference Halevi and Moed2013) present a citation context analysis for the journal of infometrics. Zhao and Strotmann (Reference Zhao and Strotmann2014) also analyze the feasibility, benefits, and limitations of in-text author citation analysis and test how well it works compared with traditional author citation analysis using citation databases. Angrosh, Cranefield and Stanger (Reference Angrosh, Cranefield and Stanger2013) present a dataset for citation context sentences and present a model for citation context identification based on Conditional Random Fields.

One of the important uses of citation context is for scientific summarization. Nanba and Okumura (Reference Nanba and Okumura1999) use the term citing area to refer to the same concept as citation context. In Nanba, Kando and Okumura (Reference Nanba, Kando and Okumura2004), they use their algorithm to improve citation type classification and automatic survey generation. Kaplan, Iida and Tokunaga (Reference Kaplan, Iida and Tokunaga2009) present a method for identifying citation contexts based on coreference analysis and their use for research paper summarization. They report F-scores of about 69 per cent, which is higher than the numbers we obtained in our experiments (54 per cent). This can be attributed to their better modeling of coreference chains, which are modeled using a simple heuristics pattern-based matching method in our system. These two approaches are complementary and it is likely that better results can be obtained by combining our MRF-based approach with features based on coreference chains.

Citation context has also been used for the literature retrieval models for scientific domains. Liu et al. (Reference Liu, Chen, Ding, Wang, Xu and Lin2014b) designed a retrieval system for the PubMed Central database using citation contexts. Yin, Huang and Li (Reference Yin, Huang and Li2011) similarly used citation context for the task of literature retrieval in the biomedical domain.

Athar and Teufel (Reference Athar and Teufel2012b) observed that taking the context into consideration when judging sentiment in citations increases the number of negative citations by a factor of three. They also proposed two methods for utilizing the context. Their experiments surprisingly gave negative results and showed that classifying sentiment without considering the context achieves better results. They attributed this to the small size of their training data and to the noise that including the context text introduces to the data. However, in our experiments, we have seen that adding context leads to a large improvement in macro F-score (close to 12 per cent). This shows that even though doing context detection jointly with sentiment detection does not lead to better results (as found by them), context does provide useful information for sentiment detection (as found by us), and it is worthwhile to pursue this direction in future work. This is further supported by encouraging results presented in Athar and Teufel (Reference Athar and Teufel2012a), where the authors present a method for automatically identifying all the mentions of the cited paper in the citing paper. They show that considering all the mentions improves the performance of detecting sentiment in citations.