1 Introduction
Over the years, scientific articles have been the primary medium for disseminating scientific knowledge. Typically, the published scientific information builds upon the previous or existing scientific knowledge and citations play a key role in defining these inter-article relationships. One of the requirements for conducting effective research is the identification of these inter-article relations, which helps in avoiding duplication and also to position one's research with respect to previous research. In recent times, a dramatic increase in research output (Gaillard Reference Gaillard2008; Research4Life 2009) has resulted in difficulties in identifying these relations. For example, by using current information services, it is possible to identify the number of documents citing a given document; but it may not be possible to identify the reasons for which the citing documents have cited the given document. As Buckingham Shum et al. point out, present day information systems fail to answer some of the basic questions of critical inquiry such as providing information about documents that either support or challenge a given document or facilitate tracing the intellectual lineage for a given idea (Buckingham Shum et al. Reference Buckingham Shum, Uren, Li, Sereno and Mancini2007). An increasing need is felt for intelligent tools capable of providing insightful services that provide value beyond quantitative measures such as impact factors and citation counts. However, the intricacies of language and the unstructured format of research articles present challenges in providing such services. Identifying contexts associated with sentences can help in designing information tools for the research community that identify inter-article relationships. For example, it would be easier to identify the gaps in the works cited in an article.
Against this background, this research is taken up to investigate the possibility of identifying context types associated with sentences and using this information for providing citation context based information services for the research community. The task of designing citation context based information services involves a series of tasks. We describe in this paper the following steps followed in our study for designing such services.
(1) Defining context types for sentences in academic articles
(2) Carrying out supervised learning methods for achieving automatic context identification
(3) Creating structured information from unstructured text
(4) Developing an application that uses structured information for providing citation context based information services
The structure of this paper is as follows. In Section 2, we identify the related work in the field of scientific discourse annotation and present our rationale for defining a new set of context type definitions and the use of sequential classifiers. In Section 3, we present our scheme of thirteen context type definitions for sentences in scientific articles and explain the procedure adopted for deriving these definitions. We also compare the definitions with similar studies in this section. In Section 4, we briefly explain our information extraction system developed for text preparation tasks. We explain, in Section 5, our experiments carried out with conditional random fields (CRFs) for achieving automatic context identification. In Section 6, we describe the different citation context based information services provided by the Semantic Web application developed in our study. In Section 8, we present an analysis of the features used in machine-learning experiments and compare the classifier results with results of similar studies. We also discuss the distinct features provided by our application. Finally, we conclude this paper by providing pointers to future work.
2 Related work
Studies on sentence classification in scientific articles have focused on identifying the functions associated with sentences and the rhetorical relations between them. Some studies have considered the full text of an article and others have limited their focus either to sentences with citations or abstracts of an article. We describe in the following sections different studies in these areas.
2.1 Annotation of full-text documents
Over the years, several studies have investigated the task of annotating scientific texts. Teufel (Reference Teufel1999) focused on analyzing scientific text with what she called ‘Argumentative Zoning’ and proposed a classification scheme of seven categories (argumentative zones) for sentences in scientific articles, based on the argumentation involved. Mizuta and Collier (Reference Mizuta and Collier2004a) extended Teufel's (Reference Teufel1999) scheme for carrying out zone analysis in biology texts. Mizuta and Collier (Reference Mizuta and Collier2004b) further provided theoretical and practical support for their zone analysis. Langer, Lüngen, and Bayerl (Reference Langer, Lüngen and Bayerl2004) noted that newer applications in areas such as the Semantic Web required richer and more fine-grained annotation of documents and presented a scheme involving sixteen topic types. Considering the scheme of Mizuta and Collier (Reference Mizuta and Collier2004a) as complex, Wilbur, Rzhetsky, and Shatkay (Reference Wilbur, Rzhetsky and Shatkay2006) devised an annotation scheme of five categories for biomedical texts focusing on the semantic content of the sentence and presented guidelines for annotation of text without using the semantic content of the sentence.
Shatkay et al. (Reference Shatkay, Pan, Rzhetsky and Wilbur2008) used the scheme of Wilbur et al. (Reference Wilbur, Rzhetsky and Shatkay2006) and conducted machine-learning experiments for multidimensional classification of biomedical texts. Liakata et al. (Reference Liakata, Teufel, Siddharthan and Batchelor2010) presented and applied two complementary annotation schemes for scientific papers in Chemistry: the Core Scientific Concepts (CoreSC) annotation scheme based on the CISP (the proposed Core Information about Scientific Papers) metadata (Soldatova and Liakata Reference Soldatova and Liakata2007) and Argumentative Zoning-II scheme (AZ-II) (Teufel Reference Teufel1999). They concluded with a recommendation for merging the two schemes. Liakata (Reference Liakata2010) further presented results of using the CoreSC annotation scheme for identifying negative and speculative statements in scientific papers.
2.2 Annotation of sentences with citations
Parallel to this line of investigation, there have been studies that have specifically examined sentences with citations in scientific articles. Two different streams of research can be identified in these studies, based on the approach involved: manual citation classification and automatic citation classification.
2.2.1 Manual citation classification
Baldi (Reference Baldi1998) identified that ten different classification schemes were proposed between the years 1965 and 1980 and gently mocked this line of investigation as a cottage industry. These studies included those of Garfield (Reference Garfield and Stevens1965); Lipetz (Reference Lipetz1965); Hodges, (Reference Hodges1972); Moravcsik and Murugesan (Reference Moravcsik and Murugesan1975); Chubin and Moitra (Reference Chubin and Moitra1975); Spiegel-Rosing (Reference Spiegel-Rosing1977); Lindsey and Lindsey, (Reference Lindsey and Lindsey1978); Oppenheim and Renn, (Reference Oppenheim and Renn1978); Finney, (Reference Finney1979); and Frost (Reference Frost1979). The categories defined in these schemes characterized various reasons for citations and the methodology for classification involved manual examination of the passage with citations, also known as ‘citation context’ using human judgment. Small (Reference Small, Dervin and Voigt1982) called this line of investigation ‘citation context analysis’. In recent times, the focus has shifted to directly learning from citing authors' motives for citations through surveys and interviews (Brooks Reference Brooks1985, Reference Brooks1986; Prabha Reference Prabha1986; Shadish et al. Reference Shadish, Tolliver, Gray and Sen Gupta1995; Case and Higgins Reference Case and Higgins2000). White (Reference White2004) calls this stream of research ‘citer motivations’.
2.2.2 Automatic citation classification
Studies have also focused on the automation of citation classification in scientific documents. These studies have either proposed a new or a modified scheme of citation classification in order to suit the automation process. Garzone (Reference Garzone1997) developed an automated citation classifier with several components for carrying out citation classification. Nanba and Okumura (Reference Nanba, Okumura and Dean1999) proposed a simplified classification scheme of three categories based on the fifteen reasons identified by Garfield (Reference Garfield and Stevens1965) and developed a semi-automatic citation classification system for achieving multi-paper summarization.
Garzone and Mercer (Reference Garzone, Mercer, Hamilton and Yang2000) claim to have developed the first fully automatic citation classifier system. This extracted and processed only sentences with citations and used a pragmatic grammar for classification. Nanba, Kando, and Okumura (Reference Nanba, Kando and Okumura2000) used citation classification for achieving automatic generation of a review article. Pham and Hoffmann (Reference Pham, Hoffmann, Gedeon and Fung2003) developed KAFTAN, the “Knowledge Acquisition Framework for TAsks in Natural language”, which rapidly acquired cue phrases for classifying citations into four citation types. The authors compared the results of their classifier with the results of Nanba et al. (Reference Nanba, Kando and Okumura2000) and showed that their approach performed better.
Teufel, Siddharthan, and Tidhar (Reference Teufel, Siddharthan and Tidhar2006) presented an annotation scheme for citations involving twelve categories, based on the categories proposed by Spiegel-Rosing (Reference Spiegel-Rosing1977), and used supervised learning experiments for automatically classifying citation sentences. Le, Ho, and Nakamori (Reference Le, Ho and Nakamori2006) used graphical probabilistic models – Hidden Markov Models (HMMs) and Maximum Entropy Markov Models (MEMMs) – for extracting citation types from scientific articles. Radoulov (Reference Radoulov2008) carried out a study exploring automatic citation classification and redesigned Garzone's scheme of thirty-five categories from a perspective of usability and usefulness.
2.3 Motivation and rationale
Our work draws motivation from both types of studies discussed above. The objective of our study is to provide information services for the academic community using citation contexts, which requires a typology that describes scientific discourse along with different citation functions and a mechanism for identifying these functions. We explain in the following sections the rationale for defining a new set of context type definitions for sentences and also the reason for using sequential classification models for classification.
2.3.1 Defining new set of context type definitions – the rationale
Though there are various annotation schemes readily available, it is not easy to use them as they are highly subjective. Baldi (Reference Baldi1998) observed that citation classification typologies developed during 1960–1980 were designed in an ad hoc manner and virtually isolated from one another. White (2006) described the earlier classification schemes as idiosyncratic and difficult to code, resulting in difficulties in reusing them across disciplines. However, the importance of these classification systems cannot be ignored. Though critical, Baldi (Reference Baldi1998) does not undermine the importance of these classification schemes and regards them as essential in understanding the contextual nature of citations. Schemes proposed by Teufel (Reference Teufel1999) and Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006) have achieved good inter-rater reliability measures indicating the usefulness of the schemes and several studies have extended these schemes successfully to suit different domains.
Guo et al. (Reference Guo, Korhonen, Liakata, Silins, Sun and Stenius2010) analyzed three annotation schemes and concluded that the selection of a scheme should be task oriented and the optimal scheme for use should depend on the level of detail required by the application at hand. The principal focus of our study is to develop a tool that identifies and presents citation contexts to users. Thus, the basic typology required for this task is a scientific discourse scheme that identifies contexts for sentences. Furthermore, the contexts for sentences with citations should represent finer details of citation functions. However none of the available annotation schemes provide such functionality. While some have specifically focused on defining different types of citation contexts, others that focus on scientific discourse do not consider finer details of citation functions. We therefore resorted to developing a new set of context type definitions for sentences, specifically defined for providing citation context based information services for the research community. The context type definitions are presented in Section 3.3. We also show in Section 3.6 how our definitions relate to the annotation schemes of Teufel (Reference Teufel1999), which describe scientific discourse, and Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006), which define various categories for citation functions.
2.3.2 Use of conditional random fields – the rationale
Most of the studies discussed above view the sentence classification task as a nonsequential classification problem, where a given sentence is considered independently from other sentences. However, sentences are rhetorically related to each other and the above studies capture these rhetorical relations mainly using cue phrases. It also needs to be noted that authors follow a sequential pattern in the writing process. Swales (Reference Swales1984) observed a sequential pattern in paragraphs of the introduction section in 50% of the articles, he surveyed. He noted that in these paragraphs, authors usually start with sentences that establish the research field and continue to report on the previous research. After this, sentences prepare the readers for the present research and continue on to introduce that research. Our earlier work identified a sequential pattern based on the generic rhetorical pattern in paragraphs of related work sections in the research articles (Angrosh, Cranefield, and Stanger Reference Angrosh, Cranefield, Stanger and Hunter2010). Thus, a sequential pattern exists in research writing and the task of sentence classification can be considered as a sequential classification task that can be addressed by using sequential classification models.
Several sequential classification models such as HMMs, MEMMs, and CRFs have been used for exploiting sequential patterns for improving classifier accuracy (Rabiner Reference Rabiner1989; McCallum, Freitag, and Pereira Reference McCallum, Freitag and Pereira2000; Lafferty, McCallum, and Pereira Reference Lafferty, McCallum and Pereira2001). Though HMMs are used for solving various sequential problems such as recognition of speech (Rabiner Reference Rabiner1989), handwriting (Hu, Brown, and Turin Reference Hu, Brown and Turin1996), gesture (Tanguay Reference Tanguay1995; Elmezain et al. Reference Elmezain, Al-Hamadi, Appenrodt and Michaelis2008), and part-of-speech tagging (Kupiec Reference Kupiec1992), they are observed to suffer from two key problems (McCallum et al. Reference McCallum, Freitag and Pereira2000). First, HMMs do not provide for incorporating features, which allow for richer representation of observations, and second, the model follows a more traditional approach of employing a generative joint model in order to solve a conditional problem with given observations. McCallum et al. (Reference McCallum, Freitag and Pereira2000) introduced MEMMs for solving these problems and showed that MEMMs achieved good performance compared to HMMs.
Though MEMMs performed better, Lafferty et al. (Reference Lafferty, McCallum and Pereira2001) identified ‘label bias problems’ in these models and introduced CRFs as a solution to this problem. CRFs have properties of both generative and classification models. As classification models, they are able to accommodate many statistically correlated features of the inputs and are trained discriminately, and as generative models they can trade off decisions at different sequence positions for achieving a globally optimally labeling (Sha and Pereira Reference Sha and Pereira2003). Lafferty et al. (Reference Lafferty, McCallum and Pereira2001) showed that CRFs performed better than HMMs and MEMMs on synthetic and natural language data.
Various studies have successfully used sequential classification models for sentence classification. Hachey and Grover (Reference Hachey and Grover2005) tested various classifiers such as C4.5, Naive Bayes, Winnow, Support Vector Machines (SVMs), and Maximum Entropy Models (MEMs) for classifying sentences in the legal domain and concluded that the SVM classifier and the maximum entropy sequence tagger are suitable classifiers for the task. Hirohata et al. (Reference Hirohata, Okazaki, Ananiadou and Ishizuka2008) looked at the task of identifying sections in abstracts as a sequential classification task and used CRFs for achieving better accuracy compared to SVMs. Chung (Reference Chung2009) used CRFs and achieved good accuracy in classifying sentences in abstracts of randomized controlled trials. Kim et al. (Reference Kim, Martinez, Cavedon and Yencken2011) used CRFs for classifying sentences in medical abstracts and to develop a system for supporting evidence-based medicine.
In the light of these studies, we consider sentence classification as a sequential classification task, where the goal is to predict a label sequence <Yi>, given an input sequence <Xi>. This implies that the label for a given sentence may depend not only on the features of an individual sentence but also on the adjacent labels.
3 Context types for sentences in academic articles
The Oxford English Dictionary (Oxford University Press (OUP) Reference Stevenson2010) defines the word ‘context’ as “the parts of something written or spoken that immediately precede and follow a word or passage and clarify its meaning”. Thus, the context of a given word can be obtained by analyzing its surrounding words. However in the words of the famous economist, Alfred Marshall “every word has many shades of meaning, and therefore needs to be interpreted by the context” (Marshall Reference Marshall2009: 43). Thus, in order to identify the context of a sentence, we may have to consider each word carefully, which could result in a complex process. However, a context type for a sentence can be defined by identifying key terms and phrases that contribute in defining its context. For example, a phrase such as “This paper shows” indicates that the author in the sentence is referring to the results of the paper discussed. Similarly a phrase such as “limitations of the study” could refer to research gaps. There could be exceptions, as in spite of the presence of these terms, the sentence may hold a different meaning. Therefore, we follow an approach of identifying key terms and phrases that contribute in defining the context of a sentence and also consider that the context type of a sentence is partly dependent on the context types of preceding and following sentences. The following sections explain the context types defined for sentences following this approach.
3.1 Methodology for identifying context types
In order to identify context types for sentences, we used the first 40 articles from a dataset of 70 articles taken from Lecture Notes in Computer Science (LNCS). As the key objective of our study is to develop a citation context based information system, we focused only on paragraphs that contained citations. Each of the sentences in these paragraphs was manually analyzed to identify the context associated with the sentence.
As a first step, we classified sentences into two categories: citation sentences and non-citation sentences. Citation sentences are those that have a reference to a published or unpublished source. Specifically, this is an expression in the sentence that points to an entry in the bibliographic references section of the article for the purpose of acknowledging the cited work. This expression can either be a numeric expression such as “[1]” or “[1, 2]” or author names used in the sentence for referring to the cited work. Non-citation sentences are those that do not have any such expressions. We focused on identifying contexts for sentences belonging to these two categories. The process of analyzing sentences resulted in developing a decision tree shown in Figure 1, which contains a set of questions for identifying the context associated with a sentence. The decision tree served as a tool for classifying sentences.
Decision tree for identifying contexts of sentences.
3.2 Categories of terms
While identifying the context type of a sentence, we also focused on identifying terms and phrases that contribute in defining the context of the sentence. The objective was to use these terms and phrases as features for our machine-learning experiments (see Section 5.3.3) Accordingly, we defined ten categories of terms that contribute to defining the context of sentences. Table 1 provides a description of each of these categories along with a listing of example terms.
Categories of terms and phrases in sentences
The analysis of sentences following the above methodology resulted in identifying the following context types for citation and non-citation sentences.
3.3 Context types for citation sentences
The contexts associated with citation sentences are defined based on reasons for using the cited work in the current article. Accordingly, we distinguish between the contexts for citation sentences shown in Listing 1.
Context types for citation sentences
3.4 Context types for non-citation sentences
The contexts associated with non-citation sentences are defined based on the overall objective of the sentence. Listing 2 provides the definitions of various context types defined for non-citation sentences.
Context types for non-citation sentences
3.5 Framework for modelling the context types
In order to define relations between the above context types, we have proposed a framework shown in Figure 2. The framework considers that a paragraph with a citation is a set of reference areas, where a reference area is defined as the text with a citation sentence and the surrounding sentences (Nanba and Okumura Reference Nanba, Okumura and Dean1999). Several patterns can exist in reference areas that can involve one or more sentences defined in the framework.
Comparison of our annotation scheme with other schemes.
3.6 Comparison of our context types in relation with other schemes
We compare in this section the context types defined in our study with schemes proposed by Teufel and Moens (Reference Teufel and Moens1999) and Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006). The comparison is shown in Figure 2 and is described in the following sections.
3.6.1 Context types of sentences without citations
Some of the categories defined in our scheme can be directly compared to the categories proposed by Teufel and Moens (Reference Teufel and Moens1999). For example, their class of Background is equivalent to the BGR class of our scheme. Similarly the categories Aim and Contrast, which refer to sentences describing research goals of the current article and weaknesses in cited works, are the same as CWO and SCCW in our scheme. However, their classes Own and Basis can be seen as the unions of classes of our scheme. The class Basis of Teufel and Moens (Reference Teufel and Moens1999) consists of sentences that use cited work as a basis or starting point or gets support from them. While this is a broad definition, the classes of CCW, CWIG, CWOG, IRCW, RWCW, and UOCW of our scheme are specific instances for using the cited work as a basis. Thus, a union of all these classes can be considered equivalent to the class Basis. Similarly, the class Own is equivalent to the union of classes of DES, ISSUE, FW, and GAPS, which describe aspects of the own work presented in the paper. We do not use the Textual and Other class, as we do not define anything similar to this class in our scheme. We further compare our classifier results with this scheme in Section 7.1.2.1.
3.6.2 Context types of sentences with citations
Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006) have proposed a classification scheme for citation sentences. In this section, we compare our context type definitions for citation sentences with their scheme. The authors define the class Weak with sentences that refer to weaknesses of previous research. The class SCCW defined in our study is equivalent to the Weak class. The next four categories are used to describe comparisons or contrasts between the author's own and other work, which is equivalent to CCW in our scheme. The authors also define a set of categories for referring to statements that mention the active use of other works in the current article. Among them, the categories PBas and PMot identify the use of cited works as a basis or starting point or as a motivation factor in the current work, respectively. The categories IRCW, CWIG, and CWOG defined in our study represent these notions and thus, the union of these classes can be considered equivalent to the union of PBas and PMot. Similarly, the categories PSim and PSup are used to represent similarity and compatibility between the current work and the cited work. The category RWCW defined in our study signifies these notions and thus, a union of PSim and PSup is equivalent to RWCW. The categories PUse and PModi identify statements where the author uses and adapts or modifies tools, algorithms, or data from the cited work. The category UOCW signifies these characteristics. We do not identify any of our categories with Neutral defined in the other study. We further compare our classifier results with this scheme in Section 7.1.2.2.
4 Text preparation
The task of sentence classification involves various tasks for extracting structured information from unstructured text. These include (a) processing PDF documents to obtain usable content; (b) identifying section information for sentences; (c) sentence segmentation; (d) parsing sentences to identify terms and phrases; (e) generating features based on the presence of terms and phrases; (f) identifying keywords in sentences; and (g) parsing reference sections to obtain information about cited documents. In order to solve these tasks, we developed an information extraction and management system with various modules. The architecture of the developed system is shown in Figure 3. The key functions of the different modules are as follows.
System architecture of our information extraction system.
The Download Manager uses XPath expressions and Java HTTP URLs for web scraping. The PDF Document Manager module is used for downloading PDF files, cropping PDF files to remove headers and footers, and converting PDF to text and XML files. The Paragraph Extractor extracts paragraphs with citation sentences from text files along with section information and the Keyword Manager identifies keywords from citation sentences. The Feature Manager is responsible for parsing paragraphs and segmenting sentences. This module identifies the presence of key terms and phrases and generates features for each sentence. The MALLET Classifier labels paragraphs using the classifier model. The Relation Manager is used to relate results obtained from the Document Manager, the Feature Manager, and the Classifier.
The Reference Manager extracts references, segments references, identifies entities such as author names, the article title, and article source in each reference, and the Bibliographic Data Manager stores bibliographic metadata of research articles. The Database Manager stores data in a relational database management system and the D2R Server is used to generate RDF data from the relational database using a mapping file, configured according to the SENTCON ontology (Angrosh, Cranefield, and Stanger Reference Angrosh, Cranefield and Stanger2011). Finally, the resulting RDF data from the D2R server are used as input for the Semantic Web application.
5 Machine-learning experiments using sequential classifiers
In Section 2.3.2, we presented the rationale for using sequential classification models in our study. We report in this section the experiments carried out with CRFs for achieving the task of automatic context identification. These experiments were performed for learning probabilities of sentence labels, given features that encode observed properties of sentences. This resulted in a trained CRF that we use for generating maximum likelihood sentence labels for new (unlabelled) paragraphs.
5.1 Conditional Random Fields
CRFs are undirected graphical models that define a single log-linear probability distribution over label sequences given an observation sequence (Lafferty et al. Reference Lafferty, McCallum and Pereira2001). The structure of the graph in a CRF encodes independence relationships between labels and not the observations. This graphical structure facilitates a functional form of the distribution. This function combines several different terms known as clique potentials into a single product, in which each term forms a subset of the variables drawn from the full model.
The conditional probability of the labels gives the observations in a CRF with a linear chain structure (where the probability of each state yi depends only on the probability of the state y i−1 and y i+1, and the observed data sequence) factors according to the following equation (Vail 2008):
\begin{equation*}
P(Y|X) = \frac{1}{{Z_x }}\prod \limits _t {\psi _t (y_{t - 1}, y_t, X)},
\end{equation*}
\begin{equation*}
Z_x = \sum \limits _{Y^{\prime }} {\prod \limits _t {\psi _t (y_{t - 1}, y_t, X)} }
\end{equation*}
\begin{equation*}
\psi _t (y_{t - 1}, y_t, X) = \exp (w^T \,f(y_{t - 1}, y_t, X)),
\end{equation*}
5.2 Dataset and experimental setup, training, and evaluation
5.2.1 Dataset and experimental setup
We conducted experiments with a dataset of 1,000 paragraphs with citations extracted from seventy research articles chosen from LNCS. We followed the following steps in our experiments.
(1) Feature Definition – We manually examined sentences in paragraphs with citations in a training dataset of forty articles and manually labelled them according to the annotation scheme described in Section 3.3. The features defined for the sentences are described in Section 5.3.
(2) Feature Selection – We performed experiments for feature selection and identified the optimal set of features. The feature selection process is described in Section 5.4.
(3) Training the Classifier – We trained the classifier model with the training dataset of forty articles using the feature set identified during feature selection. To this end, we conducted tenfold cross-validation experiments. Section 5.4 provides further details about training the classifier.
(4) Testing – Finally, we tested the developed classifier model on a test dataset of thirty articles. The results obtained with the test data are provided in Section 5.5.
For training the CRF model, we used MALLET (McCallum Reference McCallum2002), a Java-based package that provides an implementation of linear chain CRF algorithms for working with sequential data.
We computed precision, recall, and the F-score for measuring classification accuracy for each label. The F-score is computed as follows:
\begin{equation*}
P = \frac{{\left| {{\rm TP}} \right|}}{{\left| {{\rm TP}} \right| + \left| {{\rm FP}} \right|}};\quad R = \frac{{\left| {{\rm TP}} \right|}}{{\left| {{\rm TP}} \right| + \left| {{\rm FN}} \right|}};\quad F = \frac{{\left| {{\rm 2PR}} \right|}}{{{\rm P} + {\rm R}}},
\end{equation*}
5.3 Feature definition
The following features were defined for sentences in our experiments.
5.3.1 Citation features
Citation features indicate whether a given sentence has a citation or not. The distinction is made based on the presence of a citation reference in the sentence. The application uses regular expressions for identifying the presence of citation reference expressions. In addition, references to citations can be made using the names of authors listed in the references section in the sentence without using an explicit reference. In such cases, the application looks for author names and terms such as ‘et al.’, to decide about the status of the sentence.
5.3.2 Section features
Section features indicate the section of an article to which the sentence belongs. To define section features, we adopted a model where the content of an article is divided into three general blocks: introduction, body, and conclusion. Sections of an article with the titles ‘Introduction’, ‘Related Work’, ‘Motivation’, or ‘Overview’ are considered part of the Introduction block. Sections with the titles ‘Conclusion’, ‘Conclusions and Future Work’, or ‘Future Work’ are considered part of the conclusion block. Sections with other titles are considered under the Body block. It needs to be noted that the Related Work section in the article may appear anywhere in the article. Irrespective of its position, this section is considered part of the introduction block.
5.3.3 Term features
Term features for sentences are defined based on the presence of certain kinds of terms and phrases in the sentence that contribute to the context of the sentence. Table 1 lists the ten categories of terms and phrases that may be relevant to the context of the sentence. Table 1 also provides a description of each of these categories along with examples identified in each of them.
5.3.4 Normalization
Each sentence was normalized to a set of features based on the presence of the features defined above. For example, for a sentence that has a citation, a feature ‘sentHasCitation’ is created for the sentence. The features that are created for different cases are provided in Table 2. We developed Python scripts by using regular expressions for identifying and creating features for sentences.
Features defined for sentences
5.4 Feature selection and training the classifier model
In order to identify the optimal set of features, we followed the backward selection method, where the performance of the classifier was tested on the test data by removing one feature from the dataset at a time. However, we did not evaluate the absence of the features – sentHasTerm=EXT and sentHasTerm=COM as these features occurred very few times in the training data and thus would not have impacted on the classifier model.
Thus, for this purpose, we divided the training dataset in the ratio of 9:1 and used the first part for training and the second for testing. We conducted experiments using two CRF structures – a first-order linear chain and a linear chain with additional zero-order features. The reason for using the second structure was due to the fact that our earlier experiments with related works sections show that a first-order linear chain performs poorly for classes that occur less often in the training data (Angrosh et al. Reference Angrosh, Cranefield, Stanger and Hunter2010). A first-order CRF with additional zero-order features provides a ‘back-off’ prediction capability, thus increasing the accuracy for classes that appear less often. Table 3 provides details of the feature selection process.
Results of feature selection experiments
Note: CC – Correctly Classified; WC – Wrongly Classified; A – Accuracy.
As seen in Table 3, the removal of ISSUE features resulted in the maximum discrepancy by achieving only 73.74%. The use of all the features resulted in achieving a good accuracy of 88.80%. Interestingly, the removal of the feature prevSentHasCitation resulted in achieving the maximum accuracy and also a higher accuracy of 89.18% than the accuracy of 88.80% achieved using all the features. However, the difference between the two does not seem to be very significant. Thus, in order to further ascertain the role of the prevSentHasCitation feature, we conducted a tenfold cross validation on two training datasets: one that used all the features and the other without prevSentHasCitation feature.
In both the cases, we conducted experiments by using a CRF first-order linear chain and a CRF with additional zero-order features. The results of the classifiers are tabulated in Table 4 and the performance is graphically shown in Figure 4. In both cases, the first-order linear chain CRF with additional zero-order features performed better than the first-order linear chain, particularly for classes with a lower number of instances such as CCW and RWCW. Furthermore, the first-order linear chain CRF with zero order features scored almost the same for both training datasets with an accuracy of 91.70% and 91.58%, respectively. The difference between the two was very small. Therefore, in order to measure the statistical significance between the classifiers, we performed McNemar's test on the results.
Performance of classifiers for training dataset.
Results of the classifier for training dataset
Note: P – Precision; R – Recall; F – F-Score.
The contingency tables and results of McNemar's test for both the datasets are shown in Figure 5. As seen, the p-value for McNemar's test for the dataset that had all the features is less than 0.05, indicating a significant difference between the two classifiers. The p-value for the other dataset was more than 0.07. Based on this statistical significance test, we decided to use the training dataset that included all the features for developing our classifier model and tested it on our test dataset.
Contingency tables and results for McNemar's test.
5.5 Testing
After achieving the results described above, we trained a classifier model with the first-order linear chain with additional zero-order features, using the training dataset of forty articles, including all the features. We used this classifier model to test articles in the test dataset of thirty articles. Further, in order to examine the effect of newer terms on the classification accuracy, we divided the test dataset of thirty into three sets, with each set consisting of ten articles. We conducted the following two different sets of experiments on each of this dataset.
(1) Derive term features for each dataset using the terms obtained from the training dataset of forty articles.
(2) Derive term features for each dataset by appending terms from consecutive test datasets. The new terms obtained from test dataset 1 were added to the terms of the training dataset of forty articles for deriving term features for articles in test dataset 2. This step facilitated addition of newer terms to the term list for deriving term features.
The results of the classifier on the test dataset are shown in Table 5. As seen in the table, the classifier achieved an accuracy of 92.08%, 92.92%, and 90.01% with test datasets 1, 2, and 3, respectively, with inclusion of terms for deriving features. In comparison to this accuracy, the classifier achieved a lower accuracy of 89.03% and 89.15% for test datasets 2 and 3, respectively. The considerable difference is noted for the classes CWIG and GAPS (shaded rows in Table 5). The terms that are responsible for creating features for these classes belong to the ‘Shortcoming Terms’ category, which refers to gaps or shortcomings in the sentence. This indicates that it is necessary to develop a larger collection of terms belonging to this category for effectively creating this feature for presence of such terms. The average accuracy achieved for the full test dataset of thirty articles without addition of terms was 90.09% as against an average accuracy of 91.67% with inclusion of terms, resulting in a marginal difference of 1.58% between the two. Further, our analysis provided in Section 7 indicates that the number of additional terms added at each step for deriving term features became smaller indicating that the set of terms obtained from the dataset of seventy articles need not be continually extended to be applicable for newer data.
Classifier results on test dataset
Note: P – Precision; R – Recall; F – F-Score.
6 Citation context based information services
We describe in this section a Semantic Web application developed for providing citation context based information services, using the contextual data derived in the previous steps. The architecture of the application is shown in Figure 6. As seen in the figure, the D2R Server is used for converting the data in relational form into RDF. The RDF data is also converted into JSON (Exhibit) format using the Babel Service (Babel 2010) for using the data with the Exhibit API, a lightweight framework for publishing structured data on standard web servers (Huynh et al. 2007). The Timeline feature of the Exhibit API is used for browsing information with time as a reference point. The application also uses Semantic Web Server (SEWESE) tags and SPARQL queries for querying the RDF data. The application offers citation context based information services through various interfaces: the citation sentences timeline, author timeline, citations timeline, and keywords timeline, as explained in the following sections.
Architecture of the Semantic Web application.
6.1 Browsing contexts of citation sentences in the article
The citation sentences timeline interface allows users to browse contexts of all citation sentences of an article. A screenshot of this interface is shown in Figure 7(a). As seen in the figure, the citation sentences of the article are displayed in the timeline. A zoomed-in image of the timeline alone, displaying citation sentences, is provided in Figure 7(b). The timeline feature provides an interactive interface, in which the user can move the timeline horizontally for viewing the displayed citation sentences on the timeline, placed according to the year of the cited work. As may be seen in Figure 7(b), the title of the selected article (indicated by the label ¶) and the ‘Claims of the Article’ (indicated by the label ·) are displayed as links in the timeline. These links display the bibliographic details and the result sentences of the article, respectively. Figure 8 presents a screenshot where the user has selected to view the claims of the article and the result sentences identified by the classifier are displayed in the dialog box. There are also options for selecting specific types of citation sentences as shown in an enlarged version of the facet in Figure 7(c).
Screenshot of citation sentences timeline.
Viewing result sentences of an article.
The application displays various related sentences for a citation sentence, following our Sentence Context Ontology (Angrosh et al. Reference Angrosh, Cranefield and Stanger2011). These include (1) Preceding Citation Sentence; (2) Preceding Issue Sentence; (3) Preceding Shortcoming Sentence; (4) Following Citation Sentence; (5) Following Description Sentence; (6) Following Shortcoming Sentence; and (7) Following Issue Sentence. In Figure 9, the application has identified the following issue sentences for the selected citation sentence. Further, the citation (reference) displayed in the popup dialog allows the user to navigate to the citations timeline for viewing the different contexts in which the citation is cited, as explained in the following section.
Viewing associated sentences for the selected citation sentence.
6.2 Browse contexts of citations
The citations timeline interface allows users to view the contexts of citations across articles. A screenshot of this interface is shown in Figure 10(a). A zoomed-in image of the timeline alone, displaying citation sentences drawn from different articles, is provided in Figure 10(b). The facet on the right side (an enlarged version is provided in Figure 10(c)) provides information about the citation and the number of times it was cited in different articles.
Screenshot of the citations timeline.
Each of the sentences displayed on the citations timeline can be clicked to view its associated sentences. The popup dialog box displays the citing article title that has cited the citation along with the citation sentence and its associated sentences, as explained earlier in the case of the citation sentences timeline. The citations timeline helps users to view how different articles have cited a given work. An illustration of the usefulness of the citations timeline is provided in Figure 11. This figure provides a screenshot of viewing the contexts of the citations to an article from two citing articles. As can be seen, the dialog displays the title of the article citing the selected article along with the citation sentence and its associated sentences. This helps users in quickly viewing the contexts in which different articles have cited a given work. This feature gathers significance when a given work has several citations. It avoids the laborious process of examining the citation passages in different articles and provides an easier and efficient way of examining citation contexts.
Citations timeline – viewing details of different citation sentences.
7 Discussion, analysis, and conclusion
We present in this section an analysis and comparison of our results with results of similar studies. With respect to supervised learning experiments, we discuss the use of different kinds of features in other studies and consider the usefulness of our approach. We also compare our classifier results with results of similar studies. With regard to the Semantic Web application, we discuss the citation-based information services provided by other tools and highlight the distinct features of our application.
7.1 Supervised learning experiments
7.1.1 Features
Previous studies have used various features for achieving the task of automatic citation classification. Prominent among them are cue phrases, which are linguistic indicators responsible for cohesion of text (Louwerse 2001 cited by Mancini 2005: 25) and features that identify the structure and location of a sentence. Teufel and Moens (Reference Teufel and Moens1999), apart from using structural and location features, have used syntactic features that identify the tense, modality, and voice of the sentence, semantic features that capture the action, and content features that indicate the presence of keywords. The syntactic features were identified using part-of-speech (POS) information and semantic features were identified based on the presence of verbs using a manually developed lexicon. Heuristic evaluation was used to determine the tense and negation of a sentence.
Nanba et al. (Reference Nanba, Kando and Okumura2000) created rules using cue phrases and applied n-word gram analysis for achieving classification. Garzone and Mercer (Reference Garzone, Mercer, Hamilton and Yang2000) developed a pragmatic classifier that used cue phrases for building lexical matching and parsing rules. Pham and Hoffmann (Reference Pham, Hoffmann, Gedeon and Fung2003) used human annotators for creating rules using cue phrases. Mizuta and Collier (Reference Mizuta and Collier2004b) identified multiple features through manual annotation for identifying zones in scientific articles in the biology domain. The authors observed the presence of similar features across zones and concluded that multiple features were essential in zone identification. Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006) used both shallow and linguistic features, which included cue phrases, verb tense and voice, and sentence location. Hirohata (Reference Hirohata, Okazaki, Ananiadou and Ishizuka2008) used various words, word bigrams and a mixture of words and word bigrams as features for representing the content. They also used features for sentence location and to relate previous and following sentences to the current sentence.
The studies above indicate that cue phrases form an important feature in citation classification and are used for identifying rhetorical relations. Mercer and Marco (2003), after studying the frequency of cue phrases in citations, suggested that these phrases play an important role in citations. However, manual development of cue phrases and rules using them is a cumbersome and time-consuming task. Pham and Hoffman (Reference Pham, Hoffmann, Gedeon and Fung2003) noted that certain cue phrases contradict each other, which results in a single sentence being matched by multiple cue phrases belonging to different classes, leading to conflicts. Identification of such conflicts is difficult and in order to solve this problem, they automated the system to identify conflict sentences in the initial stage.
We followed an approach of identifying terms and phrases (explained in Table 1) that characterized the context of a sentence and used regular expressions for this task, which provided a concise and flexible means for matching terms and phrases in sentences. The features were derived based on the presence of terms and phrases. Generally, most of the categories have expressions that signify the context of the sentence, while the category of ‘Connecting Terms’ comprises pronouns that help in representing the relation between sentences. We also used features related to sentence location and citation status of the sentence. The citation features also contributed in describing relations between sentences.
In order to identify the increase in the number of terms in each category for newer paragraphs, we carried out an analysis of the terms. For this purpose, we divided the dataset of 1,000 paragraphs into four sets, each comprising 250 paragraphs and analyzed terms appearing in each of these sets. The results of our analysis are shown in Table 6.
Details of number of terms identified in different categories
Note: TT – Total Terms; UT – Unique Terms; RT – Repeated Terms; NT – New Terms; TUT – Total Unique Terms; PC – Percentage Change.
As shown in Table 6 (column 1), the total number of terms identified in the first set of 250 paragraphs was 1,276. However, after removing duplicates there were 385 unique terms (column 2). The total number of terms identified in the second set, i.e., paragraphs 250 to 500 was 1,049 (column 3), of which there were 182 unique terms (column 4). Further, out of these 182 terms, 93 were already present in the first set (column 5). Thus, this set contributed ninety-two new terms (column 6). The average change in the number of terms from set 1 to set 2 was found to be 21% (column 8). The total number of unique terms in the set of the first 500 paragraphs was 477 (column 7). Similarly, the third set of 250 paragraphs contributed in adding twenty-five new terms (column 12) and the average change from the first 500 paragraphs to the third set was only 3% (column 14). The total number of unique terms in the first 750 paragraphs was 502 (column 13).
The fourth and the final set of 250 paragraphs contributed in adding only three new terms (column 18) to the total of 502, thus resulting in a total of 504 unique terms (column 19) for the dataset of 1,000 paragraphs. The average change from the first 750 paragraphs to the final set was 0.7%. This suggests that the increase in terms for new paragraphs is very low and the sets of terms gathered from these 1,000 paragraphs will not need to be continually extended to be applicable to new paragraphs.
7.1.2 Comparison of classifier results
We compare in this section the results achieved in our machine-learning experiments with similar studies.
7.1.2.1 Teufel and Moens (Reference Teufel and Moens1999)
We explained in Section 3.6.1 the comparison of our scheme with Teufel and Moens (Reference Teufel and Moens1999). We present here a comparison of our results with their scheme. The authors present precision and recall measures for classification of sentences in seven categories. The comparison of our results with that of Teufel and Moens (Reference Teufel and Moens1999) and the confusion matrix of the combined test data for different classes is provided in Table 7. As seen in Table 7, the results achieved in our study score better for all classes compared to their results.
Comparison of our results with Teufel and Moens (Reference Teufel and Moens1999)
7.1.2.2 Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006)
In Section 3.6.2, we presented a comparison of the context types defined in our study for citation sentences with the scheme of Teufel et al. (Reference Teufel, Siddharthan and Tidhar2006). The comparison of our results against the results achieved by them is shown in Table 8. As may be seen, our results score better than the results obtained by them.
Comparison of our results with Teufel and Moens (Reference Teufel, Siddharthan and Tidhar2006)
Note: TCS – Total Citation Sentences; SC-NCS – Sentences classified as Non-Citation Sentences.
7.2 Citation context based information system
Some of the popular search engines used by the academic community for searching scholarly literature include Google Scholar, Scopus, ScienceDirect, PubMed, and CiteSeerX. While Google Scholar provides search across different disciplines, Scopus provides search services in sciences and social sciences. ScienceDirect is used to search articles in sciences, and PubMed, and CiteSeerX cater to specific disciplines such as medicine and computer and information science. The prominent citation-related services provided by these search engines include citation statistics that provide information about the citing articles and citation links for accessing the citing articles. They further facilitate identifying related documents. While this information helps in identifying the number of cited works for a given article, they do not provide contexts in which the cited articles have used the article. The only way to learn these contexts is to read all cited documents, which is a labourious process.
The only search engine that provides citation context services is CiteSeer, which is a popular scientific literature digital library and search engine that automatically crawls and indexes scientific documents in the field of computer and information science (Li et al. Reference Li, Councill, Lee and Giles2006). Recently, CiteSeer has been upgraded with a new architecture and data model called CiteSeerX in order to overcome scaling problems in handling more documents, more features, and more users. CiteSeerX uses Autonomous Citation Indexing (ACI) (Lawrence, Giles, and Bollacker Reference Lawrence, Giles and Bollacker1999) that automatically creates a citation index from literature in electronic format and provides additional citation-related services such as current awareness and community features that complement commercial citation indices such as SCI. ACI systems locate articles, extract citations, and identify citations to the same article that occur in different formats and also identify the context of citations in the body of article. However, none of the systems described above provide information about the context in which a citation is used by the citing articles. In comparison to the above systems, the following are the distinct features of our application.
7.2.1 Distinct features of our application
7.2.1.1 Viewing context types of citations in an article
Currently, there are no tools that enable users to view contexts of citations in an article. Our application enables users to view the use of citations in an article by presenting their context types (as discussed in Section 6.1). This helps in distinguishing between different citations in the article. For example, if the user is interested in seeing works cited by the article, for which the author has identified gaps, or has used them as a basis or a starting point, the application provides a classification of citation sentences according to different context types. This can be used to browse these specific citations. In the absence of such a feature, the user would be required to read the full article in order to identify the contexts of different citations. Further, it would also be difficult to keep track of the use of citations as the user progresses to read the article. Our application provides a useful feature in this direction.
7.2.1.2 Viewing contexts of citing articles
The application provides a useful feature in identifying the contexts of citing articles given a target article (as discussed in Section 6.2). As discussed earlier, the present available tools provide citation link services and do not help in identifying the context of a citation. Our application differs from these tools by providing the contextual information about the citation. In the absence of such a feature, the user has to go through a laborious process of identifying the context in various citing articles. Our application provides a valuable feature in this direction by extracting only the citation paragraphs from the citing articles and presenting them to the user along with their context, which helps in easier evaluation of the target article. An example use of this facility would be a researcher searching for citations to one of his articles where the cited author has used the article for identifying or overcoming gaps in the related research area or identified gaps in the article itself. The feature provided by our application helps to learn these contexts and understand the contexts in which an individual's article is cited.
7.2.1.3 Sketching the intellectual lineage for topics
The interface provides a keywords timeline (not discussed in Section 6) that allows users to browse citation sentences for a specific topic drawn from different articles, published over a period of time. This allows users to see how different authors have addressed a specific topic over a period of time and aids in sketching the intellectual lineage for ideas.
7.2.1.4 Viewing contexts of author works
The application also provides an author timeline (not discussed earlier) that displays citation sentences drawn from different articles belonging to the specified author. Each of these citation sentences can display its associated sentences and the source article. This interface helps users to browse an individual author's work cited in various contexts in different articles. This timeline also provides a ready reference to the works authored by the selected author, by displaying their titles and links for accessing the respective article.
7.3 Conclusion
We presented in this paper our research on identifying contexts associated with sentences in research articles and using this information for providing intelligent information services for the research community. To this end, we defined various context types for sentences and compared our contextual definitions with other studies. We described machine-learning experiments on automatic context identification for sentences and provided an overview of an application developed for providing citation context based information services. We have also compared our classifier results with those from prior work and have shown that our results score well in comparison. The focus of this research has been not to replace the current existing information search systems but to provide additional services that complement the existing services. The citation context based information services discussed in this paper can be integrated with current information providing systems for designing useful information services for researchers. The different additional useful information services that can be provided were discussed in this paper. Our future work focuses on conducting an inter-rater reliability study for validating the objectivity of our chosen labels for sentences and also an evaluation study for assessing the usefulness of the web application.
