3.2.1 Surface linguistic analysis

This stage aims at carrying out a basic linguistic analysis on the input document, thus preparing it for further processing. In order to carry out this analysis, external state-of-the-art tools and resources are used. In particular, for this stage we propose the following:

• • Sentence segmentation. The text is split into sentences, which are the textual units considered for generating the summary. For this purpose, the sentence segmentation tool provided at DUC evaluation campaignsFootnote ³ is used.

• • Tokenisation. A tokeniser allows us to identify each word in the document, since we will need to compute, for instance, the frequency of each word, or distinguish between stop words and non-stop words in later stages. In order to be able to identify each word of the text, a tokeniser is used. In particular, we employ Word Splitter.Footnote ⁴

• • Part-of-speech tagging. A part-of-speech tagger assigns each word with its corresponding morphological category (noun, verb, adjective, preposition, adverb, determiner, pronoun, and conjunction). This process is useful for distinguishing different types of words, since some of them (e.g. nouns or verbs) can be more important than others (e.g. determiners). This tool will be used in the additional stages of information compression and fusion. In particular, TreeTaggerFootnote ⁵ was used as a part-of-speech tagger, because it is very easy to use, and it can be used for different languages, not only for English.

• • Stemming. This process consists in reducing words to their stem form. It is very useful in the cases where there is no need to differentiate between two inflected words that belong to the same family (e.g. running and runs, both come from the verb run). The Porter Stemmer Footnote ⁶ is employed for performing this task, which will be necessary for considering all terms sharing a common stem as a single one, for later computing their frequency.

• • Stop word identification. Stop words are words that appear very frequently in documents, but they do not carry any semantic information. They are normally not used for further processing, because they are not relevant (e.g. articles: the, a; conjunctions: and, or, etc.). In our case, this process is essential, and consequently we tag them in order not to take them into consideration when performing the remaining stages of the TS process. For instance, when computing the frequency of a word for determining the topic of a document (please see the topic identification stage), words such as “and”, “is”, or “the” could be wrongly identified as document's topics if they are not identified as stop words and treated as normal terms. For carrying out this task, a list of stop words is needed. In particular, we use an English stop word list.Footnote ⁷

We would like to note that this version of compendium does not include any coreference resolution module within the Surface Linguistic Analysis stage. Although we think it is an essential part of the process and should be incorporated in the short-term future. However, this is not a trivial task. State-of-the-art results in coreference resolution, and more concretely, in anaphora resolution, are around 65% on average (Delmonte et al. Reference Delmonte, Bristot, Boniforti and Tonelli2006). Therefore, before including a step focusing on the coreference, we would like to analyse in depth, which anaphora or coreference resolution system would be the most appropriate to be used for our purposes.

3.2.2 Redundancy detection

The aim of this stage is to identify redundant information in the source documents in order not to include it in the summary. For this purpose, we propose Textual Entailment (TE) as a technique to detect redundancy. On the one hand, attempts to study the influence of TE on TS have been focused for the evaluation of summaries (Harabagiu, Hickl and Lacatusu Reference Harabagiu, Hickl and Lacatusu2007) to determine which candidate summary, among a set of these, better represents the content in the original document depending on whether the summary entails it or not. On the other hand, TE has been combined with TS to generate a summary directly from the entailment relations found in a text (Tatar et al. Reference Tatar, Tamaianu-Morita, Mihis and Lupsa2008), or by extracting the highest scored sentences of a document, where the score of each sentence is computed as the number of sentences of the text that are entailed by it. However, in none of the already existing approaches, TE has been employed for identifying and removing repeated information, thus being novel in this sense.

Textual entailment is a suitable technique for detecting redundant information because its aim is to determine whether the meaning of a text snippet, also known as hypothesis (H), can be inferred from another one, called the text (T) (Glickman Reference Glickman2006). In order to illustrate this objective, we provide the following examples, taken from the RTE corpora.Footnote ⁸ As it can be seen, the first example shows a true entailment relation, whereas the second example shows a false entailment.

• Pair id = 50 (entailment = true)

• T: Edison decided to call “his” invention the Kinetoscope, combining the Greek root words “kineto”(movement), and “scopos” (“to view”).

• H: Edison invented the Kinetoscope.

• Pair id = 18 (entailment = false)

• T: Gastrointestinal bleeding can happen as an adverse effect of non-steroidal anti-inflammatory drugs such as aspirin or ibuprofen.

• H: Aspirin prevents gastrointestinal bleeding.

Taking the rationale behind TE, the manner in which it is employed for detecting redundancy within our summarisation process is next explained. We can compute the entailment relationship between two sentences, discarding the second one, if they both contain a true entailment. This means that the meaning of the second sentence is already embedded in the first one, thus stating the information only once. Therefore, if we repeat the process with all the sentences in a document, those sentences whose meaning is already contained in other sentences can be discarded, as the information has been previously mentioned. As a result, by applying TE we can obtain a set of sentences from the text that do not hold an entailment relation with any other, and then keep this set of sentences for further processing.

It is worth stressing upon the fact that the order in which the entailment relationships are computed is the same order that the sentences have in the original documents. In this manner, we ensure that the coherence of the resulting summary is not highly affected after this stage. Moreover, we only perform TE in one direction to avoid a very high computational cost.

Specifically in our research work, we used the TE approach presented in Ferrandez (Reference Ferrández2009) for computing the entailment relations within a document. This TE system relies on lexical (cosine similarity, Leveshtein distance), syntactic (dependency trees), and semantic measures based on WordNet (Fellbaum Reference Fellbaum1998), and although its performance is around 60%, it has been shown in previous research (Lloret et al. Reference Lloret, Ferrández, Muñoz and Palomar2008a, Reference Lloret, Ferrández, Muñoz and Palomar2008b) that this technique is appropriate for detecting redundant information when addressing summarisation.

3.2.3 Topic identification

The objective of this stage is to identify the main topics of a document, so we can later use them to determine which sentences are more important. In Section 2.2 we reviewed the most common techniques for generating summaries, including the topic-based approaches. The term frequency calculation is one of the most employed techniques for achieving this goal, as it has been shown in previous work (Montiel Soto and García-Hernández Reference Montiel Soto and García-Hernández2009). Moreover, it has been proven that frequent terms are very likely to appear in human-written summaries (Nenkova, Vanderwende and McKeown Reference Nenkova, Vanderwende and McKeown2006).

Following these ideas, in compendium summariser, the topics of a document are represented by the frequency of the terms it contains. In this manner, we assume that the most frequent terms of a document will be indicative of the topics included in it, and therefore, as we have previously stated, we will reward the sentences containing the topics (i.e. frequent terms) in the Relevance Detection stage. However, it is important to note that stop words, which were previously identified in the stop word identification stage, are not taken into consideration for calculating the frequency of a document, thus not forming part of the topics of a document.

3.2.4 Relevance detection

The relevance detection stage assigns a weight to each sentence, depending on how relevant it is within the text. Then we will be able to distinguish between the sentences that carry important information and those that do not.

For tackling the TS task, we wanted to study and analyse a discourse theory that has not been applied before for this purpose. Therefore, in this stage, we take into consideration a feature with a cognitive background, The Code Quantity Principle (CQP) (Givón Reference Givón1990), which is related to how humans understand and retain the information they read, and it has been proven to hold true in written texts (Ji Reference Ji2007). Specifically, this theory states that (1) a larger chunk of information is given a larger chunk of code; (2) the less predictable information, the more coding material; and (3) the more important information, the more coding material. This means that when an item provides a specific information within a text, it has to be assigned with a coding that it would be more or less stressed according to the relevance degree of such information within the text. In other words, the most important information within a text will contain more lexical elements, and therefore it will be expressed by a high number of units (for instance, syllables, words, or phrases) (Becker Reference Becker2002). This is also in line with The Code Quantity, Attention and Memory Principle (Givón Reference Givón1990), where it is stated that there is a proportional relation between the relevance of information and the amount of quantity through which it is coded, since the more salient and different coded information in a text, the more easily the reader's attention will be caught. As a result, readers will retain, keep, and retrieve this kind of information more efficiently.

On the basis of this, a coding element can range from characters to phrases. We decided to analyse noun-phrases, because a noun-phrase is the syntactic structure that allows more flexibility in the number of elements it can contain (pronouns, adjectives, or even relative clauses). Moreover, it is able to carry more or less information (words) according to what the writer wants to express (Becker Reference Becker2002). Furthermore, previous research (Mittal et al. Reference Mittal, Kantrowitz, Goldstein and Carbonell1999) has shown that the average length of complex noun-phrases in summary sentences was more than twice as long than those in non-summary sentences. In addition, Lloret and Palomar (Reference Lloret and Palomar2009) carried out a preliminary study of the percentage of noun-phrases contained in both source documents and model summaries of a corpus of a newswire and another one of fairy tales. This analysis showed that words belonging to noun-phrases were predominant over other types of words in the documents as well as in the summaries, representing on average more than 70% of all content words (i.e. without taking stop words into account), and approximately 30% of the total words of documents.

• S ₁: The Spanish Academy of Motion Pictures Arts and Sciences presented an honorific award for the best actor.

• S ₂: The Academy presented an honorific award.

In this case, S ₁ contains more information than S ₂. Although at first sight, the second sentence might be more appropriate for TS, since it reflects the same facts as that of the first one, but in a shorter manner the first one contains more details, and this would lead to more informative summaries, which is the purpose of our TS process.

Therefore, to consider noun-phrases as coding elements for our approach based on CQP seemed appropriate.

For computing the relevance of a sentence in compendium, we assume that when an author writes a document, he or she writes it following the CQP principle, i.e. emphasizing the most important information by means of larger coding material. Therefore, we will analyse to what extent this principle is useful for detecting relevance and generate automatic summaries. As it was shown in Lloret and Palomar (Reference Lloret and Palomar2009, Reference Lloret and Palomar2010), computing the relevance of a sentence focusing only on the length of the noun-phrases it contained should not be always appropriate since the terms contained may not be relevant. Therefore, we took into consideration the topics identified in the previous stage, and the relevance of each sentence was computed by combining CQP and the weight assigned to each term through the term frequency calculation, as it can be seen in Formula 1.

(1) \begin{equation} \label{formula1} r_{s_{i}}=\frac{1} {\#NP_i} \sum_{w\in NP}|tf_{w}| \end{equation}

r_si = the relevance of sentence i,

#NPi = number of noun-phrases contained in sentence i,

tf_w = frequency of word w that belongs to the sentence's noun-phrase.

In order to identify noun-phrases within a sentence, the BaseNP Chunker,Footnote ⁹ is employed. One important aspect to take into consideration is that the use of a chunker (as well as any other NLP-based tool) can introduce some error rate. This tool achieves recall and precision rates of roughly 93% for base noun-phrase chunks, and 88% for more complex chunks (Ramshaw and Marcus Reference Ramshaw and Marcus1995), thus being suitable for our purposes.

3.2.5 Summary generation

The objective of this stage is to generate a summary of a specific length. This length can be expressed in words or in the form of the compression rate (i.e. the percentage of information the summary contains with respect to the source document). However, the way this stage is carried out strongly depends on the summary type and purpose we want to produce. Consequently, four main types of summaries can be distinguished: (1) generic; (2) query-focused; (3) sentiment-based; and (4) generic abstractive-oriented. Type 1 is directly produced when the core stages of compendium are applied, whereas for types 2, 3, and 4, the additional stages of query similarity, subjective information detection, and information compression and fusion are also required, respectively.

Next, we briefly explain each of these strategies for generating the final summary.

1. (1) Generic summaries (COMPENDIUM_E). Once the final score of a sentence is computed by means of the relevance detection stage, the most important sentences (i.e. the ones with the highest scores) are selected and extracted to form the final summary up to a desired length or compression rate. In this case, the final summary is a generic extract.

2. (2) Query-focused summaries (COMPENDIUM_QE). Having computed the two different weights for each sentence (its relevance – r_si–, and its similarity with regard to the query – qSim_si –, in the relevance detection and query similarity stages,Footnote ¹⁰ these two values are combined within the same formula (Formula 2), which is based on the F-measure calculation. As it can be deduced, β does not have a fixed value. In contrast, it can be assigned different weights between 0 and 1, so that the TS tool is more flexible, and let the user determine the importance given to the generic relevance of sentences or to the query-related sentences,

   (2) \begin{equation} \label{formula2} Sc_{s_{i}}=(1+\beta^{2})\frac{r_{s_{i}}*qSim_{s_{i}}} {\beta^{2}*r_{s_{i}}+qSim_{s_{i}}} \end{equation}
   where:

   Sc_si = the score associated to sentence i,

   r_si = the relevance of the sentence i,

   qSim_si = the similarity between the query and sentence i.

   Taking into account the size restrictions, the top-ranked sentences will be selected and extracted, forming the final query-focused extract.

3. (3) Sentiment-based summaries (COMPENDIUM_SE). For generating these types of summaries, the same strategy as for the generic ones is followed. The main difference between these is that in this case we only focus on subjective information, which is identified and processed in the subjective information detection stage.Footnote ¹¹ As a result, extracts containing only subjective information are produced.

4. (4) Generic abstractive-oriented summaries. These summaries (COMPENDIUM_E-A) combine extractive and abstractive TS strategies, and as a consequence, the sentences of the final summary are selected following a strategy that maximises the similarity between each of the new sentences generated in the information compression and fusion stage (Section 3.3), and the ones that have been selected as most important in the relevance detection stage, given that the similarityFootnote ¹² between them is above a predefined threshold. In the cases where a sentence in the extract has an equivalent in the set of the new generated sentences, the former will be substituted for the latter; otherwise, the sentence as it appears in the extract will be kept. The resulting summaries are abstractive-oriented.

3.3.1 Query similarity

When query-focused summaries have to be produced, a query is usually associated to the source documents in order to specify the kind of information the user is interested in. From an extractive point of view, the summary should contain the most relevant sentences in the document that also contains the information in the query. Therefore, the goal of this stage is to take into account the information expressed in a given query to tailor the contents of the final summary to such information.

In order to determine which sentences may be potentially related to the given query, the cosine similarity between each sentence and the query is computed, using the Text Similarity package.Footnote ¹³ Formula 3 shows how the query similarity, through cosine similarity, is calculated:

(3) \begin{equation} \label{formula0} qSim_{S_{i}}=cosine(Q,S_{i}) = \frac{\sum_{j=1}^{n}{w_{j,Q}*w_{j,S_{i}}}}{\sqrt{\sum_{j=1}^{n}{w_{j,Q}}^{2}}*\sqrt{\sum_{j=1}^{n}{w_{j,S_{i}}}^{2}}} \end{equation}

Q = the query,

s_i = the sentence i in the document,

w_j,Q = the weight of term j in the query, and

w_j,Si = the weight of term j in the sentence i.

3.3.3 Subjective information detection

The objective of this stage is to detect and process subjective information, with the purpose of producing sentiment-based summaries. This has to be performed before the relevance of sentences is assigned. In order to be able to carry out such task, a tool capable of analysing and classifying the sentiment associated to a fragment of text (e.g. words, sentences, documents, etc.) is necessary. This manner, it is possible to know whether a fragment of text is subjective or objective, and, in addition, whether it is positive, negative, or netural about a specific topic, entity, product, etc. In particular, compendium uses the external opinion mining tool described in Balahur-Dobrescu et al. (Reference Balahur-Dobrescu, Kabadjov, Steinberger, Steinberger and Montoyo2009), which relies on opinion lexicons for identifying positive and negative words and expressions. Then it computes the number of positive and negative words in sentences and, depending on the resulting value, it assigns to each sentence in the document one of these three different scores: (i) score > 0 if the sentence has a positive nature; (ii) score < 0 if it has a negative nature; and (iii) score = 0 if the sentence is neutral (i.e. its an objective sentence). Once all the sentences have been analysed according to their associated polarity, we discarded the neutral sentences for further stages so that only those which have been identified as positive or negative according to the opinion mining tool are taken into consideration. In this manner, the resulting summary will contain the different opinions people have towards a topic, product, etc., thus being a sentiment-based (or subjective) summary.

3.3.3 Information compression and fusion

This stage aims at generating new sentences in one of these forms: either a compressed version of a longer sentence, or a new sentence containing information from two individual ones. The main steps involved in this stage are as follows:

• • Word graph generation. For generating new sentences, we rely on word graphs adopting a similar approach to the one described in Filippova (Reference Filippova2010). Specifically in our approach, we first generate an extractive summary in order to determine the most relevant content for being included in the summary, and then we apply this stage for compressing and merging relevant information, since it has been proven that this approach is the most appropriate one (Lloret and Palomar Reference Lloret and Palomar2011a).

  Therefore, taking as input the generated extract, it is represented as a directed weighted DG = (V,E), where V = v_i, v _i+1, . . ., v_i+n is the set of nodes corresponding to document's words, and E = e _i,i+1, e _i+1,i+2, . . ., e_i-n,i+n is the set of edges, which consists of adjacency relations between the words. Two words are mapped into the same node only if they have the same part of speech tag. However, it is important to stress that stop words are not mapped together; otherwise, the real meaning of the sentence could be changed when generating the new sentence. The weight of each edge is calculated based on the inverse of the frequency of co-occurrence of two adjacent words, taking also into account the importance of the nodes they link through the Pagerank algorithm (Brin and Page Reference Brin and Page1998). For implementation we used Python-graph library.Footnote ¹⁴

  Once the extract is represented as a word graph, a pool of new sentences is created. In order to produce a new sentence, we employ Dijkstra's algorithm (Dijkstra Reference Dijkstra1959) to find the shortest paths between an initial node and the remaining ones that are directly or indirectly connected with it. The initial node corresponds to the first word in each sentence, since this manner, we ensure that for each sentence in the extract, we have one derived sentence, so the whole content of the extract is covered. Moreover, we chose the shortest path algorithm because on the one hand, it has been shown to be appropriate for compressing sentences in previous work (Filippova Reference Filippova2010), and on the other hand, the shortest path will also look for minimal-length sentences that contains information from several ones, thus allowing them to include more information.

• • Incorrect paths filtering. By applying the Dijkstra's algorithm over the graph we obtain all possible shortest paths between one node and the remaining ones. This leads to a high number of resulting sentences, which are not equally good. In fact, some of the sentences might be completely incomprehensible and not correctly formed. Therefore, in order to guarantee the completeness and correctness of a new sentence, this stage is needed to discard the invalid paths (i.e. generated sentences). For instance, some of them may suffer from incompleteness. Consequently, in order to reduce the number of incorrectly generated sentences, we define a set of rules, so that sentences not accomplishing all the rules are not taken into account. Three general rules are defined after analysing manually a set of generated sentences derived from a small data set, which are as follows:

  • — The minimal length for a sentence must be three words.Footnote ¹⁵

  • — Every sentence must contain a verb.

  • — The sentence should not end in an article (e.g. a, the), a preposition (e.g. of), an interrogative word (e.g. who), or a conjunction (e.g. and).

Once the incorrect sentences have been removed, we can use the new sentences instead of the original ones (see “generic abstractive-oriented summaries” in the summary generation stage). It is important to stress upon the fact that, in compendium, this stage takes place after the relevance detection and before the summary generation stages. This is because we need to be aware of the most relevant sentences of the document first.

4.2.1 Newswire corpus

Using the different DUC corpora for news, compendium_E is evaluated for single- and multi-document summarisation. For both evaluations, we use ROUGE-1, ROUGE-2, and ROUGE-L, and we report the values for F-measure (β = 1). On the one hand, for single-document summarisation, we use the DUC 2002 corpus, on the other hand, this corpus together with the ones for DUC 2003 and DUC 2004 are used for generating multi-document summaries.

Table 2 shows the results that compendium_E obtains for different newswire corpora. As it can be seen, single-document summaries achieve better results (around 45% for ROUGE-1) than multi-document ones (30% on average). The difference in performance may be due to the fact that we do not employ any specific technique for tackling multi-document summarisation. In contrast, we merge all related documents into a single one, and this is used as input for compendium_E.

Table 2.

compendium_E results for single- and multi-document summarisation for the newswire domain, F-measure (β = 1)

An example of a single- and a multi-document summary generated by compendium_E is shown in Table 3.Footnote ²¹

Table 3.

Single- and multi-document summaries generated by compendium_E for document AP880325-0239 (DUC 2002, cluster d078b)

Furthermore, we carry out a comparison of our results with respect to the best scoring system in the respective DUC editions, as well as a Lead baseline that builds the summary taking the first sentence of each document in the case of single-document summarisation, and it takes the first sentence of the first document, then the first sentence of the second document, and so on, for multi-document. In these cases, we report the recall value for ROUGE-1. Table 4 shows such comparison.

Table 4.

Comparison of compendium_E's performance with other text summarisation systems (recall value)

The single-document summaries achieve very good performance compared with the best system at DUC 2002, and the Lead baseline. In this case, compendium_E outperforms the best system by approximately 8% (0.46008 vs. 0.42776 – please see Table 4), and an increase of 12% is obtained over the baseline (0.46008 vs. 0.41132). On the contrary, results for multi-document summarisation are not as good. The Lead baseline is improved by 33% and 40% for the DUC 2002 (0.30341 vs. 0.22771) and DUC 2003 data (0.29355 vs. 0.20967), respectively, but there is a marginal increase for DUC 2004 (0.2%, 0.31362 vs. 0.31293). Despite these improvements, the performance of compendium_E for multi-document summarisation does not surpass the best system at DUC. This difference is due to the fact that we approach multi-document summarisation as a single-document summarisation where all related documents are considered as a single one. This is an indication that this type of summarisation may require a more elaborate processing instead.

4.2.2 Image caption corpus

Generic and query-focused summaries were produced using compendium_E and compendium_QE, respectively. For the query-focused summaries generated by compendium_QE, it was experimentally established that, for this research and this corpus, the optimal value for the parameter β was 0, thus meaning that the sentences related to the query were considered more important to build the summary. As far as the generic summaries (compendium_E) are concerned, the query similarity stage was not taken into consideration.

In total, 308 summaries of 200 words each were generated, and to evaluate both approaches, we use ROUGE-2 and ROUGE-SU4. These metrics allowed us to compare the automatic summaries against the model summaries also provided in this corpus.

Furthermore, as baseline the first 200 words of the Wikipedia article describing each image were selected. Also, for comparison purposes, we generated summaries employing different state-of-the-art TS systems. In particular, these were SummGraph (Plaza, Díaz and Gervás Reference Plaza, Díaz and Gervás2008), MEAD (Radev et al. Reference Radev, Allison, Blair-Goldensohn, Blitzer, Celebi, Drabek, Lam, Liu, Otterbacher, Qi, Saggion, Teufel, Topper, Winkel and Zhang2004), and SUMMA (Saggion Reference Saggion2008). In this way, the performance for generic and query-based summarisation for such systems was also compared. The results are given in Tables 5 and 6, respectively.

Table 5.

Results for generic summarisation in the image caption corpus (recall value)

Table 6.

Results for query-focused summarisation in the image caption corpus (recall value)

From the results obtained, we can conclude that query-focused summarisation is more appropriate for these types of data. If we observed the results for the summarisers, all ROUGE scores in Table 6 (query-focused summaries) are higher than the scores in Table 5 (generic summaries). Moreover, the Wilcoxon signed-rank test is computed for assessing the significance of the results. For all summarisers, except SUMMA, the query-focused summaries are significantly better than the generic ones.

Regarding our TS approach, compendium, the results for the query-focused approach (compendium_QE) increase by approximately 4% on average with respect to the generic one, compendium_E (0.08864 vs. 0.08551 for ROUGE-2 and 0.13892 vs. 0.13371 for ROUGE-SU4), thus proving that query-focused summaries are the most appropriate ones for these type of documents.

On the other hand, Wikipedia summaries are a difficult goal to achieve, as it can be seen from the results where only two summarisers, when producing query-focused summaries (SummGraph and MEAD), obtain higher results. The reasons why it is so difficult to perform better than the Wikipedia baseline are as follows: (1) These articles have been created by humans; (2) the first paragraph in a Wikipedia article is usually a summary of the entire document content; and (3) Wikipedia articles almost exclusively contain salient information to the subject matter, and do not present other information somehow related to the topic but not important (e.g. nearby hotels, or transport services).

Although the results achieved by compendium are comparable to the best performing systems, we try to elucidate the reasons why compendium performs a bit lower than SummGraph and MEAD.

Analysing the types of documents and the resulting summaries, we realised that one of the main problems in these specific types of documents resides in the nature of the corpus, which contained some noisy information, in the sense that they included a lot of noun-phrases within the documents as categories for the objects (e.g “Mahogany, Maple, crown mouldings, multiple Viking ovens, Sub-Zero refrigerators, antique. . .”). In these cases, the method proposed in compendium for detecting relevant information did not perform as it would be expected. The reason is that, according to the CQP feature, our method gives more importance to those sentences which contains longer noun-phrases, and consequently, these types of sentences are scored higher. Therefore, they are wrongly considered as relevant, and they are final incorporated in the summary. In the end, this leads to the fact that the quality of the final summaries is directly affected by these sentences, and as a consequence, the content of the generated summary will show greater differences in comparison to the sentences extracted by other summarisers. In order to improve our results, we could have classified which of the noun-phrases may be related to the topic and which may not be.

Nevertheless, the resulting summaries are good enough to provide an idea of the most important facts about a location, as it is illustrated through the two fragments of summaries related to “Nou Camp” shown in Table 7.Footnote ²² The summary at the top corresponds to a generic summary produced using compendium_E, whereas the one at the bottom is a query-focused one (compendium_QE).

Table 7.

Generic and query-focused summaries generated by compendium for the place Camp Nou

4.2.3 Blog corpus

The comments associated to each blog in the corpus are used to produce sentiment-based summaries (compendium_SE). Different from the previous experiments, instead of generating summaries of a fixed length, we experimented with three different compression rates (10%, 15%, and 20% of the document).

The evaluation conducted also varies from the previous ones in the sense that ROUGE is not employed. Instead, a qualitative evaluation is conducted, where summaries are evaluated manually. It is worth stressing upon the fact that we focus more on the quality of the summaries rather than on their content, since the content would depend on the specific need a user has at a particular moment. Moreover, for this corpus we do not have model summaries, and to produce them manually is a very difficult and time-consuming task.

In particular, the criteria proposed for evaluating the opinion summaries are the following: redundancy, grammaticality, focus, and difficulty. Redundancy measures the presence of repeated information in a summary. Grammaticality accounts for the number of spelling or grammatical errors that a summary presents. Focus evaluates whether it is possible or not to understand the topic of the summary, that is, the main subject of the text; and finally, difficulty refers to the extent to which a human can understand a summary as a whole or not.

If we have a look at the criteria proposed in DUC and TAC conferences, we will realise that we adopt more or less the same, except the difficulty criteria which is non-conventional. The reason why this criterion is included is that it provides an idea of the summary as a whole, regarding its readability.

Furthermore, three different degrees of goodness are established for each evaluated criteria. These were non-acceptable, understandable, and acceptable. In this classification, acceptable means that the summary meets the specific criterion and therefore is good, whereas non-acceptable means that the summary would not be good enough with respect to a criterion. When assessing the difficulty, the summaries were classified with regard to high, medium, and low, being low, the better.

Table 8 shows the results for this evaluation.

Table 8.

Results for compendium_SE within the blog corpus

Analysing the results obtained, a set of interesting conclusions can be drawn. As far as the grammaticality criterion is concerned, the results show a decrease of grammaticality errors as the size of the summary lowers. We can see that the number of acceptable summaries varies from 74% to 45%, for a compression rate of 20% and 10%, respectively. This is obvious because the longer the summary, the more chances there are for containing orthographic or grammatical errors. Due to the informal language used in blogs, we thought a priori that summaries would contain many spelling mistakes. Contrary to our expectations, generated summaries are quite well written, only 4% of them, at most, being non-acceptable. Another important fact that can be inferred from the results is related to how the summaries deal with the topic. According to the percentages shown in the tables previously presented, the number of summaries that have correctly identified the topic and have therefore been evaluated as acceptable, change considerably with respect to the different summary sizes, increasing when we changes from 10% to 15%, but decreasing when changing from 15% to 20%. However, as a general trend, we can see that when taking into account the number of summaries that have not performed correctly in the focus parameter, there is a decreasing trend, reducing the incorrect summaries from 33% to 14%. This means that for longer summaries, the topic may be stated along the summary, although not necessarily at the beginning of it, whereas for shorter summaries, there is no such flexibility, and as a consequence if the topic does not appear at the beginning, the most probable thing is that it does not appear in the summary at all. Finally, regarding redundancy, results indicate that summaries of 15% and 20% contain less repeated information than shorter ones. What can be seen from the results is that the summaries of 20% compression rate obtain the best results on average over the rest of the size experimented with. This is due to the fact that this compression rate achieves higher percentage (for the understandable and acceptable degrees of goodness) in two (grammaticality and focus) out of the three criteria proposed. Only the 15% compression rate summaries obtained better results in the redundancy criterion. On the other hand, as far as the difficulty criteria is concerned, results are also encouraging. According to the evaluation performed, the longer the summaries, the easier they are to be understood in general. Grouping the percentages of summaries, we obtained that 65%, 86%, and 92% of the summaries of size 10%, 15%, and 20%, respectively, have either medium or low level of difficulty, which means that they could be understood as a whole without serious difficulties. Again, for this criterion, 20% summaries achieve the best results. This has also been proved by previous research work, which demonstrated that this compression rate is more suitable for an acceptable quality of summaries (Morris, Kasper and Adams Reference Morris, Kasper and Adams1992). It is worth mentioning that this criterion is rather subjective and depends to a large extent on different factors, such as the knowledge the person who reads the summaries, the number of grammatical errors the text contains, or the connectedness of the sentences. Moreover, it is reasonable to think that long summaries can be more difficult to understand, but our experiments show that it is actually the other way around, because longer summaries may contain more information than shorter ones, which allows the user to have more awareness of the content and what the summary is about.

In general terms, while evaluating the summaries obtained, we noted some recurrent mistakes. The first one is the punctuation; in some cases we noted some commas missing or instead of having a comma, contain a full stop (e.g. “So. One option. . .”). Also, in some cases, apostrophes are missing, in examples such as dont. The second error is that in some cases the summaries start with a sentence containing a coreference element that we cannot resolve, because the antecedent has been deleted or sentences that imply some concept previously mentioned in the original text have not been selected. It is also worth mentioning that some of the grammatical errors are due to users' misspellings, for example “I thikn”. The third error concerns the spelling mistakes found in the summaries, which are directly transferred from the initial blog posts, which also contains such kind of errors (e.g. calender). Finally, we also found some void sentences, that do not contribute to the general meaning of the summary as for example, “I m an idiot”, “Just an occasional visitor”, or “welcome back!!!”.

Table 9 shows an example of an automatic summary for blog 29 with a compression rate of 10%.Footnote ²³ In this case, the summary contains mostly positive opinions, having only the last sentence a negative charge (“My only quibble would be about the name”).

Table 9.

Sentiment-based summary generated by compendium_SE for blog 29

As it can be seen, only opinions have been considered and these are presented grouped into positives, on the one hand, and negatives, on the other. We considered it as good due to the fact that there are no objectives or useless sentences. The system presents subjective sentences with an emotional charge, and as a consequence this summary meets our purposes.

4.2.4 Medical research papers corpus

With this set of documents, we want to analyse the capabilities of compendium for generating abstractive-oriented summaries. In this particular evaluation, we generate summaries of 162 words because this was the average length of the abstracts included in the medical papers (please see Table 1), and we wanted to use them as model summaries. Therefore, since our goal is to analyse to what extent the resulting automatic summaries are valid, we set up a comparison between compendium_E and compendium_E-A, considering as model summaries the ones included in the medical papers.

Table 10 shows an example of two summaries generated with compendium_E and compendium_E-A, respectively.Footnote ²⁴ It is worth mentioning that these approaches produce generic summaries, and for generating them neither the keywords of the original paper nor the information in the titles or the abstracts were taken into consideration. As it can be seen, both summaries share some of the sentences, whereas others have been shorten in the latter.

Table 10.

Example of summaries generated with compendium_E and compendium_E-A for the medical corpus

Specifically, we set up two types of evaluation. In the first one, we use the model abstracts of the papers and we compare these with the ones generated by compendium_E and compendium_E-A, for our extractive and abstractive-oriented approach, respectively. The second evaluation aims at assessing the summaries with regard to user satisfaction with respect to a 5-level Likert scale. Next, each of these types of evaluation is explained in more detail.

• • Comparison with human abstracts. In this evaluation, the summaries generated by compendium_E and compendium_E-A are assessed with respect to the human abstracts provided in the original papers. We use ROUGE-1 and report the values of recall, precision, and F-measure (β = 1). We also compare the results obtained with the state-of-the-art summariser: MS-Word 2007 Summarizer,Footnote ²⁵ and we run a t-test to account for statistical significance of the results at a 95% confidence level (statistical significant results are marked with a star).

  Table 11 shows the results for the first evaluation. In this evaluation, compendium_E-A summaries are evaluated with respect to the human abstracts, and compared to compendium_E and MS-Word 2007.

  Table 11.

  ROUGE-1 results for compendium_E-A and its comparison with other TS approaches

  

  As it can be seen, our both TS approaches are comparable with respect to the state-of-the-art TS tool (i.e. MS-Word 2007 Summarizer). Regarding compendium_E-A, it is worth mentioning that the precision obtained is higher and statistically significant compared with the remaining approaches, thus meaning that the information contained is the right one. However, its recall is lower, so in the end the final value of F-measure is negatively affected, being compendium_E statistical significant with respect to compendium_E-A for these specific measures. This is due to the fact that for this TS approach we rely on the sentences detected as important in the relevance detection stage, and we compress or merge some information within them. Therefore, the resulting summaries are shorter than the extracts, and since no extra information is added, the recall value will be never higher than it is for compendium_E. One possible solution to address this issue would be to rely on the source document and generate new sentences from it instead of the most relevant sentences. Another strategy would be to include in the compendium_E-A summary the next highest ranked sentence in the document according to the relevance detection stage that was not included in the extract because of summary length restrictions. Regarding compendium_E, it achieves slightly better results than MS-Word 2007; however, there are no statistical differences between them. Only statistically significant results are obtained with respect to compendium_E-A.

• • User satisfaction study.

  In the last evaluation, we aim at assessing the user satisfaction with respect to the generated summaries. For this purpose, we perform a qualitative evaluation and asked ten humans to evaluate our summariesFootnote ²⁶ according to a 5-level Likert scale (1= strongly disagree . . .5 = strongly agree). For each summary, humans were asked to respond to three questions concerning the appropriateness of the summaries. The questions asked, as well as the percentage of summaries for each question in scales 1 to 5 are shown in Tables 12 and 13, respectively.

  Table 12.

  Qualitative questions to evaluate the summaries

  
  Table 13.

  User satisfaction results for the different text summarisation approaches

  

  As it can be seen from the results shown in Table 13, our abstractive-oriented approach (compendium_E-A) obtains better results than the extractive one (compendium_E). Although the evaluation concerning the information contained in the summaries generated with compendium_E-A was not as good as for the extractive approach, taking into consideration their quality from a human point of view, the abstractive-oriented summaries are much better than the extractive ones. When we have a look at the different percentages of summaries that have been rated in one of each categories, we observe that there is a higher percentage of abstractive-oriented summaries that humans agree with, compared with the extractive summaries for the same rating. Moreover, it is worth stressing upon the fact that, analogously, the percentage of summaries with lower ratings (strongly disagree and disagree) also decreases when compendium_E-A is employed.

  Furthermore, concerning the average individual scoring results (between 1 and 5), compendium_E-A achieves at most 3.37 for Q2 and 3.1 for Q1 and Q3, whereas the maximum average value for compendium_E is 2.83 for Q2, the remaining questions obtaining values lower than 2.60. In light of the results obtained, it has been proved that the combination of extractive and abstractive techniques is more appropriate and leads to better summaries than extracts.