2.1 Related work

3.1 User interface

Figure 4 shows the interface of our prototype Music RS. It is a simple interface, dictated by our focus on the summarization process. It has four windows in the top part, and a bottom, pop-up window that displays details about the recommended song, or the review summary. Moving from left to right, the four windows on top display the following information:

1. (1) The top leftmost window shows the devices on which the user listens the songs. These devices are monitored with a UPNP control point, which receives messages from the devices. Based on these messages, user activities, including the metadata of the song she/he chooses, are logged.

2. (2) The second window from the left shows the list of songs available on the device selected in Window 1.

3. (3) The third window displays more information about the song the user has chosen in Window 2.

4. (4) At this point, the Expander (see Figure 3) identifies songs related to the chosen song from the Internet, including other songs from the same album, popular songs of the artist and popular songs of related artists as obtained from Yahoo! Music. Among those downloaded songs, we identify and rank the top twenty-five songs which belong to the same cluster and are closest to the given song – we use SimpleKMeans Clustering Mitchell (Reference Mitchell1997). Closeness between two songs in a cluster is measured as the number of attributes (album, artist, etc.) of the songs that match. These top twenty-five recommendations are shown in the rightmost top window (in Figure 4, only the top recommendation is shown).

If the user clicks on More Info for one of the recommended songs, the pop-up, bottom window is displayed, which contains the summary of the reviews for the specific song.

4.1 Title extraction

Song identification is the first step toward summarization of reviews. Unless songs are identified, the features cannot be made sense of, since users comment on different features for different songs, or features of the album as a whole. We identify a string of words as the title of a song to be extracted from an album review if it (1) includes some or all the words in the title of a track of that album, and (2) this string occurs in the right context. Constraint (2) is necessary because the string of words corresponding to the title may appear in the lyrics of the song or anywhere else in the review. The string Maxwell's Silver Hammer counts as a title only in sentence (a) below; the second sentence is a verse in the lyrics:

1. (a) Then, the wild and weird ‘Maxwell's Silver Hammer’.

2. (b) Bang, Bang, Maxwell's silver hammer cam down on her head.

At first approximation, extracting song titles could be approached with Named Entity Recognition (NER) techniques. However, as we noted in the Introduction, NER techniques focus on NP extraction. Since song titles range the spectrum of syntactic phrases (verb phrases, adverbs, etc), NP extraction algorithms miss many song title occurrences as shown by preliminary experiments we ran. Others have noted that NP extraction algorithms fail in the music domain, and have proposed different approaches, for example, the algorithm from Schedl et al. (Reference Schedl, Widmer, Pohle and Seyerlehner2007), which extracts band names from web sites. This algorithm is based on n-grams, and forms the basis for our approach to song title extraction. We proceed album by album. Figure 9 shows the sequence of steps for song title extraction from the reviews. Given the reviews for an album and the list of songs in that album, first we build a lexicon of all the words in the song titles. Each word in the lexicon is associated with all the IDs of the songs in whose titles it appears. We also segment the reviews into sentences via sentence boundary detection.

Fig. 9.

Flow chart for song title extraction algorithm.

All 1, 2, 3, 4-grams for each sentence in the review are generated (the upper-bound 4 was determined experimentally, as we discuss below). First, n-grams that contain at least one word with an edit distance greater than one from a word in the lexicon are filtered out. Including those n-grams with an edit distance of one will help to identify titles with spelling mistakes. Second, if higher and lower order n-grams overlap at the same position in the same sentence, lower order n-grams are filtered out. While finding the lower order n-gram we take into consideration the position of the words of the n-gram in the sentence. Only those lower order n-grams that represent the same word position as that of the higher order n-grams are filtered. For example, if given the n-gram [Song, That, Jane, Likes] we filter out all the other n-grams (<4) that occur in positions overlapping with the longer n-gram, in the same sentence – for example, [Song], [That, Jane, Likes]. Third, the remaining n-grams are merged if they occur sequentially in a sentence. Fourth, the n-grams are further filtered to include only those where (i) the n-gram is within quotation marks; and/or (ii) the first character of each word in the n-gram is upper case. This filters n-grams such as those shown in sentence (b) above. All the n-grams remaining at this point are potential song titles. Finally, for each n-gram, we retrieve the set of IDs for each of its words and intersect those sets. This intersection always results in one single song ID, since song titles in each album differ by at least one content word. Recall that the algorithm is run on reviews for each album separately and there is at least one content word that distinguishes every song title from others in the same album. The filter in step 4, which eliminates n-grams that are neither surrounded by quotes nor capitalized, may appear too restrictive, since the writing in these reviews is informal. Nonetheless, reviewers appear to adhere to these unspoken conventions, as shown by the recall we obtain on song title extraction, 99.2 percent (see Section 5.1).

As mentioned, we established n=4 as the upper bound for our n-grams based on observations and a small experiment. First, we observed that song titles longer than three or four words are less frequent. Using as an example three albums, each with ten tracks, by three different artists: in Remember Two Things (Dave Matthews Band), only two songs have titles longer than three words (one has four words and the other, five); likewise, Abbey Road (Beatles) has only two songs whose title is longer than three words (one has four words and the other, six); War (U2) has one song whose title is longer than three words (five words total). Second, we observed that even for longer titles, most reviewers refer to them in an abbreviated form (see, e.g., The song that Jane, for The song that Jane likes in Figure 1). These observations suggest that an upper bound of n=4 should be sufficient to capture the vast majority of references to song titles. The question is whether we could lower it to, e.g., n=3. However, the lower the bound on n, the higher the number of mergings of sequential n-grams the algorithm needs to perform (this is the most expensive step in the algorithm). In a small experiment on four reviews each for Abbey Road and Rubber Soul, we found that using n=3 as the upper-bound on n-gram length generated sixty-seven mergings, whereas using n=4 only thirty-eight mergings were necessary. Hence, we converged on n=4.

4.2 Feature extraction

After song titles are identified, each sentence that contains one song title can be used as an anchor to identify segments of texts that talk about that specific song, and to extract the feature(s) that the pertinent text segment discusses. This task roughly corresponds to identifying the flow of topics in a review. In principle, it could benefit from sophisticated topic models or from reference resolution algorithms. However, given the still lacking performance of state-of-the-art algorithms for these tasks, we chose a simpler method. We devised an approach that combines TextTiling Hearst (Reference Hearst1994) and domain heuristics. The TextTiling algorithm divides the text into coherent discourse units to describe the subtopic structure of the given text. TextTiling assumes that a set of words is in use while a certain subtopic is being discussed, and when that subtopic changes, a significant proportion of the vocabulary changes as well.

Figure 10 shows in more detail the sequence of steps performed during the feature extraction process. First, the reviews (pre-processed by demarcating sentence boundaries as discussed above) are divided into discourse units (DUs) using the TextTiling algorithm. Let [DU ₁, DU ₂, . . ., DU_k] be the units obtained. The segments that contain potential features of a song are identified using the following heuristics:

1. (1) Determine anchors: Include DU_i in segment s_j if it contains a song title (as identified by the title extraction algorithm). These DUs are more likely to contain features of songs as they are composed of the sentences surrounding the song title.

2. (2) Build segments: Include DU _i+1 in segment s_j if DU_i is included in s_j and DU _i+1 contains one or more feature terms.

Fig. 10.

Flow chart for feature extraction algorithm.

Building of segments rests on the notion of feature terms. Since we have no a priori inventory of features (the feature annotation will be used for evaluation, not for development), we use WordNet Fellbaum (Reference Fellbaum1998) to identify these. We consider as feature terms those noun phrases whose synonyms, direct hypernym or direct hyponym, or the definitions of any of these, contain the terms ‘music’ or ‘song’, or any form of these words, such as ‘musical’, ‘songs’, etc, for at least one sense of the noun. Feature terms do not include the words ‘music’ or ‘song’ per se, or the artist/band/album name as these are likely to occur across a whole album review.

We just mentioned that feature terms include only noun phrases. The exclusion of other grammatical phrases rests on the following analysis. We randomly chose ten reviews from the corpus; those ten reviews contained 1,473 noun phrases and seventy-eight features. Of these seventy-eight features, only two (3.4%) are not noun phrases. One of those two is long, in the sentence, ‘Clocking’ is not only long, it is also one of the most difficult songs. Those non-NP features could potentially be identified with natural language understanding techniques, for example, linking an adjective, such as long, to a noun such as length. However, given the heterogeneity of those non-NP features, the inherent limitations of the current state-of-the-art Natural Language Processing techniques and, above all, the small number of non-NP features we found in our data, we only took into account features of a song that are nouns or noun phrases. This is consistent with most of the works that performs explicit feature extraction from product reviews such as Hu and Liu (Reference Hu and Liu2004), Popescu and Etzioni (Reference Popescu and Etzioni2005), Carenini et al. (Reference Carenini, Ng and Pauls2006), Higashinaka et al. Reference Higashinaka, Prasad and Walker2006, and Zhuang et al. (Reference Zhuang, Jing, Zhu and Zhang2006) – work, such as Jin, Ho and Srihari (Reference Jin, Ho and Srihari2009), includes few rules aimed at features not expressed as noun phrases; however, their evaluation does not assess the impact of these rules on their system's performance.Footnote ¹ Likewise, the comments that the user provides on these features, and that we analyze, can only be adjectives or noun phrases.

The choice of limiting feature terms to noun phrases explains why Figure 10 includes a step concerned with part-of-speech (POS) tagging and NP chunking. We use Ramshaw's POS tagger and Noun-Phrase Chunker Ramshaw and Marcus (Reference Ramshaw and Marcus1995).Footnote ² Once feature terms have been identified, sentences that do not contain any feature terms are filtered out. At this point, the algorithm has identified segments of the album review such that each segment pertains to one song, each sentence in that segment contains a feature term, and all feature terms in those sentences are taken to be features of the song described by that segment. Since our interest is in extracting appropriate individual sentences for the summary, segments will not be used further; however, the sentences contained in those segments are then grouped based on the feature term, and on the ID of the song to which the feature term corresponds to. If a sentence contains more than one feature term, it will appear in as many groups as feature terms it contains. Duplicates will be eliminated as a by-product of sentence partitioning, to which we turn now.

4.3 Sentence partitioning and regeneration

After extracting the sentences containing the features, the next step is to divide the sentences into two or more ‘sub-sentences’, if necessary. For example, ‘The slightly awkward wording of the lyrics is endearing, and Tim Reynolds’ guitar is impressive’ discusses both features, lyrics and guitar. Only a portion of the sentence describes the guitar. This sentence can thus be divided into two individual sentences:

1. (1) ‘The slightly awkward wording of the lyrics is endearing’.

2. (2) ‘Tim Reynolds’ guitar is impressive’.

Only the second sentence is relevant to the feature guitar, whereas the first sentence is relevant to the feature lyrics. Removing parts of sentences that describe another feature will have no effect on the summary as a whole as the portions that are removed will be present in the group of sentences that describe the other feature.

To derive n sentences, each concerning a single feature f, from the original sentence that covered n features, we need to

1. (1) identify portions of sentences relevant to each feature f (partitioning);

2. (2) regenerate each portion as an independent sentence, which we call f-sentence.

Figure 11 depicts the entire process of sentence partitioning and regeneration. First, to identify portions of the sentence relevant to the single feature f, we use the Stanford Typed Dependency Parser Klein and Manning (Reference Klein and Manning2002); de Marneffe and Manning (Reference de Marneffe and Manning2008). Typed Dependencies describe grammatical relationships between pairs of words in a sentence. Starting from the feature term f in question, we collect nouns, adjectives, and verbs that are related to it in the sentence, either directly or recursively, but without including relative clause modifier (rcmod) dependencies on modifiers. These nouns, adjectives, and verbs become the components of the new f-sentence.

Fig. 11.

Flowchart for sentence partition and regeneration.

Example: Consider the following sentence: A slow, methodical drum line begins the song, that matter this album strongly.Footnote ³ The typed dependency parse for this sentence is shown in Figure 12. Because of the filtering module, only the dependencies in bold (corresponding to the words [a, slow, methodical, drum, line, begins, the, song]) are identified as components of the f-sentence for drum line – the rcmod dependency on song and all its attendant dependencies are filtered out.

Fig. 12.

Typed dependencies, to be filtered to generate f-sentence.

To generate a grammatical sentence from the words identified as components of the f-sentence, we may need to perform adjustments to their form, for example, to the number of nouns and verbs. This is a natural language generation task, specifically, sentence realization. We use McRoy, Channarukul and Ali (Reference McRoy, Channarukul and Ali2003), a sentence realizer that uses templates augmented with rules to express text structures corresponding to fragments of the target language. In each YAG template, slots represent variables to be filled with values, whereas rules are declarative statements that define how inputs to the template should be realized as text. An example of the instantiated clause template, the root of the top–down realization process, is shown in Figure 13 (the rules that have been fired to fill it are not shown). The dependencies obtained from the Typed Dependency Parser, such as subject and object, identify the slots and the template each slot follows, since slots can be templates themselves – see the subject and dobject slots in Figure 13. We developed a mapping between the Stanford Typed Dependencies and the slots that YAG needs (see Tata Reference Tata2010, Table V). The words in the dependency fill the slot.

Fig. 13.

YAG template to generate ‘A slow methodical drum line begins the song’.

We use a morphological tool Minnen, Carroll and Pearce (Reference Minnen, Carroll and Pearce2000) to obtain the base form from the original verb or noun so that YAG can generate grammatical sentences. YAG regenerates as many f-sentences from the original sentence as many feature terms are contained in it.

By the end of the partitioning and regeneration steps, for each feature f of a certain song s_i, we have generated a set of f-sentences. This set also contains every original sentence that only covered the single feature f. Figure 14 shows the regenerated review from Figure 1.

Fig. 14.

f-sentences corresponding to Figure 1.

4.4 Grouping

f-sentences are further grouped, by sub-feature and polarity. Sub-features are noun phrases that include a feature term, but define either sub-concepts or properties of the concept described by the feature term. Consider the following f-sentences for the feature violin:

1. (a) The violin solo is all well done.

2. (b) It has a nice violin solo by Boyd.

3. (c) The lead violin is noteworthy.

The first and second sentences talk about the violin solo and the third sentence about the lead violin. This step will create two subgroups, with sentence (a) and (b) in one group and sentence (c) in another. We identify subgroups via common n-grams between f-sentences, and make sure that only n-grams that are related to feature f are identified at this stage, as detailed in Figure 15. For two f-sentences, if the Longest Common N-Gram (LCN) is a phrase containing the feature term under consideration, but is not the feature term alone, then the two f-sentences belong to the same subgroup sub^k_f, otherwise they are in different subgroups. If the LCN of an f-sentence with all other f-sentences is the feature term alone, then the f-sentence belongs to the feature f-group.

Fig. 15.

Grouping sentences by sub-features.

When the procedure described in Figure 15 is applied to the three sentences above, step 2 will group (a) and (b) together since violin solo properly contains the feature term violin.

In step 3, the LCN between (c) and all other sentences contains only the feature term violin; hence, sentence (c) is left on its own. Note that steps 2 and 3 ensure that, among all the possible LCNs between pair of sentences, we only consider the ones containing the feature in question.

As concerns polarity grouping, different reviews may express different opinions regarding a particular feature or sub-feature. To generate a coherent summary that mentions conflicting opinions, we need to subdivide f-sentences in each group or subgroup sub _f,i according to polarity.

We use SentiWordNet Esuli and Sebastiani (Reference Esuli and Sebastiani2006), an extension of WordNet where each sense of a word (each synset from WordNet) is augmented with the probability of that sense being positive, negative, or neutral. The overall sentence score is based on the scores of the adjectives contained in the sentence. Although this is a simplification, it is supported by findings in the literature that show that adjectives are considered among the core clues to polarity, even if other types of words also contribute to the perceived polarity of a sentence. First, adjectives correlate with the subjectivity of a sentence (Hatzivassiloglou and McKeown Reference Hatzivassiloglou and McKeown1997; Bruce and Wiebe Reference Bruce and Wiebe1999; Wiebe, Bruce and O'Hara Reference Wiebe, Bruce and O'Hara1999), the first step toward determining its orientation (objective versus subjective, with subjective comprising positive or negative). Second, adjectives strongly contribute to the polarity of a sentence: Esuli and Sebastiani (Reference Esuli and Sebastiani2006) found that adjective and adverb synsets are evaluated as at least partially subjective three times more frequently than verb or name synsets. Esuli and Sebastiani (Reference Esuli and Sebastiani2006) note that [this] seems to indicate that, in natural language, opinionated content is most often carried by parts of speech used as modifiers (i.e., adverbs, adjectives) rather than parts of speech used as heads (i.e., verbs, nouns). Third, the added value for polarity determination of other classes of words beyond adjectives is still a subject of research Benamara et al. (Reference Benamara, Cesarano, Picariello, Reforgiato and Subrahmanian2007); Ding, Liu and Yu (Reference Ding, Liu and Yu2008).

Since there are a number of senses for each word, an adjective a_i in a sentence is scored as a normalized weighted score for each orientation of the adjective (recall that SentiWordNet includes probability for each orientation – neutral, positive, and negative – for each sense of a word in WordNet). For each a_i, we compute three scores, positive, as shown in Formula 1, negative, and objective, which are computed analogously:

(1) \begin{equation} pos(a_i)=\frac{freq_1 \: * \: pos_1+\ldots +freq_n \: * \: pos_n}{(freq_1 +\ldots .+freq_n )} \end{equation}

Fig. 16.

Polarity calculation.

4.5 Selection and ordering

We are now ready to undertake the last step in our summarization pipeline (see Figure 5): selection of the sentences to be included, and ordering them in a coherent fashion. This step has in input groups of f-sentences, where each group pertains to the feature f, one of its sub-features, and one polarity type (positive, negative, and objective). We need to select one sentence from each subgroup to make sure that all essential concepts are included in the summary. Note that if there are contrasting opinions on one feature or sub-feature, one sentence per polarity will be extracted, resulting in potentially inconsistent opinions on that feature to be included in the review. We did not observe this happening frequently, and no user commented on noticing inconsistencies in the reviews (see Section 5.3).

Recall that at this point most f-sentences have been regenerated from portions of original sentences (see Section 4.3). Each f-sentence in a subgroup is assigned a score which is equivalent to the number of features in the original sentence from which the f-sentence was obtained. The sentence which has the lowest score in each subgroup is chosen as the representative for that subgroup. If multiple sentences have the lowest score in a subgroup, one sentence is selected randomly, since they are equivalent from the point of view of which feature they mention, and of their polarity. Our assumption is that among the original sentences, a sentence that talks about one feature only is likely to express a stronger opinion about that feature than a sentence in which other features are present. Figure 17 shows an example list of f-sentences to be ordered to get the final summary.

Fig. 17.

f-sentences to be ordered.

We experimented with three different strategies for ordering the sentences in the summary:

1. (a) Noun-Phrase Centrality (NP-Centrality) Xie (Reference Xie2006); Xie, Di Eugenio and Nelson (Reference Xie, Di Eugenio and Nelson2008)

2. (b) Wordnet Fellbaum (Reference Fellbaum1998)

3. (c) Music Ontology Raimond et al. (Reference Raimond, Giasson, Jacobson, Fazekas, Gängler and Reinhardt2010)

In the development of our approach, we systematically moved from one strategy to the next (in the order given) by informally evaluating the summaries that a strategy would produce, and assessing that the results were unsatisfactory, even without running any user study. Our goal now is to give the reader a glimpse into the kind of summaries generated by each strategy. Hence, in what follows, we will demonstrate those three different strategies by showing how the sentences in Figure 17 are arranged by each strategy. Please keep in mind that only the third strategy was fully developed and used in the evaluation of SongRecommend.

Our first approach to ordering sentences was based on NP-Centrality proposed by Xie (Reference Xie2006) and Xie et al. (Reference Xie, Di Eugenio and Nelson2008) to represent the importance of a noun phrase in a given text. The value of this measure is obtained by applying centrality measures from Social Network analysis Wasserman and Faust (Reference Wasserman and Faust1994); Corman et al. (Reference Corman, Kuhn, McPhee and Dooley2002) on an NP network. The construction of the NP network is done both at the sentence level (if two noun phrases are sequentially parsed in a sentence, a link is added between them) and the document level (a link is added to every pair of noun phrases that are parsed out in succession).

According to Xie (Reference Xie2006), NP-Centrality is a correlate of local coherence. Hence, we expected NP-Centrality to help make the summary coherent. We calculated the NP-Centrality of each noun phrase in the review. The centrality of a noun phrase in all the reviews is the sum of its centralities in individual reviews. The sentences are then ordered in descending order of the NP-Centrality of the feature terms they contain. Figure 18 shows the ordering of the sentences from Figure 17 using NP-Centrality. The summary feels incoherent, for example, the description of the instruments is disconnected.

Fig. 18.

Example summary (version 1): ordered via NP-Centrality.

We then moved to a second ordering approach that uses WordNet Fellbaum (Reference Fellbaum1998), since the WordNet hierarchy has been extensively used to provide and/or to expand lexical items in various Natural Language Processing tasks, including in Natural Language Generation Bangalore and Rambow (Reference Bangalore and Rambow2000); Kleinbauer, Becker and Becker (Reference Kleinbauer, Becker and Becker2007); Mairesse and Walker (Reference Mairesse and Walker2010). Since our features are terms that contain music or song as their synonym or hypernym or hyponym in their definitions, we used this information to group the features, as follows:

1. (1) Hypernym Group: The features that contain music or song in their hypernyms.

2. (2) Synonym Group: The features that contain music or song in their synonyms.

3. (3) Hyponym Group: The features that contain music or song in their hyponyms.

4. (4) Hypernym Definition Group: The features that contain music or song in their hypernym definition.

5. (5) Synonym Definition Group: The features that contain music or song in their synonym definition.

6. (6) Hyponym Definition Group: The features that contain music or song in their hyponym definition.

After the f-sentences have been grouped into the above six groups according to the features they contain (clearly not every group will be present for every song), the summary is built by including the groups in the order given above, namely, f-sentences in the Hypernym Group will be included first, then those in the Synonym Group, and so forth. Figure 19 shows the summary as ordered using WordNet, namely, including sentences in the order of the six groups listed above. We see a slight improvement in that the musical instruments are mentioned together. The ordering of the six groups of features also allows us to provide better formatting (the NP-Centrality measure does not lend itself to natural paragraph breaks). However, the summary still lacks in coherence. Hence, we moved to a third ordering approach that exploits a music ontology (Raimond et al. Reference Raimond, Giasson, Jacobson, Fazekas, Gängler and Reinhardt2010). We first extended this music ontology to include few additional concepts that correspond to features identified in our corpus such as chorus, vocals. Also, we extended each of the classes by adding the domain to which it belongs. We identified a total of twenty different domains for all the features and defined a specific order of these domains in order to generate a coherent summary. Table 1 shows the list of domains with examples taken from the ontology – note that not all of these are identified as features by our algorithm. f-sentences are grouped and ordered according to the domain of the features, in the order shown in Table 1. The ordering of the domains is such that first we present the general features of the song (e.g., Song) domain, then present more specific domains (e.g., Sound, Instrument). f-sentences of a single domain form one paragraph in the final summary. However, in case feature domains that are considered as sub-domains of another domain appear in the same paragraph, they are ordered next to the features of the parent domain. In Table 1, sub-domains are represented with an *.

Fig. 19.

Example summary (version 2): ordered using WordNet.

Table 1.

Domains of the music ontology

Figure 20 shows the final version of our sample summary when the extracted sentences are ordered via this method.

Fig. 20.

Example summary (final version): ordered via music ontology.

5.1 Song title and feature extraction

The song title and feature extraction algorithms (Sections 4.1 and 4.2) were manually evaluated on 100 reviews randomly taken from the corpus (2 or 3 reviews for each album). This relatively small number is due to the need to conduct the evaluation manually. The 100 reviews contained 1,304 occurrences of song titles and 898 occurrences of song features, as previously annotated.

In all 1,294 occurrences of song titles were correctly identified; in addition, 123 spurious occurrences were also identified. This results in a precision of 91.3 percent, and recall of 99.2 percent. The ten occurrences that were not identified contained multiple spelling mistakes in the same word – this case also covers abbreviations such as Dr. for Doctor. Recall that we do not clean up mistakes in the reviews, and filter out words that have edit distance greater than one from words in titles.

Most of the additionally identified occurrences follow all the criteria of being a song title, as discussed earlier in Section 4.1. However, they are not song titles when the context is taken into consideration. For example, our algorithm recognizes both occurrences of Are You Experienced? as song titles in sentences (a) and (b):

1. (a) ‘Are You Experienced?’ is an excellent pop album.

2. (b) The lyrics of the track ‘Are you Experienced?’ are wonderfully written.

Of the 898 occurrences of song features, 853 were correctly identified by our feature extraction algorithm, with an additional forty-one spurious occurrences. This results in a precision of 95.4 percent and a recall of 94.9 percent. Note that a feature is considered as correctly identified if its head noun is annotated in a review for the song with correct ID. We speculate that both spurious and non-identified occurrences may be due to

1. (1) TextTiling: As discussed in Section 4.2, we use the discourse units returned by TextTiling to indirectly identify if a feature term is attributed to a song and do not perform actual reference resolution.

2. (2) WordNet: Since feature terms are extracted with the help of WordNet, some terms are wrongly identified as features, and some terms are not identified.

Our extraction results for both song titles and features are summarized in Table 2.

Table 2.

Extraction results (100 reviews)

As a baseline comparison, we implemented the feature extraction algorithm from Hu and Liu (Reference Hu and Liu2004).Footnote ⁶ We compared their algorithm with ours on ten randomly chosen reviews from our corpus, for a total of about 500 sentences. Its accuracy (40.8 percent precision, and 64.5 percent recall) is much lower than ours. For a fairer comparison, we incorporated TextTiling and our heuristics into the Hu and Liu algorithm (Reference Hu and Liu2004), increasing its precision to 56.04 percent but with no effect on recall. On this subset of the data, our algorithm still performs much better (93.2 percent precision and 91.5 percent recall) – see Table 3. In addition, the performance of the algorithm from Hu and Liu (Reference Hu and Liu2004) on our data is much lower than their original results on product reviews (72 percent precision, and 80 percent recall). We believe that this is due to the different nature of song reviews with respect to product reviews, as we discussed in Section 2.

Table 3.

Hu and Liu (Reference Hu and Liu2004) versus SongRecommend (feature extraction, ten reviews)

5.2 Informativeness of the summaries

Having evaluated the extraction components of our work, we set out to evaluate the summaries in their entirety. In the last fifteen years, summary evaluation has been a hot topic of research Mani and Maybury (Reference Mani and Maybury1999). Intrinsic evaluation of summaries generally concerns quality (how the summary reads) and informativeness. Quality is mostly evaluated by asking subjects to score the summaries (our user study will touch on judging the quality of the summaries; see Section 5.3). Informativeness can either concern how much the source information is preserved at various compression ratios or how much the summary is similar to an ideal output. For the latter kind of evaluation, a completely automatic method, called ROUGE Lin (Reference Lin and Marie-Francine Moens2004), uses n-gram recall between a candidate summary and a set of reference summaries. In contrast, the Pyramid method Nenkova and Passonneau (Reference Nenkova and Passonneau2004); Harnly et al. (Reference Harnly, Nenkova, Passonneau and Rambow2005) focuses on semantic coverage, but calls for manual annotation of Summary Content Units (SCUs) in reference summaries. Those SCUs are arranged in a pyramid of tiers of importance. The informativeness of a generated summary is then evaluated in terms of its coverage of SCUs, which are partially ordered according to their importance, and in relation to the length of the summary.

ROUGE and Pyramid sit at the two ends of a spectrum as concerns evaluation measures: ROUGE is rather coarse, since it is based on n-gram recall, whereas Pyramid is much more semantic oriented; however, ROUGE does not require further extensive manual annotation as Pyramid does. Both measures, though, rest on the availability of a corpus of reference summaries, which does not exist for our task. Whereas manual creation of corpora of reference summaries has been undertaken, it generally happens with strong institutional support: e.g., the DUC competitions in the United States are backed by the Advanced Research and Development Activity (ARDA), and are run by the National Institute of Standards and Technology (NIST). Note that in our case, we would need to generate at least one reference summary per song in our twenty-seven albums, for a total of at least 391 summaries (our twenty-seven albums contain 391 songs altogether, see Section 4).

Hence, we decided to evaluate the informativeness of our summaries in an indirect way, by comparing the information they provide to that included in high-quality reviews as those written by professional critics. We chose two or three critic reviews per album taken from www.pitchfork.com, www.rollingstone.com, and www.allmusic.com. Please note, similar to the reviews in our corpus, these critic reviews discuss the whole album, hence we do not have high-quality reviews for each individual song: if they existed, we could use critic reviews of each individual song as the gold standard to compare our summaries against. However, given the nature of the critic reviews, we assume they cover important features of the songs in those albums, and hence we use them for our indirect evaluation of informativenes.

We randomly selected ten albums, and from each album we selected five songs, and generated the corresponding fifty summaries. Summaries for two additional songs were added, since they had already been generated for other reasons. This resulted in a test set of fifty-two summaries, which corresponds to summarizing about 600 album reviews, since to create each of these fifty-two summaries, fifty reviews are summarized (recall that each album has fifty reviews, see Section 4).

We manually annotated both critic reviews and the automatically generated summaries for song titles and song features. Altogether 302, i.e., 91.2 percent of the features identified in the critic reviews are also identified in the summaries (recall that a feature is considered as identified if the head-noun of the noun phrase is identified by both the critic review and the summary, and attributed to the same song). Sixty-four additional features were included in the generated summaries, for a recall of 82 percent. It is not surprising that additional features may appear in the summaries: even if only one of the fifty album reviews talks about that feature, it is included in the summary. Potentially, a threshold on frequency of feature mention could increase recall, but we found out that even a threshold of two significantly affects precision. In addition, we believe that in this case precision should be more important than recall, since the gold standard texts we are using are not summaries tailored to specific songs.

5.3 Between-Subjects user study

Our intrinsic evaluation gives satisfactory results. However, we believe that the ultimate measure of such a summarization algorithm is an end-to-end evaluation to ascertain whether it affects user behavior, and how. We conducted a between-subjects user study, where users were presented with two different versions of our Music RS. For each of the recommended songs, the baseline version provides only whole album reviews, the experimental version provides the automatically generated song feature summaries, as shown earlier in the paper in Figure 4. The interface for the baseline version is similar, but the summary in the bottom window is replaced by the corresponding album review. The presented review is the one among the fifty reviews for that album whose length is closest to the average length of album reviews in the corpus (478 words; see Figure 6).

Each user was presented with five songs in succession, with three recommendations each (only the top three recommendations were presented among the available twenty-five; see Section 3). Users were asked to select at least one recommendation for each song, namely, to click on the URL where they can listen to the song. They were also asked to base their selection on the information provided by the interface. The first song was a test song for users to get acquainted with the system. We collected comprehensive timed logs of the user actions, including clicks, when windows were open and closed etc. After using the system, users were administered a brief questionnaire, which included questions on a 5-point Likert scale. Eighteen users interacted with the baseline version and twenty-one with the experimental version. All users were students at our university. About 85 percent of them were graduate students, about 3/4 were males, and about 2/3 listened to music daily.

Our main measure is time-on-task, the total time taken to select the recommendations from song 2 to song 5 – this excludes the time spent listening to the songs. A t-test showed that users in the experimental version took less time to make their decision when compared with baseline subjects (p=0.019, t=2.510). We also examined differences in time-on-task due to gender, previous knowledge of music, or education level (via t-test or one way ANOVA). We found that none of these factors affected time-on-task (this lack of dependence should be considered as tentative, since some of the groups, e.g., undergraduate students, are very small).

We consider the decrease in time to selection as a positive result, given that music collections can include millions of songs. However, time-on-task represents the time it takes users to peruse the review or summary, which in turn is affected by the length of the review or summary: the number of words in the summaries is significantly lower than the number of words in the reviews (p<0.001, t=16.517).

Hence, we also analyzed the influence of summaries on decision-making to see if they have any effects beyond cutting down on the number of words to read. Our assumption is that the default choice is to choose the first recommendation. Users in the baseline condition picked the first recommendation as often as the other two recommendations combined; users in the experimental condition picked the second and third recommendations more often than the first one, and the difference between the two conditions is significant (χ²=8.74, degrees of freedom (df)=1, p=0.003). If we examine behavior song by song, this holds true especially for song 3 (χ²=12.3, df=1, p<0.001) and song 4 (χ²=5.08, df=1, p=0.024). We speculate that users in the experimental condition are more discriminatory in their choices, because important features of the recommended songs are evident in the summaries but are buried in the album reviews. In particular, negative opinions are much more evident in the review summaries.

For example, Figure 21 shows the song ‘Sunday Bloody Sunday’ from the album War by U2 (first row in the figure) and the recommendations presented to the user (rows 2 through 4): ‘Surrender’ from the same album, and ‘Pride’ and ‘Bad’, both from Unforgettable Fire, another album by U2. On the right, we show the summaries generated by SongRecommend. On the left, we include the relevant portions of the corresponding album reviews, and indicate how many words occur in the album review before the portion on the song in question, and how many occur afterwards (note that users were presented with the whole album review, not only with the excerpt that we highlight here). As the reader may notice, the same album review is shown for both ‘Pride’ and ‘Bad’; this derives from our choice of showing the album review closest to the average length of the album reviews in our corpus. In this specific case, since both ‘Bad’ and ‘Pride’ are mentioned in the same sentence, the specific excerpts that we highlight are the same as well.

Fig. 21.

Users’ choices between album reviews (left), and review summaries (right).

The middle column in Figure 21 (rows 2–4) shows how many users in the baseline and experimental conditions chose that specific recommendation. Thirteen users in the baseline condition picked the first recommendation (‘Surrender’), while only three users picked it in the experimental version. These numbers are reversed for the second recommendation (‘Pride’, chosen by four users in baseline condition but by twelve in experimental condition); and for the third recommendation (‘Bad’, six users in baseline condition but thirteen in experimental condition). This example also shows that whereas users need to wade through a potentially high number of words in the album reviews before getting to the song of interest, the number of words in the summaries is much lower (31, 7, 41, 36 words respectively). Also, for each recommended song, only a few lines talk about the actual song and those lines may be buried deep inside the album review. On the other hand, the summaries make it apparent which features are shared between the original song (‘Sunday Bloody Sunday’) and the recommended songs: e.g., the political theme and the word anthem (for ‘Pride’), and Larry Mullen (for ‘Bad’).

Since the summaries are so much shorter than the original reviews, and length affected time-on-task in the between-subject comparison, a natural question arises: Does the relative length of different summaries affect users’ decisions? In other words, could users simply choose the song associated with the shortest summary? The answer is no.

We categorized three recommendations for each song as longest, medium, and shortest based on the length of the review summary. Table 4 shows how many times a summary of relative length was chosen – recall that there are twenty-one users and four songs for which to choose one or more recommendations. The table shows that users chose the shortest summary less frequently, and a χ² test confirms that this difference is significant (p = 0.02, χ²=7.77, df = 2).Footnote ⁷ This indicates that users do not merely follow a recommendation because it has the shortest summary.

Table 4.

Length of summary versus number of selections

Usability measures. As mentioned, after using the system, users were administered a brief questionnaire which included questions on a 5-point Likert scale. The questionnaires included three common questions between the two conditions, which differed only as concerns the item being asked about:

1. (1) Did you feel that the album review/summary helped you quickly decide if you want to listen to that song?

2. (2) Did you feel that the album review/summary is of decent length?

3. (3) Did you feel that the album review/summary has too much information about the song?

The experimental subjects gave a more positive assessment of the length of the summary than the baseline subjects of the length of the album review (question (2), p=0.003, t=−3.248, df=31.928). There were no significant differences on the other two questions.

A multiple Linear Regression with time-on-task as dependent variable and as predictors, the number of words in the album review/review summary, and the three questions they all answered, revealed only one, not surprising, correlation: the number of words the user read correlates with time-on-task (R ²=0.277, β=0.509, p=0.004). This correlation is not surprising, since we had already established that there is a significant difference between the number of words in the original reviews and in our summaries.

Users in the experimental version were also asked to rate the grammaticality and coherence of the summary (together with the question on length, these assessments can be considered as an intrinsic evaluation of the quality of the summary, as mentioned earlier). The two questions users were posed are: ‘Did you feel that the summary is grammatical?’ and ‘Did you feel that the overall flow of the summary is good?’ The average rating was 3.33 for grammaticality, and 3.14 for coherence. Whereas these numbers in isolation are not too telling, they are at least suggestive that users did not find these summaries badly written. We found no significant correlations between grammaticality and coherence of summaries and time-on-task.

Finally, users could add open comments. Whereas the numbers are too small to draw other than tentative conclusions, we note that twelve of the baseline users commented that they would have liked the Music RS better if they had been presented with information specific to the song; and ten of the experimental users commented that the song review summary was helpful to their decision-making process.