1 Introduction
The purpose of this paper is to review the performance of major influential opinion retrieval techniques on different datasets. The paper shows how opinion retrieval techniques have propagated from the foremost techniques to the recent ones. It enables easy identification of performance and limitations with respect to different techniques.
Since the research on opinion retrieval is relatively new, we identified state-of-the-art opinion retrieval techniques on the Association for Computing Machinery (ACM) portal and Google Scholar in early 2011. Identified techniques were then classified under text classification approach, lexicon-based approach, probabilistic approach, and other emerging approaches. The text classification and lexicon-based approaches have been the two conventional approaches discussed in the literature on opinion retrieval. Recently, probabilistic and other new approaches emerged and have also become popular in the literature. For all the techniques identified, we did not only consider published works that conducted the original opinion retrieval experiments, but recent papers that show the significance of the techniques.
Recently, some review papers have discussed opinion retrieval to a certain extent. However, these papers have only been successful in describing the theoretical foundations of opinion retrieval and the potential applications of opinion retrieval to wide range of natural language problems. For example, a survey paper on opinion mining and sentiment analysis was written by Pang and Lee (Reference Pang and Lee2008). The paper discusses fundamental knowledge on major opinion retrieval and summarization approaches. Another survey paper was written by Liu (Reference Liu2010). The paper introduces a theoretical description for the concepts and approaches in sentiment analysis and subjectivity. In fact, the main idea was to facilitate learning and teaching of opinion retrieval in an academic environment.
In contrast to the review papers written by Pang and Lee (Reference Pang and Lee2008) and Liu (Reference Liu2010), our purpose is to discuss the performance, challenges, and limitations of recent opinion retrieval techniques evaluated on different datasets. For this purpose, we provide performance summary tables for leading techniques under major opinion retrieval categories. We believe this will aid decision on suitable and appropriate opinion retrieval technique to be considered for further research and development tasks or actual implementation for commercial applications.
The organization of this paper is as follows: the remaining part of Section 1 discusses the importance of opinion retrieval, its trends in the global research community, and overview of existing opinion retrieval approaches. Sections 2, 3, 4, and 5 discuss text classification, lexicon-based, probabilistic, and other emerging opinion retrieval approaches, respectively. In each section, we give an overview of the approach supported by a component diagram. We show an example state-of-the-art work with a description of the algorithm, its performance, and limitations. We then present a performance comparison table for techniques proposed on different datasets. The table also shows the dataset used for the experiments, selected evaluation metrics, and the performance results. Section 6 discusses future research directions, and Section 7 draws summary and conclusions.
1.1 Opinion retrieval and its importance
For the purpose of this review, we define opinion retrieval as a systematic and computational process that identifies subjective information from human-generated textual contents in order to retrieve textual information that contains opinion. Further inference on opinion retrieval describes extraction and judgement analysis on various aspects of opinionated contents (Pang & Lee, Reference Pang and Lee2008; Liu, Reference Liu2010). However, the arrival of web 2.0 interactive media such as web logs (blogs) and other Social Network sites creates an avenue whereby humans show their diversities in terms of writing styles. When people express opinions on blogs, for example, they write with self-perception often conveyed by different psychological inferences; they present opinions with different languages and styles, and even play upon words. These challenges among many have made opinion retrieval task difficult. It has also led to active research on opinion retrieval, and had since recorded a huge number of research works since early 2000 (Pang et al., Reference Pang, Lee and Vaithyanathan2002; Nasukawa & Yi, Reference Nasukawa and Yi2003; Hu & Liu, Reference Hu and Liu2004; Pang & Lee, Reference Pang and Lee2004; Liu et al., Reference Liu, Hu and Cheng2005; Wilson et al., Reference Wilson, Hoffmann, Somasundaran, Kessler, Wiebe, Choi, Cardie, Riloff and Patwardhan2005; Eguchi & Lavrenko, Reference Eguchi and Lavrenko2006; Zhang et al., Reference Zhang, Yu and Meng2007b; Abbasi et al., Reference Abbasi, Chen and Salem2008; Ding et al., Reference Ding, Liu and Yu2008; Gerani et al., Reference Gerani, Carman and Crestani2009). Some of the challenges highlighted above can be attributed to the different ways of expressing opinions or sentiments by humans. This is common to complex interactive opinion-driven sources such as collection of blogs called blogosphere (e.g. Google Blogs SearchFootnote 1 and Technorati Blog DirectoryFootnote 2) and some popular review Websites such as reviewcenterFootnote 3.
Interestingly, there is huge need and growing demand for mining different kinds of knowledge from opinion-driven sources. For example, organizations wish to know relevant and qualitative opinions toward products or services rendered (Liu, Reference Liu2010), individuals wish to retrieve relevant information for personal use (Lee et al., Reference Lee, Lee, Song and Rim2010), and the need for understanding human behaviors through trends at real-time is also becoming more apparent and necessary (Munson & Resnick, Reference Munson and Resnick2010; Zafarani et al., Reference Zafarani, Cole and Huan2010). The New York TimesFootnote 4 published an article in late 2009 relating the emergence of opinion mining or sentiment analysis to translating human emotion from social media (e.g. blogs) into hard data. Such data can be used to improve the quality of a product; enhance efficiency and productivity of a company; and understand political demands of the masses. Other important usage of opinion retrieval include, analysis of questionnaire or survey responses, personalized search engines, opinion or sentiment analytic, and understanding consumers sentiments from products or movie reviews.
1.2 Global trend in research community on opinion retrieval
Text REtrieval Conference (TREC) has been a leading research community on opinion retrieval techniques (Ounis et al., Reference Ounis, Macdonald and Soboroff2008; Macdonald et al., Reference Macdonald, Ounis and Soboroff2009). TREC introduced the Blog Track in 2006 at the University of Glasgow, United Kingdom. Majority of the proposed opinion retrieval techniques are evaluated on TREC blog datasets using the standard TREC blog baselines. The TREC blog dataset contains collection of blog posts from 2006 to 2008. The TREC baselines are usually selected as the best techniques in each year's TREC blog track participation on opinion retrieval task, which is performed on the standard TREC blog dataset.
Alongside opinion retrieval task, the TREC blog track also focuses on other research tasks. These include splogs (spam blogs) detection and/or spam comments detection from blogs, blog distillation, and top stories/news identification. Recent TREC blog tasks include faceted blog distillation (a blog feed search task) and top stories identification. Other than TREC blog datasets, some techniques have evaluated their performance on other domain-specific datasets such as Internet Movie Database (IMDb) dataset for retrieving sentiments from movie reviews, AmazonFootnote 5 product reviews for retrieving sentiments on consumer products, and UsenetFootnote 6 newsgroups for retrieving sentiments from news. For each opinion retrieval approach, we will show performance comparison on TREC blog datasets alongside other domain-specific datasets.
The selection of dataset for opinion retrieval task may vary according the proposed technique or a domain of interest. However, it is advisable that any proposed opinion retrieval technique should be evaluated on a standard dataset whereby the proposed technique can be easily compared with the available standard baselines.
Finally, performance evaluation of opinion retrieval techniques involves the use of standard information retrieval (IR) evaluation metrics such as Mean Average Precision (MAP) for measuring retrieval precision over a set of queries, precision at K (e.g. P@5 and P@10) for measuring retrieval precision at a fixed level of results, relevance precision (R-Precision) for measuring the best precision from a set of relevant documents, and binary preference (Bpref) that considers non-relevant and relevant documents only from a set of judged documents from the retrieved results. The choice of evaluation metrics may, however, vary according to the proposed opinion retrieval techniques or according to the selected baselines. In any case, the selected evaluation metric must be part of the standard IR evaluation metrics. Researchers are therefore referred to background information on traditional IR evaluation metrics (Manning et al., Reference Manning, Raghavan and Schtze2009).
1.3 Overview of existing opinion retrieval techniques
Opinion retrieval techniques can be broadly classified into four categories as shown in Figure 1.
Categories of opinion retrieval approaches
Early research works treated opinion retrieval as text classification (TC) problem thereby using machine learning (ML) approach to classify text in documents that contain opinions. The TC technique enables the introduction of both traditional supervised and unsupervised ML techniques with reasonable efficiencies recorded in most research works (Ounis et al., Reference Ounis, Macdonald and Soboroff2008). Lexicon-based approach was introduced with the aim that it can compensate for the inefficiencies of the TC techniques, and many research works have combined both lexicon-based and TC techniques for effective opinion retrieval (Lee et al., Reference Lee, Na, Kim, Nam, Jung and Lee2008; Zhang & Ye, Reference Zhang and Ye2008; Bollen et al., Reference Bollen, Pepe and Mao2010). Probabilistic approaches have also been used to retrieve and rank opinions from documents using statistical inferences (Gerani et al., Reference Gerani, Carman and Crestani2010). The idea is to automatically detect large collection of documents that contain opinion without any learning technique. This technique has performed reasonably in ranking opinionated documents from a huge collection. Finally, because it was difficult for the three existing approaches to outperform the standard TREC baselines, other emerging opinion retrieval approaches have been proposed with the hope of outperforming the standard baselines (Du & Tan, Reference Du and Tan2009a, Reference Du and Tan2009b; Thet et al., Reference Thet, Na, Khoo and Shakthikumar2009). Some of the approaches are either completely independent of the three main approaches highlighted earlier or combine some of the main approaches with other newly formed techniques. In the following sections, we will describe the four categories in detail.
2 Text classification approach
2.1 Overview of the approach
The TC approach for opinion retrieval is generally termed sentiment classification (Pang & Lee, Reference Pang and Lee2008; Liu, Reference Liu2010). In this case, a classifier is built based on the occurrences of opinion words and linguistic features (semantic and syntactic) to identify if content from a particular document is subjective or not (Jia et al., Reference Jia, Yu and Zhang2008; O'Hare et al., Reference O'Hare, Davy, Bermingham, Ferguson, Sheridan, Gurrin and Smeaton2009; Siersdorfer et al., Reference Siersdorfer, Chelaru and Pedro2010). For example, comments to political news can be classified as either ‘positive’ or ‘negative’ having trained an opinion retrieval system with both positive and negative words. However, the TC approach to opinion or sentiment classification differs from the generic topic-based Web or document classification. In the generic topic-based classification, the amount of query words present in the retrieved documents is important. On the contrary, in opinion or sentiment classification, the amount of subjective words in the retrieved documents is important. Sentiment classification appears to be frequently studied by using different methods within the traditional TC approach. This has attracted a huge number of research works recently (Zhang & Yu, Reference Zhang and Yu2006; Zhang & Zhang, Reference Zhang and Zhang2006; Joshi et al., Reference Joshi, Bayrak and Xu2006; Jia et al., Reference Jia, Yu and Zhang2008; Gerani et al., Reference Gerani, Carman and Crestani2009; Jin et al., Reference Jin, Ho and Srihari2009; Kobayakawa et al., Reference Kobayakawa, Kumano, Tanaka, Okasaki, Kim and Tsujii2009; O'Hare et al., Reference O'Hare, Davy, Bermingham, Ferguson, Sheridan, Gurrin and Smeaton2009; Taboada et al., Reference Taboada, Brooke and Stede2009; Siersdorfer et al., Reference Siersdorfer, Chelaru and Pedro2010). We will discuss sentiment classification and its sub-classes as shown in Figure 2. We will also identify limitations in using the TC approach for opinion or sentiment retrieval tasks, hence propagating the need for other opinion retrieval approaches.
Structural overview of sentiment classification approach
Sentiment classification include document-level sentiment classification and sentence-level sentiment classification (Lin et al., Reference Lin, Wilson, Wiebe and Hauptmann2006). The two classification types differ by input to the respective algorithms. In document-level classification, a document (input) is classified to differentiate between the polarities contained in the document (e.g. positive, negative, and objective). In sentence-level classification, each sentence (input) in a given document is analyzed to show subjectivity or objectivity. We will discuss the two techniques in detail in the following section.
2.2 Document-level classification for opinion retrieval
In document-level classification, documents in a given collection are classified as either ‘positive’ or ‘negative’ or in some cases ‘objective’ with the assumption that the documents contain opinion (Pang et al., Reference Pang, Lee and Vaithyanathan2002; Liu, Reference Liu2010). However, this particular type of classification may only be applicable to documents that focus on a unique or independent discussion, for example, news review (Koppel & Shtrimberg, Reference Koppel and Shtrimberg2006). In this case, the opinion retrieval system is able to classify the news (an entire document) as ‘positive’ or ‘negative’ or ‘objective’. In some cases, some domain-dependent classification may require documents to be classified as ‘bad’ or ‘good’ (Koppel & Shtrimberg, Reference Koppel and Shtrimberg2006; Lin et al., Reference Lin, Wilson, Wiebe and Hauptmann2006). Also, like the traditional topic-based text classification (Baharudin et al., Reference Baharudin, Lee and Khan2010), sentiment classification can be categorized as using supervised and unsupervised methods (Liu, Reference Liu2010). We will outline some opinion retrieval techniques that fall under these methods and highlight their performance and limitations on different datasets.
2.2.1 Supervised document-level sentiment classification
This is similar to the traditional topical text classification. Supervised document-level sentiment classification can be achieved by training and testing documents with focused opinions (Liu, Reference Liu2010). For example, product review with thumbs-up and thumbs-down (Siersdorfer et al., Reference Siersdorfer, Chelaru and Pedro2010), or star ratings (Lim et al., Reference Lim, Nguyen, Jindal, Liu and Lauw2010). In contrast to understanding topical orientation in text classification, opinions or sentiments about the document are derived from opinionated features such as good, beautiful, notorious among others (Nguyen et al., Reference Nguyen, Phung, Adams, Tran and Venkatesh2010). A description of features that can be used for effective document-level classification can be found in Pang & Lee (Reference Pang and Lee2008) and Liu (Reference Liu2010). Also, research works such as Pang et al. (Reference Pang, Lee and Vaithyanathan2002), Lin et al. (Reference Lin, Wilson, Wiebe and Hauptmann2006), Zhang and Zhang (Reference Zhang and Zhang2006), Jia et al. (Reference Jia, Yu and Zhang2008), Lee et al. (Reference Lee, Na, Kim, Nam, Jung and Lee2008), Jin et al. (Reference Jin, Ho and Srihari2009), Kobayakawa et al. (Reference Kobayakawa, Kumano, Tanaka, Okasaki, Kim and Tsujii2009), O'Hare et al. (Reference O'Hare, Davy, Bermingham, Ferguson, Sheridan, Gurrin and Smeaton2009), Siersdorfer et al. (Reference Siersdorfer, Chelaru and Pedro2010), and Nguyen et al. (Reference Nguyen, Phung, Adams, Tran and Venkatesh2010) have applied classification algorithms such as support vector machines (SVM) (Sun et al., Reference Sun, Lim and Ng2002) and Nave Bayes (Zhang et al., Reference Zhang, Xue, Yu and Zha2009) for document-level sentiment classification. The results of these works show classification using SVM gave best performance in most of the experiments. We will discuss an example state-of-the-art supervised technique for opinion retrieval (Figure 3).
Component architecture for supervised document-level opinion/sentiment classification
2.2.2 Example work (supervised document-level opinion/sentiment classification)
Jia et al. (Reference Jia, Yu and Zhang2008)
In this work, a four-step procedure was used for document-level opinion retrieval task at TREC blog track 2008. In the first step, documents that are relevant to the given query are selected based on term or concept similarity. In the second step, common abbreviations are identified to enhance the efficiency of the opinion retrieval technique. In the third step, SVM is used to classify relevant opinions from sentences. Finally, documents are scored and ranked based on the degree of relevance shown by the previous steps. Figure 4 describes the flow of the algorithm.
Flow chart describing the algorithm in Jia et al. (Reference Jia, Yu and Zhang2008)
The work is an improvement on Zhang and Zhang (Reference Zhang and Zhang2006), which was proposed to solve sentence-level opinion retrieval problem. The concept of NEAR operator was introduced to determine the degree at which subjective sentences are relevant to the given query. The NEAR operator technique simply checks whether proximity of query terms to each subjective sentence occur at a five-sentence window. In the end, the overall opinion polarity of a document is determined based on the degree of relevance shown by using the NEAR operator. The result of their experiment recorded 0.4461 and 0.4473 MAPs with a slight improvement of 0.01 and 0.02 over the given baseline. It also recorded 0.4822 and 0.4822 R-Precisions in two different runs, respectively.
Discussion
Despite its significance in the opinion research community, the performance result of the above work shows that it is very unlikely that learning classifiers to retrieve opinions would outperform baselines significantly. A problem with sentiment classification task is the identification of subjective sentences according to the domain for the purpose of training the classifiers. This is still an active area of research. For example, in the iPhone domain, a subjective sentence can read ‘The iphone lacks some basic features’, whereas in the movie domain, a subjective sentence can read ‘The movie is not so interesting’. If a classifier was trained on the iphone domain, such classifiers would have low performance on movie domain as movies cannot lack basic features. Therefore, for multiple domain opinion retrieval, it becomes inappropriate to train classifiers based on the subjective expressions from a single domain. Another problem we observed is the use of word-concept relationship to identify subjective sentences. There is need to establish whether such words must occur independently without any dependency on other words in proximity or in the original sentence. For example, in the sentence ‘she is amazingly beautiful’, the word beautiful in this context depends on the word amazingly in order to determine the degree of subjectivity of the sentence. However, we think the word beautiful and amazingly have been considered independently.
2.2.3 Unsupervised document-level sentiment classification
Another approach to sentiment or opinion classification is automatic document-level classification (Lin et al., Reference Lin, Wilson, Wiebe and Hauptmann2006). The literature generally refers to it as unsupervised sentiment classification (Pang & Lee, Reference Pang and Lee2008; Liu, Reference Liu2010). Unlike the supervised approach, the unsupervised approach does not require manual labeling of documents for training, thus it is not labor intensive. Documents that contain opinions are systematically and automatically categorized according to the degree of subjectivity. Recently, some techniques have combined topic model with sentiment model to form a joint sentiment/topic model. For example, Lin and He (Reference Lin and He2009) proposed a joint sentiment/topic model (JST) by adding a sentiment layer to the popular state-of-the-art topic model, Latent Dirichlet Allocation (LDA; Blei et al., Reference Blei, Ng and Jordan2003). The idea is to detect both sentiments and topic simultaneously from documents unlike the supervised techniques that detect both topic and sentiment as a two-stage process. JST is fully unsupervised and without explicit knowledge of labeled sentiment classes. The approach has similar performance with a popular sentiment classification technique (Pang et al., Reference Pang, Lee and Vaithyanathan2002). Other similar sentiment classification approaches using topic model include Lu et al. (Reference Lu, Ott, Cardie and Tsou2011) and Paul and Girju (Reference Paul and Girju2010). Both techniques detect multiple aspects of sentiments and topics using various topic models, including LDA.
A technique called lexicon creation has also been used for unsupervised document-level sentiment classification. A lexicon contains collection of words regularly used in a particular domain, thus lexicons are created for different opinion domains. Some lexicons are created based on different categories of state of feelings such as emotion, thoughts, and expressions. An example is the Linguistic Inquiry and Word Count (LIWC), which assign words into psychologically meaningful categories (Tausczik & Pennebaker, Reference Tausczik and Pennebaker2010). LIWC was originally developed to identify positive and negative emotion words within documents. It later considered identifying categories of words that show self-reflections and simple opinions. Some lexicons are also created to determine opinion or sentiment polarity of words or sentences. An example is SentiWordNet (Esuli & Sebastiani, Reference Esuli and Sebastiani2006). SentiWordNet determines positive, negative, and neutral values for words. Such values are summed over words in a given sentence to determine the polarity of the sentence or its subjective orientation (Esuli, Reference Esuli2008). We will discuss a top performing opinion retrieval technique, which use unsupervised approach for document-level classification (Figure 5).
Component architecture for unsupervised document-level opinion/sentiment classification
2.2.4 Example work (unsupervised document-level opinion/sentiment classification)
Tausczik & Pennebaker (Reference Tausczik and Pennebaker2010)
The work uses LIWC to empirically show if a certain word or a subject of discussion belongs to an opinion category. For example, consistent use of first person personal pronouns such as ‘I’ and ‘Me’ emphasizes on a subject of attention in the given sentence. Also, the use of positive emotion words such as like, interesting and good are used in writing about positive events. Therefore, once each word in the document is identified with a category, the percentage of each category is calculated and identified with a psychometric fact gathered through empirical study, such that the leading category describes the likely opinion contained in the document (Figure 6).
Component description of using LIWC for unsupervised opinion detection technique. LIWC = linguistic inquiry and word count
For example, Chung et al. (Reference Chung, Jones, Liu and Pennebaker2008) used LIWC to discover the level at which individual word or phrase can predict success and failure of weight loss as mentioned in blogs. The LIWC technique claimed significant success at predicting weight loss success by higher percentage of sadness words within the blog.
Discussion
The LIWC technique is dictionary-based and it requires human prior knowledge or assumption for respective sentiment categories that can be associated with each given word. The prior knowledge of the human judges also depends on their perceptions or experiences with the use of such words, which can vary greatly among humans. A major limitation observed in this technique is the use of word counts to represent opinion without considering the grammatical context at which each word had occurred (Krahmer, Reference Krahmer2010). In Chung et al. (Reference Chung, Jones, Liu and Pennebaker2008), it is unclear whether individual words or phrase matched with ‘sadness’ lexicon category had specifically occurred in sentences particular about weight loss or failure. For example, these two sentences, ‘the fight for academic success’ and ‘I will fight you to finish’, have regular occurrence of the word ‘fight’, which may imply violence as an opinion target after a certain frequency threshold, whereas, the word ‘fight’ has appeared in two different contexts, respectively. That is, the sentence ‘the fight for academic success’ may imply ‘passion for academic excellence’, and the sentence ‘I will fight you to finish’ may imply ‘violence’.
Therefore, for a more effective unsupervised opinion or sentiment retrieval, we suggest context-driven subjectivity (subjective words must occur in the context wanted by the user). This can be achieved by using natural language processing or grammatical analysis to differentiate between the orientations of expressions. A good example is demonstrated in Hatzivassiloglou and McKeown (Reference Hatzivassiloglou and McKeown1997) and similar work is also presented in Turney and Littman (Reference Turney and Littman2002).
2.2.5 Research issues in supervised and unsupervised document-level classification
The contributions of document-level classification toward opinion retrieval have recorded some level of success (Turney & Littman, Reference Turney and Littman2002; Jia et al., Reference Jia, Yu and Zhang2008; O'Hare et al., Reference O'Hare, Davy, Bermingham, Ferguson, Sheridan, Gurrin and Smeaton2009; Siersdorfer et al., Reference Siersdorfer, Chelaru and Pedro2010). However, some limitations involved in both supervised and unsupervised approaches show the need for more reliable approaches, which can provide effective opinion or sentiment retrieval.
In supervised document-level sentiment classification such as Pang et al. (Reference Pang, Lee and Vaithyanathan2002), the manual labeling of words or phrases with a particular polarity is labor intensive. SentiWordNet (Esuli & Sebastiani, Reference Esuli and Sebastiani2006) is now playing an active role in this regard. Also, according to Pang and Lee (Reference Pang and Lee2008), subjectivity is a two-state task and can be interpreted differently in some cases. It comes with different levels of challenges, such as ‘relevant’ or ‘not relevant’ (Lee et al., Reference Lee, Jung, Song and Lee2010), ‘agreement’ or ‘disagreement’ (Bermingham & Smeaton, Reference Bermingham and Smeaton2009), and ‘winner’ or ‘looser’ (Tumasjan et al., Reference Tumasjan, Sprenger, Sandner and Welpe2010). However, it is still very challenging to effectively determine appropriate subjectivity state in many documents that contain opinion.
Also in Sarmento et al. (Reference Sarmento, Carvalho, Silva and Oliveira2009), the use of ironic phrases and inverted polarity in opinionated documents led to lower precision for positive opinions with 77% accuracy. This resulted from the difficulty in determining polarity in opinionated documents (Abbasi et al., Reference Abbasi, Chen and Salem2008; Pang & Lee, Reference Pang and Lee2008; Choi et al., Reference Choi, Kim and Myaeng2009). An approach to solve this problem is to consider semantic analysis at phrase and sentence level (Kobayakawa et al., Reference Kobayakawa, Kumano, Tanaka, Okasaki, Kim and Tsujii2009). Natural language processing techniques such as word-sense disambiguation using Wikipedia and Wiktionary may also be helpful to understand the orientation of opinion contained in a given document (Torsten et al., Reference Torsten, Christof and Iryna2009).
Another problem with supervised document-level sentiment classification is feature selection (Gamon, Reference Gamon2004). In product and news reviews, opinions or sentiments can be easily polarized as ‘positive’ or ‘negative’ and ‘good’ or ‘bad’, respectively (Liu, Reference Liu2010), thus it is very easy to select positive and negative features for training. However, in opinion sources such as blogs, subjective discussions can be highly diverse (Agarwal & Liu, Reference Agarwal and Liu2008; Pang & Lee, Reference Pang and Lee2008; Liu, Reference Liu2010); therefore, selecting training features from such documents for opinion or sentiment classification becomes a daunting task.
In unsupervised document-level sentiment classification manual creation of lexicon by set of human judges is often a challenging task (Krahmer, Reference Krahmer2010; Tausczik & Pennebaker, Reference Tausczik and Pennebaker2010). It suffices to say opinion categories such as self-reflection, perception, feeling, and emotion may be too abstract to be detected by simple opinion word count or presence of certain opinion words. Since words are commonly dependent on other words in a sentence (e.g. adjectives that modifies nouns and arguments that describe predicates), a better approach is to consider word–word dependency in the document. For example, proximity between words (Santos et al., Reference Santos, Ben, Macdonald and Ounis2009; Gerani et al., Reference Gerani, Carman and Crestani2010) and concept similarity (Jia et al., Reference Jia, Yu and Zhang2008) can be used to further understand subjectivity in phrases or sentences of documents that contain opinion (Table 1).
Performance comparison for document-level techniques across different datasets
2.2.6 Performance comparison of document-level techniques across different datasets
The expression of opinions or sentiments differs according to the domain. This is shown in the above comparison table. For classification-based techniques, it is practically challenging to use the same set of features across different domains due to the way people express opinion or sentiments on different domains. However, it could be easier to compare between works that have used the same dataset. For example, Pang et al. (Reference Pang, Lee and Vaithyanathan2002) and Whitelaw et al. (2005) used the same IMDd movie reviews. Whitelaw et al. (2005) had improved performance due to an automatic extraction of appraisal features that includes orientation of appraisal adjectives (e.g. very beautiful). Since opinions and sentiments are expressed differently in the datasets, it is almost certain that the performance will vary due to the nature of feature extracted to train classifiers. Overall, domain-specific unsupervised techniques are likely to outperform supervised techniques because of their ability to automatically use semantic and syntactic features, see Gamon (Reference Gamon2004).
2.3 Sentence-level classification for opinion retrieval
Sentence-level classification of opinion goes beyond identification of individual opinionated words as studied in some document-level opinion classification tasks (e.g. unsupervised technique). In sentence-level opinion classification, rather than individual words, each sentence in a given document is analyzed and checked to be subjective (Narayanan et al., Reference Narayanan, Liu and Choudhary2009). There are basically two tasks involved in sentence-level classification. These include subjectivity or opinionated classification and sentence-level sentiment or polarity classification (Jindal & Liu, Reference Jindal and Liu2006; Lin et al., Reference Lin, Wilson, Wiebe and Hauptmann2006). In subjectivity or opinionated classification, a sentence is checked for subjectivity (contains opinion) or factual (does not contain opinion) (Zhang & Yu, Reference Zhang and Yu2006; Ganapathibhotla & Liu Reference Ganapathibhotla and Liu2008; Jin et al., Reference Jin, Ho and Srihari2009; Santos et al., Reference Santos, Ben, Macdonald and Ounis2009; Zettlemoyer & Collins, Reference Zettlemoyer and Collins2009). In sentence-level sentiment or polarity classification, a sentence is checked for positive or negative or neutral sentiment polarity (Jia et al., Reference Jia, Yu and Meng2009; Narayanan et al., Reference Narayanan, Liu and Choudhary2009). Figure 7 shows the component architecture of sentence-level classification for both opinionated and polarity tasks. We will discuss an example state-of-the-art work that has used sentence-level classification.
Component architecture for sentence-level classification of opinionated and polarity tasks
2.3.1 Example work (sentence-level classification for opinion retrieval)
Zhang and Zhang (Reference Zhang and Zhang2006)
This work performed an opinionated classification task. It is also among the top performing techniques highlighted in TREC 2006 blog track (Ounis et al., Reference Ounis, de Rijke, Macdonald, Mishne and Soboroff2006). The work combines different IR and NLP techniques to achieve a two-step approach for opinion retrieval system. In the first step, the given query is pre-processed to determine relevant documents. In the second step, documents that contain opinions are retrieved.
The first step involves two query pre-processing techniques (e.g. query expansion) to ensure broader scope of retrieval results (Huang & Efthimiadis, Reference Huang and Efthimiadis2009). However, we suggest that the choice of query expansion must be carefully decided. Query expansion techniques may change the meaning of the original query drastically (Santos et al., Reference Santos, Macdonald and Ounis2010). This negatively affects the intent of information need and thereby leads to low precision.
The first pre-processing technique used in Zhang and Zhang (Reference Zhang and Zhang2006) is based on concept identification and retrieval from the given query. This involves an algorithm that combines some NLP tools such as Minipar (Lin, Reference Lin1998), WordNet (Fellbaum, Reference Fellbaum2010), Wikipedia (Torsten et al., Reference Torsten, Christof and Iryna2009), Collins Parser (Collins, Reference Collins1996), and GoogleFootnote 7 Search Engine. Further explanation of the algorithm is provided in (Zhang et al., Reference Zhang, Yu and Meng2007b). The second query pre-processing technique combines traditional query expansion using pseudo-relevant feedback (Yu et al., Reference Yu, Cai, Wen and Ma2003), query segmentation using Wikipedia (Tan & Peng, Reference Tan and Peng2008), and query expansion using web feedback (Huang & Efthimiadis, Reference Huang and Efthimiadis2009).
The second step involves retrieval and ranking of relevant documents regarding the pre-processed query. The technique combines concept similarity derived by using Wikipedia concepts to measure similarity between query and documents, and term similarity derived by using Okapi formula (Robertson & Walker, Reference Robertson and Walker1999). The work gave high priority to the concept similarity and the concept similarity algorithm is described in Liu et al. (Reference Liu, Liu, Yu and Meng2004). Sentence classifier is trained using subjective and objective sentences derived by using WikipediaFootnote 8 and Rateitall.comFootnote 9. Sufficient description for selecting subjective and objective sentences using the two medias is presented in Zhang and Zhang (Reference Zhang and Zhang2006) itself. Two relevance ranking and scoring methods were used in this technique, the first method retrieves only documents that contain opinion, and the second method determines sum of the classification scores for relevant sentences within the retrieved documents. We will show the two scores, respectively:
$$Scor{{e}_{rank}}(D,Q)\, = \,Sim(D,Q)\,\times \,I(D,Q)$$
where Sim(D,Q) is the relevant score for retrieving each document. I(D,Q) equals 1 if the document contains relevant sentences and 0 otherwise:
$$no{ \qquad\qquad\qquad\qquad Scor{{e}_{rank}}(D,Q)\, = & \,\mathop{\sum} [Scor{{e}_{classification}}(s)\,\times \,relevant(s,Q)] \cr & S \in \{ opinionated\,relevant\,sentences\,in\,D\} $$
where Scoreclassification(s) is the SVM classifier score for each sentence and relevant(s,Q) = 1 provided the sentence meets all relevance criteria. Using scoring method highlighted in Equation (1), the technique achieved MAP value of 0.1636 and P@10 value of 0.4380. Using the scoring method highlighted in Equation (2), the technique achieved MAP value of 0.1885 and P@10 value of 0.5120
Discussion
The principal concern is the impact of the query expansion techniques on the nature of opinion retrieved. It is very likely that the use of query expansion techniques may lead to poor performance in terms of context-dependent opinion retrieval. For example, using query expansion techniques on user's query logs, previous queries might be for different query intents, therefore affecting the context of the opinion to be retrieved. Also, the computational power required to combine the two query pre-processing techniques may be high. For example, using pseudo-relevance feedback for query expansion amount to very high computational power and many research works have shown concerns about the technique (Jones et al., Reference Jones, Rey, Madani and Greiner2006). More importantly, the use of Wikipedia-based concept similarity technique is still an active area of research, thus it is still difficult to find a benchmark to weigh its performance and contribution to opinion retrieval.
2.3.2 Research issues in sentence-level classification
The main issue in opinion or sentiment classification is the insufficient or domain-independent training features. Some research works have addressed the problem (Abbasi et al., Reference Abbasi, Chen and Salem2008; Pang & Lee, Reference Pang and Lee2008). In opinion retrieval, feature selection can be tedious especially in sentence-level classification where each sentence need to be annotated manually for the classifier (Liu, Reference Liu2010). In fact, one major concern is how the training features can be consistently identified for a well-diverse opinionated document, which is likely to have sentences with diverse concepts or multi-concepts (Pang & Lee, Reference Pang and Lee2008). We think opinion detection in sentences is a natural language problem as many opinions may be too abstract to be detected without proper linguistic analysis. This makes research in opinion retrieval more challenging as opinion retrieval task requires various NLP pre-processing techniques in order to ensure qualitative results (e.g. word sense disambiguation and syntactic analysis; Zhang & Zhang, Reference Zhang and Zhang2006; Santos et al., Reference Santos, Ben, Macdonald and Ounis2009).
In addition, some sentences can be too ambiguous for complete opinion retrieval processes (e.g. idioms, metaphor, and sarcasm). This makes it difficult to retrieve absolute opinion that a document may contain. Some research works avoid such sentences to avoid ambiguities in experiments. However, this omission means that the complete opinions or sentiments in the document cannot be fully realized, thus creating bias in the overall opinion retrieval process (Sarmento et al., Reference Sarmento, Carvalho, Silva and Oliveira2009). Therefore, retrieving opinions or sentiments from ambiguous sentences could be an active area of research (Table 2).
Performance comparison for sentence-level techniques across different datasets
2.3.3 Performance comparison of sentence-level techniques across different datasets
The sentence-level techniques also attracted different datasets like the document-level techniques. However, the use of TREC Blog datasets (i.e. TREC Blog 06, 07, and 08) was persistent as a result of the growing influence of the TREC Blog tracks. Sentence-level techniques seem to perform relatively low on TREC blog datasets. The reason could be because TREC Blog datasets have multiple domains (i.e. it contains blog posts discussing on different domains such as music, movies, products, leisure, etc.), thus there is indeed the possibility of inappropriate feature selection for all the domains that might exist in the documents. Again, unlike techniques used on multi-domain datasets, we think domain-specific opinion or sentiment classification techniques are more likely to have better performance due to familiar and fixed features used to train the classifiers. If we must compare between techniques used on TREC Blog dataset, it will be appropriate to compare between techniques that have used TREC blog dataset of the same year and also with the same performance metrics. For example, Jia et al. (Reference Jia, Yu and Zhang2008) used their supervised technique on TREC Blog 2008 and outperforms both Santos et al. (Reference Santos, Ben, Macdonald and Ounis2009) and Zhang and Zhang (Reference Zhang and Zhang2006); Zhang and Ye (Reference Zhang and Ye2008) techniques using TREC Blog 2006 and 2007 datasets, respectively. While comparison cannot be made directly due to the difference in the years of the datasets (TREC datasets differs by number of blog posts over the period of collection; Macdonald et al., Reference Macdonald, Santos, Ounis and Soboroff2010), we think Jia et al. (Reference Jia, Yu and Zhang2008) is an effective algorithm to retrieve substantial opinions at sentence level.
3 Lexicon-based approach
3.1 Overview of the approach
The presence of opinion words in documents creates an easy process for selecting and combining such words to be used as lexicon for automatic opinion or sentiment retrieval tasks. The process is called lexicon or dictionary generation (Esuli & Sebastiani, Reference Esuli and Sebastiani2006; Lee et al., Reference Lee, Na, Kim, Nam, Jung and Lee2008; Liu, Reference Liu2010). The lexicon generated can then be used to identify opinion or sentiments in documents that contain such opinionated words. This process is called lexicon-based opinion or sentiment retrieval. A lexicon-based approach is commonly used when a learning-based technique is to be improved by domain specific opinion words. This has improved domain-specific opinion retrieval (Kanayama & Nasukawa, Reference Kanayama and Nasukawa2006; Yang et al., Reference Yang, Lin and Chen2007; Ding et al., Reference Ding, Liu and Yu2008; He et al., Reference He, Macdonald, He and Ounis2008; Tan et al., Reference Tan, Wang and Cheng2008; Zhang & Ye, Reference Zhang and Ye2008; Du & Tan, Reference Du and Tan2009a; Na et al., Reference Na, Lee, Nam and Lee2009; Santos et al., Reference Santos, Ben, Macdonald and Ounis2009; Pan et al., Reference Pan, Ni, Sun, Yang and Chen2010; Vechtomova, Reference Vechtomova2010). It has also shown significant improvements in detecting opinions that are commonly expressed with adjectives (e.g. beautiful, bad) or adverbs (e.g. severely, permanently; Ding et al., Reference Ding, Liu and Yu2008; Strapparava & Mihalcea, Reference Strapparava and Mihalcea2008), thereby making it suitable for domain-independent opinion retrieval as well. Similar technique is also used in Tausczik and Pennebaker (Reference Tausczik and Pennebaker2010), constructing a lexicon of 80 different sentiment categories. The sentiment categories include words that describe personal reflection, personal emotion, and personal feelings. Other widely used lexicons for opinion retrieval include General Inquirer (Stone et al., Reference Stone, Dunphy and Smith1966) and Chinese Network Sentiment Dictionary (CNSD) or NTU Sentiment Dictionary (Ku et al., Reference Ku, Liang and Chen2006). Figure 8 shows the structural overview of the lexicon-based opinion retrieval approach.
Structural overview of lexicon-based opinion retrieval approach
It is also possible to create polarity-based lexicon for opinion or sentiment retrieval (e.g. SentiWordNet) (Esuli & Sebastiani, Reference Esuli and Sebastiani2006). SentiWordNet is a publicly available lexical resource that shows appropriate numeric polarity values for positive, negative, and objective words. It has made significant contributions in lexicon-based opinion or sentiment retrieval (Gerani et al., Reference Gerani, Carman and Crestani2010; Siersdorfer et al., Reference Siersdorfer, Chelaru and Pedro2010). In many lexicon-based opinion retrieval techniques, opinion words with positive orientation are used to denote agreement, while opinion words with negative orientation are used to denote disagreement. However, the presence of ambiguous sentences in documents that contain opinion has been one of the limitations of the lexicon-based approach. It is often challenging to understand the orientation of ambiguous words or to determine whether the words contain opinion or not. For example, there has been limited success in retrieving opinion from phrases of inverted polarities, irony, idioms, and metaphor (Sarmento et al., Reference Sarmento, Carvalho, Silva and Oliveira2009).
3.2 Example work (lexicon-based opinion retrieval)
Ding et al. (Reference Ding, Liu and Yu2008)
In this work, a holistic lexicon-based approach is used to determine sentiment orientation of product reviews using external evidence and linguistic rules. This work addresses the problem of context-dependent semantic orientations of opinion words. It also addresses the methodology for combining opinion words found within the same sentence. The idea is to use the opinion words at proximity to each product feature in a sentence to detect the orientation of the opinion. The main steps involved in their algorithm are described as follows:
1. Determine opinion words or phrases that express negative and positive opinion.
a. Use opinion lexicon comprising of adjectives, adverbs, nouns, and verbs.
b. Use list of context-dependent opinion words.
c. Perform part-of-speech tagging for opinion words in the lexicon.
d. Identify positive, negative, and dependent idioms.
e. Identify non-opinion phrases containing opinion words.
2. Aggregating opinions for a product feature.
a. Using the above list of context-dependent opinion words, identify positive, negative, and neutral opinion orientation on product feature in each sentence.
b. The opinion orientation score is computed as
$$\[--><$>score(f)\, = \,{{\sum }_{{{w}_i}.{{w}_i}\, \in \,S\, \wedge \,{{w}_i}\, \in \,V}}\frac{{{{w}_i}.SO}}{{dis({{w}_i},\,f)}}$$$
c. Where wi is the opinion word, V is the set of all opinion words and idioms, s is the sentence that contains product feature f, dis(wi, f) is the distance between feature f and opinion word wi, and wi.SO is the semantic orientation of the opinion word wi.
d. Using the multiplicative inverse in the formula, low weights is assigned to opinion words that are far away from the feature f.
e. With a positive score, the opinion on the feature in the given sentence is positive, otherwise negative.
3. Determine context-dependent opinions using linguistic rules.
a. Use intra-sentence conjunction rule which shows that a sentence only expresses one opinion orientation.
b. Use pseudo-intra (indirect) sentence conjunction rule to detect implicit conjunction in opinionated sentences.
c. Should (a) and (b) not hold, use inter sentence conjunction rule to determine opinion orientation for the previous or next sentences.
d. Use synonym and Antonym rule to identify implicit opinion orientation.
Discussion
The lexicon-based work described above shows a detailed algorithm that can capture the orientation of opinions regardless of the domain. The strength is particularly in the context-dependent component, which ensures opinion orientation is detected in the context needed by the user. This is an advantage over a classification-based opinion or sentiment retrieval technique that uses random frequency of words to train its classifier. Some recent works have also introduced domain specific techniques for detecting opinion or sentiments from a specific domain. This is quite effective as opinion words mostly used in the domain are also used to create the needed lexicon. For example, an attempt to retrieve opinions or sentiments from digital camera sales review may require creating lexicon that consist of words or terminology commonly used in the digital camera sales business (e.g. megapixel, memory, and lens), see Du and Tan (Reference Du and Tan2009a).
3.3 Research issues in lexicon-based approach
Generated lexicons are mostly based on heuristic rules that limit the words in the lexicon to certain lexical categories. For example, some lexicons only contain adjectives and adverbs (Hu & Liu, Reference Hu and Liu2004). The lexicon-based approach introduced in Ding et al. (Reference Ding, Liu and Yu2008) considered external evidences and the analysis of natural language expressions to understand the semantic orientation of opinion words. However, the work explicitly relies on set of certain opinionated words selected based on heuristic rules to generate the lexicon. Using the heuristic rules, opinionated words are selected from each sentence and combined to form lexical features. There are two major concerns about this technique. First, although some research works have shown significant improvements by considering adjectives and adverbs as part of their retrieval strategies, we think it rather limits the scope of sentiment or opinion that can be retrieved. Second, it is worth studying a model that finds a common relationship between all the lexical and semantic resources in a given sentence. This could include function words, which contribute greatly to the context or orientation of a sentence (Krahmer, Reference Krahmer2010). We believe opinionated words derived from a certain sentence may not completely and independently express the overall opinion or sentiment in the sentence. Consideration must be given to inter-dependencies between function words and opinion words toward the overall opinion. For example, object to subject linguistic relationships that may exist between words in the sentence may be considered (Krahmer, Reference Krahmer2010).
3.4 Performance comparison of lexicon-based techniques across different datasets
The use of TREC Blog dataset is consistent as shown in the comparison table above. In terms of MAP, one could make direct comparison between He et al. (Reference He, Macdonald, He and Ounis2008), Zhang and Ye (Reference Zhang and Ye2008), and Na et al. (Reference Na, Lee, Nam and Lee2009) since they all use the same TREC Blog 2006 dataset. He et al. (Reference He, Macdonald, He and Ounis2008) outperforms Zhang and Ye (Reference Zhang and Ye2008) by almost 40% and outperforms Na et al. (Reference Na, Lee, Nam and Lee2009) by almost 16%. We think the significant performance by He et al. (Reference He, Macdonald, He and Ounis2008) is due to the statistical approach that automatically detect evidence of subjectivity. On the other hand, Zhang and Ye (Reference Zhang and Ye2008) performed significantly with TREC blog 2007 topics using a generation model that unifies topic relevance and opinion generation through a quadratic combination. Another interesting observation is that opinion retrieval techniques tend to perform better on higher versions of TREC Blog datasets and topics (i.e. TREC Blog 2007 and 2008). We think this could be due to the wider coverage of blog documents collected between year 2006 and 2008.
It could also be observed that other than the TREC Blog dataset, some lexicon-based techniques have been experimented on reviews datasets. In terms of precision, Kanayama and Nasukawa (Reference Kanayama and Nasukawa2006) outperforms Ding et al. (Reference Ding, Liu and Yu2008) but with very low recall. However, Ding et al. (Reference Ding, Liu and Yu2008) showed substantial performance for both precision and recall. This makes Ding et al. (Reference Ding, Liu and Yu2008) the likely suitable lexicon-based technique especially on review datasets (Table 3).
Performance comparison for lexicon-based techniques across different datasets
4 Probabilistic approach
4.1 Overview of the approach
Probabilistic approaches have been used to measure the degree of opinion in opinionated documents (Elsas et al., Reference Elsas, Aguello, Callan and Carbonell2007; Kim et al., Reference Kim, Li and Lee2009; Gerani et al., Reference Gerani, Carman and Crestani2010; Jiang et al., Reference Jiang, Zhang, Yang and Niu2010). In this approach, opinionated documents are ranked in order of relevance by calculating the probability that each document contains opinions using the frequency of opinionated words. A typical generative model in IR is followed by estimating the likelihood of generating a document given a query, that is, p(d∣q). For improved opinion retrieval performance, the probabilistic approach is usually combined with lexicon-based approach. One would also prefer the probabilistic approach over the sentiment classification approach if a high number of documents are to be retrieved. In other words, a probabilistic approach is likely to give high recall more than precision.
The probabilistic approach is also known as probability ranking principle, formulated by Robertson (1977). According to the early literatures (Hiemstra, Reference Hiemstra2000; Sparck Jones et al., Reference Sparck Jones, Walker and Robertson2000a, 2000b), the probabilistic approach is dichotomous and independent of every other document within a chosen collection. For example, the relevance of a document to a given query should be either relevant or not relevant, and must be derived within each document without the need to consider other documents, see Lavrenko (Reference Lavrenko2010). For example, in a collection of 100 documents, if we assume 20 out of 100 documents are relevant to the word ‘happy’, then the probability of relevance for a document taken at random is 1/20 = 0.05. However, for queries with more than one terms, the binary independence assumption is considered (Hiemstra, Reference Hiemstra2000). For example, a query with two terms ‘happy’ and ‘family’ would have four possible binary states. That is, happy AND family, happy NOT family, family NOT happy, and happy NOT family and NOT (happy or family). Figure 9 shows the structural overview of probabilistic opinion retrieval approach.
Structural overview of probabilistic opinion retrieval approach
The idea of probabilistic opinion retrieval is that words within a given document are ranked in terms of their contributions to searching and returning opinionated information in order of relevance. For this purpose, a two-state process is involved. First, relevant document is identified based on frequency of query words. Second, from the set of relevant documents, words that express opinion and their frequency of occurrence or proximity to query words are essential to determining the opinionated relevance of each document. Lexicons of words that express opinion are used to determine the frequency of opinionated words. A precise description for the probabilistic approach to opinion retrieval is to assume user information is represented as query Q, which is a set of query term t. Ranking of relevant document d in document collection D is largely based on the frequency of t 1 … tn in d. The degree of opinion contained in d is largely based on the frequency of words w 1 … wn that express opinion as contained in the lexicon.
4.2 Example work (probabilistic opinion retrieval)
Gerani et al., Reference Gerani, Carman and Crestani2010
In this work, a probabilistic model for assigning relevant opinion scores to blog documents was proposed. The work used general opinion lexicon and proposed the use of proximity information to capture opinion term relatedness. The proximity information forms opinion density that signifies the probability of opinion about the query term. The work combined probabilistic and lexicon-based approaches and requires that opinion weights be determined based on a list of generated lexicon. The steps involved in their algorithm can be described as follows:
1. Determine proximity opinion score.
a. Use opinion lexicon to identify opinionated words.
b. Calculate the opinion density function at each position in the document by accumulating the opinion density from different opinion terms at a particular position.
c. The accumulated density is the probability of the opinion to be expressed on a term at a particular position.
d. The probability that a term at position j is about the query term at position i is computed as:
$$p(j|i,d)\, = \,\frac{{k(j,i)}}{{\mathop{\sum}\nolimits_{j^{\prime}\, = \,1}^{|d|} k(j^{\prime},i)}}$$
e. Where k(j, i) denotes the kernel function that gives the weight of opinion from position j to position i.
f. The probability that the document contains opinion at position i is computed as:
$$p(o|i,d)\, = \,\mathop{\sum}\limits_{j\, = \,1}^{|d|} p(o|{{t}_j})p(j|i,d)$$
2. Calculate the probability that the document contains opinion about the query
$$\[--><$> p(o|d,q) <$><!--$$
.a.
$$\[--><$> p(o|d,q) <$><!--$$
is computed as: b.
$$p(o|d,q)\, = \,\mathop{\sum}\limits_{i\, = \,1}^{|d|} p(o,i|d,q)p(i|d,q)\, = \,\mathop{\sum}\limits_{i\, = \,1}^{|d|} p(o|i,d,q)p(i|d,q)$$
$$\[--><$> p(i|d,q) <$><!--$$
is computed as:
$$p(i|d,q)\, = \,\left\{ {\matrix{ {\frac{1}{{|pos(q)|}}} \hfill & {if\,{{t}_i}\in q} \hfill\cr 0 \hfill & {otherwise} \right$$
3. Refine the proximity model by performing smoothing.
a. Perform smoothing with non-proximity-based estimation as follows:
$$\qquad \ {\qquad \ \qquad\qquad p(o|q,d)\, = (1{\rm{ - }}\lambda )p(o|q,d)\, + \,\lambda p(o|d) \hskip 11pc$$
b. The smoothing process ensures proximity is determined at different ranges.
Discussion
The above work showed significant performance on TREC Blog 2006 dataset with about 2.5% improvement in terms of MAP over the best run in TREC Blog 2008 opinion finding task. An interesting research outcome in this work is that proximity information of opinionated words to query words does contribute greatly to determining the degree of opinion in a particular document. However, it is important to normalize the topic-relevance score before applying it to the proximity-based opinion retrieval model. Such normalization strategy would also depend on the selected technique for the topic-relevance retrieval (e.g. BM25).
4.3 Research issues in probabilistic approach
A common problem about the probabilistic approach is the parameter tuning for the smoothing processes. One needs to compare between different parameter values before deciding on the best parameter that gives best performance. Normalization is often another problem in probabilistic approaches. Care must be taken if existing relevant scores are to be transformed to their respective probability estimates. An appropriate normalization technique would depend on the selected relevant scores to be used as baselines. While probabilistic approaches would always be an essential retrieval strategy for opinion retrieval, we think it is more likely to be effective where there are high chances of frequency of observations as applicable to common probabilistic approaches such as BM25 and Divergence from Randomness probabilistic models (Amati & Rijsbergen, Reference Amati and Rijsbergen2002; Blei et al., Reference Blei, Ng and Jordan2003; Pickens & MacFarlane, Reference Pickens and MacFarlane2006; Ounis et al., Reference Ounis, Macdonald and Soboroff2008; Macdonald et al., Reference Macdonald, Ounis and Soboroff2009; Robertson & Zaragoza, Reference Robertson and Zaragoza2009; Xu et al., Reference Xu, Liu, Xu, Yu, Song, Guan, Peng and Cheng2009; Vechtomova, Reference Vechtomova2010) (Table 4).
Performance comparison for probabilistic techniques across different datasets
4.4 Performance comparison for probabilistic techniques across different datasets
The TREC Blog 2006 dataset is commonly used by most probabilistic approaches. It is important to note that there are variations as to the probabilistic approaches used by individual works. What is more important is that the techniques follow the same fundamental probability ranking principle as we had discussed earlier. From the comparison table above, it was observed that most techniques do not have significant performance in terms of MAP. Kim et al. (Reference Kim, Li and Lee2009) gave the most significant performance on TREC Blog 2006 with MAP value of 0.4242. We think the introduction of semantic and local context proximities as features could be responsible for the improvement over other techniques. Hannah et al. (2007) also gave similar performance with MAP value of 0.4160 with little improvement on P@10 value compared to the P@10 given by Kim et al. (Reference Kim, Li and Lee2009). This improvement is reported to have been influenced by the introduction of statistical approach such as divergence from randomness (DFR) weighting model that uses weighted dictionary for opinion retrieval tasks (McCreadie et al., Reference McCreadie, Macdonald, Ounis, Peng and Santos2009). Gerani et al. (Reference Gerani, Carman and Crestani2010) also showed substantial performance on TREC Blog 2006 dataset with MAP value of 0.4292 by using the best proximity method to calculate opinion proximity density at different positions in a document. The performance is still encouraging considering the fact that it is really challenging to significantly outperform opinion retrieval baselines on TREC Blog datasets, see Macdonald et al. (Reference Macdonald, Santos, Ounis and Soboroff2010).
5 Other emerging opinion retrieval approaches
5.1 Overview of the approach
Other than the existing opinion retrieval approaches described in the previous sections, there are other emerging approaches for retrieving opinions. These approaches are designed to overcome the shortcomings of the existing main approaches since study has shown that it is very difficult to significantly outperform the baselines by using the existing approaches that were discussed earlier (Ounis et al., Reference Ounis, Macdonald and Soboroff2008). More importantly, it is difficult for us to show a component diagram that shows the overview of these approaches as they vary depending on the retrieval task and domain of retrieval. These emerging approaches have either proposed a completely new technique or combine two or more existing techniques. We will discuss an example state-of the-art work and then highlight the performance and limitations.
5.2 Example work (other opinion retrieval approaches)
Du and Tan (Reference Du and Tan2009)
In Du and Tan (Reference Du and Tan2009), an information theoretic approach to ‘finer-grained’ opinion retrieval was proposed. Rather than traditional template extraction method, the technique used an iterative based heuristics to identify hidden sentiments that are not explicitly clear or observable within each sentence in the review document. In addition, the technique allows document features and opinion targets to converge simultaneously and iteratively. Co-occurrence of terms and similar contextual information is considered for this purpose. The steps involved in their algorithm are as follows:
1. Identify hidden sentiment association.
a. Detect the sentiment association between set of product features F and set of opinion words O with polarity labels.
b. Build a weighted bipartite graph from F and O denoted by G = F, O, R, where R = [rij] denotes the m * n link weight matrix, which contains all the pair-wise weights between set F and O.
c. Calculate rij as the frequency of co-occurrence of fi and Oj at clause level.
d. Take F and O as two random variables, and then find the compression C between F and O such that the mutual information between C and O is sufficiently large.
e. To find the compression C, use iterative reinforcement approach that employs improved information bottleneck algorithm.
2. Use improved information bottleneck algorithm.
a. Unlike the traditional information bottleneck approach, the divergence between two data objects is known by using co-occurrence information between the two variables F and O.
b. Do a linear combination of the co-occurrence information and the semantic information as the final distance between two data objects F and O.
Discussion
The strength of this work lies in the interpolation of semantic information with the co-occurrence information between domain specific features and opinionated words. We think the work could be very useful for context-dependent opinion retrieval whereby the content of the opinionated documents retrieved has the same semantic context with the given query topic. However, the precision attained by the work is 78.90%, which gives more room for further improvements. Since this work was evaluated on review dataset, it is difficult to assume that the heuristic created can perform efficiently on domain independent opinionated documents such as blogs.
5.3 Research issues in other emerging opinion retrieval approaches
Emerging opinion retrieval techniques still perform lower than the state-of-art techniques that use the existing approaches that have been discussed in the previous sections. We think this is because of the difficult nature of opinion retrieval task. However, emerging approaches that tend to use NLP techniques using semantic and syntactic information seems promising. A common challenge across many opinion retrieval techniques is their ability to detect implicit opinions or sentiments, which could be difficult without effective NLP technique (Table 5).
Performance comparison for emerging techniques across different datasets
5.4 Performance comparison for emerging techniques across different dataset
The performance of these techniques varies on different datasets. However, the emerging techniques experimented on TREC Blog datasets can be easily compared for performance improvements over the baselines. Based on the comparison table above, He et al. (Reference He, Macdonald, He and Ounis2008) significantly outperforms Bermingham and Smeaton (Reference Bermingham and Smeaton2009) by almost 60%. However, performance results in terms MAP for all the emerging techniques stated in the table above are significantly low compared with the performance results given by the existing approaches discussed earlier (i.e. text classification, lexicon-based, and probabilistic approaches). As the research on opinion retrieval gets mature, these emerging techniques are more likely to use new and efficient methods that can outperform the various state-of-the-art techniques.
6 Future research directions
From this review, we believe opinion retrieval could be treated as natural language problem. Recently, Krahmer (Reference Krahmer2010) suggested a linguistic approach toward providing effective solution to some of the research issues identified in this paper. As mentioned earlier, opinionated documents are diverse such that there is need for proper understanding of what each sentence means (Munson & Resnick, Reference Munson and Resnick2010). Moreover, opinions are context-dependent, for example, one sentence may have multiple meaning, and it would be difficult to extract opinion from such sentence without effective linguistic approach. Therefore, we think semantic and syntactic features could play important role in opinion retrieval (Pang & Lee, Reference Pang and Lee2008). From the series of literatures discussed in this paper, we could observe that the success of any opinion retrieval technique would depend on high degree of semantic relevance of opinionated documents to the underlying meaning of the given query topic.
An important development is to see emerging opinion retrieval approaches divert from the conventional approaches that use frequency of query words or frequency of opinion words. The nature of problems that come with new opinion retrieval tasks require multi-faceted approach to detect opinions at different inclinations (Liu, Reference Liu2010). Inclination describes what kind of opinionated documents are to be retrieved (Macdonald et al., Reference Macdonald, Santos, Ounis and Soboroff2010). For example, some people may prefer to retrieve opinionated documents that discuss about personal life experiences rather than quality of a product. In this case, the inclination of interest is personal rather than commercial as in product reviews. To solve the multi-faceted opinion retrieval problem, it could be less effective to rely on approaches that use frequency of query words or frequency of opinion words alone. We think multi-sentence semantic dependencies approach should be used to partition opinions that occur at different inclinations.
Detecting sentences that contain opinion is still a key challenge in opinion retrieval. This affects selection of training features for the text classification approach. We think it is worth studying an automatic and effective approach to detect sentences that contains opinion regardless of the domain. Many attempts have been made to use the lexicon-based approach to solve this problem (Ding & Liu, Reference Ding and Liu2007; Tan et al., Reference Tan, Wang and Cheng2008; Na et al., Reference Na, Lee, Nam and Lee2009). However, it is not yet clear how the lexicon-based approach can be domain-independent. For example, the domains from which the words in the lexicon are selected are limited in most cases to certain domains of interest. Certain lexicons created by set of human judges, for example, LIWC (Tausczik & Pennebaker, Reference Tausczik and Pennebaker2010), have also been limited to few sentiment categories. It is obvious that with the fast growing rate of activities on the World Wide Web, documents that contain opinion are getting more diverse and dynamic. This creates possibility that lexicon created for certain domains could be outdated within a very short period. We think it could be very useful to explore how lexical relationships between words in a given sentence can be used to detect opinions.
Finally, we are aware that proximity between query and opinion terms has been explored (Santos et al., Reference Santos, Ben, Macdonald and Ounis2009; Gerani et al., Reference Gerani, Carman and Crestani2010). However, many words in proximity might be independent of each other. We think it is better to identify words proximity in the semantic context at which opinionated information has been requested and not words proximity by ordinary presence of query or opinion words.
7 Summary and conclusions
Current review papers on opinion retrieval discuss theoretical foundations and possible applications of opinion retrieval systems. In this paper, we have discussed the major opinion retrieval approaches and their performance on different datasets. We have also highlighted some limitations and questions that demand answers in future research efforts. Some of the approaches discussed in this paper have contributed immensely to solving opinion retrieval research problems. However, there are still many challenges as most emerging techniques still perform significantly lower than the state-of-the-art techniques.
Based on the performance summary tables presented in this paper, majority of the opinion retrieval techniques are evaluated on TREC Blog datasets, which aid the performance comparison. However, regardless of the approach used, we could observe most opinion retrieval techniques in the research community are still largely based on frequency or presence of individual query or opinion terms within documents. The use of frequency or presence of query terms negatively affect context-dependent opinion retrieval (Krahmer, Reference Krahmer2010). We also note that state-of-the-art techniques perform similarly except for few instances. This shows research on opinion retrieval is still very open to developing effective opinion retrieval techniques that can perform with better precision.
From our review work, it is still difficult to conclude that a particular approach is the best. Even though there may be some limitations in some of the works, we have seen instances where the text classification approach showed good performance (Turney & Littman, Reference Turney and Littman2002; Jia et al., Reference Jia, Yu and Zhang2008; Zagibalov & Carroll, Reference Zagibalov and Carroll2008; O'Hare et al., Reference O'Hare, Davy, Bermingham, Ferguson, Sheridan, Gurrin and Smeaton2009; Siersdorfer et al., Reference Siersdorfer, Chelaru and Pedro2010); lexicon-based approach (Kanayama & Nasukawa, Reference Kanayama and Nasukawa2006; Yang et al., Reference Yang, Lin and Chen2007; Ding et al., Reference Ding, Liu and Yu2008; He et al., Reference He, Macdonald, He and Ounis2008; Tan et al., Reference Tan, Wang and Cheng2008; Zhang & Ye, Reference Zhang and Ye2008; Du & Tan, Reference Du and Tan2009; Santos et al., Reference Santos, Ben, Macdonald and Ounis2009; Vechtomova, Reference Vechtomova2010); probabilistic approach (Eguchi & Lavrenko, Reference Eguchi and Lavrenko2006; Bermingham & Smeaton, Reference Bermingham and Smeaton2009; Elsas & Dumais, Reference Elsas and Dumais2010; Gerani et al., Reference Gerani, Carman and Crestani2010); and other emerging approaches (Du & Tan, Reference Du and Tan2009b; Huang & Croft, Reference Huang and Croft2009; Santos et al., Reference Santos, Ben, Macdonald and Ounis2009; Thet et al., Reference Thet, Na, Khoo and Shakthikumar2009; Xu et al., Reference Xu, Liu, Xu, Yu, Song, Guan, Peng and Cheng2009; Yu et al., Reference Yu, Liu, Huang and An2010).
We believe that regardless of the approach used for opinion retrieval, effective opinion retrieval with sufficient semantic relevance between the opinionated documents and the query topic is paramount. More importantly, performance evaluation can be based on appropriate IR performance metrics as discussed earlier in this paper. Experimentation should also be conducted on popular and standard datasets such that the performance results can be compared with other techniques (Ounis et al., Reference Ounis, de Rijke, Macdonald, Mishne and Soboroff2006; Macdonald et al., Reference Macdonald, Ounis and Soboroff2007; Ounis et al., Reference Ounis, Macdonald and Soboroff2008; Macdonald et al., Reference Macdonald, Ounis and Soboroff2009; Macdonald et al., Reference Macdonald, Santos, Ounis and Soboroff2010). In conclusion, Figure 10 shows how the discussed opinion retrieval approaches have trended over the past 8 years.
Frequency of opinion retrieval approaches over 90 research works published in research communities between year 2002 and 2010
The above figure shows trends of different approaches used in opinion retrieval over a period of 8 years as collected from ACM and Google Scholar Digital Libraries. It could be observed that machine learning or text classification approach moved at a very high pace since the inception of opinion retrieval in the early 2000 (Pang & Lee, Reference Pang and Lee2008; Liu, Reference Liu2010). However, the use of the approach dropped around year 2008 as other approaches such as lexicon-based and probabilistic approaches began to receive attention of the research community. Since 2006, the probabilistic approach has received considerable attention in opinion retrieval research community due to its scalability. Currently, other independent approaches (i.e. approaches other than machine learning or text classification, lexicon based, and probabilistic) are now getting more attention within the opinion retrieval research community. However, as mentioned earlier, none of the independent approaches has significantly outperformed the state-of-the-art baselines for opinion retrieval.
Acknowledgement
This work is supported by MONASH University Sunway Campus Higher Degree by Research (HDR) Scholarship.
