1 Introduction
Aspect-based sentiment analysis (Liu and Zhang Reference Liu and Zhang2012) has been extensively studied in the recent years because of the valuable information available in online reviews as well as their impact on consumers. Extracting sentiments on each aspect of the entity is one of the main tasks under aspect-based sentiment analysis. While there have been several works in aspect extraction (Hu and Liu Reference Hu and Liu2004; Popescu and Etzioni Reference Popescu and Etzioni2007), opinion term extraction (Dave et al. Reference Dave, Lawrence and Pennock2003), aspect-based summarization (Dave et al. Reference Dave, Lawrence and Pennock2003; Zhuang et al. Reference Zhuang, Jing and Zhu2006; Titov and McDonald Reference Titov and McDonald2008), joint aspect and sentiment models (Lin and He Reference Lin and He2009; Jo and Oh Reference Jo and Oh2011; Zhao et al. Reference Zhao, Jiang, He, Song, Achananuparp, Lim and Li2011; Mukherjee and Liu Reference Mukherjee and Liu2012; Fu et al. Reference Fu, Yang, Huang and Cui2015, Reference Fu, Wu and Cui2016), polarity identification (Somasundaran et al. Reference Somasundaran, Namata, Wiebe and Getoor2009), the task of phrase-level sentiment analysis has received limited attention. By phrase level, we refer to extracting the aspect-specific opinion expression (also referred to as aspect–sentiment phrases) along with its associated aspect rating and clustering/grouping those aspec–sentiment phrases under coherent aspect categories. The proposed fine-grained phrase level sentiment analysis can yield us aspect-specific sentiments with users’ opinions in the actual sentence context which is otherwise unavailable. Throughout this work, we interchangeably use the terminologies aspect–sentiment phrase/expression with opinion expression/phrases which also refers to an aspect-specific opinion phrases.
One of the closest related works to our task that can also be included in the fine-grained analysis is the research thread of multi-aspect sentiment analysis. Multi-aspect sentiment analysis refers to extracting the sentiment value (via polarity or rating score) for each aspect at the review level. To our knowledge, the most closely related works on multi-aspect sentiment analysis are those in Snyder and Barzilay (Reference Snyder and Barzilay2007), Titov and McDonald (Reference Titov and McDonald2008), Wang et al. (Reference Wang, Lu and Zhai2010), Lu et al. (Reference Lu, Ott, Cardie and Tsou2011), Xianghua et al. (Reference Xianghua, Guo, Yanyan and Zhiqiang2013), Poria et al. (Reference Poria, Cambria and Gelbukh2016) and Poria et al. (Reference Poria, Chaturvedi, Cambria and Bisio2016). However, these works either rely on aspect keyword supervision or only produce an aspect-based summary of an entity using latent topic attribution as opposed to the proposed fine-grained aspect–sentiment expressions at the sentence level including an aspect rating score. The individual differences of other related works to ours are detailed in Section 2.
Another common approach we often see in review sites (e.g., Amazon) is to display the aggregated overall rating of the product which does not give any information about the properties of the product. While some websites (e.g., Tripadvisor) provide the rating for product-specific aspects such as ‘Room’, ‘Service’ and ‘Location’, etc. for the hotel but this is superficial extraction and does not explain why particular aspect is rated well or poorly (e.g., by finding the exact aspect-specific sentiment expression). Therefore, it would be beneficial if a system can extract aspect-specific opinion expression with rating score attached to its aspect which can be considered as intensity of phrases. Table 1 shows some example aspect–sentiment phrases in the sentence context along with its aspect rating score. We assumed the rating score to take values from [1, 5] to make it consistent with standard rating scale. The aspect is italicized and aspect–sentiment phrases is shown within [[ ]]. Typically, a traditional approach for aspect-based sentiment analysis follows pipelined manner. First, the opinion words and phrases are identified and then polarity score of the aspects are determined. However, this sequential approach can be improved upon. The goal of our work is to provide a mechanism for jointly extracting aspect-specific opinion expression and associated aspect’s rating score.
Example aspect–sentiment phrase with aspect rating
Most aspect–sentiment models extract aspect only (Titov and McDonald Reference Titov and McDonald2008; Brody and Elhadad Reference Brody and Elhadad2010) or jointly discover aspects and sentiments (Mei et al. Reference Mei, Ling, Wondra, Su and Zhai2007; Zhao et al. Reference Zhao, Jiang, Yan and Li2010; Jo and Oh Reference Jo and Oh2011; Lakkaraju et al. Reference Lakkaraju, Bhattacharyya, Bhattacharya and Merugu2011; Moghaddam and Ester Reference Moghaddam and Ester2011; Sauper et al. Reference Sauper, Haghighi and Barzilay2011; Moghaddam and Ester Reference Moghaddam and Ester2013), but they are unigrams with no correspondence of aspect and sentiment words. However, a natural way of aspect opinion mining, is to discover the aspect-specific sentiment phrase containing both the aspect and sentiment as an ‘aspect–sentiment phrase’. This not only provides robustness of the results but also perseveres the original opinion targets and aspect evaluations within the sentence context. Aspects (both unigrams and n-gram aspect phrases) by themselves alone are also less informative as they do not explain the evaluation of that aspect expressed in the reviews. Our model preserves the sentiment phrasal structure as they were originally expressed in the reviews and additionally predicts the rating score of each aspect which can be considered as a measure of intensity of the expressed opinion (e.g., a phrase ‘staff to be attentive and friendly’ is highly positive having score of 5 while phrase ‘service was friendly though slow’ is neutral having score of 3).
In this paper, we propose a novel Latent Dirichlet Allocation–Conditional Random Fields (LDA–CRF) hybrid rating regression model which is hybrid between generative–discriminative frameworks to solve the task. The hybrid model consists of (i) an unsupervised generative component for modeling the semantic coherence of aspect–sentiment phrases based on their collocations across different documents and (ii) a supervised discriminative sequence modeling component (via CRF) for aspect–sentiment phrase extraction in the sentence context. These are further coupled with an aspect rating regression component. We investigate the effectiveness of our model by performing extensive experiments on TripAdvisor Dataset. In SemEval 2016, the aspect-based sentiment analysis task due to Pontiki et al. (Reference Pontiki2016), although the dataset has annotation for aspect phrases, there are no aspect-opinion phrase annotation. Hence, it cannot be directly employed as our focus is on discovering aspect-opinion phrases (which have both aspect phrase and opinion phrase). We could not find any standard dataset which have labeled aspect opinion phrases in the sentence context, hence decided to construct a new dataset using a semi-supervised scheme. The dataset contains both aspect-specific opinion phrases as well as aspect ratings. We will release the dataset for future research. Experimental results on real-world data shows the effectiveness of the proposed approaches that outperform several state-of-the-art competitors on both tasks: (i) aspect–sentiment phrase extraction and (ii) aspect rating regression. Additionally, we presented a heuristic approach to label the opinion phrases surrounding the Head Aspect (HA) for training the CRF model which help in extending the model without need to labeled data for new domain. We also released the associated datasets and codebaseFootnote 1 which we believe will serve as a resource for furthering research in sentiment analysis.
The remainder of the paper is organized as follows. The next section details relevant related work. Section 3 introduces the problem statement and explain the intuition regarding the model. Section 4 describes the inference for LDA–CRF. In Section 5, we describe the datasets, baselines and report the results of our experimental evaluation. Finally, Section 6 concludes the paper with a discussion.
2 Related work
There are several threads of research related to our tasks – (i) aspect–sentiment phrase extraction and (ii) rating prediction which are summarized below.
In the thread of aspect rating prediction, Lu et al. (Reference Lu, Zhai and Sundaresan2009) generated the ratable aspect summary for short comments using a structured probabilistic latent semantic analysis (PLSA). First, phrases rating were predicted using local/global prediction models where local prediction assume each phrase has same rating as average overall rating of whole collection of comments, while global prediction learns the aspect-level classifier from overall rating. Then, phrases rating were aggregated to get the final aspect rating. In Moghaddam and Ester (Reference Moghaddam and Ester2011, Reference Moghaddam and Ester2013), Interdependent LDA and Factorized LDA models were proposed, respectively, which jointly extract the major product aspects and predict their ratings based on LDA. However, the learned sentiment topics may not correspond to the predicted ratings directly which affects the prediction accuracy and regarded as the ‘Sentiment Label Alignment’ problem in Wang (Reference Wang2015). In Wang (Reference Wang2015), the Sentiment-Aligned Topic Model was proposed to rectify this problem using the sentiment lexicon where each word was associated with a sentiment polarity score. All the above methods use a preprocessing step, which parse the reviews in opinion phrases of the format [headterm, modifier] where headterm refers to an aspect and modifier is the related sentiment word using some shallow parsing techniques and reviews are modeled as bag of phrases. The works in Sauper et al. (Reference Sauper, Haghighi and Barzilay2011) and Sauper and Barzilay (Reference Sauper and Barzilay2013) use a ‘snippet’ which is short sequence of words extracted using phrase extraction system. As the rating prediction and opinion phrase extraction task is done in a pipelined architecture, it inherently suffers from error propagation, i.e., rating prediction suffers from the unrecoverable errors in the opinion phrase extraction module which is undesirable. Our approach takes a joint modeling approach to solve both the subtasks: aspect–sentiment phrases extraction and rating prediction and our experimental results demonstrate that these two task reinforce each other.
Wang et al. (Reference Wang, Lu and Zhai2010) studied the problem of latent aspect analysis to infer the aspect rating for each review. They have used aspect keywords as seed words for bootstrapping algorithm to segment the review. After aspect segmentation step, a linear regression model known as ‘Latent Rating Regression’ was proposed to generate the overall rating as a weighted sum of different aspect ratings. However, it relies on explicit aspect keywords from user as a supervision signal. In Wang et al. (Reference Wang, Lu and Zhai2011), the aspect supervision component was eliminated by employing a topic modeling to segment the reviews based on aspects. A drawback of these models is that both of them assume that in each review reviewer talks about every aspect, which is rarely true in practice for online reviews due to topical sparsity in the data. To address the aspect sparsity problem, Xu et al. (Reference Xu, Lin, Lam, Zhou, Cheng and So2014) proposed an sparse aspect coding model (SACM) using a sparse topical coding model. It does not assume aspect proportion to be normalized distribution and the sparsity of the topic representation can be controlled by a L1-regularizer thereby making the rating prediction more robust. In Diao et al. (Reference Diao, Qiu, Wu, Smola, Jiang and Wang2014), joint models of aspect extraction and rating regression were proposed and tailored for a movie review corpus. However, both of the above models assume that a user has a latent interest distribution and they learn it based on the data. However, this may not work well for users who have written very few reviews.
Our work is also related to supervised topic models (Mcauliffe and Blei Reference Mcauliffe and Blei2008; Mimno and McCallum Reference Mimno and McCallum2012; Zhu et al. Reference Zhu, Ahmed and Xing2012) where latent topic structure was used for improving rating prediction over regression on words alone. Mcauliffe and Blei (Reference Mcauliffe and Blei2008) proposed a supervised Latent Dirichlet allocation (sLDA) model in which response variable was generated with normal linear model and trained by maximizing the joint likelihood of data and response variable. In Zhu et al. (Reference Zhu, Ahmed and Xing2012), a Maximum Entropy Discrimination Latent Dirichlet allocation (MedLDA) model was proposed where the model was trained using the max-margin principle. In Mimno and McCallum (Reference Mimno and McCallum2012), a Dirichlet-multinomial regression model was proposed which used the metadata/arbitrary features of the document along with latent topic for rating prediction. We have compared our model with the above models to show that rating prediction improves with joint modeling. In Li et al. (Reference Li, Huang and Zhu2010), Sauper et al. (Reference Sauper, Haghighi and Barzilay2011) and Sauper and Barzilay (Reference Sauper and Barzilay2013), approaches were proposed that used seed words for defining asymmetric Dirichlet prior or word assignment to the sentiment topic to improve the prediction accuracy, but these are mostly only for positive or negative polarity as opposed to generic rating score prediction for multiple aspects.
In the thread of research on phrase extraction, there are several works that are related. In Choi et al. (Reference Choi, Cardie, Riloff and Patwardhan2005), Choi et al. (Reference Choi, Breck and Cardie2006) and Breck et al. (Reference Breck, Choi and Cardie2007), the task was formulated as a token level sequence labeling task and various parsing, syntactic, lexical and dictionary-based features (Kim and Hovy Reference Kim and Hovy2006; Kobayashi et al. Reference Kobayashi, Inui and Matsumoto2007; Jakob and Gurevych Reference Jakob and Gurevych2010) have been used for expression extraction. However, they mostly use the first-order assumption in their sequence models where the label of current token depends mostly on the current token and the label of previous token. This is limited and could not capture more expressive phrases. In Yang and Cardie (Reference Yang and Cardie2012, Reference Yang and Cardie2013), the issue was addressed by employing semi-CRF to describe sets of related contiguous tokens mainly known as segment features. Further in Yang and Cardie (Reference Yang and Cardie2014), a semi-CRF-based approach was employed for opinion expression intensity and polarity classification. In Johansson and Moschitti (Reference Johansson and Moschitti2011), a re-ranking model was stacked on top of a sequence labeler to extract opinion expression and was further extended for assigning polarities. Wu et al. (Reference Wu, Zhang, Huang and Wu2009) proposed a phrase dependency parsing approach to extract the relation between aspect expression and opinion expression. All above works have made important progresses. However, their main focus was on generic subjective expression extraction as opposed to joint modeling of aspect-specific sentiment expression and rating prediction which is the focus of our work.
Another thread of research in phrase extraction involves LDA for phrase mining. Most approaches (Zhao et al. Reference Zhao, Jiang, He, Song, Achananuparp, Lim and Li2011; El-Kishky et al. Reference El-Kishky, Song, Wang, Voss and Han2014; Liu et al. Reference Liu, Shang, Wang, Ren and Han2015) have addressed the phrase mining task as either a pre-processing or a post-processing step of a topic model. Zhao et al. (Reference Zhao, Jiang, He, Song, Achananuparp, Lim and Li2011) generate topical phrases through ranking the topical keywords based on tweet characteristics, while in El-Kishky et al. (Reference El-Kishky, Song, Wang, Voss and Han2014) and Liu et al. (Reference Liu, Shang, Wang, Ren and Han2015), frequency-based approaches to generate the candidate phrases were employed followed by a constrained LDA. There are works done where phrases are extracted jointly with LDA. These include the Topical n-gram model in Wang et al. (Reference Wang, McCallum and Wei2007) which extracts phrases based on the context in probabilistic manner. Lindsey et al. (Reference Lindsey, Headden and Stipicevic2012) employed hierarchical Pitman–Yor process to model phrases. In Fei et al. (Reference Fei, Chen and Liu2014), Generalized Pólya urn model (GPU) was employed on top of LDA to group the noun phrases which are considered as opinion expressions.
There is a recent thread of research which applies neural network models to the task of aspect-based sentiment analysis and opinion expression extraction. They use the distributed representation of words which represent the latent features using continuous value dense vectors. Irsoy and Cardie (Reference Irsoy and Cardie2014) and Liu et al. (Reference Liu, Joty and Meng2015) apply deep Recurrent Neural Network to extract the opinion expressions and opinion targets from sentences. The work in Wang et al. (Reference Wang, Pan, Dahlmeier and Xiao2016), Liu et al. (Reference Liu, Joty and Meng2015) and Yin et al. (Reference Yin, Wei, Dong, Xu, Zhang and Zhou2016) models target aspect term extraction as a sequence tagging task. In Liu et al. (Reference Liu, Joty and Meng2015), Recurrent Neural Network and word embedding were combined to extract explicit aspects. In Wang et al. (Reference Wang, Pan, Dahlmeier and Xiao2016), recursive neural network based on dependency tree and CRF were integrated in a unified framework to extract the aspect and opinion terms. Yin et al. (Reference Yin, Wei, Dong, Xu, Zhang and Zhou2016) proposed an unsupervised approach for training the word embedding based on the dependency paths and then the trained word embedding were used as features in a downstream CRF pipeline. These methods are mostly focused on aspect term extraction instead of aspect-specific opinion expression which is the focus of this work. Lakkaraju et al. (Reference Lakkaraju, Socher and Manning2014) extracts the aspect–sentiment pair using a hierarchical deep learning framework, but they have considered only positive–negative words as sentiment as opposed to general opinion terms.
Also, related are the works around aspect-based sentiment classification (Tang et al. Reference Tang, Qin, Feng and Liu2015, Reference Tang, Qin and Liu2016; Wang et al. Reference Wang, Huang, Zhao and Zhu2016) and the work in Nguyen and Shirai (Reference Nguyen and Shirai2015) which proposed an extension of Recurrent Neural Network that takes both dependency and constituent trees of a sentence into account for identifying the aspects sentiment. Wang et al. (Reference Wang, Huang, Zhao and Zhu2016) proposed an LSTM model with attention mechanism which focuses on different part of sentences given the aspect input. Further, in Tang et al. (Reference Tang, Qin and Liu2016), a deep memory network was proposed for explicitly capturing the importance of each context word when inferring the polarity of given aspect. These approaches provide the sentiment about the aspect but do not give in-depth information about the aspect such as the associated opinion expression.
Overall, all these approaches have some similarity with the work we present. Some methods have tried to extract the generic subjective opinion expression, while others have tried to predict the aspect rating. To our knowledge, jointly extracting aspect-sentiment expressions and determining the corresponding aspect-level rating was not attempted before.
3 LDA–CRF: A generative discriminative hybrid model
As mentioned in Section 1, we consider the task of jointly extracting aspect–sentiment phrases (i.e., aspect-specific opinion expressions), grouping them under relevant aspect categories, and also determining the rating of aspects for each review. Mathematically, our problem can be defined as follows: Given a set of reviews, let each review d = (s d, 1, s d, 2, s d, 3, . . .s d, L) has L sentences and each sentence s d, l = (w d, l, 1, . . .w d, l, N) contains N words and HA at index i, w d, l, i = HA, i ∈ [1, N]. Our task is to discover the set of aspect opinion expression {o d, l} l ∈ [1, L] and a set of aspect rating {u d, a}, a ∈ [1, K], where o d, l is a sub-sequence o d, l = (w d, l, p, . . .w d, l, q), p ⩽ i ⩽ q of sentence s d, l that contains the opinion expression surrounding the HA w HA = i and u d, a is the rating of the HA a = HA for each review d. The task involves three sub-tasks: (i) detection of phrase boundary surrounding the HA expressing the opinion about the aspect which is facilitated by a discriminative sequence model (CRFs), (ii) grouping the aspect-sentiment phrases under the appropriate aspect categories (topics) which is achieved by a modified version of the generative LDA and (iii) finally, assigning the rating to each aspect of the review using the latent features accomplished via a regression component. All the sub-components of the model are combined in a Monte Carlo EM framework providing a holistic approach for fine-grained sentiment analysis.
For ease of understanding, Table 3 lists the summary of notations used in the paper. We assume that each product domain has set of reviews and each review d is associated with L sentences and the aspect ratings in the range [1, R], where R = 5 in most of online reviews (which are based on a five-star scale scoring). Number of aspects is predefined as K. In the following, we first describe how sequence labeling task is used to extract candidate aspect–sentiment phrases and then we propose a novel generative-discriminative hybrid model to generate the aspect-sentiment words/phrases simultaneously with their rating score.
3.1 Candidate aspect sentiment phrase detection
Sequence models have been used successfully for extracting generic opinion expressions (Choi et al. Reference Choi, Cardie, Riloff and Patwardhan2005, Reference Choi, Breck and Cardie2006; Breck et al. Reference Breck, Choi and Cardie2007). We employ linear-chain CRFs (Lafferty et al. Reference Lafferty, McCallum and Pereira2001) for modeling aspect-sentiment phrases. Breck et al. (Reference Breck, Choi and Cardie2007) investigated that simple binary encoding produce better results as compared to the traditional Begin-Inside-Outside encoding for opinion expression. Thus, we tag each word as simply either in the phrase or not. We formulate the phrase detection task as follows: Let w = (w t = 1. . .w t = T) denote the sequence of observed words in a sentence s, and let each observation w t have a label y t ∈ Y, where Y = {0, 1}. If y t = 1, w t is part of a contiguous aspect-specific opinion phrase in s, otherwise not. Next, we describe the features used in our model as follows:
3.1.1 Features
Table 2 details the template of features we used in our model. Yang and Cardie (Reference Yang and Cardie2014, Reference Yang and Cardie2012) and Sauper et al. (Reference Sauper, Haghighi and Barzilay2011) had shown that word (W) and Part-Of-Speech (POS) tag features as well as their bigrams (W+POS) are effective in various sequence modeling tasks specifically in generic subjective opinion expression. However, they are unable to capture the effect of the HA on its opinion expression. Hence, we pivoted our features around the HA and considered five type of features which cover the syntactic and semantic properties of words surrounding the HA: (1) POS tag, (2) phrase chunks from parse tree, (3) prefixes such as anti, mis, non, pre, sub, un, etc., (4) suffixes such as able, est, ic, ing, ive, ness, etc., (5) sentiment polarity of words from the opinion lexiconFootnote 2 in Hu and Liu (Reference Hu and Liu2004).
Pivot Templates: Subscript i denotes the index of the head aspect, HA (italicized), subscript j denotes the index relative to i
Notations used in paper
We generated the family of features by looking in the context words in window of size up to 3. Each template generates the set of binary features which captures the various positional context around the HA. Various features from these templates coupled with the value of the current sequence tag at y t or a combination of current and previous labels y t, y t−1 and thus serve as our linear chain features, f k(y t−1, y t, w). Further, the index t for linear chain features can refer to any word in the sentence and not necessarily the HA, yielding us a very rich feature space.
3.1.2 Learning the CRF model
Given a set of training examples 
$\lbrace \mathbf {w_n},\mathbf {\overline{y_n}}\rbrace$, where 
$\mathbf {\overline{y_n}}$ are the correct sequence tags, we estimate the CRF Λ = {λk} parameters by minimizing the negative log-likelihood (NLL),
$$\begin{equation}
\Lambda = \text{argmin}_{ \Lambda }( C\sum _{n} \log (p(\mathbf {\overline{y_n}} | \mathbf {w_n},\Lambda )) +\sum _{k}\lambda _k^2 )
\end{equation}$$
where 
$p(\mathbf {\overline{y}}|\mathbf {w},\Lambda )\propto \text{exp}(\sum _{k}\sum _{t}\lambda _k f_k(y_{t-1},y_t,w)), C$ is the soft-margin parameter and the term ∑kλ2k indicates L 2 regularization on the feature weights, λk. To learn Λ, it is possible to efficiently compute the expectation of each feature function with respect to the CRF model distribution for every w n in the training data. To efficiently train the model, transition and forward–backward variables of the CRF are defined as follows:
\begin{align}
\psi _t(j,i,w)= p(y_t=j|y_{t-1}=i)p(w_t=w|y_t=j)
\end{align}
\begin{align}
\alpha _t(j)=\sum _{i \in Y}\psi _t(j,i,w_t ) \alpha _{t-1}(i)
\end{align}
\begin{align}
\beta _t(j)=\sum _{j \in Y}\psi _{t+1}(j,i,w_{t+1} ) \beta _{t+1}(j)
\end{align}
\begin{align}
p_{CRF}(y_{t-1},y_t | w) \propto \alpha _{t-1} (y_{t-1}) \psi _t(y_t,y_{t-1},w_t ) \beta _t (y_t)
\end{align}
$$\begin{eqnarray}
&&p_{CRF}(y_{t-2},y_{t-1},y_t | w) \propto \nonumber \\
&&\quad \alpha _{t-2}(y_{t-2}) \psi _{t-1}(y_{t-1},y_{t-2},w_{t-1}) \psi _t (y_t,y_{t-1},w_t ) \beta _t(y_t)
\end{eqnarray}$$
3.2 Formal definition of the generative process of LDA–CRF
LDA–CRF is a hybrid model which takes advantage of an embedded CRF model for emission of aspect and aspect-specific opinion phrases in documents. Its regression component predicts the aspect ratings from various latent variables. We perform sentence-level modeling as recommended in Titov and McDonald (Reference Titov and McDonald2008) and Fei et al. (Reference Fei, Chen and Liu2014) as it yields sharper aspects. The plate notation of the model is shown in Figure 1 and the generative process is described in Figure 2.
Plate notation of LDA–CRF.
Generative process of LDA–CRF hybrid.
First, we draw a domain-specific aspect distribution θ ~ Dir(α). Then for each aspect a, we draw the aspect-specific terms (words/phrases) distributions which are multinomials φAa from Dir(β) and aspect background distribution which are multinomials φBa from Dir(β). Each aspect-specific term (unigram aspects and aspect–sentiment phrases) and background word distribution are represented as a probability distribution over all the words/phrases in vocabulary v 1. . .V. Next, we assume that each sentence of a review can be assigned to single aspect which is generally true in case of reviews domain. So, for each review sentence (document), s d, l of a review, d we draw an aspect, z d, l ~ Mult(θ). Since there can be exponential number of candidate phrase sequences possible for each sentence s d i.e. 
$\langle w_{d,l,j} \rangle _{j=st}^{j=st+\text{len}-1}$ of arbitrary length len(len = 1 for words; len > 1 for phrases) starting at an index st ∈ [0, N] generating all the sequences is intractable. We observed that most of opinion expressions are centered around the HA thereby causing the space of potential opinion expressions to be quite sparse. We took advantage of following observation and trained a sequence model (e.g., CRF) for phrase sequence tagging as described in the previous subsection. We generated M = 5 best sequence labelings of each sentence sd(s m ∈ 1. . .Md) via forward Viterbi and backward A* search.Footnote 3 Hence, our vocabulary is the union of unigrams and n-grams discovered by CRF over M best sequence labeling, i.e., the model’s vocabulary, 
$V = \lbrace w_{d,l,j} \rbrace \cup \lbrace \langle w_{d,l^{m},j}\rangle _{j=st}^{j=st+\text{len}-1} \rbrace \ \forall d,l,j,\text{len},st,m$.
Next, we associate a latent switch variable for each word r d, l, j and each phrase 
$\langle r_{d,l^{m},j} \rangle _{j=st}^{j=st+\text{len}-1}$ of vocabulary, where each switch variable r ∈ {0, 1}. Depending upon the switch variable, word/phrases would be drawn from aspect-specific or background distribution. To generate each term 
$\langle w_{d, l^{m},j}\rangle _{j=st}^{j=st+\text{len}-1}$ of the labeled sequence s m ∈ {1. . .M}d, l, we first set the switch variables for the sentence, s d, l via the discriminative CRF model, i.e., 
$\langle r_{d,l^{m},j} \rangle _{j=1}^{j=N} \leftarrow \frac{1}{Z} \text{exp}\left( \sum _{j} \sum _{k} \lambda _{k} f_{k}\left( r_{j-1},r_{j},w_{j} \right) \right)$ by fitting a previously trained CRF model. The switch variables, r ∈ {1, 0} for a particular tagging sm d, l of sentence s d, l span over all its words and take values r = 1 for words being part of an aspect-specific opinion phrase or r = 0 for aspect background words, upon observing all words in s d, l. Finally, rating of each aspect of reviews is generated using the regression model which we describe in the next section.
3.3 Polarity score generation
We posit the prediction of rating score of aspects as a regression problem. We want to predict a real-valued observation u d, a which represent the intensity of the sentiment expressed about the aspect in a review. In Mcauliffe and Blei (Reference Mcauliffe and Blei2008), it was shown that latent topic structure can improve the regression as compared to that of words alone. We took a similar approach but instead of using the whole latent topic representation as feature, we considered the top forty terms of each aspect topic as the topic representation. Also, each term of the sentence of the aspect–sentiment phrase and its latent sequence variables are considered as features for rating regression. The response variable of our rating regression can be thought as being generated as u d, a = g(X, ω) + b which is an arbitrary regression model with bias b, set of features X and corresponding weights ω. The feature set X is rich and comprises of several elements beyond observed words. Features for generating the rating score of aspect a of review d includes features from sentences s d, l for which latent aspect assignment is a i.e. z d, l = a defined on the observed word ⟨w d, l, j⟩j = Nj = 1) of the sentences, latent switch variables ⟨r d, l, j⟩j = Nj = 1 and various combination of these. For each sentence, we generated the following features that comprises X:
• Aspect–sentiment probability: We calculated the aspect–sentiment probability of a sentence by taking maximum over all possible sentiment phrases switch probability. Each sentiment-phrase switch probability can be calculated from equation (11). We denoted it as P(r d, l = 1). Since this would be a real number, we converted it into an integer for feature. This probability represents how much confidence does the generative component have for the aspect opinion expression in a sentence, which clearly plays an important role while giving the aspect rating. There can be multiple sentences where reviewer talks about an aspect, but sentences which have higher confidence about an aspect opinion expression should be preferred for predicting the aspect rating. We later show empirically that appending every feature with this probability improves the aspect rating prediction significantly.
• Unigrams and n-grams: We considered all the words of sentence as feature, but since all possible n-gram would result in large feature space, we considered only the n-gram of sentences which are present in n-grams of top forty terms of each aspect topic. We also appended the aspect–sentiment probabilities with each words and n-grams.
• Syntactic features: Word level POS tag (e.g., VBD, ADJ) are used as used features with appending the sentences probabilities.
• Polarity count: Count of positive, negative words in a sentence according to opinion lexicon are considered as features.
Crucially, we note that aspect ratings are observed and we want to keep such opinion phrases higher in topic which explains the aspect rating as well. For that, we obtained the probabilistic version of the regression model, i.e., p(u d, a|g(·), X, ω) which explains the probability of the current response u d, a using a previously trained regression model g(·) and current assignment of the latent sequence tags ⟨r d, l, j⟩j = Nj = 1 and observed words ⟨w d, l, j⟩j = Nj = 1 for the current sentence s d, l. We leverage this conditional probability during model inference in Section 4.
3.4 Adding coherence
The posterior of generative topic models often favors terms with high frequency. This unfortunately causes non-phrasal terms to rank higher under each aspect due to their frequency which is undesirable. This section employs the GPU (Mahmoud Reference Mahmoud2008) model to optimize phrase collocations. Pólya urn sampling schemes work on colored balls and urns. In the context of topic models, a term can be seen as a ball of a certain color and a topic as an urn. The topic-word distribution is reflected by the color proportions of balls in the urn. Traditional topic models (e.g., LDA) follow the simple Pólya urn process in the sense that when a ball of a particular color is drawn from an urn, the ball is put back to the urn along with a new ball of the same color. The content of the urn changes over time, which gives a self-reinforcing property known as ‘the rich get richer’. This process corresponds to assigning a topic to a term in Gibbs sampling. The GPU model differs from simple Pólya urn in its sampling process. When a ball of a certain color is drawn, two balls of that color is put back along with a certain number of balls of some other colors. Unlike simple Pólya urn, GPU sampling not only allows us to see a ball of the same color again with higher probability, but also increases the probability of seeing balls with certain other colors. These additional balls of certain other colors added to the urn increase their proportions in the urn. Thus, GPU caters for promotion of others terms (balls) in a principled manner which we leverage as follows.
3.4.1 Proposed PSM–GPU model
We optimize the collocations of relevant aspect words and phrases in the GPU framework in two ways:
Word to phrase: Intuitively, if an aspect word is assigned to a topic, then that topic should represent that aspect and all other phrases in that aspects phrase set (i.e., phrases containing that aspect) should belong to the same topic. So, for each aspect words w, we define a ‘phrase set of w’ which contain all the phrases having aspect word w. Thus, when an aspect word w is assigned to a topic then each phrase in its phrase set is promoted with a small count in that topic.
Phrase to word: When a phrase 
$\langle w_{d,s,j} \rangle _{j=st}^{j=st+\text{len}-1}$ is assigned to a topic, each component word w d, s, j where j ∈ [st, st + len−1] in it is also promoted with a certain small count, i.e., each word of that phrase is also assigned to that topic by a certain amount.
We now define the term promotion matrix, A for the GPU framework. Every element of A, A w, w ′ refers to the promotion pseudocount, i.e., whenever a w was seen in an urn, we increment the count by A w, w ′ of w ′. w, w ′ can be word or phrase.
$$\begin{equation}
A_{w,w^{^{\prime }}} = \left\lbrace \begin{array}{@{}l@{\quad }l@{}}1 & \mathrm{\quad if }\ w = w^{^{\prime }} \\
\sigma & \mathrm{\quad if\ }w\mathrm{\ is\ an\ aspect\ word, } \\
& w^{^{\prime }} \mathrm{is\ a\ phrase} \in \mathrm{Phrase\ set\ of\ } w \\
\delta & \quad w^{^{\prime }} \mathrm{is \ a \ word} \in \mathrm{Phrase}\ w \\
0 & \mathrm{\quad otherwise} \end{array}\right.
\end{equation}$$
To improve the ranking of phrases, the value of σ is kept greater than δ. Empirically, values are given in Section 5.2.
4 Inference
The task of inference is to find the model parameters, i.e., values of latent z, r and regression parameters that most likely explain the data. We accomplish this by using Monte Carlo EM (Diebolt and Ip Reference Diebolt and Ip1996), a Gibbs sampler for the E-step to assign sentences to topics and value of switch variables. After randomly initializing the latent variables z, r and the first iteration of the trained regression model, g(·), we alternate between Gibbs sampling z, r (E-step) and optimizing the regression parameters (M-step). This scheme can also be referred to as stochastic EM.
E-step: As latent variables z and r belong to different levels, we hierarchically sample z and then r for each sweep of a Gibbs iteration. During inference, we marginalize over multinomial distribution given all the observed variable. We sample the topic of each sentence with probability:
$$\begin{eqnarray}
p(z_{d,l}&=&a \vert Z_{\lnot d,l},R_{\lnot d,l},W_{\lnot d,l} ) \propto \left[ \frac{(n_{a}^{l})_{\lnot d,l}+\alpha )}{(n_{(\cdot )}^{l})_{\lnot d,l}+ A\alpha )} \right] \nonumber \\
&&\times \left[ \left( \prod _{v=1}^{V}\frac{\Gamma (m_{a,v}^A+ \beta )}{\Gamma ((m_{a,v}^A)_{\lnot d,l}+ \beta )} \right) / \left( \frac{\Gamma (m_{a,(\cdot )}^A+ V\beta )}{\Gamma ((m_{a,(\cdot )}^A)_{\lnot d,l}+ V\beta )} \right) \right] \nonumber \\
&& \times \left[\left( \prod _{v=1}^{V}\frac{\Gamma (n_{a,v}^{B}+ \beta )}{\Gamma ((n_{a,v}^B)_{\lnot d,l}+ \beta )} \right) / \left( \frac{\Gamma (n_{a,(\cdot )}^B+ V\beta )}{\Gamma ((n_{a,(\cdot )}^B)_{\lnot d,l}+ V\beta )} \right) \right]
\end{eqnarray}$$
where nla denotes the # of sentences in review set assigned to aspect a, nB a, v denote the # of times term v was assigned to aspect a in the aspect-specific background language models, respectively, while mA a, v denotes the # of times term v assigned to aspect a in the aspect-specific language model with the promotion of terms using the promotion matrix A. It is defined as follows: mA a, v = ∑w ′ = 1V(A v, w ′*mA a, w ′). A count variable with subscript (·) signifies the marginalized sum over the latter index and ¬ denotes the discounted counts. The switch sampler has the following Gibbs conditional distribution:
$$\begin{eqnarray}
p(r_{d,l,j}&=&1 \vert z_{d,l}=a,w_{d,l,j}=v, \ldots ) \propto \nonumber \\
&&\frac{(m_{a,v}^A)_{\lnot d,l,j}+ \beta )}{(m_{a,(\cdot )}^A)_{\lnot d,l,j}+ V\beta } \times p_{\text{CRF}} (r_{d,l,j-1},r_{d,l,j}=1|v)
\end{eqnarray}$$
$$\begin{eqnarray}
p(r_{d,l,j}&=&0 \vert z_{d,l}=a,w_{d,l,j}=v, \ldots ) \propto \nonumber \\
&&\frac{(n_{a,v}^B)_{\lnot d,l,j}+ \beta )}{(n_{a,(\cdot )}^B)_{\lnot d,l,j}+ V\beta )} \times p_{\text{CRF}} (r_{d,l,j-1},r_{d,l,j}=0|v)
\end{eqnarray}$$
$$\begin{eqnarray}
p(\langle r_{d,l,j}&=&1 \rangle _{j=st}^{j=st+\text{len}-1} \vert z_{d,l}=a, \langle w_{d,l,j}=v^{^{\prime }} \rangle _{j=st}^{j=st+\text{len}-1}, \ldots ) \propto \nonumber \\
&&\frac{(m_{a,v}^A)_{\lnot d,l,j}+ \beta )}{(m_{a,(\cdot )}^A)_{\lnot d,l,j}+ V\beta } \times p_{\text{CRF}} (r_{d,l,j=st-1},r_{d,l,j=st}=1,\nonumber \\
&& r_{d,l,j=st+1}=1, \ldots r_{d,l,j=st+\text{len}-1}=1 | v^{^{\prime }}) \times p(u_{d,a} |g(\cdot ),X)
\end{eqnarray}$$
$\langle r_{d,s,j}=1 \rangle _{j=st}^{j=st+\text{len}-1}$ forming an aspect (z d, l = a)-specific opinion phrase, v ′ starting at index j = st and of length len, where 
$v^{^{\prime }}= \langle w_{d,s,j}\rangle _{j=st}^{j=st+\text{len}-1}$. The sequence probabilities, p CRF in equations (9)– (11) is computed as described in Section 3.1 while decoding the CRF model. The aspect rating probability p(u d, a|g(·), X) is computed using the probabilistic regression as described in Section 3.3. Each of the three terms in equation (11) reflects a different influence on the switch variable structure, first term is from structure of vocabulary signifying how many times a phrase is assigned to this aspect topic, second term from sequence probabilities p CRF, which bear the sequence probability and takes into account the surrounding words as well as their labels. Third term take into account the phrases which best explained their rating score.The rational here is that the probability of a response u d, a obtained from regression model trained in previous stochastic EM/Gibbs iterations conditioned on the current assignment of the latent sequence tags r as the features for the regression model can further help align the sequence tags r for this iteration having observed the aspect rating u d, a. In other words, the sequence tag distribution is shifted to those tags that best explain the given response.
M-step: After sampling the aspect for each topic and switch variable for each word and phrases in a document, we then find new regression parameters that maximize the likelihood conditioned on the current state of the sampler. In our case, we employ a support vector regression as our regression model (as it performed better than vanilla least square regression based on our pilot experiments) and compute p(u d, a|g(·), X, ω), using the Bayesian interpretation method described in Chang and Lin (Reference Chang and Lin2011) and optimize the regression parameters using the LIBSVM package. We alternate between E and M steps with a stride size of 200. First, 400 iterations of E-step were performed for topics to stabilize and after that, beyond every 200 E-step iteration, an M-step is performed for finding new regression parameters. For the first 400 iteration of the E-step, the conditional aspect rating probability was dropped from switch variable equation in equation (11). Also, during testing of aspect rating prediction task, we do not have aspect rating and hence conditional aspect rating probability term was dropped from equation (11). After learning regression parameter in the subsequent iterations completes, equation is used for switch probability. We performed a total of 1,000 iterations of stochastic EM.
5 Experimental evaluation
In this section, we describe our dataset and various parameters in our proposed approach. We also compare our models to competitive baselines to show the effectiveness of our approach in opinion expression extraction and aspect rating prediction task.
5.1 Labeling of reviews using heuristics
One of the bottleneck of CRF is it requires labeled dataset for sequences and labeling requires a lot of manual effort. Since we mainly focus on opinion expression surrounding the HA word, heuristic rule could be written with the help of POS tags and polarity of surrounding words. We followed the approach on annotation of phrases similar to Wiebe et al. (Reference Wiebe, Wilson and Cardie2005). We used the opinion lexiconFootnote 4 derived from Hu and Liu (Reference Hu and Liu2004) for positive and negative words. Same lexicon is used for constructing features of CRF as well regression component. Labeling of the phrase boundary surrounding the aspect word for positive opinions phrases is as follows:
In the first step, we searched for the positive terms (using sentiment lexicon) in the window of [− 5, 5] around the HA word. Next, to define realistic phrase boundaries, it should include the opinion about the HA word but does not include the opinion about the other aspect. So only considering the extremes with a sequence index of [HA, positive-term] would not yield good phrases. Therefore, we divided the presence of positive word in two cases. One case is when positive word is before the aspect word. Since opinion words are mostly adjectives or adverbs we considered, these two cases based on POS tag of aspect word. If aspect word is noun, then we took the farthest adjective from aspect word because most adjectives occur before noun and we want to capture all the opinion words in phrase talking about noun aspect. If aspect word is verb, then we took nearest adverb since adverbs are situated immediately before the verb in most cases. Second case is when positive word is after the aspect word. Then we take the nearest adjective (from aspect word) because later adjectives might be modifying other aspect noun aspect and generally if adjective occur after a noun then it follows the noun immediately. There might be cases when there is no positive adjective or adverb found surrounding the aspect word which is dealt later. We also included the negative phrases with negator words in surrounding window by finding negative terms in window of [−5, 5] and then finding negator terms such as ‘not’, ‘do not’ and following the same procedure described above. This process generally yields us the minimum boundaries of phrases but may omit some opinion words which we included using the method described below.
Next, we extended the phrase boundaries using the following heuristic rules that were made from the basic definition of adjective and adverb: (i) if first word of phrase is coming before HA word and it is verb, then go one index before word and see if the word is adverb then include that in phrase, (ii) if last word of phrase is adjective and next word after that is noun, then include all the consecutive noun after that, (iii) if last word of phrase is adverb include the next word if it is verb, (iv) if last word of phrase is noun include all the consecutive nouns after that.
Similar process was applied for extracting negative opinion expressions as well. Finally, after applying the above rules for automatic labeling of phrase boundaries, we corrected the phrase boundary by eyeballing the phrases. This semi-automatic heuristic method significantly reduced manual work in labeling phrases. Please note that the scale of our data is quite large as it spans over 25,000 reviews and each review has several sentences. Hence, a manual labeling of exact phrases is prohibitive. We believe that our labeled phrases were still of high quality as demonstrated by the final phrases extracted by our model which were trained on phrases labeled using the above scheme.
5.2 Dataset, preprocessing and parameter setting
We include two review datasets in our experiments: a hotel review dataset from TripAdvisor used in Wang et al. (Reference Wang, Lu and Zhai2010), and another dataset that we crawled from TripAdvisor on restaurant domain. In the hotel domain, in addition to the overall rating, reviewer also provides rating on seven pre-defined aspects in each review: value, room, location, cleanliness, check in/front desk, service and business service. We merged the ‘serivce’ and ‘business service’ aspect and considered six aspects for hotel domain. In restaurant domain, reviewer provides rating on four pre-defined aspects: food, service, value and atmosphere. All the ratings in both the dataset are in the range from 1 to 5 stars. Since we focus on opinion phrase extraction and rating prediction, we extracted the seed aspect words of each aspect using unsupervised topic models in Zhao et al. (Reference Zhao, Jiang, Yan and Li2010) and Mukherjee and Liu (Reference Mukherjee and Liu2012) to tag the index of the HA in sentences. Table 4 provides the statistics of data for both the domains. Reviews in both domain are highly skewed toward positive review rating which can be observed by having average aspect rating (average of all the aspect ratings of all reviews) to be greater than 4.
Dataset statistics
First, we performed a pre-processing step such as removing the stopwords, punctuation, special characters and words appearing less than five times in each domain. All the words are converted into lower case. For all models, posterior estimates of latent variables were taken with a sampling lag of fifty iterations post burn-in phase (of 200 iterations) with 1,000 iterations in total. Dirichlet priors were set to α = 50/K, where K is the number of topics (empirically set to ten via pilot) and β = 0.1. The CRF parameter C = 1 and GPU parameters σ = 0.05 and δ = 0.01 were estimated using cross-validation. We used the software packageFootnote 5 with RBF kernel for regression component whose parameters C = 10 and g = 0.01 were also set using cross-validation. We experimented with epsilon-support vector regression and nu-support vector regression in LIBSVM and there was not much difference in performance so for all the experiments we used epsilon-support vector regression with value of ε = 0.1 which was set using cross-validation.
Dataset dissection: Table 4 describes the statistics of data used for evaluation. We labeled around 3,000 sentences spanning both domains. The labeled phrases in column 3 of Table 4 are taken from reviews which are not included in the training set (column 2 in Table 4) of the LDA–CRF model to avoid overfitting. For aspect–sentiment expression evaluation, LDA–CRF and baseline models were evaluated on the reviews in column 2 of Table 4. For aspect rating prediction, these reviews (column 2 in Table 4) were further split randomly into eighty per cent train and twenty per cent test. Only after splitting the reviews, those sentences which had HA present are considered as part of the reviews. Columns 4 and 5 of Table 4 are the total number of instances used for training and testing of the aspect rating prediction task. Each aspect instance can have more than one sentences of a review. The number of aspect instances is less than the number of reviews because we took the aspect instances from reviews for which at least two opinion sentences are present for that aspect and ground truth aspect rating is given in the dataset. Each review have aspect ratings associated with it which were used as a supervision signal during training of LDA–CRF and other rating regression baselines. The original aspect ratings in the test data were used as the ground truth for computing MSE. The test set was kept same for LDA–CRF and other baselines for an impartial comparison.
Next, we evaluated our approach on both the tasks of opinion expression and aspect rating prediction and compared their performances with strong baselines on the individual tasks.
5.3 Aspect-specific sentiment expression evaluation
We compared our model qualitatively by assessing the quality of extracted phrases as well as quantitatively on various evaluation metrics.
5.3.1 Baselines
We consider the following most relevant phrase extraction models as our baselines:
• Topical n-gram (TNG): The Topical n-gram model in Wang et al. (Reference Wang, McCallum and Wei2007) extends LDA to model n-grams of arbitrary length. As aspects often appear close to their opinion in the sentence, topical n-grams for each aspect form a natural baseline. We used the authors’ original implementation in the MALLET toolbox.
• LDA-P with GPU (LDA-P–GPU): This model is due to Fei et al. (Reference Fei, Chen and Liu2014) and is tailored for phrase extraction in opinion mining. It considered all noun phrases as candidate aspect opinion expression. It employs LDA in the GPU framework to rank the aspect phrases higher in their topics. We used the noun phrase discovered by the OpenNLP Parser as separate term in vocabulary for implementations of LDA-P–GPU.
• semi-Markov CRF with GPU (sMC–GPU): This model builds over the model of Yang and Cardie (Reference Yang and Cardie2012) that used dependency tree features and semi-CRF to model the arbitrarily long expressions. We used these expression spans as multiword in the entire vocabulary. Then we employ GPU-based sampling with LDA proposed by Fei et al. (Reference Fei, Chen and Liu2014) to collocate opinion expressions.
5.3.2 Result analysis
Qualitative evaluation of phrases: To evaluate the quality of extracted aspect-specific opinion expressions, we labeled the topic following the instruction of Mimno et al. (Reference Mimno, Wallach, Talley, Leenders and McCallum2011). First, each topic is marked as coherent or not based on the top words of topic. If most of the words of topic belong to same semantic concept or aspect, then a topic is marked as coherent. We considered the top forty terms of topic in descending order of their probabilities to check for coherence of topics. In second step, each term of coherent topics are marked as correct if that term convey the similar meaning as that of topic otherwise it is marked incorrect. Two human judges were used in the annotation. Agreements being high (κ > 0.78), disagreements were resolved upon consensus among judges.
Figure 3 shows the top ten words/phrases of aspect ‘room’, ‘location’, ‘cleanliness’ for hotel domain and ‘food’ and ‘service’ for restaurant domain. Each topic phrases also have corresponding predicted aspect rating and each topic is converted into bins as shown in Figure 3 based on the aspect rating of phrase. Each bin’s aspect rating are in range indicated by arrow. We observed that topics were highly coherent for each aspect mainly due to promotion of aspect phrases in each topic based on aspect words and incorrect terms are mainly aspect words or opinion words such as ‘food’ and ‘experience’ which got promoted in GPU due to their high frequency. One important behavior of reviewer we observed from topical phrases is that if reviewer talk about specific aspect property they tend to give extreme rating, e.g., ‘parking free’ have five rating and ‘bathroom mirror filthy’ have one rating, while if they comment on whole aspect they generally use adjective words like ‘good’, ‘nice’, etc. for rating four and for aspect rating five, they uses adjectives like ‘excellent’, ‘very good’, etc. similar for negative rating reviews. We have not shown the aspect-specific phrases for baselines because none of them are able to extract the aspect rating along with opinion phrases, but we compare the opinion phrases extracted by all baselines quantitatively in the next subsection using two metrics: average precision@n and # of coherent topics.
Quantitative evaluation of phrases
Example of topical aspect-specific opinion phrases delimited by comma with corresponding aspect rating discovered on the scale by LDA–CRF model. Incoherent phrases of topics are marked in red, while incorrect phrases whose predicted aspect rating is incorrect are marked in blue.
Average precision@n: Figure 4 shows the average precision@n(p@n) where n is 40, 20 and 10, respectively, of all coherent topic for all the models of both domains. We note that LDA–CRF model outperform all the baselines significantly for all the p@n in both domain. Nearest baseline is sMC–GPU whose phrases are lengthy and have overlapping aspect opinion such as ‘nice hotel great location friendly staff decent price’ because they form a bigger segment units by combining the individual segments of parse tree, while LDA–CRF uses its discriminative model based on HA to detect sharper phrase boundary. LDA-P–GPU is next in terms of performance because all noun phrases might not be talking about the aspect opinion expression, hence its phrases are not as expressive as compare to sMC–GPU and LDA–CRF. We observed that precision at higher n is higher compare to lower n due to presence of some aspect or opinion words which have huge frequency in phrases and GPU promotion of words of phrases increases their probabilities but number of such terms is very low.
Charts from left to right are average precision@n where n are 40, 20 and 10, respectively, for coherent topic and last chart is the number of coherent topic in each model for both domains.
# of coherent topics: Last chart in Figure 4 shows the number of coherent topic for each model in both domains. It shows that LDA–CRF and sMC–GPU have same number of coherent topic but precision is better for LDA–CRF which shows the effectiveness of our approach. In general, LDA–CRF have more or equal number of coherent topic compared to baselines in both domains. We have not compared our approach with baselines using the Topic Coherence (Mimno et al. Reference Mimno, Wallach, Talley, Leenders and McCallum2011) metric because it uses co-document frequency of words which is good metric for comparing unigram terms for coherence. Since the frequency of phrases are sparse, the Topic Coherence metric is not directly applicable. The qualitative results of phrases in Figure 3 does reflect substantial topical coherence. Additionally, we choose to directly evaluate the discovered phrases using their predicted aspect rating. As we will see below, on this aspect rating prediction metric also the proposed model outperforms state-of-art baselines thereby showing that discovered phrases have both topical and rating coherence.
Evaluation of opinion phrase We evaluated our model’s ability to extract the opinion phrases with a deep learning model. One of the major advantage of a deep learning model is automatic learning of features in the hidden layers. We implemented a model due to Huang et al. (Reference Huang, Xu and Yu2015) which uses bidirectional LSTM–CRF model for sequence tagging task. We used word-based micro precision, recall and F-score to evaluate the quality of the model. Yang and Cardie (Reference Yang and Cardie2013) has shown that it is difficult to get exact opinion expression boundaries even for human annotators and hence focused on precision, recall measures at word level instead of complete expression level. Precision is defined as 
$\frac{|C \cap P|}{P}$ and recall as 
$\frac{|C \cap P|}{C}$, where C and P are the correct and predicted set of word labels, respectively.
We reported our result in Table 5. We use Stanford’s publicly available GloVeFootnote 6 hundred-dimensional embeddings (Pennington et al. Reference Pennington, Socher and Manning2014). Other hyperparameters such as dimension of the hidden states of LSTM are d = 100, # of layers as 3, batch size as 10 and maximum length of sentence was set to 50. We used five-fold cross-validation to evaluate the model and only considered labeled data for training and testing the models.
Comparison of results for phrase detection
Table 5 shows that precision of CRF and LSTM model is similar, while recall is low for CRF model. We believe that considering top n-phrases during decoding in CRF is able to extract most of the useful phrases in the dataset. We also considered sMC–GPU whose recall is better than CRF as a baseline and see that the eventual model outperforming other baselines significantly (see Figure 4). Even though LSTM-based model or semi-Markov CRF is performing best in the overall F-score measure of opinion phrase extraction, there are few advantages of using CRF in clustering the semantic similar phrases.
For clustering, we are extracting the top n-phrases and marginal probabilities of words and phrases using the dynamic programming technique. While we can get top n-phrases for LSTM–CRF model using the beam search, but it is often used for discrete state. It is important to take a note that it is computationally both expensive and harder to use beam search efficiently as pointed in Bengio et al. (Reference Bengio, Vinyals, Jaitly and Shazeer2015) for the continuous state model, since there is no way to factor the path in a continuous space. Hence, the actual number of candidate phrases that can be kept during beam search decoding must be reduced significantly if the scheme has to work efficiently which would directly impact downstream phrase extraction and grouping in our proposed architecture.
5.4 Aspect rating prediction
We used the following metrics for evaluation of aspect rating prediction:
MSE: Mean square error is used to measure the difference in ground truth aspect rating and predicted aspect rating. Lower the MSE value better is the model prediction.
$$\begin{equation}
\text{MSE} = \sum _{d=1}^{D} \sum _{a=1}^{K} (u_{d,a} - \hat{u}_{d,a})^2 / K*D
\end{equation}$$
where u d, a is the predicted aspect rating and ûd,a is the ground truth aspect rating for aspect k and review d.
ρreview : To measure how much predicted rating of aspect k can preserve the ground truth aspect rating across all the reviews of hotel/restaurant. Pearson correlation across hotels/restaurant (Wang et al. Reference Wang, Lu and Zhai2010) is defined as
$$\begin{equation}
\rho _{\text{review}} = \frac{\sum _{a} \rho (u_a,\hat{u}_{a})}{K}
\end{equation}$$
where u a is the predicted aspect rating vector for all reviews on aspect a, ûa is the ground truth vector of all reviews on aspect a.
5.4.1 Baselines
We consider the following baselines to compare our model in the task of aspect rating prediction:
• sLDA: Mcauliffe and Blei (Reference Mcauliffe and Blei2008) proposed an sLDA model in which response variable was generated for each document with normal linear model. Since each sentence is considered as a document in the generative process of LDA–CRF. We first annotated the sentences to aspects using the aspect keywords supervision. Then we combined the sentences which belong to same aspect and review as single document to sLDA model. We assigned the aspect rating of review as supervised response signal in training (column 4 in Table 4) phase. During testing (column 5 in Table 4), the response variable of combined sentences (single document in sLDA) was predicted and considered as aspect rating.
• MedLDA: Zhu et al. (Reference Zhu, Ahmed and Xing2012) proposed a MedLDA model which generate a response variable for document similar to Mcauliffe and Blei (Reference Mcauliffe and Blei2008) but model was trained using the max-margin principle. We took the similar approach as in sLDA baseline to compare it with our model. For both sLDA and MedLDA baseline, we used the implementation provided by this packageFootnote 7 of the original author.
• Latent aspect rating analysis (LARA): Wang et al. (Reference Wang, Lu and Zhai2010) proposed to predict the latent aspect rating for each review but they assumed each review should talk about all the aspect. So we combined all the review of the product into single review following their approach. We combined the reviews which belong to same hotel or restaurant in single review and called it ‘h-review’. Aspect and overall rating of ‘h-review’ was average of all review’s aspect and overall rating, respectively, which belong to same hotel or restaurant. For fair comparison, during training, we combined the reviews which are used in training of LDA–CRF model and similarly for testing data. Also, we separated test and train set of reviews before performing any pre-processing to perform the aspect segmentation of reviews as described in Wang et al. (Reference Wang, Lu and Zhai2010). We used the implementationFootnote 8 provided by authors.
• GG: Snyder and Barzilay (Reference Snyder and Barzilay2007) proposed the Good Grief (GG) algorithm which jointly rank the individual aspect by analyzing the meta-relation among aspect like agreement and contrast. They describe rank as assigning integer between 1 to k which can be referred as rating prediction on each aspect. We used code of the original authorsFootnote 9 for experiments.
• LASSO: We also compared our model to a standard text regression model with L1 regularizer (lasso) in which words and bigrams are considered as features. We used MATLAB implementation for experiments.
• LDA–CRF–WR: We also test simplified version of our model in which features are constructed without using the switch probability. We considered only top forty terms of the topics, its n-grams and words of sentences as features.
5.4.2 Discussion
Table 6 shows the result of all the baselines on two metrics defined above for hotel and restaurant domain. We have highlighted the best performing method for each domain in two metrics. We have grouped the GG and LARA baseline differently because LARA uses only overall rating of review and does not require aspect rating, while GG assign a rank (an integer) between [1, R] which makes them not directly comparable. One downside of LARA is the assumption that each review talks about every aspect which is not true generally and that is why they combine all the review of each hotel which make the comparison unfair due to aspect rating noise normalization of reviews. One can argue that their model is completely unsupervised in terms of aspect rating, while their MSE performance is very close to our supervised aspect approach. We believe that averaging the aspect rating of various reviews reduces the noise significantly. Also, as our model can discover aspect-specific opinion phrase along with its aspect rating, it can be used to compare the reviews of user for two different hotel on same aspects which might be useful for users who wish to avoid biases that might appear in a particular person’s review.
Experimental results on aspect rating prediction. Bold values correspond to best performance for each measure
We observe that our complete LDA–CRF model achieves the best MSE among all the baseline. LDA–CRF outperformed significantly our model’s baselines LDA–CRF–WR which does not include the features of switch variable of phrases. It shows that the joint learning of phrases which takes the latent switch variable assignment features with aspect rating substantially improves the aspect rating task. We believe that LDA–CRF outperformed the MedLDA and sLDA because of inclusion of features of candidate phrases produced by the embedded discriminative component in our model. We believe that MSE value of GG is very low due to discrete nature of rank prediction of model. We are able to outperform the LASSO which uses unigram and bigram as features. It shows that our model was able to extract informative sentiment words in top terms of aspect topic (n-grams of top phrases are considered as features). Since aspect-specific opinion phrases are mined, extracting sentiment words from phrases is trivial by simply selecting words after removing the stop words and aspect.
We are not able to outperform LARA and GG in correlation metric but outperformed other baselines. Part of the reason for this is the fact that our model does not use overall review rating as a supervision signal but only considers aspect rating independently, whereas LARA and GG heavily relies on overall rating of the review. ρreview indicates how well the relative order of predicted aspect rating is maintained across reviews and GG, LARA focuses on the aspect rating prediction based on overall rating and predicted ratings of other’s aspects which enforces the order. Hence, their performance is better than ours which assumes independent aspect rating prediction. We further investigated the aspect-level correlation and found that there are certain aspect such as ‘room’ in hotel domain which are talked most of time and their correlation is higher than 0.6, but there are aspect such as ‘check-in’ which are discussed very less hence our regression model was not able to perform well on those aspects. Hence, overall average correlation is low but our main focus was on developing a model which jointly learns aspect-specific opinion phrases as well as their corresponding aspect rating. And, MSE directly evaluates this per aspect phrase predicted rating (as opposed to the relative rating ordering for all aspects) where we are outperforming all the baselines significantly.
6 Conclusion
In this paper, we presented a novel joint model (LDA–CRF) for solving two challenging problem in fine-grained sentiment analysis, i.e., aspect rating regression and aspect-specific opinion expression extraction. We presented a model LDA–CRF whose embedded CRF component generates the candidate phrases, the generative topic model component clusters those phrases in coherent aspect topics, and its regression component generates the aspect rating for each aspect based on the phrases and latent variable assignments. We experimentally showed that the inclusion of latent variable features improves the aspect rating prediction significantly on two datasets. We also showed that our model was able to extract coherent and expressive aspect-specific opinion expression using the GPU sampling. Empirical results on real-world datasets showed that LDA–CRF outperformed several strong baselines in the task of opinion expression and aspect rating prediction.
Further, investigation of our results showed that the aspect rating prediction can be improved in correlation metric by incorporating rating of other aspects using overall rating. We wish to pursue this in our future work.
