1 Introduction
Collecting peoples' opinions, particularly evaluations of products, services or brands, is becoming more important in the commercial activities of both companies and individual consumers, because they want to sort out such opinions from multiple perspectives due to the diversification of items and people's various preferences. Due to the emergence of massive online text resources such as product review articles and messages in bulletin boards on the WWW, now the deeper analysis of people's opinions from text documents is not only feasible, but even necessary, especially for enterprises.
Beyond estimating the overall preference for a product or a service by a writer, detecting the individual utterances that specify each positive or negative property is very important for enterprises that continually seek information to improve their products or services while estimating the future demands for various products.
For example, a customer's review of a specific digital camera may contain the Sentences (1) and (2). In Sentence (1), an evaluation with positive (+) polarity (3) is given, and the two evaluations with opposite polarities (4) and (5) are found in the single sentence (2). The evaluations should be output in more structured forms as at the right side of (3)–(5) for the deeper analysis that can help users aggregate the frequently appearing features or target products.
(1) I think the contrast is sharp.
(2) While the new model doesn't have a zoom lens, the price is low.
(3)
(4)
(5)
Therefore, the task we address here is the clause-level detection of expressions that indicate positive and negative evaluations, in other words, a fine-grained type of opinion mining or sentiment analysis. The clause-level approach is desirable not only for the high-precision detection of evaluation expressions relying on predicate–argument structures, but also for the construction of contextual structures that enable the automatic acquisition of a new lexicon of evaluation expressions. An unsupervised lexicon induction method is the main contribution of this article.
The clause-level detection of semantic expressions requires multiple types of syntactic and semantic knowledge. First, we need lexical entries such as ‘to be sharp’ and ‘to have a zoom lens’ that describe favourable features in the digital camera domain. Formally, we describe a lexicon for the evaluation expressions with the set of minimal syntactic structures specifying the polarity in an expression, as in examples (6)–(8). In the rest of this article, we call them polar atoms.
(6)
(7)
(8)
Also we need systematic ways to appropriately handle such syntactic phenomena as the negation in (4). A previous work (Kanayama, Nasukawa and Watanabe Reference Kanayama, Nasukawa and Watanabe2004) showed that this approach has good precision in the detection of evaluation expressions when using a domain-independent lexicon and syntactic pattern.
Now the challenge is to increase the recall, namely, to cover new domains and extract informative knowledge. In light of this, building of a domain-dependent lexicon is very important, but it is difficult because the possible lexical entries are too numerous and they may differ in each domain. For example, the adjective ‘predictable’ has a positive polarity in the software domain, while it has a negative polarity in the domain of movie reviews. Even within a single domain, the required knowledge to assess polarity tends to be very complex: while ‘the face was in focus’ is a good result when taking a picture, ‘the fence (behind my friend) was in focus’ expresses a bad experience.
To build a domain-dependent lexicon without tedious manual work, we have devised an unsupervised method where a user only needs to collect unannotated corpora in the target domain. Making use of a domain-independent lexicon and syntactic patterns, which are already available, our method exploits context coherency in terms of polarity, an assumption that polar clauses with the same polarity will appear successively unless the context is changed with an adversative expression. For example, suppose the two Sentences (1) and (2) appear in this order. These two sentences contain three clauses and they are semantically connected as in Figure 1.
An example of the context coherency observed in the two Sentences (1) and (2). The main clauses of the sentences are connected vertically, and the subordinate relation is represented by the horizontal connection. ⊗ indicates that the polarity was changed by an adversative conjunction.
Exploiting this tendency for context coherency, we can collect candidate expressions with their tentative polarities as those adjacent to the polar clauses that have been identified by the initial lexicon that consists of domain-independent polar atoms as seeds for learning. Though there are intrinsic noises in such candidates, appropriate polar atoms are selected through statistical tests, and high precision is achieved. In addition, our learning method does not require any manual tuning of the threshold values.
In Section 2, we review related work, and Section 3 describes our runtime detection systems for clause-level evaluations in Japanese documents and the structure of the lexical entries for evaluation, the polar atoms. Section 4 explains our primary assumption for unsupervised learning, context coherency. To assess the strength of our assumption, we introduce the key metrics of coherent precision and coherent density. Section 5 describes our unsupervised learning method. Experimental results are shown in Section 6, and we conclude in Section 7.
2 Related work
The detection and classification of evaluations and opinions have been extensively studied in recent years as reviewed in Pang and Lee (Reference Pang and Lee2008), and these technologies are actually desired in the business world (Ahmad et al. Reference Ahmad, Taskaya–Temizel, Cheng, Gillam, Ahmad, Traboulsi and Nankervis2004; Wright Reference Wright2009). This section reviews previous work related to two of the main features of this article: clause-level methods for extracting evaluation expressions and unsupervised learning methods for lexical acquisition.
The technologies to analyse people's opinions are roughly classified into document-level classification and mention-level extraction. Typical document-level approaches classify a document into positive and negative (Pang, Lee and Vaithyanathan Reference Pang, Lee and Vaithyanathan2002; Turney Reference Turney2002; Dave, Lawrence and Pennock Reference Dave, Lawrence and Pennock2003; Gamon Reference Gamon2004), or finer ratings expressed by a number of ‘stars’ (Pang and Lee Reference Pang and Lee2005; Goldberg and Zhu Reference Goldberg and Zhu2006) where a document is assumed to have a single sentiment or a level of preference. These approaches have merits in that predefined evaluations of documents can be used for the supervised learning and the performance can be evaluated relatively easily. However, as mentioned in Section 1, our goal is to grasp individual favourable or unfavourable features of the products or services in question. Even sentence-level classification (Hu and Liu Reference Hu and Liu2004; Meena and Prabhakar Reference Meena and Prabhakar2007) is not sufficient because a single sentence may contain multiple positive or negative opinions, as pointed out in Nasukawa and Yi (Reference Nasukawa and Yi2003) and Wilson et al. (Reference Wilson, Wiebe and Hwa2006). Therefore, in this paper, we focus on the extraction of each utterance at a finer level as phrases or clauses (Popescu and Etzioni Reference Popescu and Etzioni2005; Wilson et al. Reference Wilson, Wiebe and Hwa2006). The clause-level sentiment detection relies on syntactic and semantic operations analogous to those in machine translation (Kanayama et al. Reference Kanayama, Nasukawa and Watanabe2004) for high precision. This method is the basis of this work described in Section 3. The expressions to be detected vary depending on the task. For example, Yu and Hatzivassiloglou (Reference Yu and Hatzivassiloglou2003) separated facts from opinions and assigned polarities only to opinions. This distinction is important for the analysis of news articles and leads to the task of subjectivity analysis (Riloff and Wiebe Reference Riloff and Wiebe2003; Pang and Lee Reference Pang and Lee2004; Wiebe et al. Reference Wiebe, Wilson, Bruce, Bell and Martin2004). In contrast, in documents such as product reviews, even rather objective opinions tend to have polarities. They are called evaluative factuals (Nigam and Hurst Reference Nigam and Hurst2004) and provide tangible information on the properties.
Unsupervised learning for sentiment analysis is being studied. For example, Hatzivassiloglou and McKeown (Reference Hatzivassiloglou and McKeown1997) labelled adjectives as positive or negative, relying on semantic orientation. Turney (Reference Turney2002) used collocations with ‘excellent’ or ‘poor’ within ten words from the given polar words, to obtain positive and negative clues for document classification.
Many other learning methods for sentiment analysis have been proposed, and they will be compared with the proposed method in Section 6.5 after a detailed description of our method.
3 Methodology for clause-level evaluation detection
Our approach towards clause-level analysis is aimed at fine-grain, high-precision sentiment analysis. This concept was originally discussed as a transfer-based extraction method analogous to machine translation (Kanayama et al. Reference Kanayama, Nasukawa and Watanabe2004). The MT method introduced there detected evaluation expressions with high precision using syntactic and semantic operations, and also constructed more informative semantic structures for opinion mining than the surface expressions which were extracted by the previous methods.
This section describes the first of the two advantages of the MT method: the use of deeper syntactic knowledge for the precise detection of evaluation expressions. Figure 2 illustrates the flow of our evaluation detection system, which involves two major steps. The first step is proposition detection. A proposition is a clause that can form a polar clause, and is identified by applying patterns to the syntactic tree corresponding to the input sentence. The next step is polarity assignment: the polarity of each proposition is examined by considering polar atoms, the lexicon for this task.
The flow of the clause-level evaluation detection.
The remainder of this section describes these two processes, the definitions of the polar atoms, and the accuracy of the clause-level evaluation detection.
3.1 Proposition detection
‘Clause-level’ means that we focus on the predicative verbs and adjectives, rather than handling all of the words or phrases that may convey the sentiment of the writer as word-level or phrase-level approaches have done. In the Japanese implementation, the predicative verbs and adjectives such as in (9) are detected, and adnominal (attributive) usages of verbs and adjectives as in (10) are ignored, because ‘
’ (‘beautiful’) in (10) does not convey a positive polarity.
(9)
(10)
Here, we use the notion of a proposition as a clause without modality, led by a predicative verb or a predicative adjective. The propositions clarify the syntactic units which are subject to the assignment of polarities.
Basically, we detect a proposition only at the head of a syntactic treeFootnote 1. However, this limitation reduces the recall of evaluation detection to a very low level. In the example (9) above, the adjective ‘
’ (‘beautiful’) is the head of the tree, while those initial clauses in (10) and (11)–(15) below are not. To achieve higher recall while maintaining high precision, we apply two types of syntactic patterns, modality patterns and conjunctive patterns Footnote 2, to the tree structures from the full-parsing.
(11)
(12)
(13)
(14)
(15)
Modality patterns, exemplified in Table 1, match some auxiliary verbs or corresponding sentence-final expressions, to allow for specific kinds of modality and negation. One of the typical patterns is [
] (‘I think 
’)Footnote 3, which allows the clause led by ‘
’ (‘beautiful’) in (11) to be a proposition. Note that the expression ‘I think ’ can be regarded as the same meaning as just saying ‘’. This treatment is specific to opinionated types of documents, but it can be applied to any domain, so we can use these patterns as domain-independent knowledge.
Japanese modality patterns used to identify propositions. ‘+neg’ denotes the proposition has the opposite polarity
Negation is also handled with a modality pattern, such as [
] (‘not ’). In this case, a neg feature is attached to the proposition to identify ‘
’ in (12) as a negated proposition. In contrast, no proposition is identified in (13) due to the deliberate absence of a pattern [ -
] (‘It is good if ’) and more complex expressions such as in (14). We used a total of 103 domain-independent modality patterns, most of which are derived with the MT method and selected from a set of frequently used sentence-end expressions which come after verbs and adjectives.
Another type of pattern is a conjunctive pattern, which allows multiple clauses in a sentence. We used a total of tewnty-two conjunctive patterns, also derived with the MT method, as exemplified in Table 2. In such coordinative clauses and causal clauses, both clauses are regarded as propositions. For example, two propositions are detected in (15) because the coordinative particle ‘
’ is found in the conjunctive patterns. On the other hand, no proposition is identified in a conditional clause such as in (13) due to the absence of corresponding conjunctive patterns. In this manner, our clause-level approach does not explicitly split a sentence into clauses, but processes clauses which can potentially be polar clauses by using the conjunctive patterns.
Japanese conjunctions used for conjunctive patterns
Both types of patterns are applied recursively and therefore two or more patterns (e.g. [
] and [ -
]) can match the combinations of the expressions (e.g. ‘-
’ (‘I don't think ’)) and the appropriate propositions are identified.
The entire syntactic trees with full parsing are needed for these operations because the relations between the clauses, especially the roles of subordinate clauses, have to be captured correctly. Also, the full parser increases precision for the detection of evaluations expressed by verb/adjective clauses that may contain embedded clauses, as shown by Nasukawa and Yi (Reference Nasukawa and Yi2003). In the Japanese-to-English translation engine, Japanese sentences are parsed by a full parser that exploits hand-crafted grammar and statistics (Kanayama et al. Reference Kanayama, Torisawa, Mitsuishi and Tsujii2000). Its current accuracy is about 90 per cent for the dependencies, which is not perfect but is the state-of-the-art level, and this apparently gives richer structures for complex sentences than shallow parsing. Parsing errors seldom affect the precision of the evaluation detection because errors may lead to failure of matching of the syntactic subtree with the patterns, but this merely lowers the recall.
3.2 Lexicon: polar atoms
We exploit polar atoms as the lexicon for the assignment of polarities to propositions. A polar atom consists of a polarity, the key declinable word (a verb or adjective) and optionally, its arguments. Example (16) is a simple polar atom, where no argument is specified. This atom matches any proposition whose head is ‘
’ (‘beautiful’). Example (17) is a complex polar atom, which assigns a negative polarity to any proposition whose head is the verb ‘
’ (‘lack’) and where the accusative case is ‘
’ (‘attraction’). This matching method is based on the MT method, in which the roles of each argument are appropriately estimated using case frames.
(16)
(17)
Compared to surface n-grams which tend to be the output of statistical learning as clues of sentiment analysis, the polar atoms introduced here are much more understandable for humans because polar atoms form the predicate–argument structures from which we can easily grasp their meanings by imagining the finite predicative form, and thus the lexicon itself can be a knowledge base for the favourable and unfavourable features. We prepared a total of 3,275 domain-independent polar atoms such as (16) and (17), most of which are originally derived from an English sentiment lexicon (Yi et al. Reference Yi, Nasukawa, Bunescu and Niblack2003). The polar atoms will be outputs of learning for domain adaptation in this study.
3.3 Polarity assignment using polar atoms
To assign a polarity to each proposition, polar atoms in the lexicon are compared to the proposition. A polarity is assigned if there is a polar atom for which the verb/adjective and the arguments coincide with the proposition, and otherwise no polarity is assigned. When the proposition is compared with a complex polar atom, the children nodes that modify the verb/adjectives with Japanese case markers (postpositional particles) are examined, and the polarity is assigned when the noun and the case marker coincide with the polar atom. The opposite polarity of the polar atom is assigned to a proposition that has the neg feature.
Here, the matching process of Japanese case markers (postpositional particles) handles several syntactic phenomena. For example, the proposition in the example (18) matches the polar atom (19) even though the postpositional particles are different (‘
’ versus ‘
’). With the knowledge that a topic marker ‘
’ can behave like the nominative case marker ‘
’, this matching can be done beyond the surface form. This process may be much more complicated when a verb is transformed into special forms, e.g. passive or causative. The details of this process were described using an analogy between sentiment extraction and machine translation (Kanayama et al. Reference Kanayama, Nasukawa and Watanabe2004).
(18)
(19)
3.4 Effects of the clause-level approach
In order to see the reliability of the clause-level evaluation detection using the syntactic patterns and domain-independent polar atoms, the previous study (Kanayama et al. Reference Kanayama, Nasukawa and Watanabe2004) compared the evaluation detection method using syntactic patterns and a more superficial approach relying only on a verb/adjective lexicon. Table 3 shows the experimental results in Section 5 of that paper to review the performance of polarity assignment in the digital camera domain. The results were compared by using these metrics, where #ab denotes the system's output is a and the manual output is b, and p, n, x stand for positive, negative and nonpolar clauses, respectively:
Weak precision:
. The coincidence rate of the polarity between the system output and the manual output when both the system and the human evaluators assigned either a positive or negative clause.Strong precision:
. The coincidence rate of the polarity between the system output and the manual output when the system assigned either a positive or negative clause.Recall:
. The detection rate of polar clauses within the manual output.
. The coincidence rate of the polarity between the system output and the manual output when both the system and the human evaluators assigned either a positive or negative clause.
. The coincidence rate of the polarity between the system output and the manual output when the system assigned either a positive or negative clause.
. The detection rate of polar clauses within the manual output.Precision and recall for the detection of polar clauses from 200 sentences
According to the observations, high precision was achieved because the modality patterns appropriately handled the syntactic phenomena such as negation and the polar words in the embedded clauses and the conditional clauses were appropriately excluded from the polar clauses. Also, the complex polar atoms contributed to limit the detection of polar clauses taking the semantics of the child node into consideration.
However, the recall was lowered by a prudent detection method based on syntactic trees and the lack of some domain-dependent expressions. Therefore, we want a method to obtain the domain-dependent knowledge to raise the recall, in a systematic way.
4 Context coherency
This section introduces the intra- and inter-sentential contexts in which we assume context coherency in terms of polarity of evaluation, and describes some observations based on this assumption.
4.1 Intra- and inter-sentential context
As some previous work pointed out, sentiment words with the same polarity (positive or negative) tend to collocate (Hatzivassiloglou and McKeown Reference Hatzivassiloglou and McKeown1997; Turney Reference Turney2002), and they can be clues to acquire new knowledge for polarities. Here is an example discourse (20) from the review of a digital camera, which contains four positive evaluations and two negative ones.
(20)
The first clause of the third sentence and the fourth sentence convey the negative evaluations, and the others convey the positive evaluations. The identification of the propositions described in Section 3.1 clarifies this distribution of the polar clauses. Figure 3 illustrates the context coherency in (20) from our viewpoint with two types of contexts: intra- and inter-sentential contexts.
The concept of the intra- and inter-sentential contexts, where the polarities are perfectly coherent. The symbol ‘⊗’ denotes the existence of an adversative conjunction.
The intra-sentential context is the link between propositions in a sentence, which are detected as coordinative or causal clauses. If there is an adversative coordinative conjunction such as ‘
’ (
-‘though’-
) in the third sentence in (20), a flag is attached to the relation, as denoted by ‘⊗’ in Figure 3. Though there are differences in syntactic phenomena, this is similar to the semantic orientation proposed by Hatzivassiloglou and McKeown (Reference Hatzivassiloglou and McKeown1997).
The inter-sentential context is the link between propositions in the main clauses of pairs of adjacent sentences in a discourse. The polarities are assumed to be the same in the inter-sentential context, unless there is an adversative expression such as those listed in Table 4. If no proposition is detected as in a nominal sentence, the context is split. This means that there is no link between the proposition of the previous sentence and that of the next sentence.
Inter-sentential adversative expressions
Note that the proposition ‘splendid’ in the first sentence and the proposition ‘light’ are close in the surface strings, but they are not connected in the context based on propositions. Our clause-level operation enables this kind of appropriate handling of the coherency of polarities.
4.2 Observations of context coherency
We believe these two types of context can be used for unsupervised learning as clues to assign a tentative polarity to unknown expressions, and this holds true in various domains. To test our assumption, we studied several corpora.
4.2.1 Corpora
In this section and the next one, we use Japanese corpora from discussion boards in four different domains, whose relevant features are summarised in Table 5. All of the corpora have clues to the boundaries of the posts in the form of HTML tags, so they were suitable to identify the discourses. We did not use any other structural information such as the date, author or human ratings that might have depended on the source documents.
The corpora from the four domains used in this article. The ‘#Posts’ and ‘#Sentences’ columns denote the numbers of posts and sentences, respectively. ‘Length’ is the average length of the sentences in Japanese characters
4.2.2 Coherent precision
How strong is the coherency in the context described in Section 4.1? Using the polar clauses detected by the evaluation detection system with the initial lexicon, we observed the coherent precision of domain d with lexicon L, defined as
where #(Coherent) is the occurrence count of the adjacent pairs of polar clauses that have the same polarities, as found in the discourse. Similarly, #(Conflict) is the occurrence count of the adjacent pairs of polar clauses that have opposite polarities.
There are always some errors in the automatic detection of evaluation expressions, but these errors do not affect the value of (21) because the weak precision is quite high, as described in Section 3.4. The strong precision was 89 per cent, so only 79 per cent of the co-occurrences of two polar clauses are estimated to be reliableFootnote 4, but this kind of error merely reduces both #(Coherent) and #(Conflict) assuming the errors are independent, so we can use the cp value in (21) for the coherent precision.
As strategies to identify the two polar clauses in these relationships, we compared two methods:
Window. A polar clause and the nearest polar clause found in the preceding n sentences in the discourse.
Link. Two polar clauses in the intra- and/or inter-sentential links as described in Section 4.1. This is the viewpoint of the context in our method.
Table 6 shows the frequencies of coherent pairs, conflicting pairs and the coherent precision for half of the digital camera domain corpus. ‘Baseline’ is the percentage of positive clauses among the polar clauses.
Coherent precision with various viewpoints of the contexts
For the ‘Window’ method, we tested for n = 0, 1, 2 and ∞. ‘0’ means two propositions within a sentence. Apparently, the larger the window size, the smaller the cp value. The window size ‘∞’ implies that the nearest polar clause is picked up from anywhere within a discourse, where the average length is seven sentences. In this case, the ratio is larger than the baseline by only 2.7 per cent, and thus the coherency between distant clauses is not reliable even though the number of clues is relatively large.
‘Link’ shows the coherency of the two types of context that we considered. The cp values are much higher than those in the ‘Window’ methods, because the relationships between adjacent pairs of clauses are handled more appropriately by considering syntactic trees, adversative conjunctions, etc. The cp values for inter- and intra-sentential contexts are almost the same, and thus both contexts can be used to obtain 2.5 times more clues for the intra-sentential context. This result has an interesting implication because the inter-sentential context is observed in any of the successive sentences that has a proposition in the main clause, while the intra-sentential context relies on the overt occurrences of coordinative or causal conjunctions. In the rest of this article, we will use both contexts to exploit more clues for learning.
We also observed the coherent precision for each domain corpus. The results in Table 7 indicate that the number varies slightly among the corpora. Although context coherency is an intuitively reasonable assumption, there are still many discordant examples in all of our domains, even if we adopt our viewpoint of context coherency.
Coherent precision for each domain
We selected one hundred cases of polarity conflicts in the digital camera domain. From our analysis, the causes of the conflicts can be classified into five groups (with the number in parenthesis indicating the percentage):
Errors in sentence delimitation (11%): Typically, the two sentences were not divided correctly due to the omission of a period between the sentences.
Undefined adversative expressions (19%): Besides the expressions in Table 4, there were some adversative expressions like ‘
’ (‘even so’), with which an opposite sentiment occurred.Changing topics (18%): Besides the adversative expressions, a topic change may link opposite polarities. For example, ‘A was really bad. B has much better quality’.
Postings without discourse (28%): Blunt writing (e.g. ‘I like A. I hate B’.), and non-narrative writing such as itemizing.
Others (24%): Other errors including parsing or evaluation detection, but most of these errors are due to typos or extremely informal writing and are too difficult to parse.
The last three types (70 per cent) of the conflicting cases are intrinsic noise and cannot be eliminated easily, therefore we need to exploit the tendency of context coherency in a way that is robust against errors.
4.2.3 Coherent density
Besides the conflicting cases, there are many other cases where a polar clause does not appear in the polar context, because even more sentences do not contain evaluation expressions. We also observed the coherent density of the domain d with the lexicon L, defined as
This shows the ratio of polar clauses that appear in the coherent context, among all of the polar clauses detected by the system. Therefore, when the detected polar clause is isolated and has no adjacent polar clauses with the same polarity, this value decreases.
Table 8 shows the coherent density in each domain. The movie domain has notably higher coherent density than the others. This indicates the sentiment expressions are more frequently used in the movie domain. In non-opinionated documents such as newspaper articles, this value should be much lower.
Coherent density for each domain
Note that the coherent density does not reflect the true distribution of the following evaluation expressions because the recall of the automatic evaluation detection is not high enough. This value only indicates the observed ratio of coherence based on the current system with the initial lexicon, but this value can be used for the unsupervised learning method described in the next section. We assume the coherent density equals the probability that a polar clause detected with a newly defined polar atom appears in the polar context, because this value remains approximately unchanged even if a polar atom is added to the lexicon, as long as the lexicon is large enough.
5 Unsupervised learning for acquisition of polar atoms
In Section 4, the polarities tend to be consistent both in the intra- and inter-sentential contexts, but the coherency is far from perfect. This section proposes an unsupervised learning method, which consists of a linguistic approach to acquire candidates as new polar atoms, and a statistical method to pick up appropriate polar atoms from among the candidates.
Figure 4 shows the flow of the method. First, the runtime evaluation detection system identifies the polar clauses, then the candidate polar atoms are collected from the surroundings of the polar clauses. Next, each candidate atom is validated using its frequency count and the two metrics in the previous section, cp and cd, which are calculated from all of the polar clauses found in the domain corpus.
The flow of the unsupervised learning process.
5.1 Counts of candidate polar atoms
From each proposition to which a polarity is not assigned, a candidate polar atom in the form of a simple atom (just a head verb or adjective in the proposition) is extracted. When the proposition has arguments consisting of the pair of a noun and a postpositional, a complex atom (a verb or adjective and its rightmost argument) is also extracted.
For example, from the discourse (23), a simple atom (24) and a complex atom (25) are extracted as candidates when the second sentence in (23) is not detected as a polar clause due to the absence of the corresponding polar atom. Note that no polarities are yet assigned to these atoms at this stage.
(23)
(24)
(25)
For each candidate polar atom a, the total appearances in the corpus f(a) is counted. Also for each a, its occurrences in positive contexts p(a) and negative contexts n(a) are counted, based on the context of the adjacent clauses, using the method described in Section 4.1. Again, in the example (23), assuming ‘
’ (‘splendid’) has a positive polarity in the lexicon, a positive polar clause is detected in the first sentence and thus the proposition in the second sentence is regarded as the positive context. Therefore, f(a) and p(a) are incremented for the two atoms (24)–(25) due to the appearance of the discourse (23). Naturally, if the proposition has the neg feature, the count of the opposite polarity is incremented.
Table 9 shows examples of candidate polar atoms with their frequencies.
Examples of candidate polar atoms and their frequencies. ‘*’ denotes that it should not be added to the lexicon
5.2 Determination whether add to the lexicon
How can we assign the polarities to the located candidate polar atoms? In other words, how can we distinguish the true polar atoms, which should be added to the lexicon, from the false polar atoms, which should be discarded? As shown in Section 4, both the coherent precision (72–77 per cent) and the coherent density (7–19 percent) are so small that we cannot rely on each single appearance of the atom in the polar context.
One possible approach is to set threshold values for the frequency in a polar context, (p(a), n(a)), and for the ratio of appearances in polar contexts among the total appearances, 
. However, the optimum threshold values tend to depend on the corpus and the initial lexicon.
In order to set general criteria, here we assume that a true positive polar atom a should have higher 
than its average throughout the corpus. The average of among the positive polar atoms in the lexicon (Lp) weighted by the frequency of a is calculated as
and assuming the symmetry between positive and negative atoms (let Ln be the negative polar atoms in the lexicon), the average equals
Thus, it coincides with the coherent density defined in Section 4.2.3.
The p(a) and f(a) are observed values, and thus whether the true probability exceeds cd(d,L) can be estimated with 90 per cent confidence using the inequality (26)Footnote 5. We compute the confidence interval using approximation with the F-distribution (Blyth Reference Blyth1986)
![\begin{linenomath}\refstepcounter{enums}
\label{equ:1}
\[
\begin{array}{l} \quad q > cd(d,L), \\ \bkdm \bkdm \bkdm \mbox{where} \quad \displaystyle{\sum^{p(a)}_{k=0}} {}_{f(a)}\mbox{C}_k \times q^k(1-q)^{f(a)-k} = 0.9.\qquad \quad \end{array} \mbox{(\theenums)}
\]
\end{linenomath}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20151023080445606-0425:S1351324911000131_eqn26.gif?pub-status=live)
Similarly, a true positive polar atom a should have higher 
than its average throughout the corpus. The average of among the positive polar atom in the lexicon (Lp) weighted by the frequency of a is calculated as
and assuming the symmetry between positive and negative atoms, the average equals
Thus, it coincides with the coherent precision defined in Section 4.2.2. Again, we conduct a statistical estimation with 90 per cent confidence. When the two inequalities (26) and (27) are satisfied, a is regarded as a true positive polar atom. Obviously the criteria for the negative polar atoms are analogous
![\begin{linenomath}\refstepcounter{enums}
\label{equ:2}
\[
\begin{array}{l} \quad r>cp(d,L) \mbox{or} n(a)=0, \\ \bkdm \bkdm \bkdm \mbox{where} \quad \displaystyle{\sum^{p(a)}_{k=0}}{_{p(a) + n(a)}}\mbox{C}_k \cdot r^k(1-r)^{p(a) + n(a)-k} = 0.9.\quad \end{array} \mbox{(\theenums)}
\]
\end{linenomath}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20151023080445606-0425:S1351324911000131_eqn27.gif?pub-status=live)
These criteria simultaneously solve the problems in setting the threshold values, minimum frequency and scope of the polar atoms. In the example of Table 9, the simple atom ‘
’ (‘to be small’) (ID = 1) is discarded because it does not satisfy (27), while the complex atom ‘
’ (‘body is small’) (ID = 3) is adopted as a positive atom. The ‘
’ (‘to be firm’) (ID = 2) is adopted as a positive simple atom, even though ten cases out of sixty four were observed in the negative context. On the other hand, ‘
’ (‘to be delivered ← on Thursday’) (ID = 4) is discarded because it does not satisfy (26), even though 
, i.e. it is always observed in negative contexts.
6 Evaluation
6.1 Evaluation using polar atoms
For the evaluation of our learning method, first we propose a method of evaluation for the lexical learning.
It is costly to make consistent and large ‘gold standards’ in multiple domains, especially for identification tasks such as clause-level evaluation detection (cf. document-level classification tasks). Therefore, we evaluated the learning results by asking human annotators to classify the acquired polar atoms as positive, negative and non-polar, instead of classifying the instances of polar clauses detected with the new lexicon. This can be done because the polar atoms themselves are informative enough to imply to humans whether the expressions hold positive or negative meanings in the domain.
To justify the reliability of this evaluation method, two annotators evaluated 200 randomly selected candidate polar atoms in the digital camera domain. The agreement results are shown in Table 10. The manual classification was agreed upon in 89 per cent of the cases and the Kappa value was 0.83, which is high enough to be considered consistent.
Agreement of two annotators' judgments of 200 polar atoms (κ = 0.83)
Using manual judgment of the polar atoms, we evaluated the performance with the following three metrics.
Lexicon Precision. The coincidence rate of the polarity between the acquired polar atom (positive/negative) and the human evaluators' judgments (positive/negative/non-polar). It is always false if the evaluators judged it as ‘non-polar’.
Runtime Precision. The estimated precision of the detection of polar clauses with newly acquired polar atoms, in the runtime evaluation detection system. The results of the manual judgments are weighted by their frequency in the corpus, and multiplied by the estimated precision when the ideal lexicon is used, assuming that the newly acquired polar atoms which are judged correct by humans are as reliable as those in the initial lexicon.
Relative Recall. The estimated ratio of the number of detected polar clauses with the expanded lexicon to the number of detected polar clauses with the initial lexicon. Relative recall will be one when no new polar atom is acquired. Since the precision was high enough, this metric can be used for approximation of the recall, which is hard to evaluate in extraction tasks such as clause-/phrase-level evaluation detection.
6.2 Effects of determination method of polar atom
In order to see the effects of our method of determination of polar atoms, we conducted a learning experiment using the corpus on digital cameras described in Section 4.2.1. First, we obtained the candidate polar atoms using the domain-independent lexicon, as described in Section 5.1. Next, the polar atoms were tested using our method from Section 5.2.
To measure the two types of precision, the annotators evaluated hundred randomly selected polar atoms that were newly acquired. The lexicon precision is calculated by simply counting the correct polar atoms and dividing the number by hundred. The runtime precision is calculated as
![\begin{linenomath}
\[
\mbox{RunPrec} \simeq \frac{\sum_a \mbox{freq}(a)\mbox{correct}(a)} {\sum_a \mbox{freq}(a)} \mbox{Prec}(B),
\]
\end{linenomath}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20151023080445606-0425:S1351324911000131_eqnU30.gif?pub-status=live){kind=small}
where freq(a) is the frequency of a polar atom a in the corpus, correct(a) is 1 if a is judged as correct (or 0 otherwise) and Prec(B) is the precision of the baseline method. Here, Prec(B) is set to 89 per cent, derived from the strong precision in the experimental results shown in Section 3.4.
Relative recall is estimated by comparing the numbers of detected polar clauses with and without the acquired polar atoms. Therefore, the relative recall is estimated as
![\begin{linenomath}
\[
\mbox{RelRec} \simeq 1+ \frac{\mbox{Num}(A)\mbox{Prec}(A)}{\mbox{Num}(B)\mbox{Prec}(B)},
\]
\end{linenomath}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20151023080445606-0425:S1351324911000131_eqnU31.gif?pub-status=live){kind=small}
where Prec() and Num() denote the precision and the number of polar clauses detected by the system in the 2,000 sentences, respectively, and A and B denote the use of the newly acquired lexicon and the baseline lexicon, respectively. We used the lexicon precision for Prec(A) and set Prec(B) at 89 per cent again.
As a control experiment, we also tried preset criteria, where the candidate atom a is adopted when the frequency of the polarity, max(p(a), n(a)), was 3 or more, and the ratio of the polarity, 
, was higher than the preset threshold θ. Varying θ from 0.05 to 0.8, we evaluated the precisions and the relative recall. Table 11 shows the results, where we call the proposed method the ‘Statistical test’.
Evaluation results with our method (Statistical test) and preset criteria for the digital camera domain. The column ‘# of polar atoms’ denotes the number of polar atoms acquired in each domain
In our method, a very high runtime precision of 85.9 per cent was achieved. This is extremely high precision for this task, considering the correctness of both the extraction and polarity assignment was evaluated simultaneously. The lower value in the lexicon precision, 65 per cent is not as important because most of the errors were in infrequent polar atoms. When each polar atom is weighted by the frequency, the precision rises to 96.7 per cent. The relative recall of 1.25 in the digital camera domain means the recall increased from 43 per cent to 54 per cent, compared with the experiment with the domain-independent lexicon in Section 3.4.
The results showed both relative recall and runtime precision by the ‘Statistical test’ method significantly outperform those with preset threshold values, even for the optimum θ, 0.3.
6.3 Robustness for different conditions
6.3.1 Diversity of corpora
We also conducted the same experiments as in Section 6.2, to see the differences between the digital camera domain and the movie domain. Figure 5 shows the relationships between the runtime precision and the relative recall in the two domains.
Relative recall versus runtime precision with various preset threshold values θ for the digital camera and movie domains. The rightmost star and circle (labelled ST) denote the performance of the proposed method (ST; Statistical test).
The proposed method also achieved high precision and recall in the movie domain. Interestingly, unlike the normal precision–recall tradeoff, the runtime precision in the movie domain was reduced when the θ was high (i.e. the condition is strict). This is due to the frequent polar atoms that can be acquired at the low ratios of the polarity. In contrast, our method does not discard these important polar atoms.
Table 12 shows the precision and recall in each of the four domains, including mobile phones and cars. High runtime precisions were achieved in all of the domains, 84 per cent on average. Though the relative recall is smaller in the three domains other than the digital camera domain, the domain-dependent polar clauses are much more informative than the general ones, and thus the high-precision detection significantly enhances the system.
Evaluation results with our method for four domains
6.3.2 Size of the initial lexicon
We also tested the performance while varying the size of the initial lexicon L. We prepared three subsets of the initial lexicon, L 0.8, L 0.5 and L 0.2, removing polar atoms randomly. These lexicons had 0.8, 0.5 and 0.2 times the polar atoms, respectively, compared to L. Table 13 shows the precisions and recalls using these lexicons as the seeds for the learning process. Though the coherence density values vary, the precision was stable, which means that our method was robust even for different sizes of the lexicon. The smaller the initial lexicon, the higher the relative recall, because the polar atoms which were removed from L were recovered in the learning process. For example, ‘
’ (‘satisfied’), which is naturally included in L but which was removed when L 0.5 was created, was acquired automatically. This result suggests the possibility of a bootstrapping method from a small initial lexicon. However, the small seed (L 0.2) did not produce a high-precision lexicon, and the highest runtime precision (from L 0.8) was mainly due to the general polar atoms that were in the original lexicon L. Therefore, to acquire domain-dependent polar atoms with the highest possible precision, a reliable lexicon as large as possible should be used as the seed.
Evaluation results for various sizes of the initial lexicon (the digital camera domain)
6.4 Qualitative evaluation
As seen in the agreement study, the polar atoms used in our study were intrinsically meaningful to humans. This is because the atoms are predicate–argument structures derived from predicative clauses, and thus humans can interpret the meaning of a polar atom by generating the corresponding sentence in its predicative form. Table 14 shows examples of the acquired polar atoms in the four domains.
Examples of acquired polar atoms in four domains
In the evaluation process, some interesting results were observed. For example, a negative atom ‘
’ (‘to be free from vignetting’) was acquired in the digital camera domain. Even the evaluator who was familiar with digital cameras did not know the term ‘
’ (‘vignetting’), but after looking it up in the dictionary she was able to label it as negative. Our learning method discovered such technical terms and labelled them appropriately.
Also, there were discoveries in the error analysis. An evaluator assigned positive to ‘
’ (‘to have camera’) in the mobile phone domain, but the acquired polar atom had the negative polarity. This was actually an insight from recent opinions that some users want phones without camera functions because cameras tend to consume battery power or because using devices with a camera is prohibited in some buildings.
6.5 Comparison with other learning methods
As reviewed in Section 2, Hatzivassiloglou and McKeown (Reference Hatzivassiloglou and McKeown1997) and Turney (Reference Turney2002) have proposed the use of collocations of semantic orientation. In our work, we use contextual information based on the succession of sentences and multiple clauses in a complex/compound sentence, which is wider and finer than the contexts they used, and address the problem of acquiring lexical entries from the noisy clues.
Qiu et al. (Reference Qiu, Liu, Bu and Chen2009) proposed a double propagation method exploiting the attributes of products. Their work has the same objective as ours and is evaluated by the newly acquired words. Our algorithm is more general than theirs because we do not require the assumption that the polarity for a specific feature is consistent in the domain corpus.
Kaji and Kitsuregawa (Reference Kaji and Kitsuregawa2007) acquired polar phrases from a large HTML corpus by exploiting very strict syntactic patterns and layout information. They successfully detected polar expressions including multiple words which correspond to the complex polar atoms in our work, but, their work can detect only domain-independent expressions, and requires an empirical threshold value to determine the polarity of the phrases. Our work seeks to acquire knowledge for each domain to be analysed with much smaller but domain-aware sets of documents.
We do not need any information on manual ratings either, as required in other methods (Higashinaka, Prasad and Walker Reference Higashinaka, Prasad and Walker2006; Du and Tan Reference Du and Tan2009). Another difference in our work is that we aim at acquiring a finer and domain-dependent lexicon by exploiting a relatively large set of lexical entries that are generally used and proven to be reliable, as a seed lexicon, instead of acquiring a general lexicon that can be manually collected from a small seed lexicon. This feature is important from an engineering point of view because it addresses the practical real-world problem of covering more domains.
Learning methods for phrase-level sentiment analysis closely share one of the objectives of our approach. Popescu and Etzioni (Reference Popescu and Etzioni2005) achieved high-precision extraction of opinionated phrases by using relaxation labelling. Their method iteratively assigns a polarity to a word or a combination of a word and a feature, relying on the semantic orientation of the co-occurring words with specific relationships in a sentence. The unit of expressions to which a polarity is assigned is similar to the polar atom in this article, but the scope of their semantic orientation is limited within each sentence. Wilson, Wiebe and Hoffmann (Reference Wilson, Wiebe and Hoffmann2009) proposed a form of supervised learning, dividing the resources into prior polarity and context polarity, which are respectively similar to polar atoms and syntactic patterns in our work. Wilson et al. prepared prior polarities from existing resources, and learned the context polarities by using prior polarities and annotated corpora. This means the prerequisite data and learned data are reversed in comparison to our approach. Our approach helps acquire more domain-dependent knowledge, and the context polarity is easier to access in JapaneseFootnote 6. Our approach and their work can complement each other.
7 Conclusion
We devised an unsupervised method to acquire polar atoms for the clause-level evaluation detection from various domains, and demonstrated its high performance. By exploiting the tendency for context coherency and by using statistical tests, the lexicon can be expanded automatically by using unannotated corpora, and tuning of the threshold values is not required. In addition, not only a single verb or adjective, but also more complex expressions consist of a verb or adjective with an argument can be extracted automatically.
Our tuning-free method enables end-users to run the algorithm to improve the sentiment analysis. These features allow them to do on-demand analysis of more narrow domains, such as the domain of digital cameras from a specific manufacturer, or the domain of mobile phones from the female users' viewpoint.
The acquired polar atoms that consist of a verb or an adjective with an optional case are understandable by humans. This feature allows for various applications. For example, when we want to buy a product about which we do not have any background knowledge, our method allows us to identify important features for selecting products by acquiring the domain-specific polar atoms from the raw domain corpus.
