II. Krumme's Methods and Findings

The publicly available information about Krumme's (Reference Krumme2009, Reference Krumme2011) methods and findings can be summarized as follows. She wrote at the start of her popular piece that she wished to “see if I could predict the price of a bottle based on the words in the review” by seeing if certain words were “preferentially used in conjunction with expensive wines” (Krumme, Reference Krumme2011).

To do that, she collected expert descriptions from an unnamed source for some 3,000 wines ranging in price from $5 to $200 (Krumme, Reference Krumme2011). She then grouped those wines into at least two price categories, with the lowest-priced wines in one category and the highest-priced wines in another. Exactly how she grouped the wines is unclear. I will presume for simplicity that she defined two mutually exclusive groups that I will call low- and high-priced wines.

Next, she used that data to train a Naive Bayesian (NB) classifier where the words in a wine's description were used to predict whether the wine was low or high priced (Krumme, Reference Krumme2009, Reference Krumme2011). Several different specifications for a NB classifier had been proposed by the time of Krumme's study (cf. Metsis, Androutsopoulos, and Paliouras, Reference Metsis, Androutsopoulos and Paliouras2006), and exactly how she specified her classifier is unclear, but I will presume she used a Multinomial NB classifier with term frequencies and Laplace smoothing (see, e.g., Metsis, Androutsopoulos, and Paliouras, Reference Metsis, Androutsopoulos and Paliouras2006, sec. 2.2). That is a common specification for a NB classifier.

For the purposes of this paper, the relevant aspect of that classifier is that the probability of word w_i (where i indexes unique words) being used to describe a wine in price category c_j (where j is either H for the high-priced wines or L for the low-priced wines) is estimated as

(1)$$P( {w_i\vert c_j} ) = ( {n_{i, j} + 1} ) /( {n_j + v} ) , \;$$

where n_i,j is the number of times word w_i is used to describe wines in price category c_j, n_j is the number of times any word is used to describe wines in that same price category, and v is the number of unique words used to describe wines in either price category. Note that n_i,j /n_j is the frequency with which word w_i is used to describe wines in price category c_j, but Laplace smoothing adds one to n_i,j so that P(w_i | c_j) is never zero. The v in the denominator is a normalization factor.

The other aspects of a NB classifier can be ignored here because I will only be interested in the P(w_i | c_j) estimates (rather than using those estimates and additional assumptions to predict a wine's price category). It can, however, be noted that, even though a decade has passed since Krumme's initial work, studies are still applying NB classifiers to expert wine descriptions in order to try to predict aspects of wines. For example, Dong et al. (Reference Dong, Guo, Rajana and Chen2020) recently used that approach to try to predict wine ratings.Footnote ²

After training her classifier, Krumme identified “expensive” words that were “more likely to be used for expensive wines” (Krumme, Reference Krumme2011). Exactly how she identified those words is unclear, so I will again presume a simple approach. I will presume she deemed a word w_i to be “expensive” if P(w_i | c_H) / P(w_i | c_L) was significantly greater than unity and deemed a word to be “cheap” if that relative probability was significantly less than unity. Such “expensive” and “cheap” words would be the most informative ones for trying to classify wines as high or low priced.Footnote ³

Her main finding was that certain words were indicative of whether a wine was low or high priced.Footnote ⁴ She did not report all or perhaps most of the words she identified as expensive and cheap ones, but she did report some of them. Table 1 reproduces all the examples of expensive and cheap words reported by Krumme (Reference Krumme2009, Reference Krumme2011) and how she grouped them.

Table 1

Krumme's Examples of Expensive and Cheap Words

Note: This table reproduces all the examples of expensive and cheap words reported by Krumme (Reference Krumme2009, Reference Krumme2011), as well as the way in which she grouped them.

Two other intriguing findings reported by Krumme were, first, that “[there are] two nearly discrete vocabularies for cheap and expensive wines [such that] about 65% of […] words are non-overlapping” (Krumme, Reference Krumme2009) and, second, that “cheap words are more likely to be recycled [while] reviewers are more likely to create new vocabulary for top-end wines” (Krumme, Reference Krumme2011). I take the former to mean that most of the unique words used to describe high-priced wines are distinct from the ones used to describe low-priced wines. I take the latter to mean that any given unique cheap word tends to be used more often than any given unique expensive word.

III. Revisiting Those Findings

Despite some uncertainties regarding Krumme's (Reference Krumme2009, Reference Krumme2011) exact methods and data source, I can use methods that are as similar as possible to hers and a different dataset to confirm or disconfirm that the previously identified “expensive” and “cheap” words in Table 1 are indicative of whether a wine is high or low priced.

The dataset I will use here is expert wine descriptions that were scraped from the Wine Enthusiast (WE) website and made publicly available by Thoutt (Reference Thoutt2017).Footnote ⁵ That dataset has reviews of about 120,000 wines priced between $5 and $200. Most of those wines are fairly cheap (with the median, mean, and standard deviation of prices being 25, 33, and 25) and have fairly short reviews (with the median, mean, and standard deviation of the number of words per review being 41, 42, and 11). There is a moderate positive correlation between a wine's price and the number of words used to describe it (r = 0.31, p < 0.01), which is consistent with Ramirez's (Reference Ramirez2010) finding that more expensive wines have longer descriptions.

To group those wines into high- and low-priced ones, I define wines greater than $50 as high priced. Fifty dollars is arguably low for a high-priced wine, but it roughly corresponds to the upper quintile of the prices of the wines in the dataset. I define wines priced at $15 or less as low priced. Fifteen dollars is arguably high for a low-priced wine, but I use that dollar amount because it roughly corresponds to the lower quintile of the wine prices and, also, the total number of words used to describe any wine of $15 or less is almost exactly equal to the total number of words used to describe any wine over $50. Defined in that way, I have almost one million words describing about 20,000 high-priced wines and almost exactly the same number of words describing a similar number of low-priced wines.

Table 2 reports, for each “expensive” word identified by Krumme, my estimates (based on the above-described methods and data) of the relative probability of the word being used to describe a high- rather than low-priced wine. When counting instances of a word, I ignored letter case and any accents (e.g., “Cuvee” and “cuvée” would count as equivalent words), but otherwise, I only counted instances where a word was exactly the same as Krumme reported it to be.

Table 2

Relative Probabilities for Krumme's Expensive Word Examples

Notes: This table reports, for each word Krumme (Reference Krumme2009, Reference Krumme2011) identified as an “expensive” one, a point estimate and 95% confidence interval (CI) for the relative probability of the word being used to describe a high- rather than low-priced wine. These estimates are based on the dataset and methods described in the main text of this paper.

* Statistically significantly different from unity at the 5% level.

As seen in that table, 13 of the 17 words previously identified as expensive ones have relative probabilities that are statistically significantly greater than unity. Three words (“butterscotch,” “cigars,” and “pork”) have relative probabilities that are not statistically different from unity at conventional levels. Only one word (“shellfish”) has a relative probability that is significantly less than unity in both a statistical (p < 0.01) and substantive sense. Most of the previously identified expensive words are thereby reconfirmed to be expensive ones.

Table 3 reports estimates for each “cheap” word identified by Krumme. All but two of those 17 words have relative probabilities that are statistically significantly less than unity. The two exceptions (“enjoy” and “harvest”) have relative probabilities that are not significantly different from unity. Thus, almost all of the previously identified cheap words are also reconfirmed to be cheap ones.

Table 3

Relative Probabilities for Krumme's Cheap Word Examples

Notes: Same as Table 2, except these estimates are for each word Krumme (Reference Krumme2009, Reference Krumme2011) identified as “cheap” ones.

* Statistically significantly different from unity at the 5% level.

In terms of the other findings reported by Krumme, I find for the dataset studied here that there do seem to be two mostly distinct vocabularies for high- and low-priced wines. Out of the roughly 20,000 unique words used to describe the over-$50 and/or under-$15 wines, only 42% of those words overlap by being used to describe wines in both price categories. The remaining 58% of words are non-overlapping.

I also find for the dataset studied here that the words Krumme identified as cheap ones tend to be reused more often than the expensive ones. If I add up every instance of any one of the 17 cheap words from Table 1 describing either an over-$50 or under-$15 wine, and if I do the same for every instance of any of the 17 expensive words from Table 1, then those cheap words occur 2.3 times more often than the expensive ones. That same conclusion still holds if I ignore the few words from Tables 2 and 3 that did not have the relative probabilities that would be expected based on Krumme's findings.