I. Introduction
Wine consumption has been part of civilization for more than 8,000 years. During most of that time, consumption was essentially local. In 1920, Europe (including Algeria, a French territory at the time) accounted for 95 percent of the world's wine production and exported only 5 percent to other countries (Anderson and Pinilla, Reference Anderson and Pinilla2018). This changed dramatically in recent years. In 2019, according to Statista,Footnote 1 just the top three exporters (Italy, Spain, and France) shipped almost 20 percent of the world's wine production.
Similarly, although the literature on wines has existed throughout the history of wine, the widespread use of numerical rating systems is associated with the modern era of wine journalism. In 1978, Robert Parker launched the Wine Advocate. He popularized the [50 − 100]-point rating system, which is now widely used in the wine world.Footnote 2 As of today, it is difficult to find wine reviews without numerical ratings. This proliferation of ratings entails a new problem, as wine stores and consumers may face different assessments of the same wine. Amerine and Roessler (Reference Amerine and Roessler1983) were probably the first to raise the importance of seeking a consensus among tasters, exploring procedures to do so. We follow their steps here, aiming to provide a formal and comprehensive framework to obtain consensuses, building on contributions within political science, social choice, game theory, and operations research.
The choice of a specific procedure to reach a consensus might have important implications. An early instance in which this can be exemplified is the Judgment of Paris. This was a competition between American and French wines, organized in Paris in 1976 by British wine merchant Steven Spurrier. He invited 11 competent French wine connoisseurs who judged two flights of ten white wines and ten red wines. Among the ten red wines, four came from France and six from the United States. The judges had to taste and rate each of them on a scale from 0 to 20. The 11 rates for each wine were added, resulting in a ranking of the ten wines. A Californian wine (Stag's Leap Wine Cellars) was at the top of the ranking. This happened at a time in which, according to Taber (Reference Taber2005, p. 17), “France ruled the world.” The outcome challenged the French supremacy, and helped to introduce American wines to the worldwide market.
Adding rates across tasters is an obvious way to reach a consensus among them. But it is admittedly naive and flawed. For instance, each expert comes with her cultural and personal preferences, which might influence the outcome beyond desire. Ashenfelter and Quandt (Reference Ashenfelter and Quandt1999, p. 170) suggested that “converting rates to ranks guarantees that each judge has the same influence on the outcome.” This is essentially the so-called Borda's proposal in voting theory, which tallies the score of a candidate x by counting n − 1 points for each voter for whom x was the best candidate, n − 2 points for each voter who ranked x as second, and so on, until 0 points were given for each voter ranking x last.Footnote 3 In the case of the Judgment of Paris, this means awarding points from 0 to 10 to each of the wines, based on the rankings provided by the judges. It turns out that Borda's proposal would have changed the Judgment’s conclusion, leading to a tie between American Stag's Leap Wine Cellars and French Château Montrose (see Table 1, two last rows).Footnote 4
The Judgment of Paris: Original Ratings
Wines: A: Stag's Leap Wine Cellars, 1973; B: Château Mouton-Rothschild, 1970; C: Château Montrose, 1970; D: Château Haut Brion, 1970; E: Ridge Vineyards Monte Bello, 1971; F: Château Léoville Las Cases, 1971; G: Heitz Wine Cellars, 1970; H: Clos du Val Winery, 1972; I: Mayacamas Vineyards, 1971; J: Freemark Abbey Winery, 1969.
As of today, wine rankings are closely scrutinized by wine consumers and wine investors alike. The impact of expert reviews on the demands for wine and wine returns has been highlighted in many publications (Jaeger, Reference Jaeger1981; Ali, Lecocq, and Visser, Reference Ali, Lecocq and Visser2008; Friberg and Grönqvist, Reference Friberg and Grönqvist2012; Fogarty and Sadler, Reference Fogarty and Sadler2014; Masset, Weisskopf, and Cossutta, Reference Masset, Weisskopf and Cossutta2015; Ashton Reference Ashton2016; Cyr, Kwong, and Sun, Reference Cyr, Kwong and Sun2017). A tiny difference in terms of ranking at the top of the distribution may have striking consequences on wine sales and investment returns. The previous story on the Judgment of Paris, together with the fact that wine investors aim to diversify their portfolio, illustrates the importance of scrutinizing the alternative procedures to seek a consensus among tasters. We aim to do so in this article, providing a formal and comprehensive framework, which rests on previous contributions in political science, social choice, game theory, and operations research. Our starting point is a group of experts who assign (numerical) ratings to the same set of wines. The objective is to produce a consensual rating (and ranking). The inputs consist of a matrix, the rows of which are experts (i = 1, 2, …, I), and the columns are wines (j = 1, 2, …, J). Element a ij of this matrix represents the rating of wine j by expert i. The output will be a column that represents a consensual rating for each wine, which we suggest obtaining after three stages: normalization, approval, and aggregation.
The first stage (normalization) converts absolute ratings into relative (i.e., quantile-based) ratings. This is meant to have ranges of ratings equalized across experts and ratings themselves homogenized. In doing so, we address Ashenfelter and Quandt's argument stated previously. It is also what Cardebat and Paroissien (Reference Cardebat and Paroissien2015) suggested to standardize scores, which is also endorsed by Global Wine Score.Footnote 5
The second stage (approval) determines a quantile that can be interpreted as a threshold that wines have to meet to be approved.Footnote 6 The use of thresholds is widespread in many instances of real life. It is, for example, a common practice in education, where students pass a test or a course only if they reach a certain threshold. In the case of wine, the term Parker effect was coined to claim that a rating of 90 points or less in Parker's rating system can cause a tipping point for buyers.Footnote 7 We suggest that only approved wines pass to the next stage with a positive weight. In other words, non-approved wines are all equally discriminated. As for approved wines, we allow for two options: to discriminate among them (using their normalized ratings as their weights in the next stage) or not (assigning the same positive weight to all approved wines). On the one hand, it is frequently argued that dichotomous decisions to signal whether a wine is approved or not are much easier to make and may be sufficient, while ratings are much more difficult. On the other hand, it might be considered unfair not to distinguish among (approved) wines with very different ratings once they are available. Thus, we shall consider both options (which will obviously yield different outcomes).
The third stage (aggregation) concludes by aggregating the information obtained from each expert after the previous stages. The first option is to simply add the ratings (obtained after the previous two stages). The second option is to introduce a second normalization before aggregating. More precisely, we divide each rating by the sum of the ratings each judge confers to all wines, and then we aggregate. As before, the choice between both options results from a value judgment: in this case, deciding whether the ratings given by an expert should depend on the number of wines that passed the approval stage or not. Thus, we shall consider both options too (which will obviously yield different outcomes as well).
Our three-stage procedure is inspired by contributions in other fields.
The normalization stage mentioned previously was already proposed by Cardebat and Paroissien (Reference Cardebat and Paroissien2015) to standardize wine scores. It had already been used in diverse areas such as the design of equal-opportunity policies (Roemer, Reference Roemer1998; Moreno-Ternero, Reference Moreno-Ternero2007), the economic evaluation of health care programs (Bleichrodt, Herrero, and Pinto, Reference Bleichrodt, Herrero and Pinto2002; Herrero and Moreno-Ternero, Reference Herrero and Moreno-Ternero2009), or the evaluation of scientific performance (Albarrán et al., Reference Albarrán, Crespo, Ortuno-Ortin and Ruiz-Castillo2010, Reference Albarrán, Crespo, Ortuno-Ortin and Ruiz-Castillo2011).
As for the other two stages, political scientists suggested using Cumulative Voting more than half a century ago (Glasser, Reference Glasser1959; Sawyer and MacRae, Reference Sawyer and MacRae1962). This allows voters to distribute points among candidates in any arbitrary way. An interesting case is the one in which every agent is endowed with a fixed number of votes that are evenly divided among all candidates for whom she votes. Approval Voting (Brams and Fishburn, Reference Brams and Fishburn1978) is another voting method in modern social choice theory (currently in practice in some U.S. local elections, as well as to elect officers in numerous professional organizations). It allows each voter to cast a vote for as many candidates she wishes; each positive vote is counted in favor of the candidate. The votes are then added candidate by candidate, and the winner is the one who gets the largest number of votes.Footnote 8 Under plausible assumptions, Approval Voting compares favorably with both the plurality rule or Borda's rule (Weber, Reference Weber1995).
Another intuition is derived from the Shapley value (Shapley, Reference Shapley, Kuhn and Tucker1953), a natural way to allocate the total surplus generated by the coalition of all players involved in a joint venture, or a cooperative game, based on the marginal contributions players produce. Ginsburgh and Zang (Reference Ginsburgh and Zang2003) used this concept to study the problem of sharing the total revenue collected from selling museum passes, which give access to several museums.Footnote 9 Later on, Ginsburgh and Zang (Reference Ginsburgh and Zang2012) applied the same theoretical model to aggregate ratings from the Judgment of Paris.Footnote 10 In their case, the protocol is different from ours because, instead of a threshold, they (exogenously) fixed a number of wines for each judge.
Finally, there is a connection with the classical knapsack problem in operations research (Martello and Toth, Reference Martello and Toth1990). It refers to a camper who would like to carry objects that have different utilities and different weights. His optimal choice results from maximizing his total utility under the constraint that he can carry only a given total weight. This integer problem has many applications. In participatory budgeting, for example, it can be formulated as follows. A (local) government asks residents to vote on proposals for how a certain fraction of their total budget should be spent (Cabannes, Reference Cabannes2004). Each voter can specify a subset of these projects, such that the total cost in the subset is bounded by the total budget (regardless of how many projects are in the subset). Thus, each voter has to solve an individual knapsack problem. An advantage of knapsack voting is that if the algorithm rates each project by the number of votes it receives and chooses projects greedily in descending order of ratings until the budget is exhausted, then knapsack voting is a partially truthful mechanism (Goel et al., Reference Goel, Krishnaswamy, Sakshuwong and Aitamurto2019). In the context of wine, Conrad, Gómez, and Lamadrid (Reference Conrad, Gómez and Lamadrid2011) posed several knapsack problems to select the optimal subset of wines subject to budget and quantity constraints.
Our proposal is, therefore, a unifying framework to accommodate a variety of options and provide a consensus among wine tasters. It ranges from the basic approach followed by Global Wine Score, which aggregates scores after standardizing them as suggested by Cardebat and Paroissien (Reference Cardebat and Paroissien2015), to more sophisticated approaches arising after setting thresholds to approve wines and weighing them proportionally to the ratings of approved wines. It is preferable to propose a unique rule because, as is well known, no single consensus rule is flawless (see Arrow (Reference Arrow1951), Gibbard (Reference Gibbard1973), and Satterthwaite (Reference Satterthwaite1975)).
The rest of the article is organized as follows. In Section II, we formalize the three stages discussed earlier. Section III illustrates these methods using the data from the Judgment of Paris. Section IV turns to apply them to a set of 114 en primeur (early) Bordeaux wines rated in 2018 by five international experts. Section V concludes.
II. The Model
We consider a set of judges or experts who rate wines. Formally, there exists a matrix A, the rows of which are experts (i = 1, 2, …, I), and the columns are wines (j = 1, 2, …, J). Element a ij thus represents the rating of wine j by expert i. Our objective is to summarize the information from matrix A into a unique column containing the consensual rating for each wine. An obvious one, which we call the usual rating consensus, would simply take the average rating of all experts. That is, for each j = 1, 2, …, J,
We try to improve this (somewhat naive but widely used) method in several ways. To do so, we introduce a procedure that involves three stages (subsections A to C). Subsection D illustrates the computations using a very simple example.
A. Normalization Stage
As suggested by Cardebat and Paroissien (Reference Cardebat and Paroissien2015), ratings are first normalized so that the ranges of ratings are equalized across experts, and ratings themselves are homogenized. To do so, we consider the cumulative distribution function (CDF) of each expert's ratings. This reduces to computing a certain number of quantiles, that is, proportions of wines rated above a given level, by each expert. If, say, expert 1 rated a wine 95, whereas expert 2 rated it 90, but both considered that 20% of the wines they rated themselves separately were better than this one (and 80% were worse), then the normalized rating for this wine will be 80 (for both experts). If instead, expert 1 considered that only 5% of the wines he rated were better than this one (and 95% were worse), whereas expert 2 considered 10% and 90%, respectively, then the normalized ratings for this wine will be the original 95 and 90, respectively.
This first stage thus converts the original matrix A into the associated normalized ratings matrix A n, where each entry $a_{ij}^n$
reflects the quantile associated to a ij in the CDF of expert i.
After this normalization, Cardebat and Paroissien (Reference Cardebat and Paroissien2015) simply suggest averaging the normalized ratings that each wine receives from all experts. That is, for each j = 1, 2, …, J,
Instead of endorsing the above proposal, we suggest replacing averaging with two additional stages: the approval stage and the aggregation stage.
B. Approval Stage
This second stage sets a quantile π that can be interpreted as a threshold that wines have to meet in order to be approved or to avoid being censored.Footnote 11 This stage converts the matrix of (normalized) ratings A n into a new matrix in which the rating of a censored wine (i.e., a rating below the quantile π) is replaced by zero. As for approved wines, we consider two options:
a) Either ratings are replaced by a constant (1 without loss of generality); this leads to a matrix A 1π, in which $a_{ij}^{1\pi } = 1$
if $a_{ij}^n \ge \pi$ and 0 otherwise,b) or ratings are stored in a matrix $A^\pi$
such that $a_{ij}^\pi = a_{ij}^n$ if $a_{ij}^n \ge \pi$ and 0 otherwise.
In the first option, the ratings given by experts to approved wines are not used to reach a consensus, whereas, in the second, they are. Which option should one choose? On the one hand, it is frequently argued that dichotomous decisions (1 or 0) to signal whether a wine is approved or not are much easier to make and may be sufficient, while ratings are much more difficult. Difficulties to exactly express quality by a score would support (a) although the method may end up with many ties in the aggregation stage that follows. On the other hand, it might seem unfair not to distinguish among (approved) wines with very different ratings once they are available, which would support (b).
C. Aggregation Stage
The last stage of our procedure aggregates the information from matrices A 1π or $A^\pi , \;$
which both construct approval ratings. The first option is to simply compute averages across wines from one or both matrices, thus mimicking what we did for matrices A and A n. We shall refer to them as the Approval and Proportional Approval consensus, respectively, which implies that for each j = 1, 2, …,J, we compute:
The second option is to introduce a new normalization in which, before aggregating, ratings are considered relative, rather than absolute: This means that in matrices A 1π and $A^\pi$
, we divide each entry by the aggregate amount in the corresponding row (which represents the sum of the ratings each judge confers to all wines), and then we aggregate. We shall refer to them as the Relative Approval and Relative Proportional Approval consensus, respectively. Formally, for each wine j = 1, 2, …, J,
Again, here the choice between both methods results from a value judgment. It amounts to the critical issue of deciding whether the ratings given by an expert should depend on the number of wines that passed the approval stage or not.
D. An Example of the Computations
Consider the following example, in which five judges rate three wines, as described in the following matrix.
We assume that the judges rated many more wines and, thus, we can obtain the CDFs of their ratings. Suppose the resulting matrix from A, after normalizing ratings making use of their CDFs, is the following:
Protocols (1) and (2) discussed earlier yield the following ratings from which one can also compute rankings:
• Usual Protocol (1):
$$ U( 1 ) = \frac{1}{5}( {100 + 95 + 96 + 96 + 99} ) = 97.2$$$$U( 2 ) = \frac{1}{5}( {95 + 100 + 97 + 99 + 100} ) = 98.2$$$$ U( 3 ) = \frac{1}{5}( {95 + 100 + 98 + 99 + 99} ) = 98.2$$• Normalized Protocol (2):
$$ N( 1 ) = \frac{1}{5}( {96 + 90 + 96 + 96 + 94} ) = 94.4$$$$ N( 2 ) = \frac{1}{5}( {90 + 96 + 97 + 99 + 96} ) = 95.6$$$$ N( 3 ) = \frac{1}{5}( {90 + 96 + 98 + 99 + 94} ) = 95.4$$
Suppose now that the threshold is set at π = 95. Matrices A 1π and $A^\pi$
easily follow from matrix A n:
The analog to the previous two protocols, but using matrices A 1π and $A^\pi$
instead of matrices A and A n, are:
• Approval Protocol (3):
$$ A( 1 ) = \frac{1}{5}( {1 + 0 + 1 + 1 + 0} ) = 0.6$$$$ A( 2 ) = \frac{1}{5}( {0 + 1 + 1 + 1 + 1} ) = 0.8$$$$ A( 3 ) = \frac{1}{5}( {0 + 1 + 1 + 1 + 0} ) = 0.6$$• Proportional Approval Protocol (4):
$$ PA( 1 ) = \frac{1}{5}( {96 + 0 + 96 + 96 + 0} ) = 57.6$$$$ PA( 2 ) = \frac{1}{5}( {0 + 96 + 97 + 99 + 96} ) = 77.6$$$$PA( 3 ) = \frac{1}{5}( {0 + 96 + 98 + 99 + 0} ) = 58.6$$
The last two protocols in which matrices A 1π and $A^\pi$
are normalized further so that each entry is divided by the overall entry in the corresponding row, are:
• Relative Approval Protocol (5):
$$ RA( 1 ) = 1 + 0 + 1/3 + 1/3 + 0 = 10/6$$$$ RA( 2 ) = 0 + 1/2 + 1/3 + 1/3 + 1 = 13/6$$$$ RA( 3 ) = 0 + 1/2 + 1/3 + 1/3 + 0 = 7/6$$• Relative Proportional Approval Protocol (6):
$$ RPA( 1 ) = 1 + 0 + ( {96/291} ) + ( {96/294} ) + 0 = 1.656$$$$ RPA( 2 ) = 0 + 1/2 + ( {97/291} ) + ( {99/294} ) + 1 = 2.318$$$$ RPA( 3 ) = 0 + 1/2 + ( {98/291} ) + ( {99/294} ) + 0 = 1.174$$
III. Application 1. The Judgment of Paris
The Judgment of Paris alluded to in the Introduction, invited 11 well-known French experts to evaluate ten red wines. The detailed results of the contest are shown in Table 1. Each wine appears in a column and each expert in a row. The flight consists of four French wines (denoted by F) and six American wines (US). The results of the tasting are reported in the last rows “average rating” of each wine and “final ranks” based on the average ratings. This is how wine tastings usually end the performance. We added one row (the very last one, in bold characters), which is the ranking that one obtains by transforming the ratings of each expert into ranks and compute average ranks, as suggested by Ashenfelter and Quandt (Reference Ashenfelter and Quandt1999). The result is quite different: (a) American wine A ties with French wine C; (b) the ranks of wines F and G change, and (c) so do wines H, K, and J.Footnote 12
We now turn to the alternative protocols considered in this article. The number of wines tasted in this context is too small to normalize ratings. We thus go immediately to stage 2 and (arbitrarily) set the threshold at 10.Footnote 13
Table 2 contains all the rankings described in this article. The rankings yield similar results. Indeed, wines A, B, and C appear among the first three in all cases, in some cases they are tied, in other cases they are not. Wine D is almost always number 4, except that it is tied once with wine F. Wines H, I, J are always the last ones. Differences mostly occur in the middle, where rankings switch among wines.
The Judgment of Paris: Comparative Rankings
Wines: A: Stag's Leap Wine Cellars, 1973; B: Château Mouton-Rothschild, 1970; C: Château Montrose, 1970; D: Château Haut Brion, 1970; E: Ridge Vineyards Monte Bello, 1971; F: Château Léoville Las Cases, 1971; G: Heitz Wine Cellars, 1970; H: Clos du Val Winery, 1972; I: Mayacamas Vineyards, 1971; J: Freemark Abbey Winery, 1969.
To conclude, changing the threshold in the previous analysis leads to some minimal changes (Table 3). For instance, if the threshold moves up to 15, the tie among wines A, B, and C (which occurred before for the third and fifth protocols) breaks. Wine C (French) actually falls to fourth place, whereas wine D (French) is third. As for wines A (American) and B (French), each one comes first with one of the two protocols. As for Protocol (6), it would also have Wines F (French), I (American), and J (American) tied at the bottom (with no votes), whereas the rest of the wines would be ranked (with no ties) as follows: B (French), A (American), D (French), C (French), E (American), G (American), and H (French).
The Judgment of Paris: Comparative Rankings (Threshold 15)
Wines: A: Stag's Leap Wine Cellars, 1973; B: Château Mouton-Rothschild, 1970; C: Château Montrose, 1970; D: Château Haut Brion, 1970; E: Ridge Vineyards Monte Bello, 1971; F: Château Léoville Las Cases, 1971; G: Heitz Wine Cellars, 1970; H: Clos du Val Winery, 1972; I: Mayacamas Vineyards, 1971; J: Freemark Abbey Winery, 1969.
IV. Application 2. Bordeaux 2018 Future Wines
We now consider the more interesting and recent tasting of the 2018 Red Bordeaux called Bordoverview.Footnote 14 Bordoverview contains ratings for 114 (Bordeaux 2018 future) wines produced by five (international) experts: Jancis Robinson (JR), Tim Atkin (TA), Revue du Vin de France (RVF), Decanter Magazine (DM), and Parkers’ Wine Advocate (WA).Footnote 15 Appendix 2 provides the list of 114 wines by alphabetic order (first column). The next five columns contain the ratings given by each expert, while the five last columns contain the normalized rates.Footnote 16
A first aspect to notice is the striking differences that exist among some of these experts’ ratings, which indicates how important the normalization stage is. More precisely, from Figure 1, we can observe how different their rating distributions are.
Rating Distributions of Experts
Table 4 shows the ratings for the six protocols introduced earlier when the (normalized) threshold is set at 90.Footnote 17 This threshold eliminates 76 of the 114 wines that are thus given a zero rating (with an exception for the first two protocols, which do not consider the approval stage), and 38 pass the test and are “approved.” Table 4 also contains the associated rankings for each of the six protocols and leads us to some interesting conclusions.
Ratings and Rankings According to Each Protocol
First, the top of the rankings seems to be quite robust. Lafite-Rothschild comes first in all cases (although tied with nine other wines in the third and fifth protocols, which, as mentioned earlier, may generate many ties). Léoville-Las-Cases seems to be a solid second follower, except for the first protocol, where it appears as number three (and the caveat for the ties in the third and fifth protocols). The first protocol actually awards the second position to Palmer, although it goes to eighth place after normalization. This is probably the first interesting difference among protocols. Another interesting case, in the opposite direction, is Ausone, which is ranked eighth in the first protocol but goes to the fifth position for protocols after normalization.Footnote 18 Other specific and somewhat striking differences between the second, fourth, and sixth protocols can be observed; for instance, le Pin ranked number 24 in the second protocol, 11 in the fourth, and 13 in the third. And, according to the fourth and sixth protocols, Margaux and Vieux Château Certan switch positions.
As mentioned previously, the third and fifth protocols introduce many ties, as the rates are dichotomous (1 for approved wines, 0 for all others). One may argue that rating wines is a difficult business and that decisions should be simplified and simply be dichotomous: one likes a wine or not. If one accepts this position, only the third and fifth protocols would be valid. Breaking the ties requires distinguishing among approved wines, as the remaining protocols do.
Note, however, that if one sets a higher threshold for letting wines pass, then fewer wines will be approved, and some ties vanish. To illustrate this aspect, if the threshold is set at 95 (instead of 90), then only two wines (instead of eight) tie for the first place: Lafite-Rothschild and Léoville-Las-Cases. They also happen to be the top two wines in other protocols, except for the first (non-normalized) protocol, in which Palmer comes second and Léoville-Las-Cases third.
To have a more general view on the differences in the protocols (except for the third and fifth, which, as we already said, have too many ties), we calculated Pearson correlations, as well as Spearman rank correlation coefficients on rankings (see Table 5).
Correlations Between Protocols
It is remarkable to see that the correlations between Protocol (1) (the usual average ratings of all five experts) and Protocols (2), (4), and (6) are all quite large (between 0.89 and 0.97). It is also noteworthy to observe that Protocol (2) (normalization) is not very different from Protocol (1) (between 0.93 and 0.97) which implies that normalizing does not change much. In addition, Protocols (1) and (6) (the most sophisticated ones) are very close to each other. Still, these observations do not mean that these small changes are innocuous. There is indeed a large difference in the perception of which wine is first—note that, against all odds, Lafite Rothschild is always number 1 or 2—but Château Ausone, ranked eighth in the usual protocol, may strongly benefit, getting from rank eighth to rank fourth, in all other protocols. It also makes a very large difference being among the top 10 and the top 20. This is also so for prices: a small difference in ranking may have large effects on the decisions made by many buyers and thus on prices. This complex relationship between talent, or quality, essentially driven by experts, and prices are illustrated by Ali, Lecocq, and Visser (Reference Ali, Lecocq and Visser2008).
In short, changes of ranks (even from one to two, or five to six) are quite significant in this profession, as well as among wine experts. The Judgment of Paris that we discussed earlier is a good example of this very unusual behavior. American wines gained prestige because one of them, Stag's Leap Wine Cellars, overshadowed three prestigious French wines: Mouton-Rothschild, Montrose, and Haut Brion.
To conclude, Table 6 illustrates the results of the ratings for the sixth protocol (which we find the most interesting), with thresholds ranging from 90 to 95. They show that there is consensus at the top of the ranking. Lafite-Rothschild and Léoville-Las-Cases always come first and second. Margaux is third in all cases except when the threshold climbs to 95, where it falls to sixth place. Vieux Château Certan goes in the other direction, climbing to the third position at the 95 threshold, whereas it ranks between fifth and tenth in the other protocols.
The Effects of Changing Approval Thresholds on Consensus Rankings
Note: In most columns (with the exception of Column 1) there are ties among wines (in italics).
V. Discussion
Numerical wine ratings are very popular. The scientific literature has paid attention to aspects such as (1) inconsistency of ratings in blind tastings (Lindley, Reference Lindley2006; Hodgson, Reference Hodgson2008; Bodington, Reference Bodington2017, Reference Bodington2020), (2) consensus among experienced wine experts (Ashton, Reference Ashton2012, Reference Ashton2013; Cao, Reference Cao2014; Luxen, Reference Luxen2018), (3) variations in the severity of experts (Masset, Weisskopf, and Cossutta, Reference Masset, Weisskopf and Cossutta2015; Stuen, Miller, and Stone, Reference Stuen, Miller and Stone2015), and (4) consumers’ demand for wine ratings (Ashenfelter and Jones, Reference Ashenfelter and Jones2013; Marks, Reference Marks2015, Reference Marks2020).
In this article, we aimed at providing a comprehensive framework to reach a consensus among tasters’ opinions (expressed via wine ratings). We consider three (consecutive) stages: normalization, approval, and aggregation. The first stage converts absolute ratings into relative ratings, as suggested by Cardebat and Paroissien (Reference Cardebat and Paroissien2015). The second one puts thresholds on normalized ratings, below which wines get no credit. The third one shares the credit from each expert among approved wines, equally or proportionally. The last two stages are inspired by classical contributions in political science, social choice, game theory, and operations research.
Our framework can accommodate six different protocols, depending on the stance one takes in each of the three stages. We illustrate our protocols using the Judgment of Paris and 2018 en-primeur Bordeaux wines, rated by five international experts.Footnote 19 Our analysis concludes that how we build the consensuses is quite different from the usual simple aggregations of ratings or rankings. We nevertheless believe that it is important to notice that different (plausible) decisions to build a consensual ranking generate different outcomes.
All our protocols have pros and cons, some of which are described in our article. We argued that normalizing ratings are important, which makes the first (standard) protocol unreliable. We also believe that a simple average of (normalized) ratings is unsatisfactory. The remaining four protocols, obtained by implementing both the approval and aggregation stages, teach us something. If one is ready to simplify ratings, converting them to dichotomous choices, then the third and fifth protocols are preferable because they make the choices of experts much easier.
But as more precise numerical ratings are available (0 to 20, or 50 to 100), we cannot ignore them just distinguishing approved and non-approved wines and directly go for the fourth or the sixth protocol. We nevertheless believe that the sixth protocol is more satisfactory as it also involves a second normalization in which the ratings of each expert are considered: When an expert approves a set of wines, her credit is split proportionally to her ratings, among those wines. This is in line with many other problems in real life where the proceeds from a joint venture are only allocated among those contributing to the venture (Bergantiños and Moreno-Ternero, Reference Bergantiños and Moreno-Ternero2015, Reference Bergantiños and Moreno-Ternero2020). In summary, we endorse Protocol (6) as the most appropriate one.
A final comment is in order. In all the protocols, the opinions of experts were equally weighted. But it may be the case that certain experts are more influential than others. The protocols could easily be extended to account for unequal weights of experts.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/jwe.2021.15.
Appendix 1. Experts Selected from Bordoverview
• Jancis Robinson is a British wine writer and critic who became the first Master of Wine (MW) outside the wine trade. She studied mathematics and philosophy at the University of Oxford. She writes a weekly column for the Financial Times. See https://www.wine-searcher.com/critics-1-jancis+robinson or https://www.jancisrobinson.com/.
• Tim Atkin is a UK-based MW and wine journalist. After training in modern languages at the University of Durham, Atkin soon moved into a career in wine writing. See https://www.wine-searcher.com/critics-34-tim+atkin.
• The Revue du Vin de France is a monthly French wine publication that started in 1927. It specializes in French wines.
• The Decanter magazine was founded in London in 1975. It is the oldest consumer wine publication in the United Kingdom. The Decanter World Wine Awards, established in 2004 by English wine critic Steven Spurrier (who was at the origin of the Judgment of Paris, discussed earlier), awards trophies and medals, as well as wine ratings. See https://www.wine-searcher.com/critics-44-decanter+world+wine+awards.
• The Wine Advocate is a bimonthly wine publication based in the United States, created in 1978 by Robert Parker, regarded as one of the most widely known and influential wine critics in the world. See https://www.robertparker.com.
