I. Introduction
Experts in many fields are so named because of superior performance exhibited in a competitive setting, in academic training or sports, to name a few. The criteria for being named an expert wine judge are not so clear. Although the reliability of wine judges has been questioned by others (Ashenfelter and Quandt, Reference Ashenfelter and Quandt1999; Ashton, Reference Ashton2012; Cicchetti, Reference Cicchetti2004), we are not familiar with any work that actually tests the reliability of wine judges, with two exceptions: that reported by Gawel and Godden (Reference Gawel and Godden2008) and the efforts undertaken by the California State Fair Commercial Wine Competition (CSFCWC) (Hodgson, Reference Hodgson2008). For more than a decade, at the California State Fair venue judges have been regularly evaluated by serving replicated samples of wine, poured from the same bottle and served on the same tasting flight.Footnote 1 Based on this accumulated data, we propose two criteria that an expert wine judge must meet: (1) the ability to award similar scores to replicated samples, and (2) the ability to employ a range of scores over many wines large enough to discriminate between them. However, we do not consider consensus among judges to be an essential criterion. What follows might be considered a guide for evaluating judges' performance that would be useful for competition managers as they strive to improve the reliability of medals awarded at their competitions.
The cornerstone of the experimental design that follows rests on the ability to serve replicated samples of an identical wine among a flight of wines of similar character (all Zinfandels, Chardonnays, etc.). This has been done at the California State Fair since 2003 with no disruption to the logistics involved in serving thousands of wines to about a hundred judges. For a panel of four judges, there is sufficient wine in a bottle to serve each judge three samples poured from the same bottle. One of these samples is designated as a “mother,” and the other two as “ghosts.” They all have different identifying codes and are placed randomly within the flight. To be fair, only the scores of the “mother” count for determining medals while the “ghosts” are used to provide data to evaluate judge performance. We postulate that an expert judge will mark the identical samples with similar scores and call such an event a successful one. For the purposes of this study, it becomes necessary to calculate the probability that such successful events occur by chance alone. This leads to a decision rule as to whether a judge may be credited with expert status with a known level of statistical error. This scheme is not critical of judges who disagree with others, only that an expert judge is consistent, marking identical wines with similar scores.
In a typical wine competition, the task before a judge is to evaluate a flight of wines by rating them according to an ordinal set of categories, for example: No Award, Bronze, Silver, or Gold. The protocol as to whether judges confer with each other or by what method the individual scores are aggregated varies among competitions. The first author, who also owns a small winery, became concerned about the disparity of awards given to his own wines at various competitions. Through circumstance, he was able to initiate an experimental program at the California State Fair to explore which factors might explain the disparity. This led to several papers examining the results of wine competitions and pointing to judges' inconsistency as the root cause (Hodgson, Reference Hodgson2008, Reference Hodgson2009a, Reference Hodgson2009b). Here, we focus on a different objective, to offer a framework for evaluating wine judges. We know of no criteria for defining an expert wine judge, but we assert that consistency is a necessary condition.
II. Theory
Consistency. Our strategy for identifying expert judges rests on their ability to discriminate among a flight of wines and to award similar scores to the replicated samples. However, it is also necessary to determine the likelihood that this could be done by chance alone. We imagine n=10 wines, of which k=3 are identical and we score these wines into m=4 categories of quality: No Award (NA), Bronze (B), Silver (S), or Gold (G). There are m (n – k +1) ways to randomly score the wines into the m categories such that the k identical wines have the same score. There are mn ways to distribute all the wines, regardless of identity. By chance, the probability of giving the k identical wines the same ordinal score, which has m levels, is
$$P_1 (m,\;k) = m^{(n - k + 1)} /m^n = m^{1 - k} $$For the example containing four quality categories, the probability of giving a set of triplicate wines the same award by chance alone is 1/16 or about 6%. Note that P 1(m,k) is independent of the number of wines in the flight.
We now address the issue of relaxing the criterion that a judge grades the identical samples with the same score. The California State Fair Commercial Wine Competition (CSFCWC) employs an eight-level ordinal scale: NA, B, B+, S–, S, S+, G–, G, that is, m=8. If a judge were to award the same score to the replicated samples, we classify this as a successful event, for example, (S–, S–, S–). But is there a significant difference in scoring a wine B+ versus S–, or G– versus G? If not, we might relax our criterion for a successful event to include wines with similar scores. So, now how do we calculate the likelihood that such events could occur by chance? First, we need to define what we mean by similar scores. We take T 1 to mean awarding all the replicates the same score. T 2 means all replicates are given scores that span at most two adjacent grades, such as B+ to S– or S– to S. T 3 means the scores of the replicates encompass a maximum of three classes, such as B+, S–, S or S–, S, S+, and T 4 allows a span of at most four adjacent grades, such as B to S or S to G.
Having assigned parameters T i to what we mean by similar scores, we proceed to calculate the likelihood that a judge could place the replicated wines within one of these similar categories by chance. For each degree of tolerance identified by a tolerance index, T 1, T 2, and so on, we calculate the frequency of successful outcomes, f i(m,k) and compute the probability that a judge could meet the Tolerance criterion by chance, P i(m,k), summarized in Table 1.Footnote 2
Frequency and Probability of Successful Outcomes by Chance for m=8, k=3
For example, there are 212 possible permutations of scores that a judge might assign the three replicates in a single flight that meet the T 4 criterion by chance alone. The probability of doing so by chance, 0.414, is excessive by normal statistical standards of significance. To reduce the likelihood that a judge might pass the test by chance, we can repeat the test in another flight. At the CSFCWC, each judge receives four flights of wines containing triplicate samples. Thus, the probability of awarding similar scores in all four flights becomes [P 4(m=8, k=3)]4. This defines the Type I error according to the framework of statistical hypothesis testing, the likelihood of rejecting a null hypothesis when the hypothesis is true. In our case, the null hypothesis is that a judge is not an expert. We reject that hypothesis if a judge is successful in awarding replicate samples a similar score, realizing that this decision, conferring expert status, carries with it an error.
To summarize, we assert that an expert judge will be able to correctly assign similar scores to identical samples of the same wine embedded in a flight of wines. Next, we calculate the probability that this could happen by chance alone on a single flight and extend the result to multiple flights. Depending on the tolerance index, we say that a judge has met the necessary qualifications to be named an Expert with a known probability that our decision could be erroneous by chance at a specified level of significance.
The level of discrimination (vis-à-vis the tolerance parameter) that we ascertain to be a marker of excellent performance is a matter of opinion. Once this level is chosen, this work provides the foundation for establishing proof of competence within a certain significance level. However, we might ask, “What is a reasonable level of discrimination?” A discussion of this question is postponed until later, but to proceed, we arbitrarily set T 4 as a marker of excellence.
We now look at the relationship between varying tolerance levels, number of replicates in a flight, and the number of flights necessary to establish significance at a given tolerance level.
Table 2 calculates the Type I error for naming a judge Expert depending on different tolerance levels, assuming that a judge receives triplicate samples in two to four flights of wine. The first row contains the probabilities, P i(8,3), that a judge would assign scores with the tolerance limits on a single flight. The following rows contain the probabilities of successfully awarding similar scores, by chance, to the replicates on two, three, or four flights. These probabilities represent the Type I errors associated with naming a judge Expert when such a decision could be made by chance alone. For example, for T 4 there is only a 3% probability that a judge would award similar scores by chance to the triplicate samples in all four flights.
Type I Error for Triplicate Samples
Table 2 illustrates the tradeoff between tolerance levels and the number of flights in which replicates are served to reach a particular level of significance. If we establish T 3 as a necessary level of discrimination, we may test for significance with a Type I error of around 5% by serving triplicate sample on just two flights. If we relax the degree of discrimination to T 4, triplicate samples must be served on four flights to reach a Type I error of less than 5%. By relaxing discrimination levels beyond T 4, we not only degrade the level of discrimination but simultaneously allow a judge to pass the test by chance alone at elevated significance levels. In other words, the test does not make a significant distinction between those who pass and those who fail.
Table 3 calculates the Type I error for naming a judge Expert depending on different tolerance levels, assuming that a judge receives duplicate samples in two to four flights of wine. The first row contains the probabilities, P i(m 8, k=2), that a judge would assign scores with the tolerance limits on a single flight. The following rows contain the probabilities of successfully awarding similar scores, by chance, to the replicates on two, three, or four flights. Note that statistical significance of less than 5% is reached only at T 1 or T 2 for duplicate samples.
Type I Error for Duplicate Samples
Inconsistency. We have looked at performance characteristics for a judge to be called an expert. So, what level of inconsistency are we to accept? At what level do we yellow cardFootnote 3 a judge? If a range of Bronze+ to Silver+ is acceptable (B+, S–, S, S+), what about a range of B+ to G–? What about Bronze to Gold, a range encompassing seven categories? Certainly, a range from NA to Gold is unacceptable. For the purpose of this paper, we “yellow card” a judge whose range of scores on any flight encompasses six or more categories (NA to S+, B to G–, B+ to G) and “red card” a judge whose range encompasses all categories (NA to G). As yellow and red cards result from inconsistent awards being given on any flight, a judge might receive several such cards, assuming that replicates are served on more than one flight. While an occasional card might be attributed to a judge having a bad day, competition managers should be aware of those who often award erratic scores to replicated wines.
Discrimination. Before meeting the qualifications of expert status based on consistency, it is necessary to verify that the judge does not achieve expert status by awarding all wines (not just replicates) the same score. This only requires a comparison of the range of a judge's scores to that of the other judges sitting on the panel.
We now apply these criteria in some examples.
1. Judge A, during the course of the competition, rates triplicate samples served on four flights (B+, S, S–), (NA, NA, NA), (G, S+, G) and (S, B, S–). This judge scores all four of these triplicates within the T 4 criterion and thus would merit expert status at the 3% level of significance.
2. Judge B awards the following: (B, S, S–), (G, S, S–), (NA, B+, B), and (S, S, S). This judge does not meet the T 4 criterion. The second flight covered five categories (S–, S, S+, G–, G), falling outside the T 4 criterion, but the judge does not merit a yellow card (scores encompassing six or seven categories).
3. Judge C awards the following: (B, G–, S), (B–, B, B), (NA, NA, NA), and (B, B+, S–). This judge is flagged (yellow card) because of excessive inconsistency (first triplicate covered six categories).
4. Judge D awards the following: (B, S, G), (G, NA, G), (S, B, G–), and (S, B+, S). These scores merit two yellow cards and a red card.
III. Data
We now apply the above to the data obtained from the CSFCWC for 2010 to 2012. This competition has had a continuous program of judge evaluation in place for a decade. Typically about 3,000 wines are evaluated by 70 judges. Each panel consists of four judges who evaluate about 15 flights of wines, with each flight consisting of about 10 wines. The judging is spread over two days, so each judge sees about 75 wines per day. Embedded in four of these flights are triplicate samples of an identical wine, poured from the same bottle and presented on the same flight. The judges score these wines on an ordinal scale: No Award (82 points), Bronze (84 points), Silver (90 points), and Gold (96 points). Plus and minus modifiers are allowed and are worth 2 points (an S– would obtain a score of 88 points). The judges assess the wines twice. In the first round, the judges independently score the wines and communicate their scores to a steward. Afterward, the judges confer with one another and are able to modify their initial scores if they so desire. For the analysis that follows, only the initial scores are analyzed.
Table 4 is an example of the data from the 2010 CSFCWC referenced earlier. The column ID identifies each wine, whereas the wine codes are randomly generated numbers that determine the order in which the wines are presented to the judges. Columns J1 to J4 contain the scores given the wines by the four judges. The last two rows show the range of scores given the replicated samples by each judge as well as the range of scores given to all the wines.
Example of Panel Score Card from CSFCWC
In Table 4, note that wine #380 appears three times. These wines have separate codes as shown in the second column. The wines are presented to the judges in order of their codes, with the placement of the replicates being randomized. Note that J1 scored the three replicated samples with the same score, 82 points (No Award). The fourth judge scored the replicates with a range of six points, No Award to S–. Both these judges meet the T 4 criterion of scoring a wine within four adjacent grades. Judge 2 would be red carded under our system, and Judge 3 would be yellow carded. Judges 1 and 4 would need to repeat their performance at the T 4 criterion on the three additional flights to be considered Expert as defined in this paper.
IV. Results
We now revisit the question of what might be a reasonable level of discrimination. Looking at the 2012 CSFCWC data, the authors counted the number of times a judge awarded a triplicate sample the same score (T 1). This occurred 24 times out of the 304 triplicate samples served. In only two instances did a judge meet the T 1 criterion twice out of the four triplicate sets. No judge met the T 1 criterion in three or four triplicate sets. Thus, the T 1 criterion seems to be beyond human ability. In addition, for the 2012 data, meeting the T 1 criterion once was often followed by a yellow card or red card performance on subsequent flights.
Another way to establish a reasonable discrimination standard is to examine Figure 1 in a previous publication (Hodgson, Reference Hodgson2008). As there is a correspondence between maximum range and consistency at the T 1 parameters, we can infer that about 20% of the judges met the T 4 criterion in any given year.Footnote 4 Thus we set the T 4 level as a moderate marker of excellence. Admittedly, this is arbitrary.Footnote 5
At the other end of the discrimination scale, judges who grade a wine with scores that range from Bronze to Gold (T 7) or, worse, offer little or no information at all to the consumers.
So, we use the results of the CSFCWC to evaluate judge performance at the T 4 index of tolerance. Our expert criterion is that a judge scores the replicated wines with similar scores (at T 4) for all four flights. Judges who fail to meet expert status are grouped into three categories: yellow carded (T 6, T 7), red carded (T 8), or T 5. The T 5 category consists of those who are not carded but fail to meet expert criteria at the 5% level of significance.
We now ask the question of how to treat a judge who meets the criterion of expertness (T 4) on three flights but fails on one, scoring the wines at the T 5, T 6, or T 7 (yellow card) or T 8 (red card). We may use the binomial theorem to calculate the probability that a judge, by chance, will be successful in three of four flights at the T 4 through T 7 indices of tolerance.
Table 5 calculates the binomial probabilities associated with successfully scoring triplicate samples of an identical wine, by chance, with similar scores. At the T 4 index of tolerance, there is a 20% probability the judge could award similar scores in three or four flights by chance alone. Thus for a judge who is successful in meeting the T 4 criterion in at least three flights, naming the judge Expert would carry a 20% Type I error that the judge could have done so by chance alone, well above our 5% threshold of significance.
Type I Error at Varying Tolerance Indices for Being Successful in x out of Four Flights
This analysis was performed for each year of the 2010 to 2012 competitions. Thus, it is possible for a judge to meet expert status one year but be carded in another. Also, as the judge pool changes from year to year, our results reflect some who judged all three years and others who judged only once.
To be named an expert, a judge must have met the expert criterion every year. A judge who was named Expert one year but failed in another would not make the final cut into our expert category. Table 6 is a summary of our results.
Statistics of Judges Serving at the CSFCWC, 2010 to 2012
Table 6 represents judges who served at least once in the three-year period 2010 to 2012. We set the tolerance level at T 4 to be considered an expert judge. Among the 122 judges who were evaluated, 22 judges met this criterion. However, only two of these judged all three years. Another three judged two of the three years, and the remaining 17 judged only once. Another 20 were unsuccessful at T 4, but also did not fall into the yellow or red card status. There were 40 judges who received a single yellow card, and an additional 40 who received at least two yellow cards or red cards.Footnote 6
There were 37 judges who participated all three years. As mentioned, two met the T 4 criterion all three years. Another 6, though not meeting T 4, also did not fall into yellow card status. Nine more received one yellow card, and the remaining 17 received more than one yellow or red cards.
It is of interest to ask whether there are common characteristics that separate our newly defined Experts from those who fell in the bottom third of our pool. Are our Experts academics, winemakers, or journalists? Do they have more experience in judging other major wine competitions? To answer the latter question, the first author contacted 270 individuals who had judged at the CSFCWC from 2010 to 2012, asking them to list other venues in which they had participated. Twenty-eight of the 62 respondents were differentiated based on two factors: (1) meeting the expert criterion (T 4) or receiving multiple yellow or red cards, and (2) having participated only in local county fairs or having judged in three or more major international competitions in the United States and abroad. The results, shown in Table 7, are counterintuitive. One might expect that those who are asked to judge major competitions would be among our expert judges. However, that was not the case. A Fisher's Exact Test shows no relationship between experience and judging consistency.
Survey of Wine Judges Comparing Experience and Consistency
V. Discussion
Past work by one of the authors was initiated by a belief that (1) the awards resulting from wine competitions are so unreliable that they can be explained by chance alone, (2) the source of the unreliability can be traced to the inconsistency of individual wine judges, and (3) individual judge performance varies from year to year. The current work is motivated by a belief that there have been no standards for selecting wine judges or that the criteria are based on false assumptions. Simply participating in a wine competition does not confer expert status. In fact, as seen in Table 7, wine judges with considerable experience judging international competitions are just as likely as those with much less experience to be inconsistent in evaluating wines.
This study began with a premise that an expert wine judge should be able to discriminate among a set of wines between those wines that are good and those that are unworthy of medals. To test this premise, an experiment is described in which a judge is asked to evaluate a set of wines containing replicated samples, and a test is proposed in which we observe whether or not a judge is able to award similar scores to the replicates. We set similarity parameters over a range of awards from intolerant (the judge must give the exact award to all replicates) to completely indiscriminating (the range of awards varies from No Award to Gold, the highest award). We then set an intermediate level of tolerance, T 4, to mean a judge could meet the test's requirement when awarding scores that covered at most four adjacent levels of quality, for example (B+ to S+).
We then used this test to evaluate 122 judges from 2010 to 2012 at the CSFCWC. Many of these judges have extensive experience in the wine industry and have judged numerous international wine competitions in the United States and abroad. The results: only 22 judges passed the test of consistency, while many did not, including those who by experience and training would otherwise be considered experts in the wine industry. Since many of these judges also serve as judges at other competitions, it is reasonable to extend these results to wine competitions in general.
Ostensibly, wine competitions are designed to provide reliable evaluations of wine quality. However, reliable assessments of wine quality require judges to discriminate among wines of varying quality on a consistent basis. While past work by the authors and others (Cao and Stokes, Reference Cao and Stokes2010) quantified judge inconsistency, this work provides a foundation for establishing reasonable standards for judge performance. Unfortunately, when applied to a recognized group of wine experts, most fail to meet these standards.
Although the focus of this study covers the period from 2010 to 2012, the authors have accumulated data on 190 judges at the CSFCWC from 2005 to 2012. For those who participated in three or more competitions, only one judge consistently met the expert criterion (T 4) at every competition in which he judged (four times) and just three more consistently met the T 5 criterion (one of whom had judged in eight competitions). So, it is fair to ask whether meeting the T 4 criterion (or even T 5) is a realistic standard of discriminatory ability in a competition setting. The data suggest otherwise. The conclusion is that wine competitions do not yield reliable recommendations of wine quality.
VI. Conclusion
A mathematical model is developed that yields a decision rule to confer expert status to wine judges. The decision rule is based on the ability to award similar scores to replicated wine samples embedded within a flight of wines. Based on either an empirically derived probability distribution or an analytically derived formula (see the appendix), a Type I error is established that accompanies the decision rule. When the model is applied to those who have judged a major wine competition over several years, it is discovered that (1) only a few meet our criterion of being an expert wine taster, (2) there is no correlation between our Expert wine tasters and experience in judging major wine competitions, and (3) many judges fail meet the expert criteria described herein, who, by reputation and experience, would otherwise be considered experts.
Appendix
The likelihood that a judge could award similar scores to the three replicate wine samples can be calculated in two ways. Both involve transforming the letter grades to integer equivalents, that is, NA, B, … , G correspond to the integer values 1, 2, … , 8. A straightforward but tedious approach is to list all permutations of the possible triplicate scores: (1,1,1), (1,1,2), … , (8,8,7), (8,8,8). There are 512 (i.e., 83) possibilities. Then we count the number of triplicates that have identical digits, such as (1,1,1), (6,6,6), corresponding to the T 1 (same score) criterion. There are eight such outcomes. We then count the number of triplicates such as (1,2,1) or (5,4,4) including the same score outcomes such that the individual digits differ by at most one unit (50 outcomes). Continuing, we count the number of permutations that differ by at most two units (122 outcomes), three units (312 outcomes), etc. These empirically derived outcomes and the associated probabilities corresponding to the tolerance parameters T i are listed in Table 1.
To generalize the calculation, we introduce an analytical formula that can accommodate arbitrary k, m, and different tolerance range T i. The probability of giving the k identical wines within T i range of scores by chance is
$$P_i (m,{\kern 1pt} k) = P_1 (m,{\kern 1pt} k) + \cdots + P_i (m,{\kern 1pt} k) = \sum\limits_{\,j = 1}^i {\,p_j (m,{\kern 1pt} k)}. $$In the formula (2), p i(m,k)=(m−i +1)n i/mk and 
$n_i = i^k - \sum _{\,j = 1}^{i - 1} (i - j + 1)n_j $ (for i≥2) where n i can be derived starting from n 1=1. Note that p 1(m,k) is the probability of giving the k identical wines exactly the same score, p 2(m,k) is the probability of giving the k identical wines scores that span exactly two units (e.g., (1,2,1), (2,2,3), etc.), and p i(m,k) is the probability of giving the k identical wines scores that span exactly i units. Since T i means all replicates are given scores that span at most i adjacent grades, this leads to the above expression of P i(m,k). In addition, (m−i+1) in p i(m,k) presents how many sets of numbers construct valid combinations of scores spanning exactly i units. Take m=8, k=3, and i=2, for example, the combinations consist of (1, 2), (2,3), (3,4), (4,5), (5,6), (6,7), and (7,8), which is (m−i+1)=8−2+1=7 sets. Then within each set, there are n i possible combinations. For example, using the set (1,2), the combinations are (1,1,2), (1,2,2), (1,2,1), (2,1,2), (2,1,1), (2,2,1), which is n 2=23−2n 1=8−2=6. So (m−i+1)n i is the number of valid combinations of scores spanning exactly i units (for i=2, there are (m−i+1)n i=7×6=42 such combinations). Dividing this by mk, the total number of all possible combinations of scores for the k identical wines, it gives us the probability p i(m,k).
In the following we take m=8 and k=3, for example, to demonstrate how to calculate P i(m,k). Starting with P 2(m=8, k=3), we first have n 2=23−2n 1=8−2=6, p 1(m=8, k=3)=m/mk=8/83=0.01563, p 2(m=8, k=3)=(m−i+1)n 2/mk=7×6/83=0.08203, and then we have P 2(m=8, k=3)=p 1+p 2=0.01563+0.08203=0.0977 (we use p instead of p(m=8, k=3) for simplicity). To calculate P 3(m=8, k=3), we have n 3=33−3n 1−3n 2=27−3−12=12, p 3=(m−i+1)n 3/mk=6×12/83=0.1406, and P 3(m=8, k=3)=p 1+p 2+p 3=0.01563+0.08203+0.1406=0.238. To calculate P 4(m=8, k=3), we have n 4=43−4n 1−3n 2−2n 3=64−4−18−24=18, p 4=(m−i+1)n 4/mk=5×18/83=0.1758, and P 4(m=8, k=3)=p 1+p 2+p 3+p 4=0.01563+0.08203+0.1406+0.1758=0.414. Based on the above formula, it can be shown that n 5 to n 7 are (24, 30, 36), p 5 to p 7 are (0.1875, 0.1758, 0.1406), and P 5(m=8, k=3) to P 7(m=8, k=3) are (0.602, 0.777, 0.918), respectively.
