2. THE QUESTION

In every collective grading problem there are some graders, and there are some things for them to grade. Just who and what they are depends on the case. They might be members of a committee and project proposals. They might be teachers and course papers, or members of a review panel and bodies of medical evidence. We will fix some representative grading problem and study that. Let N be a finite set of graders, and let X be a set of things. They could be just about anyone and anything but they do have to be some particular ones. Fix also a finite set $\mathcal{G}$, the vocabulary of available grades, with a linear ordering > from better to worse.Footnote ⁵

Example 1. Let $\mathcal{G}$ be {A, B, C, D, F}, and let A > B > C > D > F. This is the language of grades that is used, with minor modifications, in colleges and universities throughout the world.

There is also an aggregator, whose job it is to rank all of the items from better to worse on the basis of the grades awarded to them. For ease of exposition let the aggregator be – you. Taking on this role doesn't mean that you cannot at the same time be one of the graders. It doesn't even mean that you are a person. In online course evaluation, the aggregator is an algorithm running on a computer.

One question straight away is this: Just how will you proceed, once the graders have told you which grades they have awarded? That is, how will you assimilate their input into a single evaluation of all the items? To study this question it will help first to make it precise, using ideas from the theory of social choice (List Reference List and Zalta2013). For any individual grader i, let an individual appraisal G_i be a function assigning to each item in the set X some grade in $\mathcal{G}$. That is, G_i(x) ∈ $\mathcal{G}$. Think of G_i as a report that this one grader i might hand you, saying which grades he has awarded to which items. A grade profile $\mathcal{G}$ is a list <G₁, . . ., G_n> of individual appraisals, one for each grader i. It is a possible report from all the graders. For instance:

Example 1 (continuation). Let X be {a, b, c}, let N be {1, 2}, and let G₁(a) = A, G₁(b) = C, G₁(c) = C, G₂(a) = C, G₂(b) = A, G₂(c) = F. <G₁, G₂> is a grade profile.

A grade domain $\mathcal{D}$ is a set of grade profiles. Intuitively, these are all the reports that, as far as you know, the graders might produce. They needn't be all conceivable ones. Perhaps only some of the grades are made available to the graders at first, the others being held in reserve to smooth over differences during aggregation if the graders disagree. A grade domain that does not include all ‘logically possible’ reports is said to be restricted.

Example 2. Let X, N and $\mathcal{G}$ be as in Example 1. Let $\mathcal{D}$ be the set of all grade profiles G such that for any i ∈ N and x ∈ X: G_i(x) ∈ {A, C, F}.

This domain is suitable for analysing a grading problem in which the two graders are allowed to award only an A, a C or an F, with the intermediate grades B and D held in reserve.Footnote ⁶

You will use a fixed procedure or rule for aggregating the grades into a ranking of the items. This will keep you consistent. Also, you will choose this rule in advance, before you find out just what grades everybody happens to turn in. This will keep you straight. You cannot go back and change your mind, if some favourite of yours happens not to come out on top. Let a grading rule be a function F that maps any given report G, taken from some domain of possibilities, to a binary relation F(G) on X. For any given x and y, the intended interpretation of x F(G) y is: were the graders to hand in report G then your judgement on that basis, using rule F, would be that x is as good as y, perhaps better. The question is now: Which grading rule will you use?

Here is a rule that will illustrate several points later on.

Example 3. Let N and $\mathcal{G}$ be as in the previous examples. For any g ∈ $\mathcal{G}$, let mid{g, g} be g. For any g₁ and g₃ ∈ $\mathcal{G}$ such that some further grade g₂ lies, in the sense of >, in the middle between g₁ and g₃, let mid{g₁, g₃} be this g₂. If on the other hand there is no such g₂ then let mid{g₁, g₃} go undefined. For instance, mid{D, D} = D, mid{A, C} = B and mid{F, A} = C; but mid{B, C} and mid{B, F} are undefined. Now, for any suitable domain, we define a grading rule F as follows: for a given profile G we let F(G) be that relation R ⊆ X ² such that xRy if mid{G₁(x), G₂(x)} and mid{G₁(y), G₂(y)} are both defined, and mid{G₁(x), G₂(x)} >mid{G₁(y), G₂(y)}.The following point is important for what is to come. Let G be any profile in the domain $\mathcal{D}$ of Example 2. Then R is a weak ordering of X, both transitive and complete.Footnote ⁷R inherits its transitivity from the linear ordering > of the grades. Completeness follows from the domain restriction. Because the grades that 1 and 2 assign to any given x are either identical or sufficiently far apart, mid{G₁(x), G₂(x)} is always defined.Footnote ⁸

So far this setup is only a slight extension of the framework that Kenneth Arrow introduced in order to study problems of social choice (Arrow Reference Arrow1951). Notice the one important difference. In Arrow's framework there is no grading of social alternatives. Instead people rank these from better to worse. This matters because we can say more by grading things than by ranking them. When for instance the Michelin company awards one restaurant three stars and the other just two, it says that the first restaurant is better than the second. It says more than just this, though, because different grades correspond to the same ranking: had the second restaurant instead received just a single star, or no star, the implicit comparison would have been the same. There is more information in the grades than in the corresponding ranking.

Notice also that, even so, there is a way in which people typically can say less when asked to grade things than when they are allowed to rank them as they see fit. Grading vocabularies tend to be small. There are just four Michelin categories, for instance, from no stars to three. The Michelin company evaluates tens of thousands of restaurants around the world, though, and so there have to be many, many ties; it is out of the question to express using Michelin stars, say, a linear ordering of all the restaurants, which would mean assigning to each one a unique grade. When grades can capture some given ranking they might do so in more ways than one. Typically, though, there are rankings they cannot capture at all.

3. CHOOSING A GRADING RULE: SOME CONSIDERATIONS

So now you are on the lookout for a good grading rule. Here are some features to consider. All are desirable in a range of grading problems. They are closely related to requirements that Arrow (Reference Arrow1951) imposed on social welfare functions.

A first feature that a grading rule F might be expected to have is:

Completeness (C). For each grade report G in the domain of F, F(G) is a complete relation.

Completeness is good when our purpose in evaluating things is to choose from among them. Say we are to rank some candidates for a job. If we can be sure to end up with a complete ordering of the different ones then, if somehow we lose the very best candidate, perhaps to another employer, there will always be a second-best candidate to fall back on.

In order to express the next feature it will help to have some notation for the ranking implicit in the grades awarded by an individual i. Given an individual appraisal G_i, let Ğ_i, or flat G_i, be the binary relation on X such that x Ğ_i y if G_i(x) >G_i(y).Footnote ⁹ We can also flatten whole profiles: Ğ = <Ğ_i> is the list of everybody's implicit rankings. Now, let S be some subset of X, and let <Ğ_i>|S be <Ğ_i|S>, the result of restricting each ranking in the list to S. The relevant feature is now:

Ordinally Unrestricted Domain with respect to S (OU-S). For each list <R_i> of |N| weak orderings of S, the domain of F includes some report G such that Ğ|S = <R_i>.

OU-S requires the grading rule to handle a certain variety among inputs. The graders N might put the items S in any order, and they might do so independently of one another; the rule has to be able to cope no matter what rankings the graders produce. This is a plausible requirement in some grading problems. When evaluating research proposals, for instance, little might be known in advance about the merit of the different ones. Then, perhaps, the graders’ appraisals could turn out to be anything, and the grading rule had better be ready for everything. Notice that it is possible for a grading rule to satisfy OU-S only if there are as many grades in the vocabulary $\mathcal{G}$ as there are items in S. With ‘pass-fail’ grading for instance there are just two grades. All implicit rankings Ğ_i are dichotomous and imposing OU-S makes sense only for pairs S.Footnote ¹⁰

Writing R instead of F(G), when the context makes clear which grading rule F and profile G are meant, let P be the strict component of R: xPy means that, on the basis of the grades reported in G, x is counted really better than y – not merely as good.Footnote ¹¹ A further condition is then:

Weak Pareto (WP). For any x, y ∈ X and any G in the domain of F, if, for each i ∈ N, G_i(x) > G_i(y), then xPy.

This says that when everybody awards one thing a higher grade than another, it must be counted strictly better.

Another feature that might be desirable, for any given set S of items, is:

Non-Dictatorship with respect to S (D-S). There is no δ ∈ N such that for any x, y ∈ S and any G in the domain of F, if G_δ(x) > G_δ(y) then xPy.

D-S rules out choosing some particular grader in advance and, when the grade report comes in, whichever it may be, just reading off the resultant ranking from the grades that this one ‘dictator’ has awarded. Indeed, that would seem to defeat a main purpose of grading in a group, which is to pool information. Depending on the domain, though, D-S also excludes some perfectly democratic ways of proceeding. A ‘dictator’ here is just someone whose grades invariably agree with the judgements of the group. That doesn't entail any power or influence over these.Footnote ¹²

Finally, writing R for F(G) and R* for F(G*), we have:

Independence of Irrelevant Alternatives (I). For any x, y ∈ X, and any G and G* in the domain of F, if G|{x, y} = G*|{x, y}, then R|{x, y} = R*|{x, y}.

I says that the final ranking of any two given items is fixed by the grades awarded just them. It does not depend on any other features of these items that are not reflected in their grades, or on which grades are awarded to items outside the pair.

Condition I is always satisfied when, as in Example 3, each item is first awarded a resultant grade on the basis of the grades awarded to it by the different graders, and then the ranking R of all the items is read off from their resultant grades. Determining resultant grades ‘on a curve’, on the other hand, with some fixed quota set in advance for each grade, may be expected to lead to violations of I. Be this as it may, appraising things separately, each according to the grades awarded just to it, seems quite natural. Arguably, it is only proper when merit is something that things have all by themselves, not in virtue of how they compare with other things. To this extent I is a reasonable constraint on grading rules.

Now it is not difficult to see that:

Example 3 (continuation). As defined on the domain $\mathcal{D}$ of Example 2, the grading rule F of Example 3 satisfies conditions C, OU-X, WP, D-X and I.

Arrow's (Reference Arrow1951) social welfare functions are rules for assimilating individual preference rankings into a single ‘social’ ordering. His ‘impossibility’ theorem says that no social welfare function satisfies a short list of conditions (for discussion of these, see Morreau Reference Morreau and Zalta2014). Now C, OU-X, WP, D-X and I are versions of Arrow's conditions, adapted to the case in which people grade things instead of ranking them; since the grading rule of Example 3 satisfies them all, it seems that grading opens up possibilities that are not there in Arrow's framework.Footnote ¹³ Balinski and Laraki see this as one advantage of their preferred method of majority judgement which, they say, ‘overcomes’ Arrow's impossibility (Balinski and Laraki Reference Balinski and Laraki2010: xiii). Critical to the solution, in their view, is that a common language of grades enables ‘absolute’ judgements of each item separately (Balinski and Laraki Reference Balinski and Laraki2010: 185).

One main point in the following is that Arrow's impossibility is still there in the background – common language, absolute judgements and satisfaction of the adapted Arrow conditions notwithstanding. It is hiding behind an implicit assumption. Uncertainty about thresholds can destroy much of the information in grades. Suppose for instance that you as aggregator have no idea where one of the graders draws the line between a B and a C. Then, when you hear that he has awarded Jack a B and Jill a C, all you've really learned is that he thinks Jack is better than Jill: you have a ranking but that is all. The hidden assumption is that enough is known about everybody's thresholds so that grades carry more information than just rankings. When too little is known we might as well be back in Arrow's framework, where individual inputs are limited to rankings right from the start; and then, as section 5 shows, it is not possible to satisfy the adapted Arrow conditions together with a further condition of ‘soundness’. That is how Arrow's impossibility reappears.

4. SOUND GRADING

We have been developing the following model of grading in groups. First, each grader estimates the merit of each item, privately and in more or less precise terms. Second, each grader awards a grade to each item, on the basis of his estimation of it. Third, all the grades go to you, the aggregator. Finally, using a grading rule, you assimilate the grades into resultant ranking of all the items. It is inherent in this model that information about merit takes an indirect path from the graders to you, passing through the linguistic medium of grades. Intuitively, your grading rule is sound if, to the best of your knowledge, the resultant ranking must always agree with the result that a direct aggregation of the underlying estimations would have produced – if only, somehow, you could have access to these.

The qualification ‘to the best of your knowledge’ is crucial. This is where uncertainty about thresholds comes in. Section 4.2 shows how to capture this qualification by quantifying over all combinations of individual estimations of merit that, for all you can tell, are compatible with the reported grades. These combinations make up what I shall call the ‘import’ of the grade report. Section 4.3 defines soundness and gives an example of a sound grading rule. Meanwhile, section 4.1 lays a foundation for all this by introducing the dimensions of merit that grades measure.

4.1. Degrees of merit

Grades measure the goodness of things, or their quality, or value. Here, the term ‘merit’ will cover these. Merit has a finer grain than grades: two things might both deserve the same grade even though one of them is better than the other. A merit structure is a pair ($\mathcal{M}$, >), of which $\mathcal{M}$ is a non-empty set, the degrees of merit. It is ordered by > from better to worse. For example, in education we sometimes assign students letter grades on the basis of their percentage scores in a separate test. Then $\mathcal{M}$ includes the numbers between 0 and 100%. There can be more numerical structure than this minimal requirement of a better–worse relation. Sometimes, we can add up degrees of merit, and average them.

I shall assume that comparative merit is a complete ordering: for any distinct degrees of merit m and n, either m > n or n > m. With many evaluation problems this is quite unrealistic. Think of two restaurants. If one has a better menu than the other, but not such a nice view, there might be no saying which of them is, overall, best. Nor do we seem to think they are equally good, though. A further slight improvement in the menu at the first restaurant – there's an additional special today! – won't tip the balance and make it quite simply the better of the two, overall. The restaurants do not seem to be comparable in terms of the standard value relations ‘strictly better’ and ‘equally good’.

So, which will it be for lunch? When alternatives are incomparable there might be no uniquely best one, nor even a tie for best, and therefore no scope either for maximizing choice. But there is a way in which two alternatives, though not comparable by the standard value relations, might still be roughly equal. Say we count both restaurants as excellent. They are to this extent comparable with one another. If in addition they are better than all the other alternatives then it would seem quite appropriate just to flip a mental coin and get on with lunch, as we might with options that really are tied for best. This idea recalls Ruth Chang's (Reference Chang2002) notion of ‘parity’ and the role she sees for it in supporting rational choice. Indeed, Chrisoula Andreou's (Reference Andreou2015) account of parity relies on categorical judgements that are very much like my grade assignments. Thus the theory of grading, when comparative overall merit is an incomplete relation, might underwrite maximizing choice in some cases where it has seemed there can be none.

Balinski and Laraki distinguish grades, messages about merit, from what they call the graders’ ‘deep preferences or utilities’. These utilities, they explain, in the context of voting, are ‘relative measures of satisfaction’. They illustrate this point with the example of the 2002 French presidential elections:

The voters of the left would have hated to see Jacques Chirac defeat Lionel Jospin: their utilities for a Chirac victory would have been the lowest possible. The same voters were delighted to see Chirac roundly defeat Le Pen in the second round: their utilities for a Chirac victory were the highest possible. On the other hand, these same voters would probably have given Chirac a grade of Acceptable or Poor . . . were he standing against Jospin, Le Pen, or anybody else. (Balinski and Laraki Reference Balinski and Laraki2010: 185)

Now, in my account, a voter's estimation of Chirac's merit is what underlies the appraisal of him as an Acceptable candidate, or a Poor one, as the case may be. This estimation is likely to have been the same, more or less, irrespective of who the other candidates were. My degrees of merit, then, are not relative measures.

I shall now again borrow the Arrow–Sen framework by introducing merit profiles. A first notion is that of an individual merit assignment M_i. This is a function mapping each x to some degree M_i(x) of merit. Intuitively, M_i is a possible fact about i’s private estimation of each item, on the basis of which i will award grades to the different ones. A merit profile M is a list <M₁, . . ., M_n> of individual merit assignments, one for each grader. Notice that merit profiles are not reports. They are not linguistic items, or any other sort of thing that can in any direct way be made public. Rather, a merit profile is a possible fact about how good or bad all the individual graders might privately take everything to be. Merit profiles will figure, here, in an account of the content of grade reports – of the information that these convey from the graders to you, the aggregator.

A merit domain is a set of merit profiles. We will assume that merit domains are unrestricted. That is, they include all ‘logically possible’ lists of individual merit assignments. This does not have any obvious empirical consequences. It does not constrain the set of items to be evaluated, or how the graders will determine which grades to assign to them, or any other special features of the grading problem being studied.

The unsoundness theorem in section 5 relies on two assumptions about merit. First, the merit ordering is dense. That is, whenever m₁ > m₃ there is a further degree of merit m₂ in between: m₁ > m₂ > m₃. Second, it is unbounded above and below: there is no m such that, for each other n, m > n; and there is no n such that, for each other m, m > n. These assumptions are used in the proof of the crucial Lemma 11. They are strong. Either entails that $\mathcal{M}$ is infinite, which limits the applicability of this framework. Sometimes, presumably, grades are awarded on the basis of estimations of merit that are themselves coarse grained, only less so, and finite in number. Also, these assumptions are psychologically unrealistic. I expect that the first can be done away with by requiring instead that, in between any two degrees of merit that an individual function M_i actually assigns, there are in $\mathcal{M}$ as many other degrees of merit as there are grades in the (finite) vocabulary $\mathcal{G}$. The second assumption could be replaced by a requirement that no M_i shall assign an extreme value. No grader will ever find any of the items under consideration to be so utterly good or so excruciatingly bad that it is not even imaginable that something could be better, or worse.

A merit function F $^\mathcal{M}$ maps each merit profile M in its domain onto a weak ordering of X. Intuitively, it is a procedure for aggregating several estimations of merit into a single collective comparison. One equitable way to do so is to average them:

Example 4. Let X and N be as in the previous examples. Let $\mathcal{M}$ be the real interval (0, 1) with the obvious ordering. This fixes the merit domain: {<M₁, M₂>: M_i: X → (0, 1)}. Now for any such M = <M₁, M₂> in this domain, and for any x, y ∈ X, let F $^\mathcal{M}$(M) be that binary relation R on X such that:

$$\begin{equation*} xRy\,{\rm if}\ {\rm and}\ {\rm only}\ {\rm if}\,[{M_1}(x) + {M_2}(x)]/2 \ge [{M_1}(y) + {M_2}(y)]/2. \end{equation*}$$

The earlier grading rules have the same co-domain as merit functions, all rankings of the set X, but they take grade profiles as input, not merit profiles. For clarity, let us from now on use a superscript when referring to grading rules as well: F $^\mathcal{G}$ will be a grading rule, and F $^\mathcal{M}$ a merit function.

4.2. Import

We will now need an account of the information carried by grade reports. The basic notion is that of an interpretation. It fixes grading thresholds. That is, it tells us precisely how good or bad things have to be in order to get one grade or another. This idea is familiar from the grading keys or scales used to turn numerical scores into letter grades. Let us refer to the vocabulary $\mathcal{G}$ of grades more formally now, along with the linear ordering > from better to worse, as a grading language ($\mathcal{G}$, >). Technically, an interpretation maps each grade from $\mathcal{G}$ onto an interval of ($\mathcal{M}$, >), the better the grade the higher the interval:

Definition 5. An interpretation of ($\mathcal{G}$, >) in ($\mathcal{M}$, >) is a function I mapping each grade g from $\mathcal{G}$ onto a subset I(g) of $\mathcal{M}$, such that:

For each g ∈ $\mathcal{G}$, I(g) is convex: if m₁, m₃ ∈ I(g) and m₁ > m₂ > m₃, then m₂ ∈ I(g); and

I is orderly: for any g₁, g₂ ∈ $\mathcal{G}$, m₁ ∈ I(g₁), and m₂ ∈ I(g₂), if g₁ > g₂ then m₁ > m₂.

These are the basic requirements. We can stipulate as well that for each degree of merit there is some corresponding grade, and that for each grade there is some degree of merit:

∪_g∈$\mathcal{G}$I(g) = $\mathcal{M}$; and

For each g ∈ $\mathcal{G}$, I(g) ≠ ∅.

These further requirements are not as basic but we will impose them as well.Footnote ¹⁴ Here is an example of an interpretation:

Example 6. Let $\mathcal{G}$ and $\mathcal{M}$ be as in the previous examples. Let I(A) = [0.8, 1.0), I(B) = [0.6, 0.8), I(C) = [0.4, 0.6), I(D) = [0.2, 0.4) and I(F) = (0.0, 0.2). Then I is an interpretation of $\mathcal{G}$ within $\mathcal{M}$.Footnote ¹⁵

Say all the graders have finished their work. They produce a report G saying who has awarded which grade to what. Their report tells you, the aggregator, something about how good they take everything to be. Just what it tells you, though, depends on how much you know about their thresholds. The information about merit carried by G will now be captured in the notion of the import of G, written ⟦G⟧. This is a set of merit profiles. Intuitively, M ∈ ⟦G⟧ means that the information in M is, as far as you can tell, compatible with that in G. The less you know about everybody's thresholds, the more compatible merit assignments there are, and the more inclusive is ⟦G⟧.

First, here is a notion of import for individual graders.

Definition 7. The import ⟦G_i⟧^I of G_i, relative to I, is {M_i: for each x ∈ X, M_i(x) ∈ I(G_i(x))}.

⟦G_i⟧^I includes M_i if i’s estimation of any given x, according to M_i, is compatible with the grade that i has awarded to x, according to G_i, on the assumption that i’s thresholds are I.

Now an import function ⟦⟧ maps each profile G in its domain to some set ⟦G⟧ of suitable merit profiles.

Example 8. Let X, N, $\mathcal{G}$ and $\mathcal{M}$ be as in the previous examples, and let I be any interpretation. Consider any suitable grade domain $\mathcal{D}$. For any given G ∈ $\mathcal{D}$, let ⟦G⟧, the import of G, be:

$$\begin{equation*} \{ < {M_1},{M_2} > :{M_1} \in {\left[\kern-0.15em\left[ {{G_1}} \right]\kern-0.15em\right]^I}\ {\rm and}\,{M_2} \in {\left[\kern-0.15em\left[ {{G_2}} \right]\kern-0.15em\right]^I}\} . \end{equation*}$$

Intuitively, the import function of Example 8 says which assessments of merit by both graders are compatible with any given report G on which grades they have awarded, assuming they have in common the grading thresholds I. In some tightly constrained cases this assumption is reasonable. The case described before, where grades are awarded on the basis of percentage scores, could be one of these – if the graders have settled on a common grading key, and if you as aggregator happen to know just which one it is. Let us have a name for import functions of this sort.

Definition 9. ⟦⟧ is grounded and rafted if there is some I such that for each G in the domain, ⟦G⟧ = ⨉_{i ∈N} ⟦G_i⟧^I.

Grades are less misleading when the aggregator knows this much. Consider again the case of Jack and Jill. According to the report G you received, one grader gave Jack a B and the other gave Jill a D: G₁(Jack) = B and G₂(Jill) = D. If ⟦⟧ is grounded and rafted then, irrespective of I, the graders cannot, like Mr Easy and Ms Measly, have had different thresholds. ⟦G⟧ doesn't include any M such that M_Easy(Jack) = 63% and M_Measly(Jill) = 64%. There is no common interpretation I such that:

$$\begin{equation*} 63\% {\rm{ }} = {\rm{ }}{M_{Easy}}\left( {\rm Jack} \right) \in I\left( {{G_{Easy}}\left( {\rm Jack} \right)} \right){\rm{ }} = {\rm{ }}I\left( B \right),\,{\rm and}\ {\rm also} \end{equation*}$$

$$\begin{equation*} 64\% {\rm{ }} = {\rm{ }}{M_{Measly}}\left( {{\rm Jill}} \right) \in I\left( {{G_{Measly}}\left( {{\rm Jill}} \right)} \right){\rm{ }} = {\rm{ }}I(D). \end{equation*}$$

This is because interpretations are orderly (see Definition 5). Since B is a higher grade than D, each degree of merit in I(B) must be greater than each degree of merit in I(D), but 63% is not greater than 64%. When import is grounded and rafted, then, the false impression that Jack is better than Jill cannot arise, at least not in the same way. Your judgement, based on the grades, might accurately reflect the collective judgement of the graders. Example 8 (continuation) describes a case in which it reliably does.

At the other extreme we know nothing about the graders’ thresholds. Letting $\mathscr{I}$ be the set of all interpretations,

Definition 9 (continuation). ⟦⟧ is floating and unrafted if for each G in the domain,

$$\begin{equation*} \left[\kern-0.15em\left[ G \right]\kern-0.15em\right]\, = { \times _{i \in N}}{ \cup _{I \in \mathscr{I}}}{\left[\kern-0.15em\left[ {{G_i}} \right]\kern-0.15em\right]^I}. \end{equation*}$$

When ⟦⟧ is floating and unrafted, M ∈ ⟦G⟧ if there are any I₁, . . ., I_n such that, for each i, M_i ∈ ⟦G_i⟧^Ii. Intuitively, this is so if there is any way at all, by choosing thresholds for the different graders, to square the fine-grained estimations (of M) with the awarded grades (according to G).

Grade reports do not tell us very much when import is afloat and unrafted because they do not rule out very much. For instance, to continue with the example of Jack and Jill, we can square the offending M with G by allowing the graders to have their own interpretations I_Easy and I_Measly such that:

$$\begin{equation*} 63\% \in {I_{Easy}}\left( B \right)\ {\rm and} \end{equation*}$$

$$\begin{equation*} 64\% \in {I_{Measly}}\left( D \right). \end{equation*}$$

This is precisely how, in section 1, grades created the false impression that Jack must be better than Jill. Section 5 argues that, when import is floating and unrafted, grading in groups is in a sense radically unreliable. Under the conditions set out in Section 3, it cannot be counted on to track the collective judgement of the graders, no matter how that is reckoned.

The coming discussion will consider grading in groups at these extremes: the case in which import is grounded and rafted, and that in which it is floating and unrafted. There are many other possibilities, some of them quite realistic. We might know which interpretation each grader has, but that these are not the same for everyone. Then, for each i there is some perhaps distinct interpretation I_i such that ⟦G⟧ = ⨉_i∈N ⟦G_i⟧^Ii. Again, perhaps we know that the graders have settled on a grading key, but not which one it is. Then any grader might interpret the grades in any way at all, but we needn't reckon with the further possibility of different graders having different thresholds: ⟦G⟧ = ∪_{I ∈}$\mathscr{I}$⨉_i∈N⟦G_i⟧^I. Many other kinds of import functions can be imagined.

4.3. Soundness

Finally, here is what it is for a grading rule to be sound. I shall write R $^\mathcal{G}$ for F $^\mathcal{G}$(G), and P $^\mathcal{G}$ for the strict part of R $^\mathcal{G}$. Similarly, R $^\mathcal{M}$ is F $^\mathcal{M}$(M), and P $^\mathcal{M}$is the strict part of R $^\mathcal{M}$. Now we have:

Definition 10.F $^\mathcal{G}$ is sound with respect to F $^\mathcal{M}$, given ⟦⟧, if for any G ∈ $\mathcal{D}$ and any x, y ∈ X:

1. (a) if x R $^\mathcal{G}$y then for every M ∈ ⟦G⟧, x R $^\mathcal{M}$y; and

2. (b) if x P $^\mathcal{G}$y then for every M ∈ ⟦G⟧, x P $^\mathcal{M}$y.

Soundness means that your judgement that one thing is (strictly) better than another, based just on the grades, is sure to reflect the graders’ underlying views of the merit of these things – whatever, as far as you can tell from the grades, and given what you know about everybody's thresholds, these views might have been. Remember that when more is known about thresholds, there are fewer possibilities in ⟦G⟧. The requirements of Definition 10 are in this case correspondingly mild. Knowing more about how everybody interprets the grades therefore makes it easier to find a sound grading rule.

Say we are going to choose a grading rule for ranking some research proposals, on the basis of grades awarded by two experts. We have discussed the criteria and their relative importance. We have considered the judgements of previous panels, and have held a calibration session. As a result we know a certain amount about the graders’ thresholds. This knowledge is embodied in an import function ⟦⟧. Since it is for use under the given circumstances that we need a grading rule, we hold ⟦⟧ fixed. Now we can enquire simply whether any given candidate procedure F $^\mathcal{G}$ is sound with respect to any given merit function F $^\mathcal{M}$. Let us say we also have a preferred F $^\mathcal{M}$. We are going to average individual estimates, say, as in Example 4. With both ⟦⟧ and F $^\mathcal{M}$ held fixed, we can ask simply whether F $^\mathcal{G}$ is sound.

Why should we care whether it is or not? Soundness guarantees that there are no false impressions like those created by Mr Easy and Ms Measly, in section 1. No matter which grades the experts end up reporting, the resultant ranking of the proposals is sure to agree with what they really think of the proposals – collectively, and as reckoned by our preferred method. Clause (b) of Definition 10 concerns the case in which proposal x is ranked strictly higher than proposal y on the basis of their grades, while (a) covers the case in which the two proposals are judged to be on a par. Soundness is desirable because decisions made using sound procedures can be rationalized. If someone wants to know why their proposal was not funded there will be an answer. We can say that these (G) are the grades awarded by the experts; that these (⟦G⟧) are the only estimations the experts could possibly have of the merit of the proposals, given what we know from the training sessions about their understanding of the grades; that the comparison (R $^\mathcal{M}$) based on any of these possibilities (M), will confirm the resultant ranking (R $^\mathcal{G}$) we reckoned by the grades (G) – and that, in this very strong field of competitors, the proposal in question sadly could not be counted among the very best.

To rationalize the decision is not to justify it, because different parts of the rationalization are themselves subject to criticism. The disappointed author can still argue that our assumptions about grading standards, embodied in ⟦⟧, are incorrect (‘the training of the panel members was inadequate!’); or that agreement with R $^\mathcal{M}$ is beside the point because we haven't used an acceptable method F $^\mathcal{M}$ for reckoning collective comparisons (‘one of your so-called ‘experts’ knows almost nothing about the specific area of this proposal, and his views should carry less weight!’). Still, the rationalization is a beginning. It provides the main structure for a justification and indicates points at which it could be shored up or brought down.

Here is an example of a sound rule:

Example 8 (continuation). More specifically, let $\mathcal{D}$ be as in Example 2, and let I be as in Example 6. Consider again F $^\mathcal{G}$ from Example 3. Let the merit function F $^\mathcal{M}$ be defined as follows. For any suitable M and any x, y ∈ X, we put

$$\begin{equation*} x{R^{\mathcal {M}}}y\,{\rm iff}\,{I^{ - 1}}\left( {\left[ {{M_1}(x) + {M_2}(x)} \right]/2} \right){\rm{ }} \ge {I^{ - 1}}\left( {\left[ {{M_1}(y) + {M_2}(y)} \right]/2} \right) \end{equation*}$$

This notion of the inverse I ⁻¹ of I is natural but not standard: for example, since I(C) = [0.4, 0.6) we have I ⁻¹(0.5) = C. Intuitively, x R $^\mathcal{M}$y means that the graders would collectively award x as high a grade as they would y, perhaps a higher one, were they to proceed in the following way: First, the individual graders estimate the merit of every item; second, for each one they calculate a collective estimate, by averaging their individual estimations of it; finally, using the common grading key – it is implicit in I – for each item they look up the grade that it deserves, given their collective estimation of it.

Now we will see that F $^\mathcal{G}$ is sound with respect to F $^\mathcal{M}$ assuming ⟦⟧ (the import function of Example 8). This follows directly from:

Fact.For any G ∈ $\mathcal{D}$, x, y ∈ X, and M ∈ ⟦G⟧: x R $^\mathcal{G}$y if and only if x R $^\mathcal{M}$y.

To verify this fact, note first that for any g₁, g₂ ∈ {A, C, F}, and for any m₁ ∈ I(g₁) and m₂ ∈ I(g₂)

$$\begin{equation*} {I^{ - 1}}\left( {{m_1} + {m_2}/2} \right){\rm{ }} = mid\left\{ {{g_1},{g_2}} \right\}. \end{equation*}$$

To see this, check all the possibilities; really, there are only three. Now consider any G ∈ $\mathcal{D}$ and any M ∈ ⟦G⟧. By choice of the import function ⟦⟧, for any x, y ∈ X and i ∈ N we have M_i(x) ∈ I(G_i(x)) and M_i(y) ∈ I(G_i(y)), so in particular:

$$\begin{equation*} {I^{ - 1}}\left( {\left[ {{M_1}(x) + {M_2}(x)} \right]/2} \right){\rm{ }} = mid\left\{ {{G_1}(x),{G_2}(x)} \right\},\ {\rm and} \end{equation*}$$

$$\begin{equation*} {I^{ - 1}}\left( {\left[ {{M_1}\left( y \right){\rm{ }} + {\rm{ }}{M_2}\left( y \right)} \right]/2} \right){\rm{ }} = {\rm{ }}mid\left\{ {{G_1}\left( y \right),{\rm{ }}{G_2}\left( y \right)} \right\}. \end{equation*}$$

Therefore:

• 

This completes the demonstration of the fact and with it the soundness of F $^\mathcal{G}$ with respect to F $^\mathcal{M}$, assuming ⟦⟧.

In the discussion surrounding Theorem 14, in the next section, it will be relevant that F $^\mathcal{M}$ of Example 8 (continuation) satisfies Condition I. That is, for any x, y ∈ X, and any M, M* in the domain of F $^\mathcal{M}$, if M|{x, y} = M*|{x, y}, then R $^\mathcal{M}$|{x, y} = R $^\mathcal{M}$*|{x, y}. That it does is obvious because, for any merit profile M, whether x R $^\mathcal{M}$y just depends on M₁(x), M₁(y), M₂(x) and M₂(y), the degrees of merit that the different people attribute to x and y.

Example 8 (continuation) illustrates soundness in a case where a lot is known about grading thresholds. The next section shows how the possibilities for sound grading can turn on how much is known.

5. WHEN THERE CAN ONLY BE FALSE IMPRESSIONS

Uncertainty about thresholds seriously undermines grading in groups. It does this by destroying some of the information in grades. In extreme cases, we will see, there is only comparative information left. People then might as well forget the finer distinctions that can be made by awarding grades and just rank things from better to worse; to you, the aggregator, it will be the same. And then, as we will see, under certain conditions, no grading rule is sound.Footnote ¹⁶

Recall that Ğ_i is the ordering of X implicit in an individual appraisal G_i. Ğ_i|S is the ordering of S ⊆ X. The following lemma plays a crucial part in the coming discussion of the destruction of information:

Lemma 11.Let S ⊆ X be finite, let G_i and H_i be individual appraisals such that Ğ_i|S = Ȟ_i|S, let J be an interpretation of the grading language, and let N_i ∈ ⟦H_i⟧^J. There are I and M_i ∈ ⟦G_i⟧^I such that M_i|S = N_i|S.

There is a proof of Lemma 11 in the Appendix.Footnote ¹⁷ It relies on two earlier assumptions about the merit structure ($\mathcal{M}$, >): that > is dense, and that there is in $\mathcal{M}$ no greatest degree of merit, and no least degree.

In intuitive terms, suppose an underlying estimation of the merit of things is compatible with the grades that someone has awarded. Then, the lemma tells us, by assuming (in general) different thresholds, this same underlying estimation can be made compatible with any other grades that this grader might have awarded instead, if only everything is in the same order. More slowly: say for simplicity's sake that S is the whole set of items, X. Then the restrictions ‘|S’ can be ignored, and M_i|S = N_i|S means that M_i = N_i. Now, suppose the same ranking is implicit in appraisals G_i and H_i. That is, suppose Ğ_i = Ȟ_i. And suppose furthermore that some estimation N_i can be squared with either one of them, let it be H_i, by choosing thresholds for the different grades (there is an interpretation J of all the grades such that N_i ∈ ⟦H_i⟧^J). Then this same N_i can also be squared with the other one, G_i, by choosing suitable grading thresholds, in general different ones (N_i = M_i ∈ ⟦G_i⟧^I, for some I).

Again putting S = X, Lemma 11 has the direct consequence:

Corollary 12.Suppose X is finite and Ğ_i = Ȟ_i. Then ∪_{I ∈}$_{\mathscr{I}}$⟦G_i⟧^I = ∪_{I ∈}$_{\mathscr{I}}$⟦H_i⟧^I.

Intuitively, if i’s thresholds could be anything then any two grade reports from him can be squared with the same underlying fine grained assessments, if these reports determine the same ranking.

Now we will see the havoc that uncertainty about thresholds can wreak on collective grading. From Definition 9 and Corollary 12 we have:

Corollary 13.Suppose X is finite and ⟦⟧ is floating and unrafted. If Ğ = Ȟ, then ⟦G⟧ = ⟦H⟧.

This says that if nothing special is known about the graders’ thresholds, then their grade reports carry no more than ordinal information about merit. We are in effect back in Arrow's framework. The graders can be as fastidious as they want about what gets which grade; it will come to nothing because all their report will tell us, in the end, is how they rank everything. All information over and above the implicit orderings has been dissipated by our uncertainty about what grades mean, in the mouths and common language of the graders. Now, this can rob collective grading of all content:

Theorem 14.Let X be finite, and let S ⊆ X include at least 3 items. Let ⟦⟧ be floating and unrafted. Let grading rule F $^\mathcal{G}$ satisfy Ordinally Unrestricted Domain with respect to S, Completeness, Weak Pareto, and Non-Dictatorship with respect to S; and let a merit function F $^\mathcal{M}$satisfy Independence of Irrelevant Alternatives. Then F $^\mathcal{G}$is not sound with respect to F $^\mathcal{M}$, assuming ⟦⟧.

Theorem 14 is a consequence of Arrow's ‘impossibility’ theorem (Arrow Reference Arrow1951). The proof in the Appendix is a reductio. Given an import function, grading rule and merit function satisfying the conditions of Theorem 14, it is possible to construct an Arrow-style domain and a social welfare function. In outline, the domain is the set of all lists of weak orderings obtained by ‘flattening’ grade profiles from the domain of F $^\mathcal{G}$ (flattening is discussed in the beginning of section 3). The social welfare function is then ‘read off’ from the import of these grade profiles, using the merit function. This is where the soundness assumption comes in. It makes sure that all merit assignments compatible with any given grade report ‘speak with the same voice’ about which items are ‘socially better’ than which. That import is floating and rafted ensures, through Corollary 13, that a function is obtained in this way. It ensures that whenever flattening makes some grade profiles indistinguishable, their import – and hence, by this construction, also the derived social ordering – are the same. The social welfare function in question is shown to satisfy all assumptions of Arrow's ‘impossibility’ theorem, on the basis of the corresponding conditions in Theorem 14. Meanwhile, of course, Arrow's theorem tells us precisely that no such function exists.

Theorem 14 concerns the case of extreme uncertainty: the graders might have any thresholds at all, independently of one another. Under its conditions, grading in groups is radically unsound. Earlier, Examples 3 and 8 illustrated the possibilities at the other extreme, where all the graders’ thresholds are known to be the same. There was an example of a sound rule satisfying all assumptions of Theorem 14 – except of course the one about the import of grades. Evidently, then, the possibilities for sound grading really do turn on what is known about grading thresholds. They turn somewhere in between these extremes, where a certain amount known but not enough to pin things down completely. There remains a lot to be learned about the possibilities for collective grading in these cases.Footnote ¹⁸ Apart from any theoretical interest there might be practical consequences. Knowing more about the consequences of mismatched grading thresholds we might be able to develop training that will improve the judgement of juries, committees and expert panels.

Notice a family resemblance between Theorem 14 and Sen's (Reference Sen1970: theorem 8*2) demonstration that measuring individual utilities on a cardinal scale is not by itself enough to escape Arrow's theorem. In each case there is something that destroys information beyond mere comparisons from better to worse: with Sen, it is interpersonal incomparability of utilities; here, it is uncertainty about how people use words. Beneath this similarity there are differences of substance. I don't know that we can make utilities interpersonally comparable, but it is quite common to constrain how people use words. Committees and expert panels provide descriptions of grades in natural language, paradigm examples, grading protocols and checklists, calibration exercises and so on.

6. GRADING AND VOTING

Say voters rank the candidates in some election from better to worse on a ballot. Their rankings will be compiled by some method into a collective ranking, and whoever ends up on top has won. Arrow's theorem describes a theoretical limitation: no method for arriving at collective rankings can satisfy a short list of desiderata (Arrow Reference Arrow1951). Some have taken this to mean that there cannot be any such thing as democracy, conceived as government by a common will of the people, as revealed by voting. Supposedly the very idea of that, captured by the desiderata, is incoherent. Others draw the less dramatic conclusion that more information is needed in elections than just voters’ rankings of the candidates. Now, by grading you can convey more than just a ranking. Also, the extra information seems to be enough to solve some of the problems with traditional voting methods, such as Arrow's impossibility. That emerged here in section 3. Balinski and Laraki (Reference Balinski and Laraki2010) argue, for this reason among others, that we should stop using these traditional methods in elections. Voters should grade candidates instead of ranking them. This study raises a question about their proposal.

Grading in groups is familiar from expert panels, committees and so on, where trained people work in close contact with one another. Balinski and Laraki provide other examples and in each case, except for voting, it is the same. The graders are judges of sports competitions, or dancing, or music, or wine. Now, with a lot of shared training and culture we may expect there to be a good common understanding of how grades are to be interpreted. Professional meetings will provide opportunities for further coordination, if need be. In section 4.3 we found reason to expect that, under favourable circumstances such as these, grading in groups can be a meaningful exercise.

Voting in political elections typically involves much larger and above all much looser collections of people. The example of Jack and Jill showed how easily we can gain false impressions when graders might have different thresholds. More extremely, the unsoundness theorem of section 5 tells us that when nothing is known about the graders’ thresholds, results cannot be counted on to reflect any collective sense at all of better and worse. In large, culturally and linguistically diverse democracies, such as those of India or the United States, people surely don't have the same understanding of expressions such as Excellent, Good and Fair, and there can be few opportunities for bringing about further coordination. The question is simply this: How under such conditions can grading be a good way to pick winners?

The problem of unsoundness is in one way more pernicious than the familiar problems with traditional voting methods. That pairwise majority voting sometimes delivers up cycles of collective preference can be found out by studying it. So can the susceptibility of Borda counting to strategic manipulation. These problems are visible, so to speak, ‘from within’. They emerge from the details of the voting methods themselves. A grading rule, though, is sound with respect to some assumptions about what people mean by grades and unsound with respect to others. We'd like to know that the rule we use is sound with respect to reasonable assumptions of this sort, ideally with respect to the truth of the matter, and whether it is obviously depends on more than the internal workings of the procedure itself. It depends on which assumptions are reasonable and which are true, and that might be quite hard to find out. There are substantive disagreements about what is good and what fair. There are conceptual disagreements about what it is to be good or fair. How to tell the one kind of disagreement from the other?

Balinski and Laraki discuss at some length the idea that there will have to be a common grading language, if grading is to be used in political elections. Also, they seem to think that the people will need a common understanding of the applicable grades. They list media coverage, campaign materials and debates among the factors that, they say, will give the public a ‘good sense of what the language of grades means’, the first time it is used (Balinski and Laraki Reference Balinski and Laraki2010: 174, 251). But it is unlikely that media coverage and the rest will have this effect, if what is needed is that everyone will converge on the same grading thresholds. Public communications rely on natural languages, with all their vagueness and contextual shiftiness.

Natural language definitions do contribute to a common understanding on juries and professional panels but there they are supplemented by other methods. There is training with paradigm examples; there is discussion of difficult cases; there are calibration meetings, checklists and more. In the grade approach to evaluating medical evidence, for example, there is a protocol for assigning grades (see Balshem et al. Reference Balshem, Helfand, Schünemann, Oxman, Kunz, Brozek, Vist, Falck-Ytter, Meerpohl, Norris and Guyatt2011: tables 2 and 3). Presumably all these efforts contribute to the capacity of groups to estimate merit, but it is unthinkable that there could be any similar training of voters before elections. Turnout is often considered too low as things stand, even with no rigorous educational requirements attached! Anyway, the whole idea smacks of interference. Who is to tell me that instead of ticking the Excellent box next to my favourite candidate, I should mind my words and tick the Good box instead? Very much the same problem of linguistic differences arises in public surveys, and the possibilities for training respondents are similarly limited there. Political scientists have developed methods correcting people's responses for linguistic variation after these have been recorded, for example using anchoring vignettes (King et al. Reference King, Murray, Salomon and Tandon2004). But there can be no correcting ballots after elections. That would be tampering. Now I've in fact ticked the Excellent box, who is to count this as only a Good?

This is not to say that grading cannot be a better way to run elections. The point is rather that if it turns out to be a reliable way to identify the candidates that are best, in some independently understood sense, then this fact is interesting and deserves further study. Juries, committees and expert panels are very small compared with electorates. Perhaps when there are more graders it is enough for their grading thresholds to be distributed around a central norm, so that differences cancel out. Diversity in their thresholds might even turn out to be an advantage if for every hard grader there is another who is correspondingly easy. This would be an interesting finding in itself. We might in this case expect that, other things being equal, grading will track merit less reliably when practiced in smaller, more homogeneous groups. Contrary to what has seemed intuitive, it might sometimes not be good for panel members to coordinate their understanding of grades.