The “most probable distribution” method

The MPD is a combinatorial method suggested by Boltzmann in 1877 to derive the energy distribution of particles in a gas. Particles are allocated into energy levels $\{ \varepsilon _k \} $ (like marbles in boxes) with occupation numbers $\{ n_k \} $ under the constraints given by their total number N and energy E:

Since particles can be distributed in several ways, there are different possible combinations (and sets of frequencies $f_k = {{n_k } \over N}$ ). Some of these combinations are also equivalent, since identical particles can be swapped. To understand which combination is more likely to be observed in Nature, multiplicity is defined as the number of ways in which a particular combination is realized:

The logarithm of the multiplicity is known as Boltzmann’s Entropy. The gist of the MPD method is that the final probability distribution is associated to the combination that is realized in the maximum number of ways. To find such a probability the multiplicity (2), under the constraints (1), is maximed using the method of the Lagrangian multipliers:

were ${\rm{\lambda }}_1 ,{\rm{\lambda }}_2 \in R$ . The difficulty in maximizing (3) rely on the integer nature of $\left\{ {{\rm{n}}_{\rm{k}} } \right\}$ and in the factorials contained in the multiplicity. An approximate derivation, based on the fact that the limit $N \to \infty $ is taken, requires Stirling’s approximation for factorials and a continuous approximation for $\left\{ {{\rm{n}}_{\rm{k}} } \right\}$ . A better result, still in the continuity approximation, is obtained by replacing factorials with the Gamma function (Landsberg, Reference Landsberg1954). An elegant derivation was given by Clinton and Massa (Reference Clinton and Massa1972) avoiding both continuity and Stirling’s approximations. Since the present work relies on this derivation, it is useful to briefly sketch it for the present case: Clinton and Massa noticed that ${\rm{\Lambda }}(\{ {\rm{n}}_{\rm{k}} \} )$ is in a maximum if, moving a particle both in or out of an energy level, the following system of inequalities is satisfied:

Substitution of (3) in the system (4) leads to:

so that any occupation number (once divided by N) is bounded by:

Since these are actually the occupation frequencies, for the squeeze theorem the probability can be defined taking the limit $N \to \infty $ where the terms 1/N vanishes:

The condition $\sum\nolimits_k {{\rm{p}}_k } = 1$ is used to introduce a normalization term:

and finally the probability is:

It is immediate the similarity of (8) with the Polytomous RM if the occupation numbers $n_k $ are a measure of how many times a category response is chosen in N trials by a testee. Although with some differences in assumptions and methods, this line of reasoning is close to Ebneth’s (Reference Ebneth1993). His derivation, however, requires the concept of statistical ensemble, a collection of N copies of the system that, independently and randomly, assume all the possible states allowed to the system (Huang, Reference Huang1987). For instance, N copies of the entire gas. This concept is fundamental in statistical physics which is based on averages over ensembles, but it is not required in the MPD method (Landsberg, Reference Landsberg1954).

Boltzmann’s “molecular” perspective (the ensemble is the set of N particles) is followed in the present work, so that all the possible response patterns of the joint population of subjects and items are accounted (see section 3). Probability (8) is then derived as the most likely response pattern that can be observed in the population. The concept of MPD is well justified in a MaxEnt framework.

The principle of Maximum Entropy

Information Entropy is a measure of uncertainty (Shannon, Reference Shannon1948). The higher the entropy of a system the more unpredictable is its state. The concept was inspired by Gibbs statistical Entropy:

and takes maximum value when the distribution is uniform. Connections between statistical entropy and information theory were higlighted by Jaynes (Reference Jaynes1957). In particular, he suggested the importance of a MaxEnt principle, briefly stated as “when we make inferences based on incomplete information we should drawn them from that probability distribution that has the maximum entropy permitted by the information we do have” (Jaynes, Reference Jaynes1982). Such a method is a generalization of the usual methods of statistical inference that allows the choice of different priors. For instance, the MaxEnt solution for a set of probabilities $\sum\nolimits_k {p_k = 1} $ over some events $x_k $ is the uniform distribution (Laplace’s principle of indifference). Adding some functions $f_j \left( {x_k } \right)$ , whose expected values are constrained $E\left[ {f_j (x_k )} \right] = a_j \in R$ , the result is the is the exponential family:

where ${\rm{\lambda }}_j \in R$ . On the one side, the MaxEnt distribution works as the inverse of the Darmois-Koopman-Pitman theorem and creates a model for which the data are sufficient statistics (Jaynes, Reference Jaynes1982). On the other side, it relates Gibbs and Boltzmann Entropies:

The probability distribution that maximizes entropy is numerically identical to the frequency that possesses the greatest multiplicity (Jaynes, Reference Jaynes1982). Boltzmann’s Entropy is the limit of Gibbs’ Entropy if probabilities are equal. Working on the concept of ensemble Gibbs formulation is however more general and allows to describe interacting particles (Jaynes, Reference Jaynes1965). Hence, the Boltzmann entropy seems to be a suitable description of RM since it requires independence between subjects and items.

Measurement scale and equivalence classes

Given a set of subjects ${\rm{\nu }} \in \{ 1, \ldots ,s\} $ , a set of items $i \in \{ 1, \ldots ,m\} $ , and a random variable, whose realizations are response categories $X_{\nu i} = x_{\nu i} \in \{ 0, \ldots ,c\} $ , the response matrix of a test is $\{ x_{\nu i} \} $ . In the dichotomous case, $c = 1$ . A generalization of the response matrix to the population can be either a finite or an infinite matrix $\{ x_{\nu i} \} $ with ${\rm{\nu }} \in S$ and $i \in I$ , where S and I are the populations of all the subjects and items. The union $P = S \cup I$ , endowed with a weak order relation $ \mathbin{\lower.3ex\hbox{$\buildrel\prec\over {\smash{\scriptstyle\sim}\vphantom{_x}}$}} $ , allows comparisons between subjects and items. A common scale for latent traits and item parameters is a triple $\left\langle {P,M,\phi } \right\rangle $ where $\phi :\left\langle {P, \mathbin{\lower.3ex\hbox{$\buildrel\prec\over {\smash{\scriptstyle\sim}\vphantom{_x}}$}} } \right\rangle \to \left\langle {M, \le } \right\rangle $ , with $M \subseteq R$ , is an homomorphism that preserves the weak order (see for instance Krantz, Luce, Suppes, & Tversky, Reference Krantz, Luce, Suppes and Tversky1971; Luce, Krantz, Suppes, & Tversky, Reference Luce, Krantz, Suppes and Tversky1990; Suppes & Zinnes, Reference Suppes, Zinnes, Luce, Bush and Galanter1963). Equivalence classes of all the subjects and of all the items possessing the same position on the measurement scale can be defined as:

where ${\rm{\alpha }} \in A$ and ${\rm{\delta }} \in D$ are the values of the latent trait and of the item characteristic. Let j and k be the indexes spanning these sets, assuming they have at least countable cardinalities.

In order to build a Polytomous RM, thresholds are also needed. They are generally conceived as locations on the latent trait set that indicate a subject has exceeded a particular category response. Let $r \in \left\{ {0, \ldots ,{\rm{c}}} \right\}$ be the index spanning the category responses, hence ${\rm{\tau }}_{kr} $ is the threshold value needed for scoring the category $x_{jk} = r$ in an item of characteristic ${\rm{\delta }}_k $ . Thresholds appear then to be both levels of latent trait and item characteristic, ${\rm{\tau }}_{kr} \in {\rm{A}} \cap {\rm{D}}$ . In what follows, the situation is considered in which ${\rm{A}} = {\rm{D}}$ , so that there is always a match between levels of latent trait and of item characteristic.

Probability

An important debate in IRT concerns the source of randomness. In the stochastic subject view, probability explains variations due to the person or to the test situation, in the random sampling view, probability is the proportion of subjects with the same latent trait giving positive answers (Moleenar, Reference Molenaar, Fischer and Molenaar1995). Another important debate concerns whether latent traits and item characteristics are on an ordinal level or on a metric continuum (Michell, Reference Michell1990). In the former case, different subjects and items might be associated to the same non-metric scale value, in the latter, the probability of subjects and items to possess the same values would be zero. Interestingly, the MPD method accommodates all the previous perspectives. Equivalence classes (12) are defined independently on whether they describe an ordinal ranking or a coarse-grained description of a continuum (i.e, classes due to limited precision in measurement). Indeed, the MPD method moves from occupation numbers, so it is not important whether they result from resampling a subject or from different testees. In the most general perspective the response might be rewritten as $x_{jkr}^{\nu it} $ where the supra-indexes refer to subjects, items, and repetitions, while sub-indexes refer to latent traits, item characteristics, and category responses. Since debating about the existence of subjects with the same latent trait and item with the same characteristic is pointless in the MPD framework, $\nu ,i,t$ will be dropped.

Let $X_{jk} $ be the response variable corresponding to a subject with latent trait ${\rm{\alpha }}_j $ , answering to an item with characteristic ${\rm{\delta }}_k $ , and let $x_{jk} = r \in \left\{ {0, \ldots ,c} \right\}$ be its realization. The matrix $\{ x_{jk} \} $ can be ordered by increasing levels of latent trait and item characteristic and then partitioned into different blocks characterized by a couple ( ${\rm{\alpha }}_j ,{\rm{\delta }}_k $ ). The number of responses within each block is given by a set of numbers $\{ n_{jkr} \} $ . If $N_{jk} = \sum\nolimits_{r = 0}^c {n_{jkr} } $ is the total number of cells in the specific jk-th block, then the ratio $n_{jkr} /N_{jk} $ gives the proportion of responses for the r-th category (the number of subjects giving a response r to a certain class of items or the proportion of responses of a single subject to a single item depending on the view). The probability of drawing from the population a testee with latent trait $\alpha _j $ , answering r (whose threshold is $\tau _{kr} $ ) to an item with a characteristic value of $\delta _k $ , is then:

where the limit accounts for an infinite population. The law of total probability becomes:

Permutations

Multiplicity can be derived consideing that the total number of ways in which $n_{jkr} $ responses can fill the $N_{jk} $ cells of the block is given by the binomial coefficient:

Once the response category $r = 0$ has been filled, there is room left for $N_{jk} - n_{jk0} $ responses in the other categories, and so on. Hence:

that, after some algebra, yields:

so that all the possible ways in which all the blocks can be filled is given by:

that is exactly multiplicity (2) but generalized to the joint population of subjects and items.

Constraints

The first constraint that must be taken into account is given by the total number of cells in each block, that must sum up to the total number N of cells in the response matrix. It follows that:

The second constraint depends on the fact that the number of possible outcomes defined by the multiplicity (17) depends on ${\rm{\alpha }},{\rm{\delta }},$ and ${\rm{\tau }}$ . The number of responses $n_{jkr} $ is conditional to latent traits, item characteristics and thresholds. The constraint can be modeled as an implicit function of ${\rm{\alpha }}_j ,{\rm{\delta }}_k ,{\rm{\tau }}_{kr} $ and $n_{jkr} .$ Namely, $H\left( {\left\{ {n_{jkr} } \right\},\left\{ {{\rm{\alpha }}_j } \right\},\left\{ {{\rm{\delta }}_k } \right\},\left\{ {{\rm{\tau }}_{kr} } \right\}} \right) = {\rm{\mu }}$ , with ${\rm{\mu }} \in R$ . However, to avoid any interaction between different response categories in different blocks, additive independence is assumed:

Notice that, in each block, constraints likely depend on all the thresholds $\left\{ {{\rm{\tau }}_{kt} } \right\}_{t \le r} $ that precede the one associated to the r-th category. Any $h_{jkr} := h\left( {n_{jkr} ,{\rm{\alpha }}_j ,{\rm{\delta }}_k ,\left\{ {{\rm{\tau }}_{kt} } \right\}_{t \le r} } \right)$ is a generic constraint for the r-th category in the jk-th cluster, and is assumed to be a monotonic function of its arguments.

Derivation of the most probable distribution

The MPD can be derived by maximizing the Boltzmann Entropy given by (17) under the effect of constraints (18) and (19). As in section (2) this extremality problem under external constraints is reduced with Lagrangian multipliers method to the unconstrained maximization of the function:

where ${\rm{\lambda }}_1 ,{\rm{\lambda }}_2 \in R$ and the sets of $\left\{ {{\rm{\alpha }}_j } \right\},\left\{ {{\rm{\delta }}_k } \right\},\left\{ {{\rm{\tau }}_{kr} } \right\}$ enter in the equation as parameters. As noticed, the variables $n_{jkr} $ are positive integers so the previous equation is not differentiable. Since the derivation follows exactly the step given in section (2), apart from some minor changes, full proof is given in Appendix A. There is however a point worth to be mentioned: not all the shape of the constraint $h_{jkr} $ do define a probability. A unique solution can be achieved when the constraints are linear functions of $n_{jkr} $ , that is $h_{jkr} = n_{jkr} f_{jkr} $ where $f_{jkr} : = f({\rm{\alpha }}_j ,{\rm{\delta }}_k ,\left\{ {{\rm{\tau }}_{kt} } \right\}_{t \le r} )$ is a generic functions of latent traits, item characteristics and thresholds (addition of constants or functions unrelated to $n_{jkr} $ does not affect the multiplicity, see Appendix A). Interestingly, linear constraints are related to averages of latent traits, thresholds and item characteristics over all blocks and category responses. They are indeed the expected values of the sufficient statistics for the exponential family as in equation (10).

The probability resulting from maximizing (20) is then:

since ${\rm{\lambda }}_1 $ becomes a normalization term ${\rm{\lambda }}_2 $ and is a scale factor. As it can also be easily noticed, PCM and RSM can be obtained from equation (21) setting the appropriate constraint (19) to:

Similar constraints can be given for any form of the Polytmous RM (Rasch, Reference Rasch1961).