2.1. Instantaneous Probabilities

There are a great many things that might be meant by “the probability” of a macrostate of a system. One distinction that it will be helpful to make is between instantaneous and noninstantaneous probabilities of macrostates.

To illustrate this distinction, it will be useful to work with a specific nonequilibrium macrostate. Suppose that a (two-dimensional) box with length l and height h initially contains a barrier dividing it into two smaller identically sized chambers of length and height h and that the rightmost of these chambers is empty, while the leftmost is filled with a gas in equilibrium (see the left half of fig. 1).The barrier is then removed. Let macrostate M be the macrostate of the system at the instant at which the barrier is removed (see the right half of fig. 1). Having defined the macrostate M, we now ask for the “probability” of M occurring in a given box of length l and height h. One sort of thing that might be meant by this question is something like the following:

Q1. What is the probability that at a specific time t the system was or will be in macrostate M?

Figure 1

This is a question about the probability of something happening at a prespecified instant t, and so we call the probability that appears in this question an instantaneous probability.

The way in which we understand and answer the question Q1 may well depend on when it is asked, so we draw some further distinctions. Suppose at some time prior to t we ask what the probability is that in a particular box (with no barrier), a gas with the appropriate energy or temperature will be in the macrostate M at time t. This is a question about a prospective instantaneous probability.

This prospective instantaneous probability will depend on the time at which the question was asked, as well as the macrostate of the system at that time. For instance, if the gas is in equilibrium when the question is asked (filling the entire box), then the probability of the system finding itself in macrostate M at the later time t will be almost 0. On the other hand, if the gas starts off as in the left half of figure 1, and the barrier is removed at time for some infinitesimal ε, then the probabilty that the system will be in macrostate M at time t may turn out to be nonnegligible. Either way, the probability that the system will be in macrostate M at time t will depend on things other than M—in particular, it will depend on the history of the system. The thermodynamic entropy of M, however, depends only on M, and as such, it is difficult to see how the thermodynamic entropy can be viewed as a function of some sort of prospective, instantaneous probability of a macrostate.

To get around this sort of problem, one might take question Q1 to have been asked at some moment in the distant past in which the system was in equilibrium. This move resolves some problems but is only available when the system was actually in equilibrium at some moment in the past. If we want to forge a very general connection between entropy and probability, restricting ourselves to systems with a particular sort of history is not an attractive move.

Approaching things from a different angle, we might think about instantaneous probabilities retrospectively. So sometime after t, we might ask what the probability is that at time t, the system was in macrostate M. From one point of view, the system either was or was not in macrostate M at time t, and so the probability in question is either 0 or 1. Obviously, thinking about retrospective instantaneous probabilities in this way is not useful if our goal is to draw a general connection between probability and entropy.

A more useful way of thinking of retrospective instantaneous probabilities is as a type of conditional probability, where the condition involves some subset of our present knowledge about the system and its environment. In particular, we can ask what the probability is that—given what we know about the system now (or some subset of this knowledge)—the system was in macrostate M at time t. This probability may well lie properly between 0 and 1. This probability will depend quite sensitively, however, on exactly what present knowledge about the system we regard as admissible. Because the thermodynamic entropy is a function only of the macrostate of the system and not of the details of our broader knowledge, we will at most be able to conditionalize on very generic knowledge, such as the knowledge that the system is in some particular macrostate at some later time .

Even with this in mind, problems similar to those with prospective instantaneous probabilities quickly arise. The probability that the system was in macrostate at time given that it is in macrostate at some later time will depend on and . If we demand that be an equilibrium state in the far future, we then restrict our attention to systems that have already relaxed into equilibrium. All this is problematic in just the way it was problematic for prospective instantaneous probabilities, and so it does not seem promising to think of thermodynamic entropy as a function of some sort of retrospective instantaneous probability.Footnote ³

Although a more thorough discussion is necessary to completely refute the idea that the entropy is a function of some sort of instantaneous probability, we take the difficulties discussed here to be significant enough to warrant the investigation of other options.

2.2. Noninstantaneous Probabilities

Fortunately, there are alternatives to thinking of the probabilities of statistical mechanics as instantaneous. For instance, we can ask

where no reference is made to any time t. This is different from Q1, in much the way that “What is the probability that it will snow on a winter's day in Chicago?” is different from “What is the probability that it will snow tomorrow?” where this later question is asked on a particular day in a Chicago winter.

Much as is it natural to interpret “What is the probability that it will snow on a winter's day in Chicago?” as a question about a time-average, so too is it natural to interpret Q2 as a question about time-averaged occupation times of regions of phase space. For instance, consider our earlier example of a gas in a box in which a barrier has just been removed. Over time, this gas will wander through its phase space. Given any region O of this phase space, the gas will spend a certain proportion of its time in O. We can think of this proportion of time as the probability that the gas lies inside the region O. More precisely,

where is the amount of time the system spends in the region in the first seconds after some prespecified starting time. (Because involves only a limit as , the choice of starting time is immaterial.) Identifying the macrostate with the relevant subset of phase space, we can think of thus defined as the probability of the macrostate . Because this probability is not specific to any particular time, we call it a noninstantaneous probability.

There are, however, many different types of noninstantaneous probability. One could define by looking at the path through phase space actually traced out by the gas over time. The problem with this is that the box may well be tampered with or eventually destroyed—and so thus defined, will depend on the future history of the box and its environment. Because the thermodynamic entropy depends only on M and not on the future history of the box or the environment, this way of thinking about the probability of a macrostate is not useful to us.

It is better to think of the definition of in more idealized terms. Here, there are two natural possibilities (among many). The first possibility is to suppose that after some prespecified starting time, the box containing the gas is isolated from the rest of the universe. From this time onward, the location and dimensions of the box are fixed, and the particles of the gas interact only with each other and the walls of the box. To keep things simple, let us also suppose that the system is ergodic. Under all these assumptions, the noninstantaneous probability of thus defined is generally taken to be the (normalized) measure of the set M on the relevant fixed-energy hypersurface; that is,

where E is the energy of the macrostate , is a normalizing factor, and is a differential element of the hypersurface of fixed energy . In the case of an ergodic system, we take equation (1) to be a plausible candidate for the probability of the macrostate of our system.Footnote ⁴

In the second, more idealized way of thinking about noninstantaneous probabilities, we suppose that the box containing the gas is isolated and maintained at a temperature T after the prespecified starting time. One way this could be done is by placing the box in contact with a heat bath at temperature T. There are different mathematical ways of modeling the interaction between a system and a heat bath; probably the most natural involves adding a stochastic noise term to Hamilton's equations of motion to form a Langevin type equation. In this case, the quantity becomes a random variable, and so in the definition of we need to replace with its expectation value . Under assumptions much weaker than ergodicity, it can be shown (in special cases at least) that is the canonical measure of the set ; that is,

where is an arbitrary differential element of phase space, H is the Hamiltonian, and the partition function. We take equation (2) to also be a plausible candidate for the probability of the macrostate M of our system.

Our conclusion is that insofar as we want to have a chance of connecting the entropy of a macrostate of a system with the “probability” of the macrostate, we must focus on noninstantaneous probabilities of forms (1) or (2). Our main task is to investigate whether the entropy of a macrostate can be connected with either of these expressions.

3.1. Entropy and the Canonical Ensemble

What expression should we use for the entropy of a macrostate of a system? According to classical thermodynamics, when a system moves from a first equilibrium macrostate to a second equilibrium macrostate, the entropy difference between those macrostates is given by

where the integral is taken along a reversible path, dQ is the heat received by the system at any point along that path, and T is the temperature of the system at that point on the path. The second law of thermodynamics says that, thus defined, the entropy of an adiabatically isolated system does not decrease. This law of physics has been especially well confirmed for a very large variety of systems, and because of this, we take this definition of the entropy difference to be the correct, exact definition of the entropy difference between equilibrium states of a system.

It still takes a good deal of mathematical work to get from equation (3) to a concrete mathematical formula for the entropy difference for a given physical system. A more convenient expression for the entropy S of a system was given by Gibbs:

Here ρ is the full N-particle canonical distribution given by

where , is the Hamitonian of the system, and is a normalizing factor (the so-called partition function). It can be proven that the entropy differences calculated using this formula agree with those given by equation (3) (see sec. 3 of Jaynes [Reference Jaynes1965] for a discussion of this point).Footnote ⁵

3.2. Nonequilibrium Entropies

The formula (4) is only supposed to apply to equilibrium states. No real consensus exists on how to generalize the definition of entropy to cover nonequilibrium states. There is, however, one very special case of a nonequilibrium state that we will need to discuss. Suppose a gas is confined in some subregion of a box by a barrier and that the gas thus confined is in equilibrium. Suppose that the barrier is then removed, as in figure 2. Call the macrostate that includes the barrier macrostate 1 and the macrostate without the barrier macrostate 2. The only difference between these two macrostates is the presence of the barrier. Macrostate 1, however, is an equilibrium macrostate, while macrostate 2 is not.

Figure 2

When probabilities of macrostates are being discussed, it is typically the probability of macrostates like macrostate 2 (rather than macrostate 1) that are at issue. So, for instance, it is typically said that because the entropy of macrostate 2 is low, the gas in macrostate 2 will subsequently expand to occupy the whole box, resulting in a state of higher entropy and higher probability. In order for this sort of story to make sense, we need to assign an entropy to macrostate 2. Although macrostate 2 is not an equilibrium state, it is nevertheless very natural to say that macrostate 2 has an entropy and that it is the same as macrostate 1. The reason is that there is something very much like a reversible path connecting macrostate 1 and macrostate 2—namely, one in which the barrier is removed. This path is reversible in the sense that one can go back from macrostate 2 to macrostate 1 by simply reinserting the barrier. Because there is no exchange of heat involved in placing or removing a barrier around a gas that is already spatially localized, it follows that the entropy difference between these two states is 0; that is, their entropies are the same. For a different way of justifying the identification of the entropy of macrostate 1 with the entropy of macrostate 2, note that if the entropy of a macrostate of a gas is a function of the positions and momenta of the particles in the gas, then the presence of a barrier—which is the only difference between macrostates 1 and 2—does not affect the entropy, and so the entropy of these two macrostates should be identified. Whether either of these arguments are truly persuasive is, of course, something about which much more might be said.Footnote ⁶ For now, however, we simply proceed under the assumption that the entropies of these macrostates may be identified.

3.3. Case 1: The Ideal Gas

In this and the next section, we calculate the entropy and probability for various macrostates of a fixed system maintained at a constant temperature. We find that the entropy of a macrostate of our system is not a function of the probability of the macrostate, even for a fixed system with a fixed number of particles and fixed temperature. We show this by constructing macrostates of the given system that have the same probability but quite different entropies.

We begin by considering the case of the ideal gas, mainly to use as a point of contrast in our subsequent discussion. We are interested in the case of a two-dimensional ideal gas of N particles of mass m, contained in a square chamber of side length L. The macrostates in which we are interested are macrostates in which the entire gas lies in the rectangular subregion of the chamber (as shown in fig. 3) and which is otherwise the same as the equilibrium macrostate the system would have were it enclosed in this region by barriers and maintained at a given temperature T. We denote such a macrostate by .

Figure 3

Suppose then that we have a two-dimensional ideal gas consisting of N particles of mass m, contained in a square chamber of side length L, in contact with a heat bath at temperature T. What is the probability of finding the system in the macrostate ? Beginning with expression (2) for the probability obtained in accordance with the canonical distribution, we write

where and

Integral (5) is very difficult to compute. Part of the problem is that spelling out the precise conditions for membership in is quite complicated. Fortunately, it is easy to write down a much simpler expression that is numerically almost the same as (5) and which is much easier to compute. In particular, consider the expression:

Expression (6) gives the probability that all the particles lie inside the region . This is not the same, however, as the probability that the system is in the macrostate . In the macrostate , the particles are spatially distributed in a certain way, the velocities are distributed in a certain way, and so on. It does not follow from the fact that all the particles of the gas lie in that the system must be in macrostate .

It is nevertheless easy to argue that with probability overwhelmingly close to 1, if all the particles of the gas lie in then the gas will in fact be in the macrostate . The claim that the gas is in the macrostate is just a claim about the way in which the values are distributed. (Spatially, the particles should be dispersed in a particular way, the distribution of velocities should be appropriate for a system at temperature , and so on.) As such, it is possible to write down a function of the variables such that if the gas is in macrostate , and otherwise. The conditional probability that the gas is in macrostate given that all the particles lie in is

which (upon canceling a factor of that appears in both the numerator and denominator) is equal to

which equals

which equals the probability that the gas, when confined to the chamber at temperature T, will be in the macrostate . This final probability, however, is well known to be overwhelmingly close to 1 when the number of particles is very large. So with probability overwhelmingly close to 1 for large N, if all the particles of the gas lie in , then the gas will be in the macrostate . It follows that in the case of large N, equation (6) is an excellent approximation to equation (5). In the case of the ideal gas, expression (6) is easy to calculate, and the final answer is

where and .

As discussed in the previous section, we assume that the entropy of the macrostate is the same as the entropy of the macrostate of a gas of N particles confined by barriers to the region and maintained at temperature T. The entropy of such a macrostate may be calculated using equation (4), and the final result is

For fixed N, m, β, and L there is a straightforward functional relationship between the thermodynamic entropy and the probability of macrostates of the form . We write this relationship schematically as follows:

3.4. Case 2: The Ideal Gas under an External Force

We now consider what happens to the relationship between entropy and probability once we leave the realm of the ideal gas with no external forces. To this end, we consider the case of a two-dimensional gas of N noninteracting particles of mass m in the presence of an external force. As before, the gas is contained in a square chamber of side length L. The external force is described by the potential

where α is a constant. The system is essentially a two-dimesional gas in which all the particles are subjected to a constant downward force of magnitude α. The Hamiltonian for the system is given by

where and are the position and momentum of the ith particle, and . (It is assumed that for each i, and both lie in .)

As before, we are interested in the macrostate of the system in which the entire gas lies in the rectangular region and which is otherwise the same as the equilibrium macrostate the system would have were it enclosed in this region by barriers and maintained at a given temperature T. To calculate the probability of such a macrostate, we follow the same chain of reasoning in the previous section and use formula (6). The calculation is easy, and we obtain the following exact expression:

where and .

The entropy of the macrostate is given by expression (4) and is also easy to calculate. The final answer is

We now ask whether expression (10) is a function of expression (9). It is easy to see that it is not. For instance, letting , , , and , the macrostates and have the same probability, but the ratio of their entropies is 2.47 (for any N), which is very significant. It is also possible to construct macrostates of this system with equal entropies and very different probabilites. Our conclusion is that once we leave the realm of the ideal gas, the entropy of a macrostate is typically not a function of the (canonical) probability of the macrostate.

In fact, when dealing with systems in contact with a heat bath, it is worth noting that one does not even really have to leave the realm of the ideal gas to show that the entropy of a macrostate is not a function of its probability. To see why, note that with very low probability it is possible for an ideal gas in contact with a heat bath to fluctuate into a macrostate with higher entropy than that of the equilibrium macrostate. For instance, it is possible for an ideal gas with equilibrium energy E to briefly fluctuate into a macrostate having the same macroproperties as an ideal, isolated gas of energy at equilibrium, for a value of greater than the actual equilibrium energy E. Because the entropy of an ideal gas increases with the energy, we see that with low probability, the system can fluctuate into a high-entropy macrostate. If the entropy of a macrostate is taken to be a measure of its probability, this contradicts the relation established in equation (8), according to which low-probability macrostates have low entropy.

4.1. Entropy and the Microcanonical Ensemble

When dealing with an isolated system at a fixed energy E, the formula (4) for the entropy is not especially useful because it expresses the entropy in terms of the temperature T of the system rather than the total energy E of the system. Two formulas for the entropy that are more convenient in dealing with isolated systems are as follows:

Many modern textbooks use (the so-called surface entropy) without even mentioning (the so-called volume entropy).Footnote ⁷ Historically, however, was the preferred expression for the entropy, with Helmholtz (Reference Helmholtz1884a, Reference Hertz1884b), Hertz (Reference Helmholtz1910), Gibbs (Reference Gibbs1981), and others all developing the statistical mechanics of isolated systems using as the expression for the entropy.

The expressions and have approximately the same value for typical systems, and so the distinction between and is often taken to be unimportant. Nevetheless, there are situations in which these two expressions for the entropy give quite distinct results. For instance, in the case of a two-dimensional ideal gas of N particles of mass m contained in a region of area A and having total energy E we have

It follows that . Using this relation and the fact that states with the same value of AE have the same volume entropy (where m and N are fixed), it is easy to construct pairs of states with the same volume entropy but very different surface entropies, or pairs of states with the same surface entropy but very different volume entropies.

For the purposes of this article, it will be necessary to take a stance on which of the expressions and gives the correct value of the classical thermodynamic entropy. I will claim that the volume entropy is the correct expression for the entropy. This point of view has gained popularity in recent years (see the work of Berdichevskii Reference Berdichevskii1988; Gallavotti Reference Gallavotti1999; Adib Reference Adib2004; Campisi Reference Campisi2005), and so it will suffice to review only very briefly some of the main reasons given by these authors for preferring the volume entropy to the surface entropy.

The first (and perhaps most commonly offered) reason for preferring the volume entropy to the surface entropy revolves around a theorem originally due to Hertz and Helmholtz. Suppose that the walls of a container in which a gas is contained are very slowly moved—for instance, suppose that a gas is slowly compressed—where this is done adiabatically, that is, without any heat flowing into or out of the gas. A quantity that is unchanged in this process is known as an adiabatic invariant. From equation (3), it follows that the thermodynamic entropy of a system is an adiabatic invariant. What Hertz and Helmholtz proved is that if a system is ergodic, then the adiabatic invariants are just the functions of the volume entropy. The surface entropy is typically not a function of the volume entropy (this is so even for the simple case of the ideal gas), and so although the volume entropy is an adiabatic invariant, the surface entropy is typically not. This suggests that the volume entropy is the correct statistical mechanical analog to the thermodynamic entropy. (For more details, see Berdichevskii [Reference Berdichevskii1988] or Campisi [Reference Campisi2005].)

The second most commonly offered reason for preferring the volume entropy to the surface entropy is that only the volume entropy is exactly consistent with the equipartition of energy. To see why, note that from the general thermodynamic relation we have

Using this formula, we can define the temperature T in terms of . Consider now the case of the two-dimensional ideal gas, and replace the volume V by the area A in the preceding discussion. If we substitute the surface entropy for S in equation (15), we get , whereas if we substitute the volume entropy for S in equation (15), we get . By the theorem on the equipartition of energy, the average total kinetic energy of an ideal gas in two dimensions is kNT, and so ; that is, . So the volume entropy, and not the surface entropy, gives a value for the temperature consistent with the equipartition of energy.Footnote ⁸

For a third reason for preferring the volume entropy to the surface entropy, we note that it is the volume entropy (and not the surface entropy) that typically coincides with the canonical Gibbs entropy (eq. [4]). We illustrate this with the case of the ideal gas. Using the formula (7) for the Gibbs entropy of an ideal gas in two dimensions along with , we have

If N is large so that we may use Stirling's approximation , we have

This coincides with the volume entropy but not the surface entropy.

A full discussion of the differences between the surface and volume entropies would take us too far afield. From here onward, we will simply take the volume entropy (12), rather than the surface entropy (11), to be the correct expression for the entropy of an isolated system.

4.2. Case 1: The Ideal Gas

Let us consider the case of an isolated two-dimensional ideal gas of N particles of mass m contained in a square chamber of side length L and with total energy E. As before, the type of macrostate in which we will be interested is one in which the entire gas lies in the rectangular subregion of the chamber and which is otherwise the same as the equilibrium macrostate in which the system is isolated, has the same energy E, but is enclosed in this region by barriers. We denote such a macrostate by .

What is the probability of finding the system in the macrostate ? Considerations of the sort offered in sections 3.2 and 3.3 lead us to identify this probability with

This quantity may be computed exactly. The final answer is

in agreement with the results of section 3.3. Using equation (14), the (volume) entropy for this macrostate is

The relationship between the entropy and the probability of such macrostates is then given by

4.3. Case 2: The Ideal Gas under an External Force

We return to our study of the ideal gas in the potential , where we now assume that our system is isolated and has total energy E. Microcanonical calculations are much more complicated than canonical calculations, and so in order to obtain analytic expressions for the probabilities and entropies in which we will be interested, we need to make some assumptions. In particular, we assume that the rectangular chamber has no top, as shown in figure 4. Because the energy of the system is fixed, no particle can have a y value greater than , and so our system is equivalent to one that is contained in a rectangular region that is closed at the top, but with height (fig. 4). We will be interested in macrostates of the system in which the entire gas lies in the (semiinfinite) rectangular region and which is otherwise the same as the equilibrium macrostate the system would have if it had the same energy E and was enclosed in this region by barriers.

Figure 4

In order to calculate the probability and entropy of these sorts of macrostates, it is useful to calculate

with

We suppose . This expression may be calculated exactly, and the final result is

where . Defining

we then have

The probability of the macrostate is then given by

The (volume) entropy of the macrostate is given by

To show that entropy is not just a measure of probability, it is easiest in this case to construct states with the same entropies but very different probabilities. Note that (assuming ) for fixed E and N, macrostates with the same value of have the same entropy. Setting and , we see that the state defined by has the same entropy as the state defined by . The probabilities of these two states, however, differ by a multiplicative factor of 5. It is also easy to construct pairs of states with identical probabilities but very different entropies.