In one of the classics of behavioral psychology, Schedules of Reinforcement, Ferster and Skinner (Reference Ferster and Skinner1957) reported how changing the contingencies that link an animal's response to reward can change the rate and pattern of responses. After sufficient exposure to a fixed contingency of reinforcement, the animal's rate of response tended to reach a steady state. After the regularity changed, response rates also changed, typically increasing at first. In the case of a new regular contingency, the animal's performance could eventually reach a new steady state. We could interpret the findings of Ferster and Skinner in terms of reward-seeking behavior and its sensitivity to changes in the schedules of reinforcement. Alternatively, we could consider their results in terms of two distinct motives: the motive for the reward (i.e., food) and the motive to acquire the ability to predict environmental contingencies (Kelly Reference Kelly1963; Piaget Reference Piaget1937). These two motives correspond, respectively, to observations on a smaller timescale (e.g., a single response) and observations on a larger timescale (e.g., increased rate of response vs. reaching steady state). We believe that Anselme & Güntürkün's (A&G's) proposal would benefit from a similar emphasis on the second type of motive that corresponds to patterns of behavior along relatively larger timescales.

Consider Joe, an office worker, who buys a cup of coffee from the same coffee shop every morning before going to work. One day, after arriving at the coffee shop, he is faced with a “closed” sign at the door. He goes to work, and he finds himself thinking about the coffee shop several times during the day. He might even visit the shop a few times. Even though he finds the coffee shop open in the afternoon, he remains concerned. His preoccupation is not only because he wants his daily dose of caffeine, but also because he wants to know whether his access to the source of coffee is restored, stable, and reliable. Visiting the shop, even prior to encountering the closed sign, served two purposes: (1) getting coffee and (2) confirming Joe's implicit knowledge of his own action and the rewarding outcome. After facing the closed sign, Joe was deprived of both coffee and the ability to predict access to coffee. Receiving a cup of coffee in such a circumstance would address the first need, though addressing the second need would require repeated exposure to the new environmental contingency. Presumably, being deprived of the ability to predict access to reward is unpleasant. Therefore, Joe might increase the frequency of his visits to the coffee shop, to shorten the process of reaching the ability to predict the new environmental contingency.

Goal-directed activity is typically nested within an understanding of the context in which the activity takes place. The term action scheme (Piaget Reference Piaget1937) refers to the implicit understanding of the context in which an animal engages in an activity, including learned action-outcome associations (e.g., walking toward the coffee shop means, among other things, one is getting closer to a cup of coffee). An action scheme, thus, involves an implicit understanding of the environment in which movements can serve goals. It is on the basis of that understanding that the animal is capable of expecting certain outcomes. Indeed, in the absence of any ostensible reward, simply perceiving an expected action outcome can serve as a reward, presumably because it serves to confirm the animal's action scheme (Eitam et al. Reference Eitam, Kennedy and Higgins2013; White Reference White1959). Therefore, when the primary reward of a given activity becomes inaccessible, it is not merely the reward that is taken away. What is also taken away is the reliability of the action scheme.

A&G discuss the function of intensified foraging along two dimensions: (1) seeking what is valued and (2) uncertainty. These lead them to characterize the phenomena in terms of hope, which they define as wanting without certainty. Their characterization is not inconsistent for the two types of motivation – an animal can “want” food in the presence of a stable or reliable action scheme, but an animal can only “hope” for food in the absence of a stable or reliable action scheme. It might, however, appear that the motivation to pursue a scarce or unpredictable resource is the same type of motivation underlying the pursuit of an abundant or predictable resource. We suggest drawing a distinction between two kinds of motivation, one of which is directed at the valued resource. The other motivation is directed toward retaining or updating an action scheme for the pursuit of the resource. Drawing the distinction is somewhat counterintuitive, because the two kinds of motivation do not necessarily correspond to separate sets of activities.

Why does the rate of action increase in the case of a scarce and unpredictable resource? The animal in such a circumstance may work toward a new equilibrated action scheme. Uncertainty regarding access to food is aversive, and therefore, shortening the period of uncertainty would require that the animal expend more effort on building an action scheme. Hope is implied in the attempt to achieve a stable action scheme. After all, acting as if the contingencies will eventually become predictable is a prerequisite for trying to learn about them. Increasing the rate of responding (i.e., sampling the environment), in the form of intensified foraging, may generate a tentative action scheme. If the scheme is effective in reducing prediction error, the animal's activity will reach equilibrium. In any case, working to acquire a stable action scheme is not the same as hoping for the valued resource.

Returning to the example of Joe, it is possible that after checking the coffee shop a few times, he learns about the new opening hours, thus acquiring a new reliable action scheme in relation to his morning coffee. It is also possible that he widens the scope of his sampling, exploring the neighborhood for other coffee shops. Exploration is the basis for learning action schemes (Gozli & Dolcini Reference Gozli and Dolcini2018). That is, the purpose of exploration is not merely acquisition of reward, but also the acquisition of new action schemes that provide the animal with sensorimotor mastery and stable access to reward over relatively large timescales.