Several studies in behavioral ecology and psychology have found that certain forms of uncertainty (e.g., unpredictable access to food) promote and invigorate exploration, foraging, and food-related responses. Anselme & Güntürkün (A&G) argue convincingly that there is a motivational basis to this process. They propose that uncertainty produces an “incentive hope” that promotes exploration and seeking behavior, analogous to the ways the “incentive salience” (or “wanting”) of specific stimuli such as food promotes approach behavior (Berridge Reference Berridge2004).

It may be instructive to revisit the idea of complementary motivational and incentive systems for reward seeking and uncertainty reduction, respectively, within current formulations of exploration-exploitation in computational neuroscience. The idea that forms of uncertainty should elicit exploratory actions is encountered in various models; yet these models differ in their specific implementations. For example, various proposals about “exploration bonuses” in reinforcement learning are related to the idea of “hope” in A&G – in the sense that the bonus essentially amounts to an optimism about visiting regimes of state-space that have not been explored (at least recently). This supplement to the utility function precludes excessive or premature exploitation (Christiansen et al. Reference Christiansen, Mason and Mitchell1991; Dayan & Sejnowski Reference Dayan and Sejnowski1996; Sutton Reference Sutton, Porter and Mooney1990).

Other computational approaches, such as active inference, assume that exploration and exploitation are two aspects of the same imperative (to minimize expected free energy). This amounts to resolving uncertainty under normative considerations about the epistemic or informational value of exploratory actions (Friston et al. Reference Friston, Schwartenbeck, FitzGerald, Moutoussis, Behrens and Dolan2014; Reference Friston, FitzGerald, Rigoli, Schwartenbeck, O'Doherty and Pezzulo2016a). Clearly, in active inference, uncertainty has no value per se (and indeed, animals do not seek out uncertainty when they have a choice). Rather, it is the potential reduction of uncertainty and the associated information gain that enable (future) goal-directed behavior – which is intrinsically valuable. Pursuing this sort of value through exploratory or curious behavior has thus a strong knowledge-seeking, motivational component, which has been variously called intrinsic motivation, epistemic drive, value of information, Bayesian surprise, and salience in different contexts (Baldassarre & Mirolli Reference Baldassarre and Mirolli2013; Berlyne Reference Berlyne1960; Friston et al. Reference Friston, Rigoli, Ognibene, Mathys, Fitzgerald and Pezzulo2015; Reference Friston, Lin, Frith, Pezzulo, Hobson and Ondobaka2017; Gottlieb et al. Reference Gottlieb, Oudeyer, Lopes and Baranes2013; Oudeyer et al. Reference Oudeyer, Kaplan and Hafner2007; Schmidhuber Reference Schmidhuber1991; Singh et al. Reference Singh, Barto, Chentanez, Saul, Weiss and Bottou2005).

A recurrent idea is that the ultimate reason to reduce uncertainty, or more broadly to pursue epistemic drives, is to facilitate subsequent pragmatic, utility-maximizing behavior. Yet, active inference agents consider both epistemic and pragmatic drives on an equal footing, during action (or policy) selection, and both can be primary determinants of proximal behavior. This is evident if one considers that active inference agents strive to minimize a quantity (a free energy functional) that explicitly includes two terms: the pragmatic value and the epistemic value afforded by actions or policies (Friston Reference Friston2010; Friston et al. Reference Friston, FitzGerald, Rigoli, Schwartenbeck and Pezzulo2016b; Pezzulo et al. Reference Pezzulo, Rigoli and Friston2018; Reference Pezzulo, Rigoli and Friston2015). This is important because it equips rewards with an information measure and, conversely, means we can talk about the utility of information.

During sequential action or policy selection, the relative balance between pragmatic and epistemic imperatives depends on quantities that are inferred by the agent, such as the statistics of reward distributions and contextual uncertainty (without requiring ad hoc terms like bonuses). Limiting conditions, such as when the agent is fully satiated or has resolved all its uncertainty, advocate pure epistemic or pragmatic behavior. But between these two extremes, policy selection balances pragmatic and epistemic value, which may manifest as apparently suboptimal foraging (e.g., excessive seeking and foraging in uncertain situations), if one considers only utility maximization rather than the epistemic imperatives that are part and parcel of free energy minimization. Crucially, the balance between pragmatic and epistemic imperatives changes systematically with experience and varying environmental conditions. A progressive reduction of epistemic value reveals pragmatic (possibly habitual) policies, when environmental uncertainty resolves – or vice versa, when it suddenly increases (Friston et al. Reference Friston, Schwartenbeck, FitzGerald, Moutoussis, Behrens and Dolan2014; Pezzulo et al. Reference Pezzulo, Cartoni, Rigoli, Pio-Lopez and Friston2016). On this view, there is no need to invoke any random exploration; in other words, there are always optimal epistemic moves to select from. Indeed, one can develop epistemic habits under active inference (Friston et al. Reference Friston, FitzGerald, Rigoli, Schwartenbeck, O'Doherty and Pezzulo2016a).

This line of thinking is compatible with the information primacy hypothesis that reducing environmental uncertainties can be a primary determinant of proximal behavior (Inglis Reference Inglis2000; Inglis et al. Reference Inglis, Langton, Forkman and Lazarus2001; Woodworth Reference Woodworth1958). One implication of this view is that, because information gain is motivating per se, above and beyond reward, animals may have an incentive motivation to reduce uncertainty, or to collect valuable information, without necessarily being optimistic about the outcome of an uncertain event; for example, hoping to find reward when this is unlikely. Assessing the contributions of information gain and reward expectation (or hope) to foraging in uncertain domains is an important, open research question (Behrens et al. Reference Behrens, Woolrich, Walton and Rushworth2007; Daw et al. Reference Daw, O'Doherty, Dayan, Seymour and Dolan2006; Gallistel & Gibbon Reference Gallistel and Gibbon2001; Hayden et al. Reference Hayden, Pearson and Platt2011; Schwartenbeck et al. Reference Schwartenbeck, FitzGerald, Dolan and Friston2013).

The motivational incentives associated with uncertainty may manifest beyond their effects on action selection. In the real world, exploration comes at a cost – not just the cost of abandoning or deferring exploitable rewards, but also, for example, the greater physical effort and energetic investment required for extensive foraging (Charnov Reference Charnov1976b). It thus makes sense that the same motivational incentives promoting exploration also have pragmatic energetic effects; for example, by decreasing the “costs of effort” during cost-benefit considerations (Iodice et al. Reference Iodice, Ferrante, Brunetti, Cabib, Protasi, Walton and Pezzulo2017; Salamone et al. Reference Salamone, Correa, Farrar, Nunes and Pardo2009; Walton et al. Reference Walton, Kennerley, Bannerman, Phillips and Rushworth2006). In turn, the lower inferred costs of acting promote higher activity rates and information-seeking behavior, whose benefits (e.g., increased reward rate) may become apparent, especially under conditions where rewards are occasional.

In sum, adapting to uncertain environments may require the convergence of cognitive, motivational, and bodily factors, which jointly conspire to make the animal fit to its niche. Exploratory and novelty-seeking behavior need not necessarily be treated in terms of “bonuses,” “hopes,” or other correctives of a quintessentially reward-seeking behavior, but emerge from normative considerations within belief-based approaches, namely, planning as (active) inference (Attias Reference Attias2003; Botvinick & Toussaint Reference Botvinick and Toussaint2012; Donnarumma et al. Reference Donnarumma, Maisto and Pezzulo2016; Friston et al. Reference Friston, FitzGerald, Rigoli, Schwartenbeck and Pezzulo2016b; Maisto et al. Reference Maisto, Donnarumma and Pezzulo2015; Pezzulo et al. Reference Pezzulo, Rigoli and Chersi2013; Pezzulo & Rigoli Reference Pezzulo and Rigoli2011; Stoianov et al. Reference Stoianov, Genovesio and Pezzulo2015; Reference Stoianov, Pennartz, Lansink and Pezzulo2018).