3.1 Emotion

In the last decades, diverse theories have demonstrated the influence of emotion in the cognitive process (Damasio, Reference Damasio1994). The definition of emotions is a controversial issue since there are more than 92 definitions in the literature (Kleinginna & Kleinginna, Reference Kleinginna and Kleinginna1981). In general, it is said that emotions can be defined as the ‘evaluative judgments of the environment, the self and other social agents, in light of the agent’s goals and beliefs’ (Hudlicka, Reference Hudlicka2008). Emotions are short-lived affective states that arise suddenly in the context of a specific stimulus, having significant psycho-corporeal repercussions and producing physiological changes. Three fundamental characteristics of emotions are their high intensity, their short duration and that they are tied to a particular event. There are two principal aspects to take into account in emotion modeling: emotional representation and emotional elicitation.

Regarding emotional representation, some dimensional theories have been used in intelligent agents modeling such as Russell (Reference Russell1980) and Mehrabian (Reference Mehrabian1996b). In these models, affective states are not independent from one another, rather, they are related to one another in a systematic manner (Gunes et al., Reference Gunes, Schuller, Pantic and Cowie2011). In the case of Russell (Reference Russell1980) is a bidimensional model that represents emotional states according to two universal values: valence and arousal. The proposal of Mehrabian (Reference Mehrabian1996b) presents the PAD theory that is a tridimensional model, used not only for representing emotional states, but also for representing mood (see Section 3.2).

Emotional elicitation is the most studied aspect in emotional agent modeling; therefore, several theories of emotion have been used (Frijda, Reference Frijda1987; Zajonc et al., Reference Zajonc, Murphy and Inglehart1989; Ortony et al., Reference Ortony, Clore and Collins1990; Roseman et al., Reference Roseman, Spindel and Jose1990; Smith & Lazarus, Reference Smith and Lazarus1991; Damasio, Reference Damasio1994; LeDoux, Reference LeDoux1996; Sherer, Reference Sherer, Dalgleish and Power1999; Plutchik, Reference Plutchik2001). In general, according to the emotion elicitation process, they can be grouped into cognitive and non-cognitive theories (Reisenzein et al., Reference Reisenzein, Hudlicka, Dastani, Gratch, Hindriks, Lorini and Meyer2013). In general, non-cognitive theories of emotions follow two main perspectives regarding the elicitation of emotions. There are theories that claim that certain kinds of affect (i.e., sensory-pleasure for smelling a rose) are non-cognitively generated, because they presuppose only non-propositional or even non-conceptual representations (Reisenzein, Reference Reisenzein2006). Other theories claim that even prototypical emotions (i.e., fear, anger, joy) can be non-cognitively caused, given that they can be elicited by the visual or acoustic appearance of danger (i.e., visualize a fierce animal), without mediating any thoughts (Reisenzein et al., Reference Reisenzein, Hudlicka, Dastani, Gratch, Hindriks, Lorini and Meyer2013). These theories have been less used in intelligent agent modeling, being the cognitive ones the most used.

In cognitive theories, the emotion elicitation requires certain ‘higher-order’ mental representations (beliefs and desires) (Green & Green, Reference Green and Green1992). In the case of non-cognitive theories, emotions in a broad sense (i.e., sensory affects-hedonic pleasure, fear) are always elicited by stimuli via a more direct route that circumvents a higher cognitive processes. Cognitive theories are the most used and in most of them a cognitive evaluation process (appraisal) of the stimulus precedes emotion (Frijda, Reference Frijda1987; Ortony et al., Reference Ortony, Clore and Collins1990; Roseman et al., Reference Roseman, Spindel and Jose1990; Smith & Lazarus, Reference Smith and Lazarus1991; Sherer, Reference Sherer, Dalgleish and Power1999). Cognitive appraisal theories attempt to explain why a certain event results in one emotional response rather than another and why a certain emotion can be elicited by different events. The key concept is that the subjective cognitive evaluation of events in relation to the agent’s goals is responsible for emotion (Roseman & Smith, Reference Roseman and Smith2001). More generically, events have to be evaluated as having personal meaning or relevance (Van Reekum, Reference Van Reekum2000). This evaluation is the so-called appraisal. The most popular of the cognitive theories is the OCC theory (Ortony et al., Reference Ortony, Clore and Collins1990). This is due to its tree structure that allows implementing it in a simple way, and also because it works with concepts (beliefs, desires and standards) well-studied in agents theories. It is important to emphasize that the OCC theory does not specify the influence of the emotions in the cognitive process, but only their elicitation. The theory bases its foundation on the way individuals perceive the world and it defines three aspects in which the individual can focus its perception: events, agents and objects. If the focus is put on the consequences of events, fortune-of-others, prospect-based and well-being emotions may be elicited. If the focus is put on the actions of the agents, attribution emotions may be elicited. If the focus is put on the properties of objects, attraction emotions are obtained. This way, OCC defines 22 emotions describing their elicitation as a result of the cognitive appraisal process of the stimulus, guided by some evaluation variables (appraisal variables). The theory also defines the global and local variables that influence the intensity of the emotions. As said before, the success of this theory lies in its easy transfer to a computational model since it explicitly defines the elicitation of the emotions and the appraisal variables for its evaluation, as well as the detailed qualitative and quantitative description of both processes. Figure 1 shows the overall structure of this theory.

Figure 1

Global structure of the emotions of the OCC theory. Taken from the book ‘The Cognitive Structure of Emotions’ (Ortony et al., Reference Ortony, Clore and Collins1990)

An interesting aspect of the modeling of emotion is that they have been usually classified into three main types: primary, secondary and tertiary emotions (Damasio, Reference Damasio1994; Sloman, Reference Sloman1998). Primary emotions (being startled, terrified, sexually stimulated) are innate responses to an external event. Secondary emotions (anxious, apprehensive, relieved, pleasantly surprised) generation involves primary emotions and a more complex cognitive appraisal. Tertiary emotions are disturbing states (feeling humiliated, infatuated, guilty) that reduce self-control. Primary emotions generation has usually been modeled on intelligent agents using a cognitive reactive layer, whereas secondary emotions require a deliberative cognitive layer in addition to the reactive machinery of the primary emotions. The case of tertiary emotions is more complex and has not yet been successfully implemented since it depends on the existence of a mechanism that controls the partial loss of control of attention in the intelligent agent, requiring a sophisticated information processing architecture (Sloman, Reference Sloman and Dautenhahn2000).

Another interesting question is the ability to measure the intensity of which an emotion is experienced since it is a quantitative measure that defines a line between affective reactions and emotions. According to Ortony et al. (Reference Ortony, Clore and Collins1990), an emotion only occurs when the intensity achieves a specific threshold; this means that an intelligent agent not always will experiment an emotion. The intensity also determines how strong the response of an individual is (Scherer, Reference Scherer2000). An affective state that depends on the intensity of the emotions is mood, which will be discussed in the next section.

3.2 Mood

Mood is recognized as the global affective state of an individual (Desmet, Reference Desmet2015). This affective state tends to vary in time due to emotions, returning afterward to its initial state. Compared with emotions, mood lasts longer, has lower intensity and often appears without being associated with a specific event or object (Larsen, Reference Larsen2000). Gratch and Marsella (Reference Gratch and Marsella2004) refer the importance of modeling moods as ‘they have been shown to impact a range of cognitive, perceptual and behavioral processes, such as memory recall (mood-congruent recall), learning, psychological disorders (depression) and decision-making’ (Gratch & Marsella, Reference Gratch and Marsella2004). Several mood theories can be mentioned such as: Larsen (Reference Larsen2000), Mehrabian (Reference Mehrabian1996b), Watson and Tellegen (Reference Watson and Tellegen1985) and the PAD theory (Mehrabian, Reference Mehrabian1996b).

Most of the theories model mood in a bidimensional way, by means of two edges universally recognized as fundamental to characterize mood values: valence and arousal. In this regard, the PANAS theory (Watson & Tellegen, Reference Watson and Tellegen1985) and the Circumplex theory of affect (Russell, Reference Russell1980) stand out. In the case of Russell theory, (Russell, Reference Russell1980) postulates that the underlying structure of an affective experience can be characterized as an ordering of affective states on the circumference of a circle (Remington et al., Reference Remington, Visser, Diener, Barrett, Green, Mayer, Rusting and Watson2000). Sixteen mood states are defined in terms of valence and arousal. The valence variable is related to negative or positive affect and the arousal value measures the excitement level. The PANAS theory proposed by Watson and Tellegen (Reference Watson and Tellegen1985) does not specify particular mood states but mood categories. Mood states are also defined in two dimensions: valence (pleasant–unpleasant) and arousal (high energy–low energy). Four categories of mood arise in this model: energized–unpleasant, energized–pleasant, calm–unpleasant and calm–pleasant. In the case of the PAD model, mood is modeled in a tridimensional way. Pleasure–Displeasure that is an evaluation of predominance of positive versus negative affective states; Arousal–Non-arousal is the level of physical activity and mental alertness; and Dominance–Submissiveness that is a measure of feelings of control versus lack of control. The PAD theory is the most widely used in intelligent agents.

Mood is generally considered influenced by personality, commented in the next section.

3.3 Personality

Personality refers to characteristic thought patterns that persist through time and situations and that distinguish one person from another. It also influences the emotional response of each individual as well as its mood. Personality and emotions are different in terms of focus and duration. Emotions are always tied to a particular event or object and are short term. Personality remains constant in time and is not tied to a particular event. It can be said that an emotion is a specific and short-lived modification of personality.

In the field of intelligent agents, the most widely used personality theory is the Big Five theory (McCrae & John, Reference McCrae and John1992), also known as Five Factor Model (FFM). The model establishes five factors that are universally considered as the basic dimensions of personality: openness, conscientiousness, extraversion, agreeableness and neuroticism, frequently known out under the acronym OCEAN. The model has been implemented in several intelligent agents’ proposals (André et al., Reference André, Klesen, Gebhard, Allen and Rist2000; Yu & Choi, Reference Yu and Choi2005; Durupinar et al., Reference Durupinar, Allbeck, Pelechano and Badler2008). One popular example is ALMA (Gebhard, Reference Gebhard2005) where the personality is used to calculate the initial mood of the agent. To do so, the work of Mehrabian (Reference Mehrabian1996a) that proposes a mapping between PAD and Big Five is used.

There are other personality theories such as The four temperaments theory (Merenda, Reference Merenda1987), the P-E-N theory (Eysenck, Reference Eysenck1967) and the Interpersonal Circumplex or IPC model (Wiggins, Reference Wiggins2003), but they have been scarcely used in the modeling of intelligent agents. The Circumplex model defines a broad set of interpersonal relationships that have a direct relation with behavior in the affective and cognitive area. It is a bidimensional model that defines two orthogonal axes: a vertical axis (of status, dominance, power or control) and a horizontal axis (of solidarity, friendliness, warmth or love). The four temperaments model is one of the oldest personality models in the world and it is also known as the Hippocrates–Galen temperaments theory (Merenda, Reference Merenda1987). The model defines four temperaments: sanguine, choleric, melancholic and phlegmatic. Another interesting personality theory is the P-E-N (Psychoticism–Extraversion–Neuroticism) of Eysenck (Reference Eysenck1967). The psychoticism trait is related to normality which means that individuals with higher p scores are prone to engage in irresponsible behavior (i.e., aggression, egocentrism and impulsiveness) and to contravene social norms, motivated by a need for immediate gratification, in spite of its consequences. Extraversion trait is modeled according to the two extremes of spectrum, that is, introversion and extroversion, according to the related concepts of inhibition and excitation. Neuroticism is explained with regard to the psychological concept of emotionality where high scores on N factor reveals a low threshold of emotionality, while lower scores indicated a high threshold of emotionality (Eysenck, Reference Eysenck1991). This theory is considered together with the Big Five theory, one of the ‘giants’ in personality modeling (Scholte & De Bruyn, Reference Scholte and De Bruyn2004), but in spite of this, it has been scarcely used in intelligent agents.

3.4 Empathy

The most common definition of empathy is ‘put yourself in someone else’s shoes to understand his emotions’ (Pacherie, Reference Pacherie, Berthoz and Jorland2004). An important feature of empathy is that it has two dimensions: cognitive and affective. The cognitive dimension is about understanding the situation of the other and the affective one is about feeling an emotion produced by the other’s situation. Both may or may not appear together, it means that an empathic cognitive emotion can be felt without feeling an affective one. The modeling of empathy in intelligent agents has been considered from two perspectives: the empathy evoked by the intelligent agent (into the user) or the empathy that the intelligent agent ‘feels’. There are proposals in both directions: those that seeks to evoke empathy in users (Paiva et al., Reference Paiva, Dias, Sobral, Woods and Hall2004) and those that focused on the empathy that agent feels (Reilly, Reference Reilly1996; McQuiggan & Lester, Reference McQuiggan and Lester2006; Ochs et al., Reference Ochs, Paris, Pelachaud, Paris and Sadek2008). The latter ones are those of interest here. Usually, these works are not based on specific models but on empirical works or theoretical approaches such as the mentioned OCC theory (Ortony et al., Reference Ortony, Clore and Collins1990). In fact, OCC describes the elicitation of two empathic emotions when a goal of another intelligent agent or user is achieved: happy-for and sorry-for. To consider them, the intelligent agent needs to have a representation of other agent’s (or user) goals.

3.5 Emotional regulation and coping

Emotional regulation is defined as ‘the processes by which individuals influence which emotions they have, when they have them, and how they experience and express these emotions’ (Gross, Reference Gross1998). Other authors define it as ‘the extrinsic and intrinsic processes responsible for monitoring, evaluating and modifying emotional reactions, especially their intensive and temporal features, to accomplish one’s goals’ (Thompson, Reference Thompson1994).

Another concept regarding regulation is coping; its more accepted definition is given by Lazarus and Folkman (Reference Lazarus and Folkman1984) who express that coping ‘consists of cognitive and behavioral efforts to manage specific external or internal demands (and conflicts between them) that are appraised as taxing or exceeding the resources of the person’. It should be noted that emotional regulation focuses on the increase or decrease of positive and negative emotions (Gross, Reference Gross1998), whereas coping is more about diminishing negative emotions in stressful situations (see Lazarus & Folkman, Reference Lazarus and Folkman1984; Scherer, Reference Scherer, Scherer and Ekman1984). The most used models in the field of agent modeling are the one of Smith and Lazarus (Reference Smith and Lazarus1991), oriented to coping and the Gross theory (Gross, Reference Gross1998) oriented to emotional regulation.

In Smith and Lazarus (Reference Smith and Lazarus1991), coping is related to the way the individual interprets its relation with the environment. The theory proposes two fundamental coping strategies: problem-focused, that consists of altering the adverse situation, and emotion-focused, that consists of coping distressing emotions that arise in situations where the circumstances not going to change. An example of the use of this model in intelligent agents is EMA (Marsella & Gratch, Reference Marsella and Gratch2009), a proposal to create agents that are able to cope with negative affect.

In Gross (Reference Gross1998), emotions can be regulated in five different phases during the emotion generation process: Situation Selection (avoidance of people, places or objects), Situation Modification (active efforts to modify the situation in order to alter its emotional impact), Attentional Deployment (selection of where to put the focus on), Cognitive Change (modification of the cognitive evaluation of the situation) and Response Modulation (expression of the emotion elicited). Each phase takes part in the process according to the emotional dynamics and it is able to regulate positive as well as negative affect. CoMERG (Bosse et al., Reference Bosse, Pontier and Treur2007) and S-model (Soleimani & Kobti, Reference Soleimani and Kobti2012) are two computational models of emotion regulation that follow Gross’s paradigm.

Both coping and emotional regulation are oriented to the regulation of emotions but there are also theories to regulate more medium-term affective states like mood, such as the theory of Larsen (Reference Larsen2000). In fact, the affective regulation term has been proposed (Gross, Reference Gross1998) as a global category that includes emotional regulation, coping and mood regulation. Moreover, there are works that address the subject of empathic emotion regulation (EER) and define it as ‘an interpersonal regulation system in which an empathic response to another person’s emotional state aims to regulate both emotion within the provider and across interaction partners’ (Hamburg et al., Reference Hamburg, Finkenauer and Schuengel2014). Nevertheless, this kind of issues has not been modeled yet in intelligent agents.

3.6 Theoretical bases of the affective influence on cognitive process

The influence of emotions on the cognitive process is a field of interest in many research areas (philosophy, psychology, neuroscience) (Blanchette & Richards, Reference Blanchette and Richards2010; Angie et al., Reference Angie, Connelly, Waples and Kligyte2011). Studies have shown that emotions influence the human decision-making process (Damasio, Reference Damasio1994) and may influence all the phases of the cognitive process: perception (Zadra & Clore, Reference Zadra and Clore2011), beliefs (Frijda et al., 2000; Glore & Gasper, Reference Glore, Gasper, Frijda, Manstead and Bem2000) and behavior (Dolan, Reference Dolan2002).

Emotions, mood and personality are the affective aspects whose influence on the cognitive processes has been studied. As commented before, emotions and mood are two strongly related concepts. This is why in many works about emotional influence, they are treated interchangeably (Zadra & Clore, Reference Zadra and Clore2011).

Regarding the influence of emotions on the cognitive processes, there is no general consensus among researchers. There are studies that show that emotions influence the mediating information processing (attention, perception, memory), situation assessment, decision-making, goal management, planning and learning (Frijda et al., Reference Frijda, Manstead and Bem2000a; Dolan, Reference Dolan2002; Angie et al., Reference Angie, Connelly, Waples and Kligyte2011). For example, positive emotions induce global focus, whereas negative emotions induce a moral local focus and analytical thinking. Also, anxiety states reduce attentional and working memory capacities and biases attention toward the detection of threatening stimuli (Jeon, Reference Jeon2017).

Regarding the influence of mood, it has been studied its impact on a range of cognitive, perceptual and behavioral processes, such as memory recall (mood-congruent recall), learning, psychological disorders (depression) and decision-making (Gratch & Marsella, Reference Gratch and Marsella2004). Particularly, its influence on judgment is mediated by the individual’s subjective (positive or negative) experience about the object of judgment. This means that, when evaluating objects or belief statements, individuals act accordingly to how they feel about them, using that information for the judgment in a mood-congruent way (Schwarz & Clore, Reference Schwarz and Clore1983).

Regarding personality, studies have been focused mainly on its influence on behavior (McEachan et al., Reference McEachan, Sutton and Myers2010; Lönnqvist et al., Reference Lönnqvist, Verkasalo and Walkowitz2001). For example, a growing number of researchers have concluded that neurotic (as compared to non-neurotic) individuals have stronger negative affective reactions to undesirable stimuli, whereas extraverts (as compared to introverts) have stronger positive affective reactions to pleasant stimuli (i.e., Larsen & Ketelaar, Reference Larsen and Ketelaar1989; Larsen & Ketelaar, Reference Larsen and Ketelaar1991).

In the Section 5, the BDI proposals that deal with the affective aspects studied in this section will be reviewed. First, the main BDI concepts are reviewed.

5.1 Classification of the emotional BDI proposals

Following the classification presented in Section 2, the proposals found can be grouped into two main categories: BDI formalizations and extensions of the BDI architecture. The works are presented in Table 2 indicating the affective aspect they consider (emotions, mood, personality, empathy and regulation). The level of application is also indicated: some proposals are no implemented at all, other present just an experimental scenario or application to test its viability. Only a few present a full application with a specific objective.

The first group of proposals is formed by those belonging to the formalization group. It is quite a diverse group. Most of them are based on agent logics languages, in particular, in BDI logics. But others extend existing agents specification frameworks or agent modeling languages as it is commented below.

The proposals of Steunebrink et al. (Reference Steunebrink, Dastani and Meyer2012) and Meyer (Reference Meyer2006) extend the BDI-based KARO agent specification framework (Meyer et al., Reference Meyer, Van Der Hoek and van Linder1999). The KARO framework is based on a blend of dynamic logic with epistemic logic, enriched with modal operators for beliefs and desires concepts (Meyer & Van Der Hoek, Reference Meyer and Van Der Hoek2004). No application or scenario is presented.

The works of Adam et al. (Reference Adam, Herzig and Longin2009), Dastani and Lorini (Reference Dastani and Lorini2012) and Gluz and Jaques (Reference Gluz and Jaques2017) use BDI logic languages: the ABC, Dynamic Logic of Graded Attitudes (DL-GA) and AfPL logics, respectively. The ABC logic is an extended version of one of the first BDI logics proposed by Cohen and Levesque (Reference Cohen and Levesque1990). It formalizes the concepts of action, time, belief and choice in a simple propositional modal logic (Herzig & Longin, Reference Herzig and Longin2004). The DL-GA logic is designed to represent with special operators the beliefs, goals and intentions of the agent, where beliefs and goals have degree of plausibility and desirability, respectively. The AfPL logic is a propositional logic with the traditional logic operators that manage the beliefs, goals intentions concepts.

As we will see in the next section, in all there cases, they formalize the OCC theory, combined with Oatley and Jenkins (Reference Oatley and Jenkins1996) theories in the case of Adam et al. (Reference Adam, Herzig and Longin2009) and the Lazarus (Reference Lazarus1991) and Frijda (Reference Frijda1986) theories in the case of Dastani and Lorini (Reference Dastani and Lorini2012). Agent logic languages facilitate the transformation from specification to implementation. Nevertheless, Gluz and Jaques (Reference Gluz and Jaques2017) is one of them that presents an experimental scenario (a vacuum cleaner robot), but without implementation.

Finally, in this group, the proposals of Bosse and de Lange (Reference Bosse and de Lange2008), Bosse and Zwanenburg, (Reference Bosse and Zwanenburg2009), Bosse and Höhle (Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011) can also be found. In this case, they use as formalization tool an agent-based modeling language, called LEADSTO (Bosse et al., Reference Bosse, Jonker, Van Der Meij and Treur2007). LEADSTO language allows to model dynamic processes in terms of both qualitative and quantitative concepts, by the means of either logical or mathematical means, or a combination thereof. They formalize three Ekman’s emotions, the expectation-based emotions of Castelfranchi and Lorini (Reference Castelfranchi and Lorini2003) and two emotions based on the appraisal theory presented in Gratch and Marsella (Reference Gratch and Marsella2004). They present three experimental applications (simulation environment with two IVA, a dice game with an embodied agent and game based on RobCop, respectively) to test the viability of the proposals.

As we see, these works have focused mainly on emotional formalization and mostly on formalizing the OCC theory. Only three of them have considered other aspects such as mood (Bosse & Höhle, Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011) and regulation (Bosse & Zwanenburg, Reference Bosse and Zwanenburg2009; Dastani & Lorini, Reference Dastani and Lorini2012). Details on how they consider these three aspects are presented in the next section. Empathy and personality have never been formalized.

The second big category is formed by those works that are extensions of existing BDI cognitive architectures. They follow the generic BDI schema (see Section 4) extended with affective modules that influence different cognitive phases (as will be explained in Section 5.7). Although OCC is also widely used, many more theories are applied (as it will be commented in the next section). Mood and personality are also considered in about half of them. On the contrary, empathy (just treated in three of them) and regulation (only in two) are scarcely considered.

In these cases, as it was commented in Section 4, the existence of several implementation tools facilitates their development and application. In spite of this, some of them do not present any application or example scenario. Other just present experimental applications or scenarios to test the architecture’s viability. This is the case of Jiang et al. (Reference Jiang, Vidal and Huhns2007) that present an application developed in the Tileword platform (Pollack & Ringuette, Reference Pollack and Ringuette1990) to compare the behavior of emotional and non-emotional BDI agents. Puica and Florea (Reference Puica and Florea2013) present an experimental scenario of an agent situated in a 2D map with artifacts and traps. Hernández et al. (Reference Hernández, Déniz, Lorenzo and Hernández2004) present a more elaborate experimental application based on a humanoid robotic head with vision and audio sensor capabilities to evaluate the reaction of the robot to certain colors and the activation of primary emotions (surprise, fear) and secondary emotions (happiness, sadness and anger) given the image’s luminance value. Neto and da Silva (Reference Neto and da Silva2012) present two games as experimental applications: iterated prisoner’s dilemma and a memory card game. They focus on evaluating the influence of the affective state in the activities of perception, memory and decision-making. In the case of Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011), the experimental application is an embodied agent, called EMMA, integrated as third pattern into Max’s virtual environment (Becker-Asano & Wachsmuth, Reference Becker-Asano and Wachsmuth2008). They focus on the assessing EMMA’s capability of empathizing with Max’s emotions during the interaction with a human user. Four of the proposals present full applications. In all the cases, an embodied conversational agent is used but within different application scenarios. De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003) work has been used as the base of an interactive listening agent (Bevacqua et al., Reference Bevacqua, Mancini and Pelachaud2008) and an Interactive Storyteller (Bevacqua et al., Reference Bevacqua, Prepin, Niewiadomski, de Sevin and Pelachaud2010). In Ochs et al. (Reference Ochs, Sadek and Pelachaud2010), the application is 3D talking head that shows facial expressions according to the intensity of the elicited emotions, in order to empathize with the user. In the case of Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011) and Becker-Asano and Wachsmuth (Reference Becker-Asano and Wachsmuth2008), they present two different systems, the first one focused on empathic emotion elicitation and the second one on primary and secondary emotion elicitation. The application developed by Becker-Asano & Wachsmuth (Reference Becker-Asano and Wachsmuth2008) is a virtual human named Max that plays the role of museum guide. The application has been implemented in several scenarios (robotics, chat boot agent) and also more recently it has been extended with memory retrieval functions of the ACT-R cognitive architecture (Becker-Asano, Reference Becker-Asano2014).

In Table 1, the reviewed emotional BDIs are presented showing the affective aspects considered and their level of application.

Table 1

A classification of the proposals reviewed in this paper showing the affective aspect that model (mood, personality and regulation) and the application type (Exp_app: Experimental application; Exp_scenario: Experimental scenario; Real_app: Real application)

Exp_app = Experimental application; Exp_scenario = Experimental scenario; Real_app = Real application

5.2 Emotion modeling

Emotion is the most commonly treated emotional concept in intelligent agents. As it has been analyzed in Section 3.1, there are many emotion elicitation theories that have been used in this context and they can be grouped into cognitive and non-cognitive theories. In cognitive theories, the emotion elicitation requires certain ‘higher-order’ mental representations whereas in the case of non-cognitive theories, emotions are elicited by stimuli via a more direct route that circumvents higher cognitive processes (Reisenzein et al., Reference Reisenzein, Hudlicka, Dastani, Gratch, Hindriks, Lorini and Meyer2013; Reisenzein et al., Reference Reisenzein, Hudlicka, Dastani, Gratch, Hindriks, Lorini and Meyer2013). The emotional BDI proposals selected to be reviewed will be classified into those two groups.

Only three proposals follow non-cognitive theories. In this group, the work of Jiang et al. (Reference Jiang, Vidal and Huhns2007) stands out and is the basis of other proposals. They follow the theory of Damasio (Reference Damasio1994) but the concept of emotion is implemented in general way, without specifying the elicitation of any particular emotion. They provide the cognitive affective reasoning needed for eliciting both primary and secondary emotions in the general BDI process. Primary emotions are modeled as the first feeling that an individual experiences when an event occurs and secondary emotions may be caused directly by primary emotions, or come from more complex chains of thinking. Current beliefs and intentions originate primary emotions, and the generation of secondary emotions is influenced by beliefs, intentions and primary emotions. In contrast with the other examples, beliefs can be acquired through perception, contemplation or communication. In the case of Jones et al. (Reference Jones, Saunier and Lourdeaux2009), they maintain the cognitive and emotional modeling of Jiang et al. adding personality and physiology, which is their main contribution. Physiology (stress, hunger/thirst, tiredness, temperature, injury and contamination) is used as a moderator of the cognitive-affective process that directly influences the elicitation of primary emotions. Personality modeling is another important aspect that differences these proposals as it will be explained in Section 5.4.

In the case of Bosse and de Lange (Reference Bosse and de Lange2008), their modeling of emotions is based on Ekman et al. (Reference Ekman, Friesen and Ellsworth1972) and just three basic human emotions (anger, happiness or fear) are elicited. In contrast to the other proposals in this group, beliefs can only be acquired from observations and only one emotion can be experienced simultaneously. Besides, each emotion has an associated emotional response level (ERL) directly influenced by agent’s beliefs. In spite of the simplicity of their emotional modeling, they address an interesting and practically unexplored field in intelligent agents modeling, emotional regulation, as it will be analyzed in Section 5.5.

The proposals based on cognitive theories are the more numerous and have been divided into formalization and extension proposals. In the formalization category, four of them are based on the OCC theory and the others in Lazarus (Reference Lazarus1991), Oatley and Jenkins (Reference Oatley and Jenkins1996) and Castelfranchi and Lorini (Reference Castelfranchi and Lorini2003), respectively.

The proposals that formalize the OCC theory have in common the way emotions are elicited, because they consider that emotions depend on beliefs and—directly or indirectly—on desires. In Adam et al. (Reference Adam, Herzig and Longin2009), emotion elicitation involves two types of beliefs (strong beliefs, that are those that are truth without doubt, and weaker beliefs, that are those that are related to the probability of an event to be true) and goals that are related to the desirability/undesirability of the consequences of events. They manage intensity, but they do not deferenciate between emotion elicitation and emotion experience. This means that in the case that the conditions for an emotion are met, the agent will always experience it. In addition, they do not distinguish between emotions of the same OCC branch as anger and irritation. Finally, their formalization is restricted to their own BDI-based logical framework. The proposal of Steunebrink et al. (Reference Steunebrink, Dastani and Meyer2012) is a first attempt to differentiate between affective reactions and experienced emotions: an emotion triggered is experienced only if it has already been experienced in the past and the overall emotion intensity is positive. To do so, they implement an intensity assignment process but without detailing the complete intensity calculation. In contrast to Adam et al. (Reference Adam, Herzig and Longin2009), their formalization of the OCC theory it is not tied to a particular framework, because they provide a formalization into dynamic doxastic logic an later they translate it into a logical implementation. In contrast with the precedent works, Gluz and Jaques (Reference Gluz and Jaques2017) present a probabilistic formalization of the OCC emotion elicitation process. They also distinguish between emotion elicitation and emotion experience managing intensity, but providing all the detailed calculations of the variables affecting intensity (potential and threshold). As the other works, they formalized the process of appraisal that depends on agents’ beliefs, goals and desirability but adding a new desirability operator. In the proposal of Dastani and Lorini (Reference Dastani and Lorini2012), emotions are defined in terms of graded beliefs, graded goals, and intentions and intensity is considered. Their implementation of coping strategies represents a step forward in including affective aspects into formalizations and it will be discussed in Section 5.4. All the precedent proposals formalize OCC emotions but they differ in the number of emotions they consider: Steunebrink et al. (Reference Steunebrink, Dastani and Meyer2012)-22 OCC emotions, Adam et al. (Reference Adam, Herzig and Longin2009)-20 emotions, Gluz and Jaques (Reference Gluz and Jaques2017)-8 emotions (event-based emotions with consequences for self-based) and Dastani and Lorini (Reference Dastani and Lorini2012)-4 of the prospect-based emotions (see specific emotions in Table 2).

Table 2

The proposals reviewed showing the theory in which the emotional modeling is based on and the emotions they model

F = Formalization; Ex = BDI extension

In the formalization group, there are also proposals based on different theories of emotion, not OCC. This allows them to model other types of emotions not usually considered. The proposal of Meyer (Reference Meyer2006) is based on the emotional theory of Oatley and Jenkins (Reference Oatley and Jenkins1996) and models the elicitation of free-floating emotions, whose elicitation is not always directly related to a particular object or situation. They formalize four types of them: happiness, sadness, anger and fear. The elicitation of emotions is shaped by the success or failure of a plan and by the achievement or abandoning of the pursued goal. They do not manage the intensity of emotion. Bosse and Zwanenburg (Reference Bosse and Zwanenburg2009) proposal is based on the emotional theory of Castelfranchi and Lorini (Reference Castelfranchi and Lorini2003) and considers expectation-based emotions (before-emotions and after-emotions). Modeling this kind of emotions allows the agent not only to ‘feel’ an emotion when an event happens, but also by anticipating future events (before-emotions) and evaluating past events (after-emotions). Expectation-based emotions are generated by the combination of beliefs and desires in a particular state, considering also the intensity related to the importance of the desire. The proposal of Bosse and Höhle (Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011) is based on Gratch and Marsella (Reference Gratch and Marsella2004) work. Emotions are generated as a result of the evaluation of events through desires and are influenced by beliefs. Two types of emotion are modeled, happiness and sad, but the model can also be used to represent emotions like anger and fear. They do not differentiate between emotion elicitation and emotion experience, but they work with the concept of intensity proportional to the strength of the desire. Their proposal differentiates from all the other because they include mood modeling as it will be discussed in Section 5.3.

As most of the proposals in the formalization group are based on BDI logics, they may be not expressive enough to accommodate all assumptions of psychological emotion theories. For example, most of them are purely qualitative; however, emotions differ not only in quality but also in intensity, which is important to explain certain effects of emotions. In this regard, proposals like the one of Dastani and Lorini (Reference Dastani and Lorini2012) that combines a dynamic knowledge logic with a logic for graded beliefs and desires can provide a more wide range of constructs to enrich the modeling of emotions. Most of them miss also another important issue: the emotional influence on cognitive reasoning, that is, how the emotions can help agents to choose the most rational action to be performed, and how the emotions can improve the way that agents reason or decide or act.

Regarding the proposals that are BDI extensions, they are mainly (all except two of the selected proposals) based on the OCC theory, either alone or combined with other theories. Proposals based only on the OCC theory manage the process emotional elicitation in a similar way: an event is appraised in terms of beliefs, desires and intentions and a certain score regarding its valence (negative/positive, displeasure/pleasure) and arousal (intensity of the elicited emotion) is returned. Their main differences consist of emotions they model and the consideration of additional concepts that directly affect the cognitive and affective process. The proposal of Neto and da Silva (Reference Neto and da Silva2012) is the one that models the 22 OCC emotions. The other proposals focus on modeling a specific branch of the OCC emotions, like the case of De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003) that model the event-driven category, Van Dyke Parunak et al. (Reference Van Dyke Parunak, Bisson, Brueckner, Matthews and Sauter2006) that model only two emotions. Other works like Ochs et al. (Reference Ochs, Sadek and Pelachaud2010) and Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011) are based on the OCC theory but their main objective is empathy modeling, and therefore will be discussed in Section 5.5.

Among the BDI extensions that are based only on the OCC theory, the proposal of De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003) is probably the most relevant since it is one of the first emotional BDI proposals and it is one of the most cited. They present an interesting proposal that uses a belief network for representing the agent’s mental state. They model the elicitation of a subset of the emotions: the event-driven categories of fortune-of-others, well-being and prospect-based emotions. The elicitation of emotions is determined by the variation in the agent’s beliefs about whether a goal can be threatened or archived, depending on the importance (weight) assigned to the goal. They manage the intensity of an elicited emotion depending on the weight of the goal. This idea of assigning weights to desires is similar to the ideas of probability of Gluz and Jaques (Reference Gluz and Jaques2017) and graded goals of Dastani and Lorini (Reference Dastani and Lorini2012). The main problem is the complexity of the proposed beliefs network, since the number of nodes increases considerably with the number of emotions modeled. Nevertheless, they make up for this problem, reducing the time required to propagate evidence and update the emotion intensities, thanks to the effectiveness of the employed algorithm. They also take into account mood, personality and regulation aspects discussed in Sections 5.3, 5.4 and 5.6, respectively.

Some other OCC-based proposals stand out because they consider new concepts, providing the agent with a wide range of information and resources that improve its performance in a dynamic environment. Van Dyke Parunak et al. (Reference Van Dyke Parunak, Bisson, Brueckner, Matthews and Sauter2006) include the concepts of dispositions, triggers and tendencies. They extend the appraisal reasoning processes with an analysis process. The appraisal process assesses the beliefs in the context of agent disposition and returns the elicited emotion. Beliefs are mapped to digital pheromones, which inform the presence of other agents (or objects) in the environment. These pheromones act as triggers for emotions. The analysis process determines the intentions to be committed by the agent, drawing on the pheromone vector. As it can be seen, their emotional modeling is tied to a particular situation (domain-dependent) and only two emotions are modeled.

Neto and da Silva (Reference Neto and da Silva2012) manage a memory component for past events and a memory filter that retrieves those beliefs that are aligned with the agent’s current emotional state. They also consider that the emotional state of the agent is not only determined by the elicited emotions caused by an event, but also by the agent’s perception. To do so, every possible perception has associated a vector of 22 emotions, and the sum of its intensities has to be equal or gather than a minimum value so they can be take it into account in the emotional state of the agent. Another interesting aspect is that the affective output of the emotional component is the agent‘s mood and not the elicited emotions.

As it was mentioned above, only two proposals are not based on OCC theory. Lejmi-Riahi et al. (Reference Lejmi-Riahi, Kebair and Said2014) emotional BDI is based on the emotional theory of Loewenstein and Lerner (Reference Loewenstein and Lerner2003) and is oriented to deal with expected and immediate emotions. Expected emotions are predictions about the emotion that will be experienced if the decision is taken, and immediate emotions refer to the emotions felt at the time of the decision-making. In contrast with the proposals that model OCC, they propose three appraisal process: incidental stimulus appraisal (for immediate emotions elicitation), and emotional option and cognitive option appraisal (for expected emotions elicitation). Immediate emotions generation is influenced by beliefs and intentions and expected emotions generation is influenced by immediate emotions, desires and intentions. The proposal of Pereira et al. (Reference Pereira, Oliveira, Moreira and Sarmento2005), unlike all the previous ones, is opened to the implementation of any emotional theory; they do not detail how emotions are modeled, but just the interaction between the emotional component and the BDI process. Their main contribution is the introduction of two interesting concepts: capabilities and resources. The idea of introducing these concepts comes from identifying some disadvantages of the BDI model such as the lack of information about resource bounds, of a dynamic deliberation process and of the consideration of other mental states different than beliefs, desires and intentions. They define capabilities as a library of available abstract plans that the agent owns and resources as the way of converting these abstract plans into action plans. Resources can be either physical (disk space, available memory, etc.), virtual (energy sources on a virtual world, other agents, etc.) or both. The emotional component manages both the resources and the capabilities, and the availability or unavailability of capabilities depends on the availability of the required resources.

The proposals that combine different cognitive and non-cognitive theories of emotion are oriented to model the elicitation and representation of a high diversity of emotions (primary, secondary and tertiary) and to consider additional affective concepts such as mood, personality and regulation. Three of the proposals are based on the Damasio and OCC theories and they model primary and secondary emotions. Of them, the proposal of Becker-Asano and Wachsmuth (Reference Becker-Asano and Wachsmuth2008) stands out for being one of the most cited and has also been used in different applications domains. Their model combines the OCC and the Damasio theory for emotion elicitation and the PAD for emotional representation. Primary emotions are modeled as inborn affective states that are triggered by reflexes in case of harmful stimuli, determining the reactive behavioral responses of the agent. Secondary emotions are modeled as a more complex affective state and their elicitation depends much more on the situational and social context than that of primary emotions. Puica and Florea (Reference Puica and Florea2013) focus on simulating human mechanisms of internal resource usage and on agent personality following the ideas of Pereira et al. and of Jones et al., respectively. They combine the theory of Damasio for representing primary and secondary emotions, the OCC theory for emotion elicitation and the Russell’s theory for emotion representation. Alfonso et al. (Reference Alfonso, Vivancos and Botti2014) also follow the theories of emotion of Ortony et al. (Reference Ortony, Clore and Collins1990) and Damasio (Reference Damasio2005). They include the modeling of the surprise emotion, not considered in OCC, which can be either a primary or secondary emotion. Hernández et al. (Reference Hernández, Déniz, Lorenzo and Hernández2004) combines Damasio (Reference Damasio1994) with Sloman (Reference Sloman2002) theory (not OCC). They propose an emotional component composed of a three-level belief system that generates three emotional levels: primary, secondary and tertiary. Although tertiary emotions are not implemented, this is the first attempt for considering them. They use also model emotional representation in a dimensional space (valence; arousal) based on the ideas of (Breazeal, Reference Breazeal2002).

As a conclusion, most of the proposals are based on just one theory (being OCC very prominent) but the idea of combining several theories is interesting and should be further explored. It would allow widening the type of emotions considered to some of them that are practically unexplored, such as Sloman’s tertiary emotions. Another interesting aspect is the combination of elicitation and representation emotion theories. In this sense, the OCC theory combined with the PAD model is often used, but other interesting theories such as the Russell and PANAS theories for emotional representation may also be used. Moreover, the consideration of new concepts (resources, capabilities, triggers, tendencies, information not only acquired by perception) provides a wide range of information and resources to the agent and may improve his performance in dynamic environments.

In Table 2, the revised proposals are presented showing the emotional theory they are based on and the specific emotions modeled. The way these works consider the different affective aspects will be commented in the next section.

5.3 Mood modeling

Mood is generally recognized as the global affective state of an individual (Section 3.2). In spite of this, just 7 of the 22 proposals reviewed model mood, being one (Bosse & Höhle, Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011) a formalization and all other extended BDI architectures. In Table 3, those proposals are shown.

Table 3

Proposals reviewed that model mood showing the affective theory they follow and the mood states they model

As it can be seen, the proposals that deal with mood in the simplest way (De Rosis et al., Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003; Hernández et al., Reference Hernández, Déniz, Lorenzo and Hernández2004) do not follow any mood theory and do not model mood as an independent state but rather as the decay of the emotions in time. Bosse and Höhle (Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011) treat mood as an independent state that depends on the previous mood in combination with the weighted sum of the different emotional intensities, but without applying any explicit affective theory.

In the other four proposals, mood is modeled as an independent state applying an affective theory, being the dimensional one, and more specifically, the PAD theory the most extended. In some cases, mood is modeled directly derived from PAD space (Neto & da Silva, Reference Neto and da Silva2012; Alfonso et al., Reference Alfonso, Vivancos and Botti2014) whereas in other it is modeled as the agent’s overall positive or negative feeling (Becker-Asano & Wachsmuth, Reference Becker-Asano and Wachsmuth2010; Boukricha & Wachsmuth, Reference Boukricha and Wachsmuth2011).

In all proposals, mood persistence in time is modeled as a decay factor that is applied in the absence of another input stimulus. In some cases, the decay factor is determined directly by the agent personality as well as mood’s initial or default value (De Rosis et al., Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003; Alfonso et al., Reference Alfonso, Vivancos and Botti2014). In other cases, the decay factor is determined by a linear function (Becker-Asano & Wachsmuth, Reference Becker-Asano and Wachsmuth2010), as a change rate (Bosse & Höhle, Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011), or as an amount by which the current emotional state is divided (Hernández et al., Reference Hernández, Déniz, Lorenzo and Hernández2004).

All proposals also agree in the interconnectivity between mood and emotions which results in ‘emotion dynamics’: emotions influence mood updating and mood influences emotion elicitation. This is empirically supported by Neumann et al. (Reference Neumann, Seibt and Strack2001), who found that individuals in a positive mood are less likely to experience negative emotions and vice versa. The influence of mood in the emotional elicitation process is generally named mood congruency. This bidirectional relationship has been particularly modeled in those proposals that model primary and secondary emotions. In most of the cases, mood influences the elicitation of primary emotions (Alfonso et al., Reference Alfonso, Vivancos and Botti2014) and in some cases, it is used directly to elicited them (Becker-Asano & Wachsmuth, Reference Becker-Asano and Wachsmuth2010). In the case of Becker-Asano and Wachsmuth (Reference Becker-Asano and Wachsmuth2010), mood congruency of secondary emotions is also considered. Other proposals model the mood–emotion relationship as a modulation factor, like the case of Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011) that implement an empathizer-mood component where a negative mood increases the potential of eliciting a negative empathic emotion and decreases the potential of a positive one. In contrast, a positive mood increases the potential of a positive empathic emotion and decreases the potential of a negative one. Proposals that do not follow any theory this relationship is modeled through the decay factor.

As a conclusion, the consideration of mood as an independent state, modeled with an appropriated theoretical base that ties it to the emotional process, guarantees a mood-congruent emotions elicitation and, therefore, more believable intelligent agents. The consideration of personality is also essential.

5.4 Personality modeling

Personality refers to characteristic thought patterns that persist through time and situations and distinguish one person from another (Section 3.3). Personality modeling is an important aspect to take into account when modeling believable affective agents. Nevertheless, personality is only considered in 8 of the 22 reviewed proposals. All of them are BDI extensions: personality is not modeled in any BDI formalization. In Table 4, the proposals that considered personality are presented. They have been grouped depending on whether they model personality as traits (i.e., neuroticism), temperaments (i.e., lethargic) or behavioral dispositions (i.e., irritability). Half of them follow domain-independent theoretical approaches (Eysenck, Reference Eysenck1967; Carbonell, Reference Carbonell1980; Merenda, Reference Merenda1987; McCrae & John, Reference McCrae and John1992; Ortony, Reference Ortony, Trappl, Petta and Payr2002) which means that the considered traits/temperaments/dispositions are not tied to a particular type of situation and can be used for modeling several scenarios. Proposals that do not detail any affective theory are prone to model traits/temperaments/dispositions as domain-dependent. This is the case of Jones et al. (Reference Jones, Saunier and Lourdeaux2009) and Van Dyke Parunak et al. (Reference Van Dyke Parunak, Bisson, Brueckner, Matthews and Sauter2006), that model specific traits and behavioral dispositions, tied to crisis and combats simulation situations, respectively. In the other cases, such as in Becker-Asano and Wachsmuth (Reference Becker-Asano and Wachsmuth2010) and Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011), personality is modeled just as a parameter value that determines how emotional the agent is.

Table 4

The proposals reviewed in this section showing the affective theory in which the mood modeling is based on and the mood states they model

Personality modeling is related to emotion and mood modeling. The relationship between personality and mood is quite close, as it is generally modeled based on the idea that mood changes depending on the individual personality traits. In this regard, in most of the proposals, the agent’s default or initial mood value is set according to the agent’s personality traits, directly influencing its updating and decay factor over the interaction. On the contrary, Puica and Florea (Reference Puica and Florea2013) just model personality and emotions. In this case, the relationship is modeled through two variables, emotion intensity and decay, linked to introversion/extraversion and stable/unstable traits, respectively (Eysenck, Reference Eysenck1967). Finally, the proposal of Neto and da Silva (Reference Neto and da Silva2012) models the three affective aspects interconnected. In this case, the relationship is modeled through the well-known ALMA model (Gebhard, Reference Gebhard2005) that allows mapping between emotions, mood and personality (see Section 3.3).

As a conclusion, the use of a theoretical approach allows the obtaining of domain-independent solutions and a more realistic modeling of the relationships between emotions, mood and personality. Nevertheless, these relationships need to be more intensively explored. Only the PAD and the Big Five theory have been used to model the personality–mood relationship, but as commented in Section 3, there are others interesting proposals such as Russell and PANAS theories that can be connected to Big Five, or even to another personality theory such as Eysenck (Reference Eysenck1967). The same happens with emotion–mood–personality relationship where OCC–PAD–Big Five has been the only molded through the (Gebhard, Reference Gebhard2005) work. Thus, a formalized approach as well as a more interconnected modeling of personality in BDI agents remain to be done.

5.5 Empathy modeling

It is generally accepted that the modeling of the empathy in intelligent agents is very important in order to improve their human-like behavior, especially on improving human–user interaction. In spite of this, most of the works just model empathy through the OCC emotional theory or based on simple empirical rules, as explained in Section 3.4. In this section, we will comment the two BDI extensions that were found that deal with empathy modeling (see Table 5).

Table 5

The reviewed proposal that model empathy, showing the empathic theory in which they are based on and the empathic emotions they model

Ochs et al. (Reference Ochs, Sadek and Pelachaud2010) are based on empirical and theoretical approaches (Scherer, Reference Scherer1988; Ortony et al., Reference Ortony, Clore and Collins1990; Lazarus, Reference Lazarus1991; Frijda et al., Reference Frijda, Ortony, Sonnemans and Clore1992). They make use of their own emotional model where the emotion generation process is related to a particular event and the emotional intensity corresponds to the emotional impact of the event. For modeling empathy, the intelligent agent has a mental representation of other’s mental state. The agent deduces user’s intentions from the enunciation of user’s communicative acts. Agent’s empathic emotions elicitation depends on the success or failure of user’s intentions: when events that affect user’s intention occur, the agent feels an empathic emotion.

Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011) present a more complete mechanism to model intelligent agents’ empathic emotions. The emotional state and empathic emotions of the agent are represented in the PAD space, and primary and secondary emotions are modeled. The proposal is based on theoretical approaches (Ortony et al., Reference Ortony, Clore and Collins1990; Davis, Reference Davis1994; Hoffman, Reference Hoffman2000; De Vignemont & Singer, Reference De Vignemont and Singer2006) and presents three processes to model empathy: Empathy Mechanism, Empathy Modulation and Expression of Empathy. The Empathy Mechanism is responsible for the outcome of the empathic emotions based on perceived other’s emotional facial expressions. They define a predefined saliency threshold that represents agent’s responsiveness to the other’s situation. If the elicitation of the empathic emotion takes place, it is affirmed as belief and the second and third process occurs. The second process is the modulation of the empathic emotion intensity (Empathy Modulation component) through several predefined factors: mood, personality and social relationship to the other. Finally, the third process (Expression of Empathy) occurs and the empathic emotion expression is triggered.

As a conclusion, empathy modeling is scarcely modeled in BDI extensions and not modeled at all in formalizations: it remains an open research question.

Table 6

The reviewed proposals that model emotional regulation, the regulation theory they apply and the regulation strategies modeled

ER = Emotional regulation; C = Coping

5.6 Regulation modeling

As mentioned before (Section 3.5), there are two concepts related to regulation: coping, directed to negative emotions regulation, and emotional regulation, oriented to both positive and negative emotions. Following that distinction, the reviewed proposals are presented in Table 6. Most of the review proposals in this section are BDI extensions, but also two formalizations can be found: one related to coping, Dastani and Lorini (Reference Dastani and Lorini2012), and the other, Bosse and Lange (Reference Bosse and de Lange2008), related to emotional regulation.

Regarding coping, the proposal of Becker-Asano and Wachsmuth (Reference Becker-Asano and Wachsmuth2010) presents a simple coping mechanism, not based on any theory, that just considers reappraisal in the emotion module. The other three proposals deal with coping in different ways as they are based on different theories. Van Straalen et al. (Reference Van Straalen, Heylen, Theune and Nijholt2009) model coping in bad news scenarios. They propose a categorization of coping strategies based on the theory of Kuübler-Ross (Reference Kuübler-Ross2009) about terminal patients: Denial and Isolation, Anger, Bargaining, Depression, and Acceptance. The emotional BDI proposed by Alfonso et al. (Reference Alfonso, Vivancos and Botti2014) is based on the ideas of Castelfranchi (Reference Castelfranchi and Paiva2000); they propose a coping component that determines if the changes on current mood deserve to take actions in cognitive processes. In those cases, the way intentions are selected is changed using a plan prioritization strategy. In the formalization proposed by Dastani and Lorini (Reference Dastani and Lorini2012), coping modeling is based on Lazarus (Reference Lazarus1991) theory. Three coping strategies are modeled: strategies affecting intentions (removing/generating them), strategies affecting beliefs (increasing/reducing their strength) and strategies affecting goals (increasing/reducing their strength).

Regarding emotional regulation, juts two proposals have been found. De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003) present a simple mechanism of masking emotions formalized with context-dependent rules without explicit theoretical foundation. They consider aspects such as the interlocutors’ personalities, their relationship and the social interaction context. In the case of Bosse and Lange (Reference Bosse and de Lange2008), the regulation process is modeled following Gross (Reference Gross1998) theory. They introduce a parameter that measures the speed to return the ERL to its default state. For example, after an unpleasant event, an agent may stay in an emotional state (i.e., angry) for a long time or leave the event behind quickly, depending on the values of the regulation parameter.

As we see, coping has been modeled far more than emotional regulation. The latter has only been modeled in very simple ways that represent a starting point in this unexplored line of investigation.

5.7 Affective influence on cognitive process

As commented in Section 3.4, nowadays, it is clear that the individual affective state affects the cognitive process. Due to this connection, it is important to consider the affective influence on the cognitive process when modeling intelligent agents.

Only three of the proposals reviewed Adam et al. (Reference Adam, Herzig and Longin2009), Steunebrink et al. (Reference Steunebrink, Dastani and Meyer2012) and Gluz and Jaques (Reference Gluz and Jaques2014) do not consider the affective influence on the cognitive process. All of them are based on the OCC theory that does not describe this emotional influence. The other works implement the affective influence on different steps of the BDI cognitive process. In Table 7, the emotional BDI proposals and the affective influence on the BDI cognitive process are shown.

Table 7

The proposals reviewed and the influence of agent’s emotions (E) or (Other’s) emotion, mood (M), personality(P), empathy (Emp), coping (C) and emotional regulation (ER) on the BDI cognitive processes, as well as the theoretical bases of these influences

The first column shows the theoretical base of the affective influence management. In half of them, the modeling of the affective influence is not based on theoretical foundations but just on intuitive evidences learned from psychological literature. On the contrary, others are based on affective theories well known in intelligent agent modeling such as Castelfranchi and Lorini (Reference Castelfranchi and Lorini2003), Damasio (Reference Damasio1994), Lazarus (Reference Lazarus1991), Loewenstein and Lerner (Reference Loewenstein and Lerner2003) and Frijda (Reference Frijda1987).

As it was explained in Section 3.6, the affective influence can be modeled in several steps of the cognitive process such as beliefs, perceptions, decision-making, memory and behavior. In emotional BDIs, this is done following the steps of the BDI cognitive process (Section 4): perception, belief updating, options generation, intention filtering and actuation. In the proposals reviewed, intention filtering is the most influenced BDI cognitive process. This influence is mainly done by emotions, but in two cases mood and coping alter the process. Another cognitive aspect in which affective influence has been widely modeled is belief updating. In this case, emotions is also the affective aspect that influences the most, but mood, personality and affective capacities (coping and empathy) influences have also been modeled. The other cognitive aspect widely considered in the literature is the motivational state of the agent. In the case of the BDI architecture, this corresponds to the options generation process where the agent’s desires/goals are generated. Even though it is true that emotional influence is also the most considered aspect, this is the only step where the influence of all the affective aspects has been modeled, including affective capacities such as empathy, emotional regulation and coping. Perception (influenced by emotions and mood) and actuation (influenced by emotions, mood and affective regulation) are the processes in which the affective influence has been less considered.

As it has been already commented, emotional influence is the most widely aspect considered in the literature. It is modeled by directly affecting the way perception is achieved, by directing attention toward belief-relevant information, by prioritizing desires and/or by helping to select intentions. In our review, the proposals of Pereira et al. (Reference Pereira, Oliveira, Moreira and Sarmento2005), Jiang et al. (Reference Jiang, Vidal and Huhns2007) and Lejmi-Riahi et al. (Reference Lejmi-Riahi, Kebair and Said2014) stand out. They are the most referenced ones that consider the emotional influence on most of the BDI process. Pereira et al. (Reference Pereira, Oliveira, Moreira and Sarmento2005) take into account the emotional influence on all the cognitive process. Nevertheless, they do not explicitly detail how the influence is implemented: they make an overview on how the emotional state of the agent modulates the decision-making process according to the availability of resources and capabilities (see Section 5.1). In the case of Jiang et al. (Reference Jiang, Vidal and Huhns2007), they take into account the influence of primary and secondary emotions on belief updating, option generation and intention filtering. They use emotions to set the priority of desires and help to decide intentions. To do so, they follow the idea of some researchers (Camerer et al., Reference Camerer, Loewenstein and Rabin2003) that point out that decision-making may actually be driven by emotions, since people with emotional impairments have trouble making decisions. The proposal of Lejmi-Riahi et al. (Reference Lejmi-Riahi, Kebair and Said2014) takes into account the influence of secondary emotions in almost all the BDI cognitive processes. They follow the theory of Loewenstein and Lerner (Reference Loewenstein and Lerner2003) that establishes two ways in which emotions can influence at the time of decision-making: as anticipated consequences (expected emotions) and as feelings experienced at the time of decision-making (immediate emotions). Immediate emotions influence all cognitive processes except actuation. In the case of expected emotions, the influence is modeled in the options generation process. This way, they propose a complex cognitive-affective processing that makes it possible to model other types of behaviors that are not directly product of decision.

Another way to influence decision-making is through a more permanent affective state such as mood. Only three of the reviewed proposals consider this influence. In the case of Bosse and Höhle (Reference Bosse, Höhle, Mehrotra, Mohan, Jae, Varshney and Ali2011) (i.e., the one that presents a formalization), they do not detail the theoretical foundation of their modeling. Mood and emotional influence are modeled in the same way: as rules that are mostly domain-specific and define the impact of emotions and mood on the BDI cognitive process; for instance, a positive mood increases the desire to cooperate with teammates and a negative mood increases the desire to behave aggressively. The other two proposals model mood’s influence independently of emotional influence. Their mood’s influence modeling is based on theoretical foundations and focuses on perception and decision-making (and in memory in the case of Neto and da Silva (Reference Neto and da Silva2012)). Both proposals model the influence on perception based on the same idea: mood can intensify or blur perceptions: depending on agent’s mood, the same sensation may generate different perceptions. Regarding the mood’s influence on decision-making, they model it in different ways. In Neto and da Silva (Reference Neto and da Silva2012) proposal, it is modeled in the updating beliefs and intention filtering processes. This is implemented through emotional markers (Damasio, Reference Damasio1994) that allow tagging actions with emotional states, in such a way that actions can be selected through the corresponding emotional states that tag them, and vice versa. In contrast, Alfonso et al. (Reference Alfonso, Vivancos and Botti2014) only model mood influence on the intention filtering process: if the changes experimented in the current mood deserve to take actions, this will determine the intentions selected.

Regarding personality, there are two proposals (BDI extensions) that model its influence on the BDI cognitive process: De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003) and Jones et al. (Reference Jones, Saunier and Lourdeaux2009). In both proposals, the personality influences the options generation process, but in different ways. In the case of De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003), personality traits are related to certain goal weights directly affecting the intensity of the elicited emotions. In Jones et al. (Reference Jones, Saunier and Lourdeaux2009), they also consider the influence of the agent’s physiological state but it is not detailed. They also model the influence of personality (and emotions) on the belief updating process, in particular, in the way new beliefs are interpreted.

Affective capabilities also influence the BDI cognitive processes. As in precedent sections, we will analyze affective regulation and empathy.

Proposals that consider regulation (coping and emotional regulation) influence have all a strong theoretical foundation. In general, both emotional regulation and coping have direct influence on agent decision-making and final behavior. Coping modeling has been modeled as a mechanism that directly affects the intention filtering and/or actuation process, determining various types of responses. In Alfonso et al. (Reference Alfonso, Vivancos and Botti2014), coping influences the intention filtering process through a mechanism of plan’s prioritization according to the agent’s current mood. In the case of Van Straalen et al. (Reference Van Straalen, Heylen, Theune and Nijholt2009), as their particularly interested in the way the agent responds to bad news, coping alters the actuation process, both influencing the behavior selection and the manifestation of the selected behavior. Dastani and Lorini (Reference Dastani and Lorini2012) present a more complete model. Following the idea that coping can be seen as a cognitive mechanism whose aim is to discharge a certain emotion by modifying one or more of the mental attitudes that triggered the emotion, they consider it in beliefs updating, options generation and intention filtering. This is done by looking over an intention, by changing the agent’s interpretation of the situation, by changing a belief or a goal’s strength.

Regarding emotional regulation influence, just one proposal has been found that models it. In De Rosis et al. (Reference De Rosis, Pelachaud, Poggi, Carofiglio and De Carolis2003), the influence of emotional regulation is in the actuation process: it affects directly agent’s expression, masking elicited emotions. This does not necessarily imply the elimination of an emotion but the variation of its intensity or its duration. In the case of Bosse and Lange (Reference Bosse and de Lange2008), that also consider emotional regulation, they do not model a direct influence of regulation on the agent’s cognitive process. The influence is indirect on the agent ERL, which in turn influences the motivational state of the agent.

The influence of empathy in the cognitive process is modeled in the two proposals that consider empathy modeling (Section 3.5). In both of them, the other’s emotions influence the cognitive process in the perception process. Besides, the agent empathic emotions influence the option generation process in the case of Ochs et al. (Reference Ochs, Sadek and Pelachaud2010) and the belief updating in Boukricha and Wachsmuth (Reference Boukricha and Wachsmuth2011) proposal.

As a conclusion, the BDI cognitive phases influenced and how this influence is modeled varies greatly from one proposal to another. Emotions influence is taken into account in most of the proposals, mood and personality to a lesser extent. The belief updating process and the intention filtering process are the processes most influenced by emotions, mood and, in a lesser extent, by personality. Options generation process is frequently influenced by a combination of some affective aspects such as emotions and mood or emotions and personality. The influence of affective capacities (affective regulation and empathy) is scarcely modeled and remains nowadays an open line of research. In general, the actuation process is the most influenced by coping and emotional regulation as it is related to the final behavior of the agent. Empathy has been modeled to influence the belief processing and the options generation processes.

6.1 Emotions

Emotions are short-lived affective states that arise suddenly in the context of a specific stimulus. Nowadays, it is recognized as mandatory to endow the agent with an emotional model if believability is sought. In most of the proposals, emotions influence the cognitive process in several ways: in perception, motivation and decision-making processes or at least a combination of them.

6.2 Mood

Mood, as a more durable affective state, has been shown to impact a range of cognitive, perceptual and behavioral processes, and the emotional elicitation (mood-congruent emotions). Nevertheless, in many of the proposals, it is just considered as an unattributed affective basis to which emotions decay.

6.3 Personality

Personality is considered a long-term affective state that does not change over time. This way personality can be used to differentiate one agent from another.

6.4 Empathy

Empathy is the capacity to understand or feel what another person is experiencing from within his or her frame of reference. Empathic experience is related to the other’s cognitive-affective state. Empathy should be considered if believable agent–user interaction is sought.

6.5 Regulation

Regulation modeling is focused on coping with stressing situations and regulating both negative and positive emotions. In the proposals that model coping, the regulatory process influences the selection of the agent’s behavior and how it is expressed. On the other hand, the proposals that model emotional regulation are focused on the regulation of the emotional response.