According to the categorical approach in the investigation of facial expressions of emotion, there are six basic categories: fear, anger, sadness, happiness, disgust, and surprise (Ekman, Reference Ekman1994). Happy facial expressions are identified faster and more accurately than any other emotional face when the face stimuli are presented singly (Calvo & Lundqvist, Reference Calvo and Lundqvist2008; Loughead, Gur, Elliott, & Gur, Reference Loughead, Gur, Elliott and Gur2008; Palermo & Coltheart, Reference Palermo and Coltheart2004; Tottenham et al., Reference Tottenham, Tanaka, Leon, McCarry, Nurse, Hare and Nelson2009), and also when they are presented simultaneously with neutral faces (Calvo & Nummenmaa, Reference Calvo and Nummenmaa2009) or with other emotional faces (Calvo & Nummenmaa, Reference Calvo and Nummenmaa2011). Aguado, García-Gutiérrez, and Serrano-Pedraza (Reference Aguado, García-Gutiérrez and Serrano-Pedraza2009) also reported a faster classification of happy than angry facial expressions, with facial happiness being recognized faster in female than in male posers, whereas the opposite occurred with anger.
The advantage in the recognition of happy expressions can be attributed to the diagnostic value or distinctiveness of the smile (Leppänen & Hietanen, Reference Leppänen and Hietanen2007). Morphologically, a smile involves changes in the mouth region of a face, with lip corners turned up and pulled backwards due to the contraction of the zygomaticus major muscle, often accompanied by a raised upper lip and exposed upper teeth (Ekman & Friesen, Reference Ekman and Friesen1978). Distinctiveness or diagnostic value refer to the degree that a facial feature is associated with a particular expressive category and not with others. Whereas facial features in the other expressions overlap to some extent across categories, the smile is systematically and uniquely associated with facial happiness (Calvo & Marrero, Reference Calvo and Marrero2009; Kohler et al., Reference Kohler, Turner, Stolar, Bilker, Brensinger, Gur and Gur2004). Furthermore, whereas a smiling mouth is necessary and sufficient to categorize faces as happy, the eye region makes only a modest contribution (Calder, Young, Keane, & Dean, Reference Calder, Young, Keane and Dean2000; Leppänen & Hietanen, Reference Leppänen and Hietanen2007; Nusseck, Cunningham, Wallraven, & Bülthoff, Reference Nusseck, Cunningham, Wallraven and Bülthoff2008). Such a single feature can thus be used as a shortcut for a quick and accurate categorization of a smiling face as happy.
In addition to being highly distinctive or diagnostic at the semantic or categorical level, the smile is salient or conspicuous at the perceptual level. Visual saliency can be assessed as a combination of purely physical image properties such as local contrast, energy, and spatial orientation (Itti & Koch, Reference Itti and Koch2000; Walther & Koch, Reference Walther and Koch2006). Using computational simulation, the smiling mouth has indeed been found to be more visually salient than any other region of happy and non-happy faces (Calvo & Nummenmaa, Reference Calvo and Nummenmaa2008). By being salient, the smile can successfully compete with other face areas for attentional capture (Calvo & Nummenmaa, Reference Calvo and Nummenmaa2008). High local saliency can account for the recognition advantage of happy faces when they are presented in extrafoveal locations of the visual field (i.e., beyond 2.5° from fixation; Calvo, Nummenmaa, & Avero, Reference Calvo, Nummenmaa and Avero2010; Goren & Wilson, Reference Goren and Wilson2006). As both covert and overt visual attention modulate sensory gain on the basis of the input signal saliency (Itti & Koch, Reference Itti and Koch2000), visual saliency ensures that a smiling mouth is attended earlier and thus becomes more accessible to processing than other facial features, even before the viewer looks at the face or the mouth directly.
In the current study, we investigated (a) whether the smile projects from the mouth to the eye region before the mouth is looked at or in the absence of fixations on the mouth, and (b) whether, as a consequence, the smile can make the eyes look happy, even if they do not really convey happiness. Visual saliency could make the mouth accessible to extrafoveal vision, and thus the smile would be perceived when the mouth is not directly fixated. If so, and given that smiles are highly diagnostic signals of happiness, viewers could be misled to interpret non-happy eyes as happy. The hypothesis that, due to visual saliency, the smiling mouth radiates to adjacent facial regions implies, first, that the smile would be particularly resistant to visual acuity degradation when it appears outside of foveal vision (e.g., when the viewer is looking at the eyes of a smiling face). Second, the influence of the smile on other regions would decrease gradually with increasing distance between the extrafoveal mouth and the currently fixated facial region (e.g., the eyes).
To address the issue of whether the smile radiates outwards to other face regions and makes them look happy, we used the following methodological developments: (a) participants were presented with whole-face stimuli for 150 ms and asked to categorize their eyes as happy or not happy; (b) the viewer’s attention was visually pre-cued to the eye region of the face stimulus, and foveal vision of the mouth region was blocked by means of gaze-contingent masking (i.e., a mask covered the mouth region in case the viewer tried to look at it, although this region remained accessible extrafoveally); (c) we used composite faces with blended expressions with a smiling mouth but non-happy eyes, in addition to intact faces with genuinely happy or non-happy expressions (i.e., congruent mouth and eyes of the same expression); (d) the distance between the eyes and the mouth was varied, such that the mouth was located either close to foveal vision, in parafoveal, or in peripheral vision (relative to the fixated eye region); and (e) we assessed the visual saliency (a combination of contrast, orientation, and energy) of the eye and the mouth regions by means of computational modeling (see below). This paradigm allowed us to investigate the effects of a task-irrelevant and non-fixated mouth on the task-relevant and currently fixated eyes. If, in these conditions, the expression of the distant, overtly unattended, mouth affects the categorization of the eyes, we can infer a smile projection effect.
Three groups of predictions can be made regarding the influence of the smiling mouth. First, the presence of a smile in blended expressions will increase the probability of incorrectly judging non-happy eyes as happy, and slow down reaction times in correctly judging them as not happy, in comparison with when exactly the same eyes appear in genuinely non-happy expressions (with no smile). Second, if there is radiation outward from the mouth, the aforementioned effects of the smiling mouth on the non-happy eyes will diminish gradually as the distance between the currently fixated eyes and non-fixated mouth increases. Third, if visual saliency underlies the projection of the smile on the eyes, the smiling mouth will not only be more salient than the genuinely happy eyes and the mouth of the non-happy expressions, but the smiling mouth will also remain more salient than the non-happy eyes in all the blended expressions.
Method
Participants
Twenty-one psychology undergraduates from La Laguna University (between 19 and 25 years old; 15 female) received course credit for their participation.
Stimuli
We selected 168 digitized color photographs from the KDEF (Lundqvist, Flykt, & Öhman, Reference Lundqvist, Flykt and Öhman1998) stimulus set. The face stimuli portrayed 24 individuals (12 females: KDEF no. 01, 02, 07, 11, 14, 19, 20, 22, 26 29, 31, 35; and 12 males: KDEF no. 03, 05, 06, 10, 11, 12, 22, 23, 24, 25, 31, 35), each posing seven basic expressions (original KDEF: neutral, happiness, anger, disgust, sadness, fear, and surprise). In addition to these intact faces with genuine expressions, we constructed composite faces with six blended expressions for each of the 24 selected KDEF models, thus producing 144 new face stimuli. For these blends, we combined the upper half of each non-happy face with the lower half of the happy face of the same individual, by cutting each face along a horizontal line through the bridge of the nose and smoothing the junction by Adobe® Photoshop® CS5. We produced the following blends: NeHa (Neutral eyes + Happy mouth, i.e., smile), AnHa (Angry eyes + Happy mouth), DiHa (Disgusted eyes + Happy mouth), SaHa (Sad eyes + Happy mouth), FeHa (Fearful eyes + Happy mouth), and SuHa (Surprised eyes + Happy mouth). Figure 1, shows some examples of the genuine and the blended expressions. All the face stimuli were cropped: Nonfacial areas (e.g., hair, neck, etc.) were removed by applying an ellipsoidal mask. The face stimuli were presented against a black background.
Sample of face stimuli with truly happy expressions, non-happy expressions, and blended expressions. Blended expressions: AnHa = angry eyes + smiling mouth (i.e., angry upper part of face with happy lower part of face); DiHa = disgusted eyes + smile; SaHa = sad eyes + smile; FeHa = fearful eyes + smile; SuHa = surprised eyes + smile; NeHa = neutral eyes + smile.
For each face stimulus, we created three different sizes, to manipulate the distance between the mouth and the eyes. To avoid disrupting the overall configuration of the face, the face size was varied proportionally for all its parts. In the smallest format, each face was 4.9 cm wide by 6.5 cm high, thus equaling visual angles of 4.9° × 6.5° at a 60-cm viewing distance. This approximates the size of the real face of a person (13.8 × 18.5 cm) viewed from a distance of 1.80 m. In the intermediate size, each face was 9 cm wide by 12 cm high, which equals 9° × 12° at 60-cm viewing distance. This approximates the size of a real face viewed from a distance of 1 m. Finally, in the larger format, each face was 13.8 cm wide by 18.5 cm high, thus equaling visual angles 13.8 × 18.5 at 60-cm viewing distance. This closely approximates the size of a real viewed from a distance of 63 cm. The eye-mouth distances (from center to center) for each face stimulus size were, respectively: (a) 2.6°, where the mouth remained in the lower boundaries of parafoveal (i.e., close to foveal) vision, (b) 4.8°, which corresponds to the farther boundaries of parafoveal vision, or (c) 7.0°, where the mouth was in peripheral vision (see Wandell, Reference Wandell1995).
Apparatus and Procedure
The stimuli were presented on a 21” monitor with a 120-Hz refresh rate. Participants’ eye movements were recorded with an Eyelink II tracker (2004). The sampling rate of the eyetracker was 500 Hz and the spatial accuracy was better than 0.5°, with a 0.01° resolution in pupil tracking mode. Participants had their head positioned on a chin and forehead rest at a 60-cm viewing distance.
At the beginning of the session, participants were told that faces with different expressions would be presented briefly, preceded by a cue rectangle on the area where the eyes of the face would appear. They were instructed to look only at the eyes, ignore other areas, and respond as quickly and accurately as possible whether the eyes conveyed a happy expression or not, by pressing a left or a right button. Performance was measured by the probability of judging the eyes as happy, and by reaction times for correct responses to faces with happy eyes (i.e., eyes judged as ‘happy’) and for correct rejections (i.e., eyes judged as ‘not happy’) to expressions with non-happy eyes.
Each participant was presented with 312 randomly ordered trials (24 of each of the genuine and the blended categories). See Figure 2 for an overview of the trial structure. Each trial began with a central drift correction circle (0.5°) as a fixation point. When the participant fixated this circle, a bright cue rectangle (only the outline) appeared for 250 ms on the upcoming eye region location. The cue comprised an area between the outer edges of both eyes and eyebrows of the face stimulus. Upon the cue offset, the face appeared for 150 ms. Gaze-contingent masking was used throughout the face display (see Calvo & Nummenmaa, Reference Calvo and Nummenmaa2007): If the viewer initiated a saccade away from the cued eye region towards the mouth, a black mask covered the bottom half of the face, thus the mouth always remained inaccessible to foveal vision.
Sequence of events and overview of basic characteristics of a trial.
Design
A fully within-subjects factorial design combined eye expression (6: angry, disgusted, sad, fearful, surprised, and neutral), type of mouth (2: congruent non-happy vs. incongruent smile), and distance between the eyes (the middle point between both eyes) and the center of the mouth (3: 2.6° vs. 4.8° vs. 7.0°). In the congruent condition, the mouth was always non-happy and belonged to the same expression as the eye region (i.e., genuine non-happy expressions). In the incongruent condition, the mouth involved a smile but the eyes conveyed a non-happy expression (i.e., blended expressions). A condition involving truly happy faces (with happy eyes and a smile) was compared with each of the other categories. Eye-mouth distance was blocked, with all the faces of the same size presented in the same block, and block order was counterbalanced.
Ideally, we could have included a control condition in which the upper part (with the eye region) of a face was shown, while the lower part (with the mouth region) was masked by means of Fourier-phase scrambling. In this condition, no expressive information could be obtained from the mouth region, although low-level properties of this region would be retained. We have, nevertheless, included this control in additional research (Fernández-Martín, Avero, & Gutiérrez-Calvo, Reference Fernández-Martín, Avero and Gutiérrez-Calvo2012; Calvo, Fernández-Martín, & Nummenmaa, Reference Calvo, Fernández-Martín and Nummenmaa2013). In these studies, (a) the probability of responding that non-happy eyes were “happy” was higher, and (b) correct rejection times were slower, in the presence of a visible smiling mouth (both when it was fixated and when it remained extrafoveally) than in the aforementioned control condition where there was no visible mouth (either foveally or extrafoveally). This allowed us to attribute the effects to the smiling mouth, and consider such condition as not strictly necessary for the current study.
Assessment of Visual Saliency
We obtained the saliency values for each of five face regions (forehead, eye/eyebrows, cheek /nose, mouth, and chin) by means of the iLab Neuromorphic Vision C+ Toolkit (iNVT; Itti & Koch, Reference Itti and Koch2000; see also Walther & Koch, Reference Walther and Koch2006). This algorithm simulates which features in a given image attract attention as a function of physical image properties (local contrast, orientation, and energy), by mimicking the response properties of retinal neurons, lateral geniculate nucleus, thalamus, and V1. Such features are then integrated for a neural saliency map that is a graded representation of the visual conspicuity of each pixel in the image. After computing the saliency map for the face, average saliency values were extracted for the two most expressive areas, i.e., mouth and eyes. In our face stimuli, each of these regions subtended a 2.4° vertical visual angle.
Results
Eye-movement Data
Eye movement recordings revealed that all first fixations on the face stimulus landed within the cued (i.e., eye) region, only one fixation was made during the 150-ms display, and no saccades were launched towards the mouth region. This confirms that the task-relevant eye region was always looked at and, importantly, that any potential influence of the mouth region on the judgment of the eye expression occurred without overt attention to the mouth, with the mouth in parafoveal vision.
Probability of Categorization of Eye Expressions as “Happy”
For all the comparisons that involved multiple contrasts (i.e., the eye expression condition, and the distance condition), both for the probability of categorization responses and also for reaction times of correct responses (see below), we used Bonferroni corrections (the significance level was set at p < .05). Mean probability scores are shown in Table 1. In a 6 (eye expression) × 2 (mouth expression) × 3 (eye-mouth distance) ANOVA on the probability of judging the eye expression as happy, the main effects of eye expression, F(5, 100) = 46.63, p < .0001, ηp Footnote 2 = .700, and mouth expression, F(1, 20) = 349.32, p < .0001, ηp Footnote 2 = .946, were qualified by interactions between mouth expression and distance, F(2, 40) = 5.28, p < .01, ηp Footnote 2 = .209, and between mouth and eye expression, F(5, 100) = 13.61, p < .0001, ηp Footnote 2 = .405.
Mean Probability of Responding that the Eye Expression was Happy, as a function of Eye-Mouth Distance and Type of Expression
Note:
AnHa: angry eyes with happy mouth; DiHa: disgusted eyes with happy mouth; SaHa: sad eyes with happy mouth; FeHa: fearful eyes with happy mouth; SuHa: surprised eyes with happy mouth; NeHa: neutral eyes with happy mouth.
To decompose the mouth by distance interaction, we conducted separate one-way (3: distance) ANOVAs for each mouth condition. For blended expressions (incongruent smile condition), the probability of judging the non-happy eyes as happy decreased as a function of eye-mouth distance, F(2, 40) = 4.59, p < .025, ηp Footnote 2 = .187, with significant differences between the shortest and the longest distance levels. There was an inverse linear relationship between the eye-mouth distance (2.6° vs. 4.8° vs. 7.0°) and the probability of classifying the non-happy eyes as happy (67 vs. 62 vs. 56%; R Footnote 2 = .99). In contrast, for genuine non-happy expressions (congruent condition), the evaluation of the eye expression remained unaffected by viewing distance (see Figure 3). Differences in performance were always statistically significant between the blended and the genuine non-happy expressions at all three distance levels (all pairwise comparisons, ts (20) > 5.0, ps < .0001).
Mean probability scores (in percentage) in responding that the eye expression was happy, and reaction times (in ms) of correct responses, as a function of eye-mouth distance and type of mouth. Means with a different superscript (a and b) are significantly different.
To decompose the mouth by eye expression interaction, we first calculated difference scores between the blended-expression categories and the corresponding genuine expressions (i.e., blended–genuine). Footnote 1 These scores indicated how much the smiling mouth increased the probability of evaluating each type of eye expression as happy. In a one-way ANOVA (6: eye expression), a significant effect appeared, F(5, 100) = 13.61, p < .0001, ηp Footnote 2 = .405, on the difference scores. Post hoc comparisons showed that the neutral eyes were the most likely to be influenced by a smiling mouth, whereas the angry eyes were the least likely to be affected, with significant differences between the angry eyes and the rest, and between the neutral eyes and the rest (difference scores: M = .39, .50, .52, 55, .58, .72, for angry, fearful, disgusted, sad, surprised, and neutral eye expressions, respectively).
To compare the truly happy face category with the non-happy and the blended expression categories (see Table 1), we conducted a 13 (expression: happy, angry, sad, disgusted, fearful, surprise, neutral, AnHa, SaHa, DiHa, FeHa, SuHa, and NeHa) × 3 (eye-mouth distance) ANOVA on the probability of categorizing the eyes as happy. A main effect of expression, F(12, 240) = 169.47, p < .0001, ηp Footnote 2 = .894, showed that the truly happy eyes were more likely to be judged as happy than the eyes of all the other categories. For happy faces, the probability of evaluating the eye expression as happy was equivalent in all three distance conditions (see Figure 3).
Reaction Times of Correct Responses
Mean latencies of correct responses are shown in Table 2. In a 6 (eye expression) × 2 (mouth expression) × 3 (eye-mouth distance) ANOVA on correct response latencies, main effects of eye expression, F(5, 100) = 6.33, p < .0001, ηp Footnote 2 = .240, mouth expression, F(1, 20) = 23.40, p < .0001, ηp Footnote 2 = .539, and a mouth expression by distance, F(2, 40) = 5.66, p < .01, ηp Footnote 2 = .221, emerged. Reaction times were significantly longer for the surprised and the neutral eyes (both, M = 697 ms) than for disgusted, angry, and sad eyes (M = 650, 652, 669, respectively), with fearful eyes (M = 681) being equivalent to the rest. Also, non-happy eyes took longer to be judged as not happy in the presence of an incongruent smiling mouth (M = 711 ms) than in the presence of a congruent non-happy mouth (M = 638).
Mean Reaction Times (in ms) for Correct Responses, as a function of Eye-Mouth Distance and Type of Expression
Note:
AnHa: angry eyes with happy mouth; DiHa: disgusted eyes with happy mouth; SaHa: sad eyes with happy mouth; FeHa: fearful eyes with happy mouth; SuHa: surprised eyes with happy mouth; NeHa: neutral eyes with happy mouth.
Separate one-way (3: distance) ANOVAs for each mouth condition, revealed that, for blended expressions (incongruent smile condition), response latencies in judging the eyes as not happy decreased as a function of eye-mouth distance, F(2, 40) = 3.28, p < .05, ηp Footnote 2 = .141, with significant differences between the shortest and the longest distance level. There was an inverse linear relationship between the eye-mouth distance (2.6° vs. 4.8° vs. 7.0°) and reaction times in classifying the non-happy eyes as not happy (739 vs. 703 vs. 691 ms; R Footnote 2 = .92). In contrast, for non-happy expressions (congruent condition), reaction times were not affected by viewing distance (see Figure 3). Differences in reaction times were always statistically significant between the blended and the genuine non-happy expressions at the shortest, t(20) = 5.51, p < .0001, the intermediate, t(20) = 4.05, p < .001, and the longest distance, t(20) = 2.36, p < .05.
To compare the truly happy face category with the non-happy and the blended expressions (see Table 2), we conducted a 13 (expression: happy, angry, sad, disgusted, fearful, surprise, neutral, AnHa, SaHa, DiHa, FeHa, SuHa, and NeHa) × 3 (eye-mouth distance) ANOVA on the latencies of correct responses. A main effect of expression, F(12, 240) = 13.49, p < .0001, ηp Footnote 2 = .403, showed that the truly happy eyes were correctly categorized faster than the eyes of all the other categories, except those of genuinely angry, sad, disgusted, and fearful expressions. For happy faces, the probability of evaluating the eye expression as happy was equivalent in all three distance conditions (see Figure 3).
Analysis of Visual Saliency
A 2 (configuration: non-happy vs. blended) × 6 (expressive eyes: angry, disgusted, sad, fearful, surprised, and neutral) × 2 (region: eyes vs. mouth) ANOVA was performed on the saliency values of the eye and the mouth regions. See the mean scores for each category in Table 3, and the average scores for the non-happy, the blended, and the happy expressions, in Figure 4. Main effects of configuration, F(1, 276) = 59.98, p < .0001, ηp Footnote 2 = .179, and region, F(1, 276) = 591.16, p < .0001, ηp Footnote 2 = .682, were qualified by a configuration by region interaction, F(1, 276) = 262.89, p < .0001, ηp Footnote 2 = .488, and further by a three-way interaction F(1, 276) = 2.87, p < .025, ηp Footnote 2 = .049. The mouth was more salient than the eyes for all the blended expressions, t(143) = 51.15, p < .0001, d = 5.67, r = .94, and also for the non-happy faces, t(143) = 4.26, p < .0001, d = 0.54, r = .26, except those with sad and neutral expressions (ps > .18). The mouth-eye difference was much greater for the blended than for the non-happy expressions, as revealed by the corresponding Cohen’s d and effect-size r values: While the percentage in saliency that was accounted for by region was 88.5% (i.e., the square of the effect-size r value) for the blended faces, it was only (though still significant) 6.8% for the non-happy faces. The contrast between the mouth and the eye region for happy faces yielded also a highly significant difference, t(23) = 22.90, p < .0001, d = 6.69, r = .96 (91.8% of variance).
Visual Saliency Values of the Mouth and the Eye Regions for each Facial Expression of Face Stimuli
Note:
AnHa: angry eyes with happy mouth; DiHa: disgusted eyes with happy mouth; SaHa: sad eyes with happy mouth; FeHa: fearful eyes with happy mouth; SuHa: surprised eyes with happy mouth; NeHa: neutral eyes with happy mouth. Means with a different superscript horizontally are significantly different; means with the same superscript or no superscript are equivalent.
Mean visual saliency of the eye and the mouth regions for happy, blended, and non-happy facial expressions. Means with a different superscript (a and b, for the mouth region; x and y, for the eye region) are significantly different; means with the same superscript are equivalent.
The eye and the mouth saliency scores for happy faces were compared with those for the non-happy and the blended expressions in separate one-way (7: expression) ANOVAS for the eye and the mouth region (see Table 3 and Figure 4). For the mouth region, in the comparison of the non-happy vs. the happy faces, effects of expression emerged, F(6, 161) = 17.53, p < .0001, ηp Footnote 2 = .395. The mouth of happy faces was more salient than that of all the non-happy faces. In contrast, in the comparison of blended expressions and happy faces, all the categories with a smile had equivalent mouth saliency (F < 1). For the eye region, in the comparison of the non-happy and the happy faces, effects of expression, F(6, 161) = 3.26, p < .01, ηp Footnote 2 = .108, showed only lower scores for happy than for surprised and sad faces. In the comparison of the blended and the happy faces, no significant difference appeared (F < 1).
Discussion
Three major findings appeared in the current study. First, the presence of a smile in blended expressions increased the probability of incorrectly judging non-happy eyes as happy, and slowed down reaction times in correctly judging non-happy eyes as not happy, in comparison with when exactly the same eyes appeared in genuinely non-happy expressions. Second, these biasing effects of the smile on the eye expression varied as a function of the eye-mouth distance: The closer the non-fixated smiling mouth was located in relation to the currently fixated eyes, the greater the biasing effect (i.e., the tendency to judge non-happy eyes as happy in the incongruent smile condition). Third, the smiling mouth was more salient than the mouth of all the other expressions and all the eyes with which it was combined in the blended expressions; importantly, the presence of a smiling mouth made non-happy eyes less salient than when they appeared in their respective non-happy faces Footnote 2 .
These findings support the existence of a mechanism by which a salient smile radiates and projects to other parts of the face, thus making them look happy. Visual saliency (a combination of physical stimulus properties such as local contrast, energy, and spatial orientation) is proposed to underlie this mechanism Footnote 3 . Visual saliency affects the initial shifts of covert and overt attention (Torralba, Oliva, Castelhano, & Henderson, Reference Torralba, Oliva, Castelhano and Henderson2006). Even when the smiling mouth appears in extrafoveal areas of the visual field, the high saliency of the smile makes it resistant to visual acuity degradation, and so it remains accessible to covert attention (Calvo et al., Reference Calvo, Nummenmaa and Avero2010; Goren & Wilson, Reference Goren and Wilson2006). This extrafoveal perception advantage of the smile is consistent with another characteristic of the mechanism, i.e., radiation. The closer the smiling mouth is to the current fixation (on the eyes), the more perceptually accessible would the smile be. The pattern we have found regarding the linear relationship between eye-mouth distance and the biasing influence of the smile in blended expressions fits perfectly with this concept of visual radiation. The smile radiation intensity is assumed to decrease gradually the farther away from the current fixation the mouth appears, and hence the influence of the smile on the eyes diminishes in nearly the same proportion.
Importantly, in the current study, the smile effects on the eye expression occurred in the absence of fixations on the mouth while the eyes were under visual inspection. This suggests that the salient and “radiant” smile projected to the eyes. Projection implies that a task-irrelevant and distant mouth affects the perception of a task-relevant and overtly attended region (e.g., the eyes). As a consequence of projection of a diagnostic smiling mouth, non-happy eyes conveyed an impression of happiness to the viewer. This role of saliency, radiation, and projection of the smile is relevant to the debate about configural versus featural processing of facial expressions (Aguado et al., Reference Aguado, García-Gutiérrez and Serrano-Pedraza2009; Bimler & Paramei, Reference Bimler and Paramei2006; Calder et al., Reference Calder, Young, Keane and Dean2000; Tanaka, Kaiser, Butler, & Le Grand, Reference Tanaka, Kaiser, Butler and Le Grand2012). Faces are mainly processed configurally (e.g., Calder & Jansen, Reference Calder and Jansen2005; Richler, Mack, Gauthier, & Palmieri, Reference Richler, Mack, Gauthier and Palmieri2009), that is, they are coded as unitary objects in a representation that combines the different parts of the face. The projection of happiness from a smiling mouth towards the eye region is consistent with the view that facial expressions are perceived as integrated objects, where a salient feature (e.g., the smiling mouth) can bias the processing of other face parts. Projection (from non-fixated regions) would be a requirement for configural processing.
Our proposal of a saliency-radiation-projection mechanism is not intended to be an alternative hypothesis to the holistic or configural processing conceptualization. Rather, our proposed mechanism is compatible with and adds to this conceptualization by specifying some functions underlying holistic processing. More specifically, faces would be processed holistically because of projection of the non-fixated parts towards the fixated part. This is demonstrated when the whole face cannot be foveally fixated in a single glance—because it is larger in size than the area covered by a fixation and there is time only for one fixation (Richler et al., Reference Richler, Mack, Gauthier and Palmieri2009)—and yet perception of the fixated region is influenced by the content of non-fixated regions. This was exactly the case in the current study. Furthermore, the projection mechanism would be particularly relevant for expressions in which there is a visually salient feature, with saliency making the feature radiate to surrounding facial areas. The saliency explanation can thus be integrated with the holistic account. Salient features (e.g., the smiling mouth) would automatically attract covert attention (even if overt attention to them is blocked, and before overt attention could take place). As a consequence, a salient feature would be especially likely to be integrated in the holistic processing of a face. If such a salient feature is, in addition, highly diagnostic of a facial expression, it will be likely to bias the judgment of the overtly attended features (and possibly the whole face).
We have argued that the smiling mouth radiates to adjacent regions because of saliency, that radiation becomes gradually more intense as the mouth is closer to the current fixation, and that this explains the increasing influence of the smile on the eyes. By increasing the eye-mouth distance, we also increased the size of all the face parts proportionally, to preserve the overall face configuration. Prior research has shown that an increase in an image stimulus size reduces holistic processing of the face (McKone, Reference McKone2009), because the different face areas and features are further apart, and so perceptual integration becomes more difficult. This is also compatible with our findings and the proposed explanation. If the smiling mouths affects the eye expression through projection, and this involves integrative processing of distant parts, the smaller the face is (i.e., in the shorter eye-mouth distance conditions), the easier holistic processing will be and, therefore, the stronger the projection influence of the mouth. Accordingly, the rather metaphorical concepts of “radiation” and “projection” fit within the configural processing approach in face recognition.
In conclusion, a smile enhances the tendency to judge non-happy eyes as happy, even when only the eyes are directly looked at and the mouth remains in extrafoveal areas of the visual field. This suggests that the influence of the smile occurs through a projection mechanism of configural face processing. Projection is based on the high visual saliency of the smiling mouth, with saliency making the smile perceptually accessible to extrafoveal vision. The biasing influence of the smile on the eye expression increases gradually with decreasing eye-mouth distance. This allows us to infer that saliency makes the smile radiate and thus become more accessible the closer it is to the current fixation on the eyes. The eye expression, nevertheless, modulates to some extent the smiling mouth saliency effects in blended expressions: Angry eyes are more resistant than those of other expressions to the biasing influence of the smile, whereas neutral—non-expressive—eyes are the most vulnerable to this influence Footnote 4 .
These findings have practical implications for the discrimination between genuine and fake smiles (Ambadar, Cohn, & Reed, Reference Ambadar, Cohn and Reed2009; Johnston, Miles, & McCrae, Reference Johnston, Miles and Macrae2010; Krumhuber & Manstead, Reference Krumhuber and Manstead2009). First, as long as a smile appears in a face with non-happy eyes, an observer will be led to some extent to perceive (possibly wrongly) the person who smiles as happy, and will need additional time to decide that the smile is not genuine. Second, such a biasing influence will be greater when the person who smiles is farther away from the viewer—thus with a shorter distance (in degrees of visual angle) between the eyes and the mouth of the smiler—than when the smiler is closer. In general, this makes us wonder about the degree of truth in the folk belief that the eyes are “a window to the soul” or a “mirror of the soul”. At least, the eye expression is not impervious to projection effects from the smile. Rather, a smile modifies the eye expression, and could thus hide the smilers’ actual motivational and emotional state—if not their soul.
