1. Introduction
Gesture contributes meaning to spoken utterances. Sometimes this contribution is supplementary, as when a speaker says ‘I wrote a letter’ while producing a typing gesture (Obermeier et al., Reference Obermeier, Holle and Gunter2011); other times it is central, such as when a speaker says ‘This one’ while pointing at an object. Production studies show that speakers use gestures to disambiguate homonyms (Holler & Beattie, Reference Holler and Beattie2003), and perception studies show that such gestures are used to infer the intended meaning (Holle & Gunter, Reference Holle and Gunter2007).
What is known as the Next Wednesday Question (NWQ) provides one of the clearest demonstrations of the disambiguating power of gesture. When presented in a written version, the question ‘Next Wednesday’s meeting has been moved forward two days, on what day is it now?’ is ambiguous and elicits both ‘Monday’ and ‘Friday’ responses (McGlone & Harding, Reference McGlone and Harding1998). The ambiguity arises because the spatial adverb forward can either be interpreted from an ego-moving perspective, analogous to ‘We are approaching the meeting’, or a time-moving perspective, analogous to ‘The meeting is approaching us’ (Clark, Reference Clark and Moore1973; Moore, Reference Moore2006; Nuñez et al., Reference Nuñez, Motz and Teuscher2006). Across studies, about half of all participants favor each interpretation (Stickles & Lewis, Reference Stickles and Lewis2018), although substantial individual differences have been documented (Duffy & Feist, Reference Duffy and Feist2014; Duffy et al., Reference Duffy, Feist and McCarthy2014; Feist & Duffy, Reference Feist and Duffy2023; Li, Reference Li2024; Li & Cao, Reference Li and Cao2021), and results may be more or less biased toward either ‘Monday’ or ‘Friday’ responses depending on the precise way the question is formulated (Feist & Duffy, Reference Feist and Duffy2015). Figure 1 illustrates the underlying concept of the ego-moving and time-moving perspectives in the NWQ paradigm.
Schematic illustration of the ego-moving and time-moving perspectives.
Figure 1. Long description
The diagram consists of two horizontal panels separated by a line. Both panels feature a timeline of eight calendar tiles labeled Sat 07, Sun 08, Mon 09, Tue 10, Wed 11 (blue), Thu 12, Fri 13 (purple or orange), and Sat 14. The word PAST is on the top-left and FUTURE is on the top-right.
* Top Panel (Moving Ego): A stick figure is depicted running from left to right. The figure is currently positioned over Sun 08. Two arrows point rightward from Mon 09 toward Wed 11 and then toward Fri 13. A faint group icon sits above Wed 11, and a dark group icon sits above Fri 13, suggesting the observer is moving toward future events.
* Bottom Panel (Moving Time): A stick figure stands stationary on Sun 08. Two arrows point leftward from Fri 13 toward Wed 11 and then toward Mon 09. A dark group icon is positioned above Mon 09, while a faint group icon is above Wed 11. In this view, Mon 09 is highlighted in orange, suggesting that time or events are moving toward the stationary observer.
Previous research has shown that the ego-moving/time-moving ambiguity of the NWQ can be partially resolved by gesture. Jamalian and Tversky (Reference Jamalian and Tversky2012) demonstrated that a sagittal gesture directed away from the speaker substantially increases the proportion of ‘Friday’ responses, and vice versa, a gesture directed toward the speaker increases the proportion of ‘Monday’ responses. This finding has subsequently been replicated numerous times (Lewis & Stickles, Reference Lewis and Stickles2017; Tversky & Jamalian, Reference Tversky and Jamalian2021; Winter & Duffy, Reference Winter and Duffy2020). The effect size is substantial: although never reaching 100% ‘Monday’/‘Friday’ responses with all participants, the gesture can shift responses from near-full ambiguity (at the population level: 50%) to roughly 80% ‘Monday’ or 80% ‘Friday’ responses.
Now that we know that gesture matters in resolving the perspectival ambiguity of the NWQ, we can ask new questions. The NWQ provides an ideal test case because a directional gesture could plausibly relate both to the motion event encoded by ‘moved’ and to the directional adverb ‘forward’. The question is no longer whether gesture matters in this task, but when and where gesture is integrated with speech. In this study, we use the NWQ as a test bed for studying how the coordination of gestural, verbal and prosodic elements impacts semantic integration, and ultimately ambiguity resolution. Rather than asking whether gesture influences interpretation, we can ask how gesture is integrated with the prosodic and verbal dimensions of the NWQ, and how this depends on the relative timing of different elements with respect to each other.
2. Background
Except for emblems, which are highly conventionalized gestures such as ‘thumbs up’, gesture meaning is rarely interpreted independently of concomitant spoken language. Most co-speech gestures are semantically dependent on accompanying speech (Kendon, Reference Kendon2015). Experiments demonstrate that while emblems can often be understood in isolation (Gunter & Bach, Reference Gunter and Bach2004), iconic gestures are typically underspecified without linguistic context (Feyereisen et al., Reference Feyereisen, Van de Wiele and Dubois1988; Hadar & Pinchas-Zamir, Reference Hadar and Pinchas-Zamir2004; Krauss et al., Reference Krauss, Morrel-Samuels and Colasante1991). A circular hand movement, for example, could refer to any round object, a circular path, or a cycle, among many other things. When the movement is combined with ‘We did it again’, however, it becomes clear that it signals repetition.
This dependence of gesture on speech has also been demonstrated in the NWQ paradigm. Winter and Duffy (Reference Winter and Duffy2020) found that replacing ‘moved’ with ‘changed by’ eradicated any effect of gesture, i.e., sagittal forward and backwards movements did not meaningfully affect the proportion of ‘Monday’ or ‘Friday’ responses. They argued that when speech no longer evokes movement, listeners are less likely to interpret the gesture as depicting motion along a timeline. More generally, the evidence that gesture influences the interpretation of spoken utterances (Beattie & Shovelton, Reference Beattie and Shovelton1999; Graham & Argyle, Reference Graham and Argyle1975; Holle & Gunter, Reference Holle and Gunter2007; Kelly et al., Reference Kelly, Barr, Church and Lynch1999; McNeill et al., Reference McNeill, Cassell and McCullough1994) and that spoken utterances influence the interpretation of gestures (Hadar & Pinchas-Zamir, Reference Hadar and Pinchas-Zamir2004; Krauss et al., Reference Krauss, Morrel-Samuels and Colasante1991) shows that speech and gesture interact by mutually constraining one another during meaning construction (Cassell et al., Reference Cassell, McNeill and McCullough1999; Kelly et al., Reference Kelly, Özyürek and Maris2010, Reference Kelly, Healey, Özyürek and Holler2015).
This process of semantic integration fundamentally depends on the temporal coordination of speech and gesture: successful integration is strongest when gestures overlap with, or slightly precede, the corresponding verbal element (Habets et al., Reference Habets, Kita, Shao, Özyurek and Hagoort2011; Obermeier et al., Reference Obermeier, Holle and Gunter2011). This mirrors production patterns, where gesture strokes are tightly synchronized – usually partially overlapping or slightly preceding – with the semantically related element in speech (Butterworth & Beattie, Reference Butterworth, Beattie, Campbell and Smith1978; Kendon, Reference Kendon1980; Leonard & Cummins, Reference Leonard and Cummins2011; McNeill, Reference McNeill1992; Morrel-Samuels & Krauss, Reference Morrel-Samuels and Krauss1992). However, the NWQ presents multiple potential candidates for such integration. A directional gesture could plausibly relate to the motion event encoded by the verb ‘moved’, or the directional information encoded by the adverb ‘forward’. Determining which of these elements serves as the primary anchor for gesture interpretation in the NWQ is a central goal of this study (Experiments 1–3).
A further goal is to test how this temporal coordination interacts with prosody. Gestural and prosodic prominences are closely coordinated in production (Esteve-Gibert & Prieto, Reference Esteve-Gibert and Prieto2013; Fung & Mok, Reference Fung and Mok2018; Jannedy & Mendoza-Denton, Reference Jannedy and Mendoza-Denton2005; Loehr, Reference Loehr2012; for review, see Wagner et al., Reference Wagner, Malisz and Kopp2014). However, previous gesture studies on the NWQ either did not report how stimuli were realized prosodically (Jamalian & Tversky, Reference Jamalian and Tversky2012; Lewis & Stickles, Reference Lewis and Stickles2017; Tversky & Jamalian, Reference Tversky and Jamalian2021), or did not systematically manipulate prosody (Winter & Duffy, Reference Winter and Duffy2020). This raises the possibility that the effect of gesture on ambiguity resolution may not only depend on semantic affiliation, but also on alignment with prosodic prominence; e.g., it could make a great difference to how the NWQ is interpreted if the primary pitch accent of the sentence is placed on the verb ‘moved’ or the adverb ‘forward’. We therefore investigate both dimensions simultaneously (Experiment 3).
3. Different accounts of gesture integration in the NWQ
Across three experiments, we manipulated the temporal alignment of gesture, speech and prosody. In our conditions, gestures either coincided with the verb ‘moved’, coincided with the spatial adverb ‘forward’ or occurred after the target sentence containing the primary directional information (‘Next Wednesday’s meeting has been moved forward two days [gesture]. On what day is it now?’) has been completed (post-target; Experiments 1–2). In Experiment 3, we additionally manipulated whether the primary pitch accent of the sentence falls on the verb or the adverb.
We designed and conducted the experiments in sequence. In Experiment 1, the gesture-on-adverb condition necessarily included the adverb, but the NWQ in the verb and post-speech conditions excluded the adverb. After realizing that the baseline probability of responding Friday, regardless of gesture direction, was much higher for both conditions without the adverb, we realized that leaving out the adverb fundamentally changes the nature of the NWQ from a genuine ambiguity to complete underspecification. The effect of gesture placement cannot be interpreted independently of the presence of the adverb ‘forward’ – without this adverb, direction is not a meaningful part of the question at all. To deconfound gesture placement and adverb presence, Experiment 2 filled in the missing gesture-on-verb and post-speech conditions that also included the adverb. This means that taken together, Experiments 1 and 2 allow us to disentangle the relative contributions of gesture placement and adverb presence. Experiment 3 then proceeded with investigating gesture placements with pitch accents using a version of the NWQ that always included the adverb.
As discussed in this section, these manipulations across Experiments 1–3 allow us to test different accounts of gesture integration, which we dub 1) adverb anchoring, 2) verb anchoring, 3) gesture priming, 4) gesture salience and 5) prosodic anchoring. These accounts are introduced below.
3.1. Adverb anchoring
A first possibility is that gesture is primarily integrated with the element that is directly implicated in the ambiguity of the NWQ, the adverb ‘forward’. The spatial ambiguity inherent in the NWQ is most closely related to whether ‘forward’ is interpreted as moving forward in an ego-moving manner (away from speaker) or a time-moving manner (toward speaker). Indeed, Winter and Duffy (Reference Winter and Duffy2020) and Lewis and Stickles (Reference Lewis and Stickles2017) seem to have tacitly assumed that the spatial adverb would be the most relevant gestural anchor by coordinating their directional gestures with this element (Jamalian & Tversky, Reference Jamalian and Tversky2012 and Tversky & Jamalian, Reference Tversky and Jamalian2021 conducted their experiments face-to-face and spoke of alignment with the entire verb phrase, leaving it open where the gesture was placed exactly). If directional information is indeed the key integration point, gesture should have its strongest influence with ‘forward’. The effect of forward/backward gestures should weaken when gestures are shifted to the verb, or to the post-target position following the question.
3.2. Verb anchoring
A second possibility is that gesture is integrated with the primary motion event described by the sentence, i.e., the verb. Production studies find gestures to be most common with verbs (Hadar & Krauss, Reference Hadar and Krauss1999; Wilding et al., Reference Wilding, Winter, Littlemore and Perlman2025). Close alignment between the directional gesture and the verb is also expected under the Gesture-as-Simulated-Action framework (Hostetter & Alibali, Reference Hostetter and Alibali2008, Reference Hostetter and Alibali2019), which views gestures as externalized motor simulations of actions and events. Applied to the NWQ, this account leads us to expect that the verb ‘moved’ constitutes the primary integration point: because the gesture depicts movement, it should have the strongest influence when aligned with the verb rather than the adverb. The post-target gesture condition should have the weakest effect because here, the gesture follows the entire sentence and is therefore outside of the verb phrase.
3.3. Gesture priming
A third possibility is that earlier elements bias the understanding of later elements, i.e., gesture influences interpretation through a priming mechanism where the spatial direction implied by the gesture influences how following elements are understood. Previous work has shown that gestures can prime semantically related concepts (So et al., Reference So, Yi-Feng, Yap, Kheng and Yap2013; Vainiger et al., Reference Vainiger, Labruna, Ivry and Lavidor2014; Wu & Coulson, Reference Wu and Coulson2005, Reference Wu and Coulson2007; Yap et al., Reference Yap, So, Melvin Yap, Tan and Teoh2011). The idea that priming matters is also suggested by previous work on the NWQ itself. Boroditsky (Reference Boroditsky2000) and following this, many other studies (e.g., Boroditsky & Ramscar, Reference Boroditsky and Ramscar2002; Matlock et al., Reference Matlock, Ramscar and Boroditsky2005; Sullivan & Barth, Reference Sullivan and Barth2012) found that spatial primes, such as images showing forward movement versus images of pulling something toward oneself, prime ego-moving and time-moving perspectives when hearing the NWQ, as shown by a relative difference in ‘Monday’/‘Friday’ proportions between these priming conditions. From the perspective of our current study, the directional gestures might behave just like the spatial primes used in these experiments. Since our experiment does not feature a gesture preceding the NWQ, our experiment is not set up to differentiate gesture priming from verb anchoring. Both lead us to expect that gesture effects are strongest on the verb condition, weaker on the adverb and weakest in the post-target condition.
3.4. Gesture salience
A fourth possibility focuses on how salient a gesture is. What makes a gesture salient is presumably influenced by many different factors such as a gesture’s size and velocity, whether unusual hand shapes are used and so forth. One dimension of salience that we can easily manipulate in a way that is analogous to the verb/adverb placement is to have a gesture occur on its own. This is analogous to Ebert and Ebert (Reference Ebert and Ebert2014) and Schlenker (Reference Schlenker2018), who analyze co-speech gestures as conveying meaning that is less central to a message than post-speech gestures (for review, see Ebert, Reference Ebert2024). An account rooted in gesture salience would predict that a directional gesture following the NWQ would have the strongest impact on ‘Monday’/‘Friday’ responses because it stands out as an independent communicative signal.
3.5. Prosodic anchoring
The fifth and final possibility is that semantic integration depends on alignment with prosodic prominence. Because pitch accents signal prominence (Baumann & Röhr, Reference Baumann and Röhr2015; Baumann & Winter, Reference Baumann and Winter2018; Terken & Hermes, Reference Terken, Hermes and Horne2000; Turnbull et al., Reference Turnbull, Royer, Ito and Speer2017 for a review), listeners may attend more strongly to gestures that coincide with accented elements. Extensive research in production shows that gestures have a strong probabilistic tendency to occur together with pitch accents or other prosodically prominent elements (Baills & Baumann, Reference Baills and Baumann2025; Esteve-Gibert & Prieto, Reference Esteve-Gibert and Prieto2013; Fung & Mok, Reference Fung and Mok2018; Gregori & Kügler, Reference Gregori and Kügler2026; Im & Baumann, Reference Im and Baumann2020; Jannedy & Mendoza-Denton, Reference Jannedy and Mendoza-Denton2005; Loehr, Reference Loehr2012; Türk & Calhoun, Reference Türk and Calhoun2024; for an early review, see Wagner et al., Reference Wagner, Malisz and Kopp2014). If prosodic anchoring matters, we would expect forward/backward gestures to influence ‘Monday’/‘Friday’ responses when gestures coincide with the accented constituent, regardless of whether this constituent is the verb or the adverb. Experiment 3 crosses gesture placement (on verb, on adverb) with pitch accent placement (on verb, on adverb). In this 2 × 2 design, if the prosodic anchor were the key point for semantic integration, we would predict a match effect, i.e., an interaction between gesture position and pitch accent position.
Table 1 summarizes the predictions of the different accounts. Experiments 1 and 2 include a post-target condition and allow us to test the first four possibilities. Experiment 3 is set up to specifically test the contribution of prosody, but by virtue of having two gesture placement conditions (verb, adverb), it also provides converging evidence for the relative contribution of these elements.
Expectations about the influence of gesture on ‘Monday’/‘Friday’ responses for different accounts of semantic integration in the NWQ
Table 1. Long description
The table consists of three columns: Account, Proposed integration point, and Expectation. It contains five rows of data:
* Adverb anchoring: Proposed integration point is the spatial adverb ‘forward’. Expectation is Adverb > verb > post-target.
* Verb anchoring: Proposed integration point is the verb ‘moved’. Expectation is Verb > adverb > post-target.
* Gesture priming: Proposed integration point is earlier gesture biases later interpretation. Expectation is Verb > adverb > post-target.
* Gesture salience: Proposed integration point is independent post-speech gesture. Expectation is Post-target > adverb is approximately verb.
* Prosodic anchoring: Proposed integration point is alignment with pitch accent. Expectation is Match > mismatch.
3.6. Additional factors: gesture awareness and likeability
Across our experiments, we also asked whether participants were aware of the gesture, i.e., whether they consciously noticed them. If the gesture was indeed more salient in the post-target condition, participants should be more aware of it. The effect of the forward/backward gesture may overall be stronger for participants who consciously noticed the gesture.
In the very first gesture study on gestures in the NWQ, Jamalian and Tversky (Reference Jamalian and Tversky2012) asked participants the question while producing forward/backward gestures standing next to the participant. This disambiguates another dimension of perspective, not temporal perspective (ego-moving vs. time-moving), but whether the participant takes the speaker’s orientation with respect to gesture in general. When looking at the gesturer face-to-face, a speaker’s forward gesture is actually a movement toward the listener. This means that a participant assuming the speaker’s perspective should resolve the ambiguity toward ‘Friday’, and a participant not assuming their perspective should resolve it toward ‘Monday’. Whether participants take the speaker’s perspective is thus a latent factor that potentially weakens gestural effects when conducting the experiment online with video stimuli where the gesture cannot be produced standing next to the speaker, as in Jamalian and Tversky (Reference Jamalian and Tversky2012).
One way of probing this latent factor is via likeability, with the linking assumption that participants would be more willing to take the speaker’s perspective if they like them. This linking assumption was motivated by the results of an open-ended post-experiment debriefing question reported in Winter and Duffy (Reference Winter and Duffy2020) (Experiment 1), which showed that participants spontaneously reported very strong (dis)preferences for the speaker seen in the video stimulus. To address this possible latent factor, Winter and Duffy (Reference Winter and Duffy2020) included a standard likeability scale (Reysen, Reference Reysen2005) in their Experiments 2–4. Results showed that within each experiment, the impact of likeability on the influence of directional gestures was too subtle to pass conventional thresholds of certainty, but when entered into a combined analysis, there was enough evidence for a small but consistent effect: speakers who liked the model seen in the video more were relatively more strongly affected by gestural direction. In this study, all three experiments include a questionnaire item on likeability. This allows us to assess whether this element of Winter and Duff’s study (Reference Winter and Duffy2020) replicates.
4. Methods
As all three experiments use the same general question and gestures, we introduce their methods together.
4.1. Procedure
We used the online questionnaire SoSci Survey (Leiner, Reference Leiner2021) to collect responses to the NWQ in three pre-registered experiments in a between-participants single-trial design. The pre-registration of all studies can be found on OSF: https://osf.io/p8svw/overview.
After consenting, participants first performed a test to see whether their audio and video worked. After this, they answered three demographic questions: age, gender and which languages they spoke. After a simple attention check (15 + 7 =?), participants read the following instructions before the main trial:
On the next screen, you will see a video of a person asking a question. Focus on the screen and listen to what the person has to say. Please play the video only once and respond quickly and intuitively with whatever comes to mind first. Try to not overthink and do not change your answer: we are interested in your initial reaction.
After clicking continue, participants saw a video of a speaker producing different versions of the NWQ (see Stimuli in Section 4.2). They then entered their response into an open-ended text entry box that was displayed beneath the video on the same page. We also asked whether participants were consciously aware of the gesture, and whether they learned best with visual, auditory or haptic input (this dimension was not analyzed in the current study). In addition, they rated how much they liked the speaker on a scale from 1 to 7 in Experiment 1 (see Winter & Duffy, Reference Winter and Duffy2020). In Experiments 2 and 3, the simple one-dimensional likeability question was replaced by four questions from the Reysen likeability scale (Reysen, Reference Reysen2005), for which participants were asked whether they found the speaker in the video likeable, approachable, warm and friendly. For Experiments 2 and 3, we also added an open-ended entry box where participants could enter their thoughts on the experiment and give feedback.
The data collection for all experiments was performed on a Wednesday, as has been done in the original experiment by McGlone and Harding (Reference McGlone and Harding1998) and because Winter and Duffy (Reference Winter and Duffy2020) found a day-of-the-week effect, where participants responded ‘Friday’ or ‘Monday’ more often the closer the day of testing was to either one of these days.
4.2. Stimuli
Across all experiments, we used the following formulation of the NWQ: ‘Next Wednesday’s meeting has been moved (forward) two days. On what day is it now?’ This question was asked by a female British English native speaker and always paired with a gesture, which across all conditions was a one-handed ‘flat wall’ handshape that was moved along the sagittal plane in front of the speaker’s body (cf. Winter & Duffy, Reference Winter and Duffy2020 Experiment 1 vs. 2), as shown in Figure 2. The gesture was timed such that its stroke was produced simultaneously to a target word – either the verb ‘moved’ or the adverb ‘forward’ – or following the target sentence, i.e., ‘has been moved (forward) two days [gesture]. On what day is it now?’ Preparatory and retraction phases were performed as felt natural to the speaker. This entails a retraction movement after the gesture stroke that co-occurred with speech. Winter and Duffy (Reference Winter and Duffy2020) argue that this could create interference, since it means that the retracting hands are seen as moving backwards into the speaker’s lap after having executed a forward movement. To avoid having movements into different directions within the same stimuli, Winter and Duffy (Reference Winter and Duffy2020) used a post-stroke hold at the end of the forward or backward movement (Experiments 2–4). Our speaker, however, reported that the post-stroke hold felt very unnatural to her. We therefore accepted a retraction movement during the sentence to increase the naturalness of the stimuli.
Illustration of the gesture used in the stimuli. The example illustrates a forward gesture: (1) preparatory upward movement, (2) stroke phase with a sagittal movement away from the speaker, (3) apex of the gesture, and (4) retraction back to the resting position.
Across the three experiments, the gesture’s direction and position, as well as the position of the sentence accent, were varied. All experiments included the 2-level factor Gesture Direction: forward or backward. In Experiment 1, the question was produced in one of six conditions (= 2 Gesture Direction conditions × 3 Gesture Position conditions), as shown in Figure 3. The factor Gesture Position involved co-timing the gesture with the adverb ‘forward’, the verb ‘moved’, or an empty pause following the target sentence before the main question. The verb and post-target conditions did not feature an adverb in the verbal component. This decision was made to assess the direct disambiguating function of the gesture, without any disambiguating cue in speech. However, this meant that Experiment 1 also introduced an additional factor, Adverb Presence (adverb/no adverb). To assess the pure influence of Gesture Position, we additionally conducted Experiment 2 (see Figure 4). By and large, this is a replication of Experiment 1, except that the verb and post-target conditions now also featured the adverb ‘forward’. In Experiments 1 and 2, the main pitch accent of the sentence was placed on the adverb except for the sentences that did not feature an adverb, in which case it was placed on the verb.
Experimental conditions of Experiment 1. Gesture Direction (forward/backward) is varied across three Gesture Positions (adverb/verb/post-target). Syllables in bold and capitals carry the main pitch accent; [] parentheses indicate the gesture position.
Figure 3. Long description
The table is titled Experiment 1. At the top, a prompt sentence reads: Next Wednesday’s meeting has been moved forward two days. On what day is it now?
The table is organized into three rows representing different Gesture Positions:
1. Position Adverb: The sentence fragment is ‘... has been moved [FORward] sub g e s two days.’ The word FOR is in bold capitals, and the word forward is enclosed in brackets with a sub g e s subscript.
2. Position Verb (no adverb): The sentence fragment is ‘... has been [MOVed] sub g e s two days.’ The word MOV is in bold capitals, and the word moved is enclosed in brackets with a sub g e s subscript.
3. Position post-target (no adverb): The sentence fragment is ‘... has been MOVed two days [ ] sub g e s.’ The word MOV is in bold capitals, and empty brackets with a sub g e s subscript follow the word days.
At the bottom, a horizontal axis labeled Direction contains two categories: forward gesture and backward gesture.
Experimental conditions of Experiment 2. Gesture Direction (forward/backward) is varied across three Gesture Positions (adverb/verb/post-target). Syllables in bold and capitals carry the main pitch accent; [] parentheses indicate the gesture position.
Figure 4. Long description
At the top, the prompt reads: Next Wednesday’s meeting has been moved forward two days. On what day is it now?
The table contains three rows for Position:
1. Position Adverb: The sentence fragment is ‘... has been moved [FORward] sub g e s two days.’ with the gesture occurring on the adverb.
2. Position Verb (with adverb): The sentence fragment is ‘... has been [moved] sub g e s FORward two days.’ with the gesture occurring on the verb.
3. Position post-target (with adverb): The sentence fragment is ‘... has been moved FORward two days [ ] sub g e s.’ with the gesture occurring after the target phrase.
In all examples, the syllables F O R in forward are bolded and capitalized to indicate pitch accent. The bottom row indicates that these three positions are tested across two Directions: forward gesture and backward gesture.
Experiment 3 zoomed in on the role of prosody and applied a 2 × 2 × 2 design with the following factors: Gesture Direction as in Experiments 1 and 2, Gesture Position (adverb/verb) and Accent Position, for which the main pitch accent of the sentence was produced either on the adverb or the verb. The combination of Gesture Position and Accent Position yielded matching and mismatching accent-gesture combinations (see Figure 5). There were two types of matches: either gesture and accent on the adverb, or gesture and accent on the verb. The resulting mismatch conditions had the gesture either precede the accent (gesture-accent) or follow it (accent-gesture).
Experimental conditions of Experiment 3. Gesture Direction (forward/backward) is varied across two Gesture Positions (adverb/verb) and two Accent Positions (adverb/verb). Syllables in bold and capitals carry the main pitch accent; [] parentheses indicate the gesture position.
Figure 5. Long description
The table is titled Experiment 3. At the top, a prompt reads: Next Wednesday’s meeting has been moved forward two days. On what day is it now?
The main body of the table contains four rows representing different conditions:
* Row 1: Match Adverb. The sentence fragment is: ... has been moved [FORward] sub g e s two days. The word FORward is in bold capitals and enclosed in brackets.
* Row 2: Match Verb. The sentence fragment is: ... has been [MOVed] sub g e s forward two days. The word MOVed is in bold capitals and enclosed in brackets.
* Row 3: Mismatch accent-gesture. The sentence fragment is: ... has been MOVed [forward] sub g e s two days. MOVed is in bold capitals, while forward is in lowercase and enclosed in brackets.
* Row 4: Mismatch gesture-accent. The sentence fragment is: ... has been [moved] sub g e s FORward two days. moved is in lowercase and enclosed in brackets, while FORward is in bold capitals.
The bottom row of the table is labeled Direction and is split into two columns: forward gesture and backward gesture.
We recorded the stimuli for all experiments at the Phonology Lab of Goethe University Frankfurt am Main. When asking our speaker to produce stimuli with specific gestures, it is only natural that there are slight variations in the precise timing of speech and gesture with each utterance. We therefore collected multiple different trials within each condition and subsequently annotated the stimuli in ELAN (The Language Archive, 2024; Wittenburg et al., Reference Wittenburg, Brugman, Russel, Klassmann, Sloetjes, Calzolari, Choukri, Gangemi, Maegaard, Mariani, Odijk and Tapias2006) following the M3D annotation system (Rohrer et al., Reference Rohrer, Tütüncübasi, Florit-Pons, Vilà-Giménez, Esteve-Gibert, Ren-Mitchell, Shattuck-Hufnagel and Prieto2025). This superset of stimuli was pared down by finding the specific videos that were best aligned, i.e., where the gesture was produced as close as possible to the word we aimed to target for a specific condition. This allowed us to maximize experimental control while keeping stimulus production as natural as possible. It also helped to make sure that there were no extraneous cues (e.g., eyebrow or head movements) that were prominent in the videos. Figure 6 illustrates the annotation of one stimulus with a well-positioned gesture. We used Praat (Boersma & Weenink, Reference Boersma and Weenink2024) for word, syllable, f0-max and f0-min annotations to check accent placement and to pick the most natural but most similar accent contours across conditions. The stimuli we used were produced with L+H* or H* pitch accents within the ‘Tone and Break Indices’ system for British English accent annotation (ToBI, Silverman et al., Reference Silverman, Beckman, Pitrelli, Ostendorf, Wightman, Price, Pierrehumbert and Hirschberg1992).
Example of annotations to check the alignment of gestures with the respective constituents for choice of stimuli with PraatPicture (Puggaard-Rode, Reference Puggaard-Rode2024). A part of the stimulus file is shown here with an L+H* accent on the verb ‘moved’ (capitalized syllable), while the gesture stroke (capitalized) occurs on the adverb.
Figure 6. Long description
The visualization is divided into six horizontal panels aligned to a shared time axis from 0 to 1.977 seconds.
* Top Panel (Waveform): Displays the amplitude of the speech signal. The highest peaks occur during the capitalized syllable MOV in the word moved.
* Second Panel (Spectrogram and f 0): A grayscale spectrogram showing frequency in H z from 0 to 10000 on the left y-axis. A red pitch contour (f 0) is overlaid, measured from 120 to 350 H z on the right y-axis. The pitch rises sharply during the verb moved and falls during the adverb forward.
* Third Panel (Accent Tier): Contains two labels. V is aligned with the verb moved, and Adv is aligned with the adverb forward.
* Fourth Panel (Word Tier): Segments the speech into four blocks: MOVed, forward, two, and days.
* Fifth Panel (Gesture Tier): A single long block labeled Gesture unit spans from the middle of the word MOVed to the end of the word days.
* Bottom Panel (GesturePhase Tier): Subdivides the gesture unit into four phases: preparation (aligned with the end of MOVed), STROKE (aligned with forward), hold (aligned with two), and recovery (aligned with days).
Vertical dotted lines extend from the waveform through the spectrogram to mark the boundaries of the accented segments.
4.3. Participants
Participants were recruited via the platform Prolific (Prolific, 2014) and received a reimbursement of 0.60£ (approximately 0,71€) for their participation. We applied the following screening criteria: Participants had to be adults (18+) with no vision or hearing impairment, under the age of 70, they had to be based in the UK or the US, speak English as a first language and have a Prolific approval rate of at least 92%. The experiment was made available only to participants using a computer or tablet (phone participation was not allowed). As all experiments followed a between-participants design, we made sure that participants who took part in one condition could not participate in any other condition across all studies using Prolific exclusion filters. We excluded data points for which the attention question was not answered correctly, or for which responses to the target question were something other than uniquely identifiable ‘Monday’ or ‘Friday’ responses, such as ‘Tuesday’, ‘Wednesday’, ‘Monday or Friday’ or ‘Just give the new date’.
Three hundred and two participants were recruited for Experiment 1, with 28 participants being excluded (9.3% of total sample, four because of failing to answer the attention question correctly and 24 because of not responding either ‘Monday’ or ‘Friday’). This resulted in a total of 274 valid trials for Experiment 1 (48 forward gesture × adverb, 47 forward gesture × verb, 46 forward gesture × post-speech, 46 backward gesture × adverb, 43 backward gesture × verb, 44 backward gesture × post-speech). The sample consisted of 177 female, 96 male and 1 non-binary participant. The median age of the sample was 38 years (mean = 40.4, range = 18–70). Of these, 34 participants indicated that they speak one or more languages in addition to English.
For Experiment 2, 303 participants were recruited and 34 participants were excluded (11.2%, 6 because of the attention question and 28 because of not responding ‘Monday’ or ‘Friday’), yielding 269 valid trials (45 forward gesture × adverb, 40 forward gesture × verb, 46 forward gesture × post-speech, 46 backward gesture × adverb, 45 backward gesture × verb, 47 backward gesture × post-speech). The sample consisted of 112 female and 157 male participants (median age = 43 years, mean = 43.4, range = 18–69). Of these, 47 participants indicated that they speak one or more languages in addition to English.
Four hundred and twenty participants were recruited for Experiment 3, of which 73 were excluded (16.8%, 3 because of the attention question, 70 because of not responding ‘Monday’ or ‘Friday’), yielding 347 valid trials (41 forward gesture × match adverb, 41 forward gesture × match verb, 41 forward gesture × accent-gesture, 45 forward gesture × gesture-accent, 48 backward gesture × match adverb, 40 backward gesture × match verb, 47 backward gesture × accent-gesture, 44 backward gesture × gesture-accent). The sample consisted of 162 female and 185 male participants (median age = 43.5 years, mean = 43.9, range = 19–70). Of these, 50 participants indicated that they speak one or more languages in addition to English.
4.4. Statistical analyses
We conducted all statistical analyses with R version 4.4.2 (R Core Team, 2024). The package tidyverse version 2.0.0 (Wickham et al., Reference Wickham, Averick, Bryan, Chang, McGowan, François, Grolemund, Hayes, Henry, Hester, Kuhn, Pedersen, Miller, Bache, Müller, Ooms, Robinson, Seidel, Spinu and Yutani2019) was used for data processing; brms version 2.22.0 (Bürkner, Reference Bürkner2017) was used to fit Bayesian models. Analyses and code are available here: https://osf.io/p8svw/overview
We used Bayesian logistic regression models to analyze the proportion of ‘Friday’ responses as a function of the following predictors: Experiment 1 included the predictors Gesture Direction (forward/backward) and Gesture Position (gesture on adverb/verb/post-target). In Experiment 2, the predictors were Gesture Direction and Gesture Position (gesture on adverb/verb/post-target). In Experiment 3, the predictors were Gesture Direction, Gesture Position (gesture on adverb/verb) and Accent Position (accent on adverb, verb).Footnote 1 Combining the factors Gesture Position and Accent Position allowed us to analyze the matches or mismatches of gesture and prosody. Correspondingly, we ran separate models considering the Match (match, mismatch) and the (Mis)match Position (match adverb, match verb, mismatch acc-ges, mismatch ges-acc) as predictors.
We additionally wanted to see whether we could replicate the likeability effect found in Winter and Duffy (Reference Winter and Duffy2020), using the simple Likert scale in Experiment 1 and the sum of the Reysen likeability scale (Reysen, Reference Reysen2005) in Experiments 2 and 3. For ease of presentation and consistency with Winter and Duffy (Reference Winter and Duffy2020), we discuss proportions of ‘Monday’/‘Friday’ responses in relation to Gesture Direction with respect to an interaction with Likeability as a binary factor (median split into a higher and a lower likeability group). Winter and Duffy (Reference Winter and Duffy2020) did not find likeability effects within each experiment, but only in a conjoined analysis that aggregated results across their Experiments 2–4. We therefore also performed a conjoined analysis where z-scored likeability ratings from Experiments 1–3 was split into a higher and a lower likeability group, over which we then ran a model on the interaction between Likeability and Gesture Direction. Further, we ran a logistic regression model on the influence of Gesture Position in interaction with Gesture Awareness, based on participants’ self-reported information on whether they noticed the gesture in the video or not.
Given that the outcome variable for all main Bayesian logistic regression models was the proportion of ‘Friday’ responses, positive coefficients for a given condition indicate an increase in ‘Friday’ responses; negative coefficients indicate an increase in ‘Monday’ responses. For all models, we used weakly informative priors on the condition effects with a Student’s t-distribution with v = 5, μ = 0 and σ = 2.5. In all experiments, all 2-level factors were deviation-coded (−0.5, +0.5) and 3- and 4-level factors were treatment-coded, with releveled models run to capture all comparisons. We ran models with 8 Markov Chain Monte Carlo (MCMC) chains with 8,000 iterations each (2,000 warmup), which resulted in 48,000 post-warmup samples. Convergence diagnostics indicate that all models were estimated reliably (all R̂ values = 1.0; no divergent transitions). We report the probability of a coefficient having the same sign, based on the posterior distribution for that coefficient. Effects were considered robust if the 95% credible intervals (95% CrI) did not include 0. In addition, we used model comparison using Leave-One-Out cross-validation (LOO-CV) to test for interaction effects.
To explore the effect magnitude of the Gesture Direction factor in each factor level of the remaining factors, we calculated the absolute difference in proportion between the forward gesture and the backward gesture in each other factor level. We then used Cohen’s h to calculate effect magnitude by running the test over the difference between forward and backward direction proportions. A bigger magnitude thus indicates that the Gesture Direction effect is stronger at a factor level of one of the other factors, for instance, in a certain Gesture Position.
5. Results
Table 2 shows the proportion of ‘Monday’ and ‘Friday’ responses per experiment. Over all experiments combined, ‘Monday’ and ‘Friday’ responses were given approximately half and half.
Overall proportions of ‘Friday’ and ‘Monday’ responses for each experiment
Table 2. Long description
The table consists of four columns: Experiment number, Data points, Friday responses, and Monday responses.
* Experiment 1: 274 data points, with 182 Friday responses (66.4 percent) and 92 Monday responses (33.6 percent).
* Experiment 2: 269 data points, with 108 Friday responses (40.1 percent) and 161 Monday responses (59.9 percent).
* Experiment 3: 347 data points, with 163 Friday responses (47.0 percent) and 184 Monday responses (53 percent).
* Total: 890 data points, with 453 Friday responses (50.9 percent) and 437 Monday responses (49.1 percent).
We order our results section as follows. First, we look at the marginal effects of Gesture Direction (Section 5.1) and Gesture Position (Section 5.2), each time ignoring all other effects. Next, we look at gesture-accent matches and mismatches in Experiment 3 (Section 5.3). We then address response differences across conditions in Section 5.4. Finally, we report on Likeability and Gesture Awareness effects in Section 5.5.
5.1. Gesture direction effect
In this section, we look at Gesture Direction without considering any of the other effects. In all experiments, the forward gesture led to more ‘Friday’ responses, and the backward gesture led to relatively more ‘Monday’ responses, as shown in Figure 7.
Gesture Direction effects for each experiment; proportions of ‘Friday’ (purple) and ‘Monday’ (orange) responses per Gesture Direction (forward/backward); the horizontal line visualizes the 50% mark.
Figure 7. Long description
A multi-panel figure with three stacked bar charts labeled Experiment 1, Experiment 2, and Experiment 3. The y-axis represents Proportion from 0.00 to 1.00 with a horizontal reference line at 0.50. The x-axis for each chart compares forward gesture and backward gesture. Orange represents Monday and purple represents Friday.
* Experiment 1 (N = 274). For forward gesture, Monday is approximately 0.18 and Friday is 0.82. For backward gesture, Monday is approximately 0.51 and Friday is 0.49.
* Experiment 2 (N = 269). For forward gesture, Monday is approximately 0.46 and Friday is 0.54. For backward gesture, Monday is approximately 0.73 and Friday is 0.27.
* Experiment 3 (N = 347). For forward gesture, Monday is approximately 0.39 and Friday is 0.61. For backward gesture, Monday is approximately 0.66 and Friday is 0.34.
Across all experiments, backward gestures consistently result in a higher proportion of Monday responses compared to forward gestures.
In Experiment 1, 83.0% (N = 117) of participants responded ‘Friday’ after seeing a video with a forward gesture, in contrast to 48.9% (N = 65) who responded ‘Friday’ when seeing a backward gesture. The corresponding Bayesian logistic regression model indicated a strong effect of Gesture Direction (logit = +2.19, SE = 0.37, 95% CrI: [1.50, 2.93]), with a high probability of being of the same sign (p(b > 0) = 1.0). In Experiment 2, 45.8% (N = 71) of participants responded ‘Friday’ when seeing the forward gesture and 26.8% (N = 101) responded ‘Friday’ when seeing the backward gesture. The corresponding logistic regression model also indicated a strong effect of Gesture Direction (logit = +1.18, SE = 0.26, 95% CrI: [0.67, 1.70]), with a high probability of being of the same sign (p(b > 0) = 1.0). In Experiment 3, 60.7% of the participants (N = 102) responded ‘Friday’ in the forward gesture condition, compared to only 34.1% (N = 61) in the backward gesture condition. As with the other experiments, the corresponding logistic regression model indicated a strong effect of Gesture Direction (logit = +1.15, SE = 0.23, 95% CrI: [0.70, 1.59]), with a high probability of being of the same sign (p(b > 0) = 1.0).
5.2. Gesture position effect
The effects of the Gesture Position factor without considering any of the other factors serve the purpose to assess whether the gesture’s placement alone is crucial for its role in shifting the proportion of ‘Monday’/‘Friday’ responses, by addressing whether a certain position evokes stronger or weaker disambiguating effects. Results for Gesture Position are illustrated in Figure 8. In Experiment 1, participants responded ‘Friday’ much less often with the gesture in adverb position than in the other two conditions. However, recall that in Experiment 1, the verb and post-target conditions did not include an adverb (see Sections 3 and 4.2), and thus the difference between the adverb and the other conditions could be due to the absence of the adverb. For this reason, we do not report inferential statistics for Gesture Position in Experiment 1.Footnote 2
Gesture Position effects for each experiment; proportions of ‘Friday’ (purple) and ‘Monday’ (orange) responses per gesture position (adverb/verb/post-target in Experiments 1 and 2). Experiments 1 and 2 contain information about adverb presence; the horizontal line visualizes the 50% mark.
Figure 8. Long description
The chart is divided into three panels labeled Experiment 1, Experiment 2, and Experiment 3. The y-axis represents Proportion from 0.00 to 1.00, with a horizontal reference line at 0.50. Orange represents Monday and purple represents Friday.
* Experiment 1 (N = 274) has three bars. The adverb with A D V bar shows Monday at approximately 0.63. The post-target no A D V bar shows Monday at approximately 0.22. The verb no A D V bar shows Monday at approximately 0.15.
* Experiment 2 (N = 269) has three bars. The adverb with A D V bar shows Monday at approximately 0.54. The post-target with A D V bar shows Monday at approximately 0.61. The verb with A D V bar shows Monday at approximately 0.65.
* Experiment 3 (N = 347) has two bars. The adverb bar shows Monday at approximately 0.47. The verb bar shows Monday at approximately 0.59.
In all bars, the remaining proportion up to 1.00 is filled by the purple Friday segment.
In Experiment 2, when adverb presence was kept consistent, 46.2% of participants (N = 42) responded ‘Friday’ with the gesture in adverb position, 35.3% (N = 30) responded ‘Friday’ with the gesture in verb position and 38.7% (N = 36) responded ‘Friday’ with the gesture in post-target position. The corresponding logistic regression models did not indicate a reliable effect of Gesture Position (post-target–adverb: logit = −0.33, SE = 0.31, 95% CrI: [−0.93, 0.27]; post-target–verb: logit = +0.13, SE = 0.32, 95% CrI: [−0.50, 0.77]; adverb–verb: logit = +0.46, SE = 0.32, 95% CrI: [−0.17, 1.09]).
In Experiment 3, we compared only the adverb and verb positions. 52.5% of participants (N = 93) responded ‘Friday’ when they saw a gesture in adverb position, compared to 41.2% (N = 70) when they saw a gesture in verb position. The corresponding logistic regression model indicated a relatively weak Gesture Position effect (logit = +0.55, SE = 0.23, 95% CrI: [0.10, 1.00]), with a high probability of being of the same sign (p(b > 0) = 0.99).
To address interactions with gesture direction, an interaction model in Experiment 2 showed an interaction between Gesture Direction and Gesture Position in the post-target and adverb condition (logit = +1.64, SE = 0.61, 95% CrI: [0.46, 2.87]), with a high probability of being of the same sign (p(b > 0) = 1). The model also found an interaction in the post-target and verb condition of which the credible interval barely included zero (logit = +1.18, SE = 0.63, 95% CrI: [−0.04, 2.43]), but which had a high posterior probability of being of the same sign (p(b > 0) = 0.97). Using LOO-CV, we compared a Gesture Direction × Gesture Position model versus one without interactions. It showed that while the interaction model was slightly more predictive, the difference was small with a high standard error, which indicates that the model without interactions has a similar predictive accuracy to the interaction model (elpd difference = −2.1, SE difference = 2.8). In Experiment 3, a model without interactions outperforms a Gesture Direction × Gesture Position model in a LOO-CV comparison (elpd difference = −0.6, SE difference = 0.8), showing that the model without interactions is more predictive.
5.3. (Mis)match of gesture and accent (Experiment 3)
The gesture-accent (mis)matches in Experiment 3 showed that accent position only plays a role in disambiguation in relation to the gesture’s position. We combined the Gesture Position and Accent Position factors to assess the effects of a binary Match factor (match: accent and gesture on the same constituent; mismatch: accent and gesture on different constituents) in relation to Gesture Direction, as shown in Figure 9. While the Gesture Direction effect was visible (i.e., more ‘Friday’ responses with forward gestures, see Section 5.1), no big difference in responses between matches or mismatches was apparent. With the forward gesture, participants responded ‘Friday’ in 62.2% (N = 51) of the cases when gesture and accent matched, and similarly, they responded ‘Friday’ in 60.0% (N = 51) of the cases when the two mismatched. With the backward gesture, 34.1% of participants (N = 30) responded with ‘Friday’ in the match conditions and also 34.1% of participants (N = 31) responded with ‘Friday’ in the mismatch conditions. Model comparison via LOO-CV showed that a model without the Gesture Direction × Match interaction outperformed a model with interactions (elpd difference = −1.0, SE difference = 0.3), which is in line with the lack of any noteworthy differences between matching and mismatching conditions reported above. Apart from confirming the Gesture Direction effect (see Section 5.1 above), the no-interaction model showed that the Match factor had no effect on responses (logit = +0.06, SE = 0.22, 95% CrI: [−0.37, 0.50]).
Interaction of Match and Gesture Direction effects in Experiment 3; proportions of ‘Friday’ (purple) and ‘Monday’ (orange) responses per match condition (match/mismatch) and gesture direction; the horizontal line visualizes the 50% mark.
Figure 9. Long description
A stacked bar chart titled Responses by Match x Gesture Direction with a sample size N = 347. The vertical Y axis is labeled Proportion and ranges from 0.00 to 1.00 in increments of 0.25. A horizontal reference line is drawn at the 0.50 mark. A legend on the right identifies orange as Monday and purple as Friday. The chart is divided into two main sections.
1. Forward gesture section on the left contains two bars:
- Match bar: Monday orange is approximately 0.38 and Friday purple is approximately 0.62.
- Mismatch bar: Monday orange is approximately 0.41 and Friday purple is approximately 0.59.
2. Backward gesture section on the right contains two bars:
- Match bar: Monday orange is approximately 0.66 and Friday purple is approximately 0.34.
- Mismatch bar: Monday orange is approximately 0.66 and Friday purple is approximately 0.34.
In the forward gesture condition, Friday responses are more frequent, while in the backward gesture condition, Monday responses are more frequent.
To investigate the influence of the timing of the accent relative to the gesture, we then assessed whether results differed as a function of whether the gesture and accent occurred on the verb or on the adverb by addressing the influence of (Mis)Match Position on responses, as shown in Figure 10. In the matching cases, 49.4% of participants (N = 44) responded ‘Friday’ when the match occurred on the adverb and 45.7% (N = 37) responded ‘Friday’ when the match was on the verb. In the mismatching cases, 55.7% of participants (N = 49) responded ‘Friday’ when the accent preceded the gesture and 37.1% (N = 33) responded ‘Friday’ when the gesture preceded the accent. The corresponding logistic regression models showed only one reliable effect, namely the difference between accent-gesture and gesture-accent conditions (logit = +0.85, SE = 0.32, 95% CrI: [0.24, 1.48]), with a high probability of being of the same sign (p(b > 0) = 1). All other levels did not show reliable differences.
Match Position effects in Experiment 3; proportions of ‘Friday’ (purple) and ‘Monday’ (orange) responses per match position (match adverb/match verb/mismatch accent-gesture/mismatch gesture-accent); the horizontal line visualizes the 50% mark.
Figure 10. Long description
The y-axis is labeled Proportion and ranges from 0.00 to 1.00 with a horizontal reference line at 0.50. The x-axis contains four categories. A legend on the right indicates that orange represents Monday and purple represents Friday. Total sample size is N = 347.
* In the match adverb condition, the orange Monday bar and purple Friday bar are equal, both meeting at the 0.50 line.
* In the match verb condition, the orange Monday bar is slightly above the 0.50 line, approximately at 0.55, with the purple Friday bar making up the remainder.
* In the mismatch accent-gesture condition, the orange Monday bar is below the 0.50 line, approximately at 0.45, with the purple Friday bar occupying the larger top portion.
* In the mismatch gesture-accent condition, the orange Monday bar is significantly above the 0.50 line, approximately at 0.63, with the purple Friday bar occupying the smaller top portion.
5.4. Gesture direction effect magnitude
Table 3 shows Cohen’s h values for the effect of Gesture Direction (forward–backward) across all other conditions. The effect is rather large across most conditions, with Experiment 1 having the largest Gesture Direction effects overall, and Experiment 3 having medium-sized effects. The strongest Gesture Direction effect was obtained in the post-target position without adverb in Experiment 1 (Cohen’s h = 1.14); the weakest effect was obtained in the post-target position with adverb in Experiment 2 (Cohen’s h = 0.11). Benchmarks for effect magnitude for Cohen’s h are 0.2 for a weak effect, 0.5 for a medium effect and 0.8 for a strong effect.
Gesture Direction difference score (abs(forward-backward)) and effect magnitude of the Gesture Direction effect based on each other factor (level) for each experiment, using Cohen’s h
Table 3. Long description
The table consists of seven columns: Exp., Factor, Factor level, Forward gesture Friday %, Backward gesture Friday %, Difference score %, and Effect magnitude Cohen’s h.
* Exp. 1 (Factor: Gesture position):
- Adverb: Forward 54.2, Backward 19.6, Difference 34.6, Cohen’s h 0.74.
- Verb: Forward 97.9, Backward 72.1, Difference 25.8, Cohen’s h 0.82.
- Post-target (no adverb): Forward 97.8, Backward 56.8, Difference 41.0, Cohen’s h 1.14.
* Exp. 2 (Factor: Gesture position):
- Adverb: Forward 68.9, Backward 23.9, Difference 45.0, Cohen’s h 0.94.
- Verb: Forward 52.5, Backward 20, Difference 32.5, Cohen’s h 0.69.
- Post-target (adverb): Forward 41.3, Backward 36.2, Difference 5.1, Cohen’s h 0.11.
* Exp. 3:
- Gesture position (Adverb): Forward 69.5, Backward 37.9, Difference 31.6, Cohen’s h 0.65.
- Gesture position (Verb): Forward 52.3, Backward 29.8, Difference 22.6, Cohen’s h 0.46.
- Accent position (Adverb): Forward 55.8, Backward 31.5, Difference 24.3, Cohen’s h 0.5.
- Accent position (Verb): Forward 65.9, Backward 36.8, Difference 29.1, Cohen’s h 0.59.
- Match position/ordering (Match adverb): Forward 65.9, Backward 35.4, Difference 30.4, Cohen’s h 0.62.
- Match position/ordering (Match verb): Forward 58.5, Backward 32.5, Difference 26.0, Cohen’s h 0.53.
- Match position/ordering (Mismatch accent-gesture): Forward 73.2, Backward 40.4, Difference 32.7, Cohen’s h 0.68.
- Match position/ordering (Mismatch gesture-accent): Forward 46.7, Backward 27.3, Difference 19.4, Cohen’s h 0.41.
Comparing effects within Experiment 1, the results showed that the difference score between gesture directions was higher with the gesture on the adverb than the verb, indicating that when the gesture matched with the adverb influences responses the most (although Cohen’s h was higher for the verb condition, which is due to the response proportions being close to the end of the proportion scale). The post-target condition showed a large effect of Gesture Direction with a difference score of 41% and a Cohen’s h of 1.14. However, due to the experimental setup of Experiment 1, we cannot be sure that this is a pure Gesture Direction effect, or rather an effect of the absence of the adverb in this condition. Looking at Experiment 2, which included the adverb also in the verb condition, the condition with the gesture on the adverb was stronger (difference score = 45%, Cohen’s h = 0.94) than the gesture on the verb condition (difference score = 32.5%, Cohen’s h = 0.69). When the adverb was present in the post-target condition, the Gesture Direction effect effectively disappeared (difference score = 5.1%, Cohen’s h = 0.11). Experiment 3 showed the same Gesture Position trend as Experiments 1 and 2, in that the gesture on adverb condition elicited a bigger difference between gesture directions (difference score = 31.6%, Cohen’s h = 0.65) than the gesture on verb condition (difference score = 22.6%, Cohen’s h = 0.46).
Regarding prosody-gesture (mis)matches in Experiment 3, only minor magnitude differences emerged between matching conditions (Cohen’s h = 0.62 for match on adverb and Cohen’s h = 0.53 for match on verb), but differences were more apparent for the mismatching conditions, where the ordering of gesture and accent modulated the Gesture Direction effect: the gesture effect was stronger when the gesture followed the accent – so when the gesture was placed on the adverb (difference score = 32.7%, Cohen’s h = 0.68) and weaker when it preceded the accent (difference score = 19.4%, Cohen’s h = 0.41).
5.5. Likeability and gesture awareness effects
As discussed in Section 4.4, we performed a median split for ease of presenting percentages in a high and low likeability group. A higher likeability of the speaker increased the proportion of ‘Friday’ responses with the forward gesture compared to a lower likeability and to the general distribution in Experiments 2 and 3, but not in Experiment 1, as shown in Table 4.
Proportion of ‘Friday’ responses per likeability group, experiment and Gesture Direction with difference proportions between directions
Table 4. Long description
The table is organized into three main columns for Likeability group: Higher likeability, Lower likeability, and General. Each group is subdivided into three columns: Forward percent, Backward percent, and Difference percent.
* Exp. 1:
- Higher likeability: Forward 80.0, Backward 43.2, Difference 36.8.
- Lower likeability: Forward 84.9, Backward 51.7, Difference 33.2.
- General: Forward 83.0, Backward 48.9, Difference 34.1.
* Exp. 2:
- Higher likeability: Forward 64.8, Backward 27.0, Difference 37.8.
- Lower likeability: Forward 41.7, Backward 26.7, Difference 15.
- General: Forward 54.2, Backward 26.8, Difference 27.4.
* Exp. 3:
- Higher likeability: Forward 66.7, Backward 37.0, Difference 29.7.
- Lower likeability: Forward 55.6, Backward 30.8, Difference 24.8.
- General: Forward 60.7, Backward 34.1, Difference 26.6.
We ran logistic regression models on the interaction of Likeability and Gesture Direction for each Experiment. Experiment 1 did not indicate any interaction between Likeability and Gesture Direction (logit = +0.01, SE = 0.57, 95% CrI: [−1.10, 1.13]). Experiment 2 did indicate an interaction between Likeability and Gesture Direction in that the coefficient barely overlapped with zero (logit = +0.90, SE = 0.52, 95% CrI: [−0.11, 1.93]) and the posterior tests reveal a high probability of being of the same sign (p(b > 0) = 0.96). In Experiment 3, we found no interaction between Likeability and Gesture Direction (logit = +0.19, SE = 0.45, 95% CrI: [−0.69, 1.07]). Z-scoring the scales across all experiments before performing a median split and then running a conjoined likeability model revealed an interaction between Likeability and Gesture Direction (logit = −0.48, SE = 0.28, 95% CrI: [−1.03, 0.07]), with a high probability of being of the same sign for the interaction (p(b < 0) = 0.96).
Regarding Gesture Awareness, the large majority of participants across all experiments self-reported to have noticed the gesture (N = 817; 91.8%). Table 5 displays the proportions of participant’s gesture awareness (yes/no) across different Gesture Positions and Match Positions in all experiments. No consistent patterns are visible in different gesture positions across all experiments; differences between conditions were rather small. Regarding the match position, differences between conditions were also small, except for the match on verb condition having a slightly lower gesture awareness.
Gesture awareness proportions per experiment, gesture position and match position
Table 5. Long description
The table is structured with columns for Condition, Exp. 1, Exp. 2, and Exp. 3. Each experiment is subdivided into Yes percent and No percent awareness columns.
Under the Gesture position category:
* Adverb: Exp. 1 (Yes 80.9, No 19.1), Exp. 2 (Yes 95.6, No 4.4), Exp. 3 (Yes 96.6, No 3.4).
* Verb: Exp. 1 (Yes 84.4, No 15.6), Exp. 2 (Yes 97.6, No 2.4), Exp. 3 (Yes 95.3, No 4.7).
* Post-target: Exp. 1 (Yes 85.6, No 14.4), Exp. 2 (Yes 93.5, No 6.5), Exp. 3 data is not provided.
Under the Match position category (data only provided for Exp. 3):
* Match adverb: Yes 96.6, No 3.4.
* Match verb: Yes 92.6, No 7.4.
* Accent-gesture: Yes 96.6, No 3.4.
* Gesture-accent: Yes 97.8, No 2.2.
We ran logistic regression models to assess the interaction of Gesture Awareness and Gesture Position. In Experiment 1, 229 participants (83.6%) paid attention to the gesture and the model indicated a Gesture Awareness effect (logit = −2.40, SE = 1.19, 95% CrI: [−5.12, −0.45]), with a high probability of being of the same sign (p(b < 0) = 0.99), showing that participants who did not notice the gesture were more likely to respond with ‘Monday’. There was no interaction with Gesture Position. In Experiment 2, 257 participants (95.5%) paid attention to the gesture and the model indicated no Gesture Awareness effect or interactions. In Experiment 3, 331 participants (95.4%) paid attention to the gesture and the model indicated a weak Gesture Awareness effect (logit = −1.26, SE = 0.66, 95% CrI: [−2.63, −0.05]) with a high probability of being of the same sign (p(b < 0) = 0.98), showing that participants who did not notice the gesture were more likely to respond with ‘Monday’. There was no interaction with Gesture Position, and no interaction with Match Position.
6. General discussion
This study represents both a replication as well as an extension of the gestural effects in the NWQ paradigm. Across three experiments, we replicated the finding that gesture influences how people resolve the ambiguity of the NWQ (Jamalian & Tversky, Reference Jamalian and Tversky2012; Lewis & Stickles, Reference Lewis and Stickles2017; Tversky & Jamalian, Reference Tversky and Jamalian2021; Winter & Duffy, Reference Winter and Duffy2020). At least relatively speaking, we found that forward gestures consistently resulted in more ‘Friday’ responses than backwards gestures. Given that this study and past studies used slightly different speakers, stimuli, populations and testing contexts, the cumulative evidence for the impact of gesture is now remarkably consistent. Indeed, the NWQ now constitutes one of the clearest demonstrations that co-speech gesture can systematically shape sentence interpretation, and in a manner that is quite substantial, with relatively large effect sizes in terms of raw proportions. At this point, we are no longer asking whether gesture matters in this context; we are asking what determines when it matters.
In this study, we explored what aspects of speech, gesture and prosody determine whether a gesture successfully biased interpretation. The key indicator of this was the proportional difference between ‘Monday’/‘Friday’ responses for forward and backward gestures. For this, our most consistent finding was that a gesture had its strongest effect when it coincided with the spatial adverb ‘forward’. In contrast, gesture effects were weaker when the same gesture was aligned with the verb ‘moved’, even though the gesture itself depicted movement. Experiment 3 showed that in the grand scheme of things, manipulating pitch accent placement had relatively little impact when the gesture already coincided with ‘forward’. Only when gesture aligned with the verb – the element where gesture also produced weaker effects overall – did prosodic prominence play a marginally bigger role. One interpretation of this finding is that when gesture aligns with the key disambiguating element ‘forward’ (what we dubbed the adverb anchoring account in Section 3), a pitch accent does not confer additional prominence to it; but when gesture is in a position where it has overall weaker effects (on the verb, as suggested by the results from Experiments 1–3), its influence can be boosted by a concomitant pitch accent. Taken together, this pattern of results suggests that semantic anchoring trumps prosodic anchoring, at least in the context of the NWQ.
Since the expectations of the verb anchoring and gesture priming accounts are co-extensive in the current experimental setup (see Section 3), the results we found also provide little support for a gesture priming account. If earlier gestures exerted their influence by biasing the interpretation of subsequent linguistic material, gesture effects should have been strongest when the gesture occurred on the verb, before participants encountered the critical adverb. Instead, we found the opposite.
Likewise, the gesture salience account as formulated in Section 3 was not supported by our data. In Experiment 1, when the adverb was absent, forward/backward post-target gestures led to the biggest difference in ‘Monday’/‘Friday’ responses observed across all experimental conditions from all three experiments. Experiment 2, however, revealed that this strong gesture effect entirely depended on the presence of the adverb: when the adverb was restored, forward/backward gestures made minimal difference in a post-target context. The most likely interpretation of this pattern of results is that in Experiment 1, without the adverb, the NWQ is entirely underspecified with respect to directional information, which then allows the gesture to supply the missing directional information. In Experiment 2, however, participants had already encountered the adverb ‘forward’ and may have settled on an interpretation before the gesture occurred. Participants may be reluctant to revise their initial disambiguation, which means that the gesture is simply not integrated with the NWQ, as suggested by the absence of any meaningful effect on ‘Monday’/‘Friday’ responses in this context.
This parallels the distinction between underspecification and ambiguity. Without the adverb, direction is not under discussion; with the adverb, the direction is linguistically specified, but ambiguous between ego-moving and time-moving perspectives. The difference between the two post-target conditions therefore suggests that results depend on whether gesture contributes missing information (without adverb, strong effect) or speaks to the resolution of an already established ambiguity (with adverb, barely any effect). Clearly, the mere fact that the gesture was temporally isolated did not guarantee a stronger effect on interpretation across the board – only when the directional information was fully unspecified when no adverb was present. It should be noted, however, that gesture awareness was not higher for the post-target condition, which may suggest that the manipulation did not achieve its intended effect, i.e., making the gesture more salient.
The presence of adverbs also relates to the fact that we observed overall more ‘Friday’ responses in Experiment 1 than in Experiment 2. A closer look reveals that this is due to the post-target and verb conditions having more ‘Monday’ responses when the adverb is present (Experiment 2), compared to when it is absent (Experiment 1). What explains this result? In line with the idea that without the adverb, the question is underspecified with respect to direction rather than ambiguous, listeners may use more of their world knowledge in filling in their understanding of the NWQ. Or, in other words, if the question does not linguistically specify any direction whatsoever, participants likely rely on what they know about meetings in general. Li (Reference Li2020) showed that in real-world contexts, people experience meetings being more likely moved toward a later date, presumably because moving it toward an earlier date leads to more scheduling complications and may also be dispreferred for social reasons, given that it can cause inconvenience. The gesture itself increases the preference for the ‘Friday’ response, which is supported by the fact that participants responded ‘Friday’ more often when they noticed the gesture than those who did not.
Importantly, however, the presence of an adverb had relatively little effect on the main results relating to gestural coordination. Even though the baseline proportion of ‘Monday’/‘Friday’ responses shifted drastically as a function of adverb presence, the difference between forward/backward gestures was 32% in the gesture-on-verb condition with adverb, and 26% in the gesture-on-verb condition without adverb, a mere difference of 6 percentage points.
Taken together, the results suggest that gesture exerts its strongest influence when aligned with the constituent that is most relevant for resolving the ambiguity of the NWQ, which is the word ‘forward’ in the present paradigm. Moreover, the results from the gesture placement effects in relation to Experiment 3, which also included prosody, suggest that the crucial timing relationship in the NWQ is not gesture-to-accent timing but gesture-to-meaning timing. This indicates that the prosodic anchoring account is secondary to an account with semantic restrictions. Our results suggest that gesture matters the most where it is needed the most for disambiguation, even though in natural speech it tends to co-occur with verbs. These results also make a lot of sense in the context of the idea that iconic gestures are difficult to interpret without being further specified by concomitant spoken language (Feyereisen et al., Reference Feyereisen, Van de Wiele and Dubois1988; Hadar & Pinchas-Zamir, Reference Hadar and Pinchas-Zamir2004; Krauss et al., Reference Krauss, Morrel-Samuels and Colasante1991). A directional gesture needs to occur with ‘forward’ to be interpretable with respect to ego-moving and time-moving perspectives; since ‘moved’ itself is relatively unbiased with respect to these perspectives and does not seem to strongly imply a direction (Feist & Duffy, Reference Feist and Duffy2015), the gesture has little effect when coinciding with the verb.
Together with Winter and Duffy’s (Reference Winter and Duffy2020) finding that the effect of gesture disappears altogether when no movement is implied, and together with their finding that changing the adverb from ‘forward’ to ‘backward’ has a bigger effect than changing gestural direction, these results are in line with the idea that the interpretation of iconic gestures is largely parasitical on speech. Yes, gesture can influence whether people think the meeting has been moved to Monday or to Friday, but this effect very much depends on the concomitant language, and how gesture is timed with respect to it.
In terms of likeability, the individual experiments showed either no or only weak likeability effects. We then ran a conjoined likeability analysis, similar to Winter and Duffy (Reference Winter and Duffy2020) and replicated an overall likeability effect. This further confirms that likeability affects the perspective that a listener takes upon the speaker’s gestures: if the listener has a positive association with the speaker, the listener is more likely to interpret the gesture from the speaker’s perspective. However, the results also show that the likeability effects are altogether rather small. Both in the present study and in Winter and Duffy (Reference Winter and Duffy2020), sufficient power is needed (i.e., aggregating evidence across experiments) for these effects to be discernible.
7. Conclusions
The set of multimodal perception experiments in this article confirms that a gesture’s direction influences how people respond to the ambiguous NWQ, i.e, gesture helps in disambiguating between ego-moving and time-moving perspectives (Jamalian & Tversky, Reference Jamalian and Tversky2012; Lewis & Stickles, Reference Lewis and Stickles2017; Tversky & Jamalian, Reference Tversky and Jamalian2021; Winter & Duffy, Reference Winter and Duffy2020). In contrast, responses differed less strongly as a function of where within the question the gesture occurred. Across experiments, a clear pattern emerged where gesture produced stronger effects when it was time-aligned with the key disambiguating element (in this case, the adverb). In contrast, prosody played relatively little role. A further finding was that gesture produced a strong effect when it followed a sentence that left the perspective fully unspecified (i.e., no adverb), but this effect dissipated when the sentence included the adverb. These results suggest that even though the direction of a gesture clearly matters in this paradigm, its ability to resolve perspectives clearly depends on concomitant speech. The interpretive impact of a gesture depends on what words it accompanies.
Data availability statement
Preregistrations of the experiments, item videos, data and code for analysis and plotting are uploaded on OSF: https://osf.io/p8svw/overview.
Acknowledgments
We thank Kathryn Barnes for agreeing to record the stimuli for us and all participants of the studies for their contribution. We thank Cornelia Ebert for useful feedback on the semantics of gestures.
Funding statement
We acknowledge the financial support of the DFG priority program SPP 2392 Visual Communication (ViCom), Frankfurt am Main, Germany (Alina Gregori: project ID 502013510, KU 2323/5-1 & Susanne Fuchs: project ID 502013782 FU791/9-1). Susanne Fuchs was supported by the ILCB at Aix-Marseille University. Bodo Winter was supported by the UKRI Future Leaders Fellowship MR/T040505/1.
Competing interests
The authors have no conflicts of interest to declare.
Ethical standards
Ethics approval for the data collection has been granted to the Linguistics Lab at Goethe University Frankfurt by the German Linguistics Society Ethics Commission under No. 2024-03.
Open access
