1. Introduction
Signed languages are unique partially due to the prevalence of visually motivated mappings between form and meaning. The sign for ‘ball’ in American Sign Language (ASL) looks like holding a ball, the sign for ‘cat’ looks like whiskers, etc. Increasingly, researchers argue that iconicity as a systematic property should be viewed as a quality of all languages operating in balance with arbitrariness (Dingemanse, Blasi, Lupyan, Christiansen, & Monaghan, Reference Dingemanse, Blasi, Lupyan, Christiansen and Monaghan2015; Monaghan, Christiansen, & Fitneva, Reference Monaghan, Christiansen and Fitneva2011; Monaghan, Shillcock, Christiansen, & Kirby, Reference Monaghan, Shillcock, Christiansen and Kirby2014; Perniss, Thompson, & Vigliocco, Reference Perniss, Thompson and Vigliocco2010; Perniss & Vigliocco, Reference Perniss and Vigliocco2014).
This study attempts to replicate findings from spoken language translation recognition experiments, extending research to the visual modality to explore the lexical processing of visually motivated systematicity in a signed language. Translation recognition as an experimental paradigm is well suited to resolving questions about such interactions of form and meaning. Participants view lexical items from two languages and decide if they are equivalent. Based on the observation that second language learners often confuse words with similar forms or meanings, carefully controlled incorrect translations are used to create interference with the correct translation – slowing down processing and revealing the degree to which conceptual and lexical levels of memory are activated using accuracy and reaction time (RT) measurements. Correct translations are generally translated quickly and accurately even at lower levels of language proficiency, and are used as fillers to help disguise the key experimental manipulations from participants.
Prior research has found that that novice spoken language (unimodal) bilinguals are more sensitive to incorrect (Distractor) second language (L2) translations of first language (L1) words phonologically related to correct L2 translations, whereas expert unimodal bilinguals are more inhibited by distractors semantically related to the target (Talamas, Kroll, & Dufour, Reference Talamas, Kroll and Dufour1999). This is attributed to novices activating L1 translation equivalents as an intermediate step in accessing semantic representations (Lexical mediation), whereas experts activate semantic representations directly (Conceptual mediation) (Kroll & Stewart, Reference Kroll and Stewart1994). Stimuli in Kroll and Stewart’s seminal study were ordered in semantically related categorized lists (e.g., dress, suit, shoes) as a way to activate conceptual memory. They found that semantically related items were translated into the L2 slower than when translated into the L1. Stimuli in categorized lists were also translated into the L2 slower than in mixed lists (e.g., orange, lion, ambulance). These results indicated that L1–L2 translation was conceptually mediated, but L2–L1 translation was lexically mediated, and that the strength of the connections between L1 and L2 lexical items to conceptual memory was weaker for L2 lexical forms, particularly for novices. More recent research suggests that behavioral data may not always be sensitive to proficiency differences (Misra, Guo, Bobb, & Kroll, Reference Misra, Guo, Bobb and Kroll2012; Sunderman & Kroll, Reference Sunderman and Kroll2006; Wu & Thierry, Reference Wu and Thierry2010). Thus, it is probable that any replicable effects in spoken/signed language bilinguals will occur at later (i.e., conceptual) levels of processing.
There is reason to expect differences in translation recognition between unimodal bilinguals and spoken/signed (bimodal) bilinguals at the conceptual level of processing. Baus, Carreiras, and Emmorey (Reference Baus, Carreiras and Emmorey2013) tested processing of iconic signs and found that novice bimodal bilinguals were faster to recognize correct iconic ASL/English sign translation pairs (e.g., STIR/stir) presented simultaneously as a picture of the sign and its translation, than non-iconic sign translation pairs (SING/sing). Expert bimodal bilinguals were slower to recognize correct iconic translations than non-iconic translations. These results are difficult to interpret relative to the spoken language translation recognition paradigm because participants were responding to simultaneously presented stimuli, because responses were to correct rather than incorrect translations, and because there were no purely phonologically or semantically related translation pairs to which iconically related pairs could be compared. Nevertheless, a compelling case can be made that novices and experts do process translations differently depending on the level of iconicity. This may be partially attributable to Imageability; the extent to which a word’s referent evokes a visual or auditory image, which has been positively correlated with iconicity (Baus et al., Reference Baus, Carreiras and Emmorey2013; Thompson, Vinson, & Vigliocco, Reference Thompson, Vinson and Vigliocco2010).
Unimodal studies that have examined the processing of high and low imageability concepts in bilingual memory have found similar effects for high imageability concepts in both expert and novice bilinguals. These studies suggest that words with high imageability are translated primarily using conceptual mediation (de Groot & Poot, Reference de Groot and Poot1997; Van Hell & De Groot, Reference Van Hell and De Groot1998). Words with low imageability are translated primarily using lexical mediation for novices, but conceptual mediation for experts. This pattern can be explained by proposing that high imageability concepts activate more lexical items with shared conceptual properties than low imageability concepts, regardless of levels of L2 language experience, but that low imageability concepts activate only language specific conceptual properties for novices, favoring lexical connections in translation (Dong, Gui, & MacWhinney, Reference Dong, Gui and MacWhinney2005). Dong et al. give the example of the concept of ‘foot’ being equally as important to the high imageability English word kick as the Chinese ti, whereas the concept of ‘settlement’ is salient in the low imageability colony, but the concept of ‘exploitation’ is prominent in Chinese equivalent (2005, p. 224).
Imageability effects could behave the same in an oral/auditory language as in a manual/visual one. However, the observation that high imageability concepts facilitate conceptual processing in second language learners without regard to language experience does suggest that differences between novice and expert learners of ASL attributed to motivated signs activating connections between phonological and semantic representations also require careful interpretation. Two findings make a compelling case that iconic motivation creates a link between form and meaning that affects lexical processing in signed languages without regard to language expertise. The first is from spoken language literature on Simulation (Barsalou, Reference Barsalou2008) where phrases such as “The ranger saw the eagle in the sky” evoke a visualization of extended wings which facilitates naming a picture of an eagle with wings extended over naming a picture of an eagle with wings folded (Zwaan & Madden, Reference Zwaan, Madden, Pecher and Zwaan2005). In signed languages, this semantic facilitation effect is found for iconic signs (Grote & Linz, Reference Grote, Linz, Müller and Jena2003; Ormel, Knoors, Hermans, & Verhoeven, Reference Ormel, Knoors, Hermans and Verhoeven2009; Thompson, Vinson, & Vigliocco, Reference Thompson, Vinson and Vigliocco2009). Another related effect, apparently unique to sign language research, is the finding that iconicity inhibits RTs in form-based decision tasks, such as phoneme monitoring (Thompson et al., Reference Thompson, Vinson and Vigliocco2010). When signers are asked to indicate whether phonological parameters are present in a sign, decision latencies for iconic signs are longer than for non-iconic signs. These effects suggest that iconicity impacts processing without explicitly evoking semantics.
Psycholinguistic treatments often operationalize iconicity using non-signer ratings. Even when relying on signer ratings, studies typically assume that form/meaning connections are accessible to both those with and without prior language experience with the signed language in question. Current investigations into the differences between signer and non-signer iconicity ratings have found high global correlations, but there are key distinctions likely due to non-signers’ gestural experiences (Sevcikova Sehyr & Emmorey, Reference Sevcikova Sehyr and Emmorey2019). Non-signer ratings of iconicity are likely related to language internal motivation in ways that have yet to be completely understood, but explorations of this topic, both in signed and spoken languages (Dingemanse et al., Reference Dingemanse, Blasi, Lupyan, Christiansen and Monaghan2015; Lepic, Reference Lepic2015; Occhino, Reference Occhino2016), have suggested that many iconic signs are additionally Diagrammatic, both in their motivation and in word formation (Haiman, Reference Haiman1980; Ungerer, Reference Ungerer, Nänny and Fischer1999), rather than solely Imagistic, which is presumably the quality most easily accessed in non-signer ratings. The diagrammatic mapping of articulatory features to conceptual schemas is a documented property of signed language lexicons (Emmorey, Reference Emmorey2014; Taub, Reference Taub2001; P. Wilcox, Reference Wilcox2000; S. Wilcox, Reference Wilcox2004) and is exemplified in the relationship between a sign and a Sign family (Frishberg & Gough, Reference Frishberg and Gough2000). Lepic and Occhino (2018, p. 159) give an example of this in the signs for FAR, CHASE, and FOLLOW in which the handshape parameter is fixed but different movements disambiguate the related meanings. This example shows that the phonological construal of the hands as two articulators maps onto the semantic construal of two entities. Two-handed signs can encode figure–ground relations by foregrounding and backgrounding at the same time (Lepic, Börstell, Belsitzman, & Sandler, Reference Lepic, Börstell, Belsitzman and Sandler2016).
Diagrammatic iconicity in signed languages refers to a paradigmatic relationship within the grammatical structures of the language to a set of non-linguistic, real-world references which may themselves be conceptualized schematically (such as in figure and ground). While systematic relations can appear across multiple lexical items and may not be iconic (e.g., glitter, glimmer, glow, glint, glisten, etc.), and iconic relations may be idiosyncratic, one-off form/meaning relationships, diagrammatic signed language iconicity is both systematic and iconic. A direct imitative or resemblance relation between references and articulation exists in individual parameters of specific signs (such as the closed hand representing actual fist in the sign PUNCH, or bent fingers representing bent legs in the sign SIT) which also may also simultaneously participate in paradigmatic morphological similarities across multiple related sign forms (such as two sets of bent fingers representing multiple people sitting for COUCH and even more legs when combined with additional movement in SIT-IN-A-CIRCLE; see Figure 1). Importantly, this extremely productive morphological paradigm of two-handed signs expressing multiple referents is also diagrammatic in its schematic relationship between linguistic and real-world analogies – one hand is one set of legs, two hands are two sets of legs, two hands with additional movement are more than two sets of legs. Experimentally, this means distractors related through diagrammatic connections share similarity both phonologically and semantically with correct translations and with non-linguistic conceptualization of their referent.
ASL sign for PUNCH, SIT, COUCH, and the depicting sign (DS) for sitting in a circle which is produced with the bent 2 fingers on each hand located in front of the signer and arcing movement away from the body.
Recognizing both the language-external and language-internal properties of diagrammatic iconicity can explain why signers would perceive their own signs to be more iconic than the signs of other signed languages (Occhino, Anible, Wilkinson, & Morford, Reference Occhino, Anible, Wilkinson and Morford2017). A Cognitive iconicity approach (Occhino, Reference Occhino2016; P. Wilcox, Reference Wilcox2000; S. Wilcox, Reference Wilcox2004) also implicitly accounts for degree of language exposure, since the learning of these language-internal patterns is accumulated over the course of an individual’s experience with a signed language. In this perspective, Motivation comes from an alignment in the way an individual thinks about the articulators and the way they think about their referent. When there is structural similarity between these construals, a language construction is perceived as iconic. Critically, the proclivity for an individual to have the necessary construals of both form and meaning to perceive a motivated relationship is influenced by their assimilation into the language ecology that created these structural similarities. In other words, perception of iconicity within a signed language is contingent on how well an individual has learned to think like a user of that system.
This study hypothesizes that: (1) in a translation recognition task, the effect of language-internal motivations (operationalized as diagrammaticity) will be greater for experts than for novices; (2) the effect of language-external motivations (operationalized as imageability) will not vary by language experience; and (3) imageability should affect both experts and novices but do so at different levels when also diagrammatic because what is ‘iconic’ (meaning the combination of the language-internal aspects of diagrammaticity and purely language-external imagistic motivation) for each group is altered by language experience.
Specifically for (1), if effects of diagrammaticity in signed language processing are similar to spoken language studies that explore morphological relatedness (Feldman, Reference Feldman2000; Qasem & Foote, Reference Qasem and Foote2010), this condition will likely result in higher interference than either phonological or semantic distractors alone, and cause more interference for experts than for novices. (2) If effects of high imageability are similar to reported effects of high iconicity in signers (Thompson et al., Reference Thompson, Vinson and Vigliocco2009, Reference Thompson, Vinson and Vigliocco2010), then participants should show more interference when rejecting phonological than semantic distractors for concepts with high imageability. On the other hand, if imageability affects bimodal bilinguals the same way as unimodal bilinguals (de Groot & Poot, Reference de Groot and Poot1997; Van Hell & De Groot, Reference Van Hell and De Groot1998), both experts and novices should show more interference when rejecting semantic distractors than phonological distractors for high imageability concepts. (3) If diagrammatic lexical relations interact with imageability as suggested by theories of cognitive iconicity, the necessity of signers having first learned language-internal aspects of these patterns to show interference in this condition should be mitigated by the accessibility of language-external patterns. This will then reduce the difference in interference between experts and novices, making them more alike in their responses when the imageability of items is high than when it is low.
2. Methods
2.1. participants
Fifty-seven English–ASL bilinguals participated in the experiment. Data from two participants were excluded due to equipment failure, bringing the total number for analysis to 55 (42 female). Participants were L1 speakers of English and L2 signers of ASL 18–58 years old (M = 24.6, SD = 8.9). Participants’ experience with ASL ranged from three semesters of classroom instruction to experienced interpreters (M = 10.3 years’ experience, SD = 8.6). ASL proficiency was assessed using the ASL Sentence Repetition Task (SRT) (Hauser, Paludnevičienė, Supalla, & Bavelier, Reference Hauser, Paludnevičienė, Supalla, Bavelier and de Quadros2008), designed to assess native signers. Participants viewed 20 sentences of increasing complexity and were asked to repeat the sentence back exactly after it was signed to them. Exact repetitions were scored as 1; repetitions with replacements or errors were scored as 0. A native signer evaluated participants’ responses. Participants’ scores ranged from 0 to 10 (M = 2.98, SD = 2.56). ASL proficiency assessments often rely on participants self-rating their production/comprehension skills (cf. Anible et al., Reference Anible, Twitchell, Waters, Dussias, Piñar and Morford2015). Low average scores indicate that the test is not an ideal measure for second language learners. The SRT measure is still superior to self-ratings since (a) it is an online measure where performance during active language processing is assessed, and (b) it does not rely on participants’ subjective assessments. Figure 2 shows the distribution of scores based on whether participants were students or interpreters.
Distribution of proficiency scores by occupation status.
2.2. materials
Participants viewed 48 distractor sets. Each set included four incorrect translation distractors: unrelated, phonologically related, semantically related, and diagrammatically related in both form and meaning to the correct translation. Only filler trials were presented as correct translations. For example, the ASL/English translation pair SIT/sit was implicit in the distractor pairs SIT/problem (where PROBLEM is phonologically related in ASL to the correct translation of sit), SIT/table (semantically related), SIT/couch (diagrammatically related), or SIT/newspaper (unrelated) (see Figure 3).
ASL sign for the SIT, and corresponding unrelated (NEWSPAPER), phonological (PROBLEM), semantic (TABLE), and diagrammatic (COUCH) distractor conditions.
Phonologically related distractors included signs that overlapped in at least one of three sign parameters: handshape, location, or movement. These signs were not semantically related to the correct translation. A weighted scoring system was used to assess overlap. Overlap in movement/location was given a high rating, overlap in movement/handshape or handshape/location was given a medium rating, and overlap in just one parameter was given a low rating. Levels were based on Hildebrandt and Corina’s (Reference Hildebrandt and Corina2002) finding that participants asked to judge the phonological similarity between a target-sign and surrounding flanker-signs rated some combinations of sign parameters higher than others. That study based their rankings on “125 possible combinations of 5 different handshapes (1, 5, 0, F, 8) 5 different locations (forehead, nose, chin, chest, wrist) and 5 different movements (circular, straight to signer, straight from signer, wrist rotation, and unidirectional twice-repeated)” (2002, p. 597). In the current study, high overlap accounted for 32% of the items, mid 36%, and low 32%. ‘Appendix A’ includes a description of which parameters were judged to be overlapping for each stimuli pair.
Semantically related distractors were semantic associates of correct translations and were not phonologically related to the correct translation. The top three associates for each item were drawn from the Edinburgh Associative Thesaurus (Kiss, Armstrong, Milroy, & Piper, Reference Kiss, Armstrong, Milroy, Piper, Aitken, Bailey and Hamilton-Smith1973) and the highest ranked associate that was not part of an English collocation was selected. Associates in the top three occasionally had no single ASL equivalent and/or are translated using the same lexical item. For example, the highest ranked associate for ‘camp’ was ‘tent’ – these two terms are near homophones in ASL. In these cases, the next highest item with a single translation in ASL was selected. The average ratio of responses for the semantically related distractor given target word was within the first ten semantic associates (M = 0.09, SD = 0.1) in the database.
Diagrammatically related distractors were both phonologically and semantically related to the correct translation. The motivated mapping between form and meaning for these distractors was the same as for the correct translation. In cognitive iconicity, signs highlight individual properties of meaning as well as form. The V-handshape can potentially represent the number two, or pairs of things such as eyes or legs, etc. Semantic properties of concepts can also have multiple instantiations. The concept of ‘sit’ includes many features that are interconnected; the position of the legs or the object that is sat on. Signs can highlight these semantic characteristics. A pair of signs that are related diagrammatically share properties of both form and meaning, as in SIT and COUCH. Phonologically, both use the V-handshape (occasionally also instantiated as a bent-V allophone). Semantically, both select the way legs behave while sitting as the property that is profiled. In SIT the sign profiles one person’s legs, and in COUCH the sign profiles a row of legs. Other signs, like DESK, are semantically related to ‘sitting’, but highlight a different property like ‘flatness’. The sign PROBLEM is phonologically related to SIT, but the same properties of form connect to unrelated semantic features. All diagrammatically related distractors contained related phonological features participating in similar conceptual domains that are features of a shared morphological schema (Lepic & Occhino, Reference Lepic, Occhino and Booij2018). Diagrammaticity was operationalized as high phonological and semantic scores in relation to the target. Items were selected by reviewing entries in the American Sign Language Handshape Dictionary (Tennant & Brown, Reference Tennant and Brown2010). ‘Appendix A’ lists both the form and meaning profiled in the morphological schemas that each pair of items participates in. This list was reviewed by two coders until they agreed on all items.
Items across conditions were matched closely for word length, frequency, concreteness, and imageability (see Table 1). These measures have all been shown to affect lexical access in online processing (Hudson & Bergman, Reference Hudson and Bergman1985; Jescheniak & Levelt, Reference Jescheniak and Levelt1994; Poarch, Van Hell, & Kroll, Reference Poarch, Van Hell and Kroll2015). Scores were drawn from the MRC Psycholinguistic Database (Coltheart, Reference Coltheart1981); a dictionary of 150,837 words with up to 26 attributes per entry. When an item was not available, the average of the condition was used, accounting for 5% (13/278) of values. A series of one-way ANOVAs evaluated differences between the experimental conditions (excluding fillers). There were no significant effects of condition on length [F(3,177) = 0.91, p = n.s., MSE = 2.63], log frequency [F(3,177) = 1.36, p = n.s., MSE = 2.22], or concreteness [F(3,177) = 1.19, p = n.s., MSE = 10477.49], or imageability [F(3,177) = 1.75, p = n.s., MSE = 7058.99].
Control measures for experimental stimuli
2.3. procedure
Four lists of 96 trials were constructed containing 48 fillers and 48 critical ‘incorrect’ distractors, with twelve items per condition. No target item in a translation set was repeated in a list, but the 48 fillers were the same across lists. The order of the trials was randomized. A practice block of five trials was shown at the start of the experiment to familiarize participants with the protocol. The task was to decide whether two words were translation equivalents of each other. All stimuli were presented as a short video clip of a sign and an audio-recording of a word. The participants started each trial by pressing the ‘enter’ key on the keyboard. The fixation timer was displayed for 200ms, then the video and the timer for that trial began simultaneously. Five-hundred milliseconds after the video ended, the audio stimulus played. The participant pressed either the ‘yes’ key (on the left hand) or the ‘no’ key (on the right hand), after which the timer stopped. Participants kept their hands over the response keys and made their responses as quickly and accurately as possible. The procedure was designed using Expyriment (Krause & Lindemann, Reference Krause and Lindemann2014). The protocol, including all video- and audio-recordings used as stimuli, is available on-line (Anible & Anible, Reference Anible and Anible2015).
After completing the experimental task, participants were asked to rate the number of ASL equivalents for the English translations. For example, the English word run has at least four lexical items in ASL that express the concepts of ‘jogging’, ‘machinery functioning’, ‘liquid flowing’, and ‘a run in fabric’. A higher number of equivalents can slow lexical access in tasks involving translation (Baus et al., Reference Baus, Carreiras and Emmorey2013; Laxén & Lavaur, Reference Laxén and Lavaur2010; Tokowicz & Kroll, Reference Tokowicz and Kroll2007). The average number of ASL translations for English target items based on this post-test was 1.83 (SD = 0.72). Four items (‘break’, ‘open’, ‘turn’, and ‘wash’) were removed because the average number of translation equivalents was more than 2.5 standard deviations above this mean. Two additional items (‘haircut’, ‘music’) were removed because they were judged to be nouns rather than verbs; reducing the total number of stimuli pairs in the final analysis to 42.
3. Data analysis
Accuracy was automatically coded – ‘no’ key-presses after non-filler trials. Mean accuracy was .94; accuracy per item ranged from .75 to 1.00 (SD = 0.18); accuracy per subject ranged from .80 to 1.00 (SD = 0.21). RTs faster than 200ms or slower than 5000ms were removed as outliers resulting in twelve observations (< .3%) trimmed. Mixed-effects models with crossed random effects for participants and items (Baayen & Milin, Reference Baayen and Milin2010) were estimated using the lme4 package (Bates, Maechler, Bolker, & Walker, Reference Bates, Maechler, Bolker and Walker2013) in R (R Core Team, 2015) and p-values were calculated using the lmerTest package (Kuznetsova, Brockhoff, & Christensen, Reference Kuznetsova, Brockhoff and Christensen2013).
3.1. model construction
An omnibus RT model was estimated first, using only accurate trials. Fixed effects included the interaction between the experimental variables and linear effects of participant variables (age, and SRT-score), and stimulus variables (word length, frequency, concreteness, imageability, number of translation equivalents, and phonological relatedness). Random effects were random intercepts for participants and items. Models were trimmed to remove non-significant fixed effects at p > .05. An interaction for the subject variable SRT-score was found, but no effect was found for age. Word length, frequency and concreteness, number of translation equivalents, and phonological relatedness had no significant effects/interactions, but there were main effects and interactions for imageability. The model was subjected to criticism focusing on the residuals (Baayen & Milin, Reference Baayen and Milin2010). Datapoints (n = 48; 2% of the data) with absolute standardized residuals exceeding 2.5 standard deviations were removed from the dataset and the model was re-estimated, resulting in improved fit. Correlation of fitted values with trimmed RTs was R 2 = .54, up from R 2 = .44; BIC = 29598.89, down from 31651.79.Footnote 1
3.2. results
The experimental condition, SRT-score, and imageability interacted significantly with the intercept (the unrelated condition). Table 2 shows the results of fitting this model for the experimental variables and interactions with the significant effects. Figure 4 shows these significant interactional effects for proficiency and imageability with the unrelated baseline condition. Figure 5 shows the three-way interaction of diagrammaticity, proficiency, and imageability. These were created with the effects (Fox, Reference Fox2003) and the ggplot2 packages (Wickham, Reference Wickham2009).
Main effects and interactions with SRT-score and imageability. P-values < .001 are marked with ‘***’, < 0.01 with ‘**’, < 0.05 with ‘*’, and < 0.1 with ‘.’.
Interaction of significant fixed effects on experimental conditions by (A) SRT-Score (proficiency) and (B) Imageability. Error bars represent the standard error of the estimates. Estimates have been jittered and scales adjusted for easier interpretation of error bars.
ASL signs for KISS and CARRY, with corresponding ASL translations of the English distractors; unrelated (EMERGENCY and ROSE), phonological (FIX and CONVINCE), semantic (LOVE and CATCH), and diagrammatic (QUARREL and OFFER).
Participants were quicker to reject the phonologically related distractor than the unrelated distractor (β = –295.9, t = –1.89, p = .059). After viewing the target sign (SIT) they were faster to reject a phonologically related distractor (problem) than an unrelated distractor (newspaper). There was a significant positive interaction between phonology and imageability, indicating slower RTs when rejecting the phonologically related distractors for increasingly imageable targets (β = 0.72, t = 2.33, p = .02). When the imageability of the target was high (KISS), participants were slower to reject the phonological distractor (fix) than the unrelated distractor (emergency), but when the imageability of the target was low (CARRY) there was no difference between a phonologically related distractor (convince) and an unrelated distractor (rose).
Participants were slower to reject a semantic distractor than an unrelated distractor (β = 528.1, t = 3.34, p < .001). After viewing the target (SIT), it took longer to reject a semantically related distractor (desk) than an unrelated distractor (newspaper). There was a negative interaction for semantics and imageability, with faster RTs for rejecting a semantically related distractor for increasingly imageable targets (β = –0.82, t = –2.67, p = .008). When the imageability of the target was low (CARRY), participants were slower to reject a semantic distractor (catch) than an unrelated distractor (rose), but when the imageability of the target was high (KISS) there was no difference between a semantically related distractor (love) and an unrelated distractor (emergency).
Participants were slower to reject the diagrammatic distractor than the unrelated distractor (β = 520.4, t = 3.06, p = .002). After viewing the target (SIT), it took longer to reject the diagrammatically related distractor (couch) compared to the unrelated distractor (newspaper). There was a negative interaction between this condition and SRT-score, indicating faster RTs when rejecting a diagrammatic distractor for increasing ASL proficiency (β = –96.3, t = –2.30, p = .02). Participants with lower SRT-scores showed greater interference when rejecting a diagrammatically related distractor than an unrelated distractor, but participants with higher SRT-scores showed less interference. There was also negative interaction between diagrammaticity and imageability, indicating faster RTs when rejecting the diagrammatic distractor for increasingly imageable targets (β = –0.78 t = –2.33, p = .019).
Critically, there was a three-way interaction between diagrammaticity, SRT-score, and imageability. RTs were slower when rejecting items with high imageability ratings for participants at the upper end of the SRT-score scale compared to participants at the lower end of the scale (β = 0.19, t = 2.62, p = .02). Participants with lower ASL proficiency were slower rejecting a diagrammatic distractor than an unrelated distractor when the imageability of the target was low (CARRY/offer), but when the imageability of the target was high (KISS/quarrel) they showed no difference. Participants with higher ASL proficiency showed the opposite pattern – they were slower to reject a diagrammatic distractor than an unrelated distractor when the imageability of the target was high, but when the imageability of the target was low they showed no difference. The strength of this effect decreases at higher levels of proficiency due to lower participant numbers. Plotting this effect shows that the trend is clear, nonetheless, even for intermediate levels (Figure 6).
Three-way interaction of diagrammaticity by SRT-score and imageability. Error bars represent the standard error of the estimates. Windows 0 through 10 display the model estimates at lower (0) to higher (10) SRT-scores. The number of participants for each SRT-score (proficiency) ranking window is displayed in the bottom right.
3.3. discussion
Results for diagrammaticity failed to align with results for morphological relatedness in spoken language. Novices rather than experts showed a stronger interference effect in this condition. This result is explained by its interactions with imageability.
This study also predicted that language-external motivations (imageability) would be shared regardless of language experience. One possibility for imageability was to increase interference for rejecting phonological competitors, but decrease interference for semantic competitors, matching effects reported for signers. This effect was confirmed by the results of the model. Semantic distractors were processed slower than unrelated distractors when imageability was low, and processed faster as imageability increased. Phonological distractors were processed slower than unrelated distractors when imageability was high and processed faster as imageability decreased. This pattern was the same for both novices and experts.
This study also predicted that diagrammaticity would interact with proficiency. This was confirmed by the model, and additionally modulated by imageability, but not as predicted. Rather than high imageability reducing the interference of diagrammatic distractors for experts, they showed interference when imageability was high, and novices showed interference when imageability was low. These findings suggest that experts process diagrammatic iconicity as phonological competitors since the pattern matches that of phonological interference increasing with higher imageability. Novices, on the other hand, appear to be primarily processing diagrammatic iconicity as semantic competitors since the pattern matches that of semantic interference for low imageability targets.
3.3.1. Imageability effects
Imageability is the measurement of how easy or difficult it is to visually or acoustically imagine a concept. All imageability ratings were collected on the English tokens of the ‘correct’ translation. There is currently no database measuring imageability for the ASL lexicon, but English imageability ratings have been found to capture an effect that is cross-linguistically consistent (Rofes et al., Reference Rofes, Zakariás, Ceder, Lind, Johansson, de Aguiar and Howard2018).
There was a main effect of imageability where the more imageable a concept was, the faster the RT. This result matches imageability findings for spoken language (de Groot & Poot, Reference de Groot and Poot1997). There were two interactions between imageability and the experimental condition. First, as the imageability of items in the phonologically related distractor condition increased, so did the average RT. Second, as the imageability of items in the semantically related distractor condition increased, the average RT decreased. These results match the prediction that the visual nature of signed language causes effects of imageability to manifest differently across modalities, and are consistent with effects of iconicity.
Thompson et al. (Reference Thompson, Vinson and Vigliocco2009, Reference Thompson, Vinson and Vigliocco2010) found the same pattern of semantic facilitation and phonological inhibition for highly iconic signs, similar to the interference effects demonstrated here for highly imageable signs. Thompson et al. (Reference Thompson, Vinson and Vigliocco2010) did also control for imageability, but did not find it to be a significant predictor of variability. That study included both nouns and verbs, so it is possible that imageability may play a more significant role for verbs than for nouns. There is further evidence that ‘object’ (noun) and ‘action’ (verb) concepts interact with iconicity differently. Ortega and Morgan (Reference Ortega and Morgan2015) found that non-signers in a lexical priming task responded faster to both iconic signs that profile actions and objects, but that experienced L2-signers had their lexical decision times facilitated only by action iconicity signs. This present study also restricted stimuli to verbs to facilitate elicitation of an iconicity effect from both novice and expert sign language users.
While iconicity and imageability can point in separate directions, this study assumes that imageability ratings for verbs capture some effects typically attributed to the iconicity. This claim is bolstered by the finding that verbs are rated higher for iconicity by non-signers than signers (Sevcikova Sehyr & Emmorey, Reference Sevcikova Sehyr and Emmorey2019). Post-hoc analysis showed a weak, non-significant correlation between non-signer iconicity ratings and imageability (r = .26, p = .09), following the same collection protocol used by the ASL-LEX database (Caselli, Sehyr, Cohen-Goldberg, & Emmorey, Reference Caselli, Sehyr, Cohen-Goldberg and Emmorey2017). When one item was removed (KICK) – a sign that profiles the use of hands to represent feet which non-signers would rate as low, but signers would readily understand – the correlation reached significance (r = .30, p < .05). This appears to align with the finding that, while generally signers and non-signers agree on iconicity rating, some signs that receive high iconicity ratings are actually focused on a different iconic mapping. Sevcikova Sehyr and Emmorey (Reference Sevcikova Sehyr and Emmorey2019) report a subset of signs in their study that were rated high for iconicity, but also low for transparency (ability of non-signers to correctly guess the correct meaning when presented with a sign), such as the sign MY being incorrectly guessed as ‘chest’ or ‘heart’. Non-signer iconicity ratings likely overlap with the gestural repertoires recruited during acquisition of a signed language (Janke & Marshall, Reference Janke and Marshall2017; Ortega, Schiefner, & Özyürek, Reference Ortega, Schiefner and Özyürek2019; Sevcikova Sehyr & Emmorey, Reference Sevcikova Sehyr and Emmorey2019). Available evidence strongly indicates that more research is needed on the nature of the relationship between imageability and iconicity for L2 learners of signed languages.
3.3.2. Diagrammaticity effects
Rejecting diagrammatically related distractors was more difficult than rejecting unrelated distractors. The most proficient participants were slowest to reject the highest imageability diagrammatic distractors, and the least proficient participants were slowest to reject the lowest imageability diagrammatic distractors. This suggests that sensitivity to a signed language’s internal patterns is modulated by the imageability of concepts and by language experience.
In the cognitive iconicity perspective, iconic mappings are learned and built up over iterative exposure, like all language forms. In usage-based models, bilinguals who have more experience with a signed language are more sensitive to these kinds of conceptual representations than those with more limited exposure. While a novice may recognize that the sign for STAND in ASL profiles two legs, it is less likely they will recognize that the sign for DANCE also profiles actions of legs since they do not touch the palm/ground in a predictable manner. The former example represents language-external knowledge, and the latter represents language-internal knowledge. A proficient signer of ASL recognizes that the extended 1st and 2nd fingers (‘V’ handshape) are part of a diagrammatic pattern in the language where many signs recruit the same phonological characteristics for related meanings. For example, the signs STAND, DANCE, JUMP, FALL-DOWN, and RIDE all use this pattern (see Figure 7).
Proposed patterns of diagrammatic activation in high imageability networks for (A) experts and (B) novices in a cognitive iconicity perspective. The highly imageable STAND functions as an anchor for the network to which other lexical items connect. Solid lines indicate stronger connections between form and meaning and dotted lines indicate weaker ones.
A possible explanation of how processing functions within these patterns of form/meaning mappings is that high imageability lexical items are ‘phono-semantic anchors’ where many diagrammatically related signs attract to a single prototypical codification of an image schema. Activation of items within these dense networks would then primarily inhibit phonological access due to the many form competitors; an effect that only appears as proficiency strengthens these connections. Activation of items in sparse networks, however, would primarily facilitate semantic access because specific conceptual representations would preferentially activate fewer form competitors. This type of model has much in common with the Context Availability Model (Kieras, Reference Kieras1978; McRae, de Sa, & Seidenberg, Reference McRae, de Sa and Seidenberg1997; Schwanenflugel & Shoben, Reference Schwanenflugel and Shoben1983) which proposes that conceptual information is weakly associated with abstract concepts, but strongly associated with non-abstract ones. Activation of non-abstract conceptual information is more likely to access networks that act together. Both ideas support a perspective that language processing is impacted by differences in how the conceptual system activates these properties of meaning.
4. Conclusion
This study’s results are consistent with the hypothesis that motivated signs activate links between form and meaning. This pattern of lexical access suggests that bimodal bilinguals access translations differently than unimodal bilinguals. Rather than showing increased semantic interference when translating high-imageability concepts like unimodal bilinguals, bimodal bilinguals show increased phonological interference. This is consistent with evidence of semantic facilitation in deaf signers when processing conceptual properties for motivated signs (Thompson et al., Reference Thompson, Vinson and Vigliocco2009), but phonological inhibition when processing form-based properties (Thompson et al., Reference Thompson, Vinson and Vigliocco2010).
Results also indicate that language-internal motivations interact with external ones differently given language experience. The similarity of the experts’ responses in the diagrammatic condition to the phonological condition, where interference is greater for higher imageability, suggests that they process diagrammatic distractors as phonological competitors. Novices, however, appear to process diagrammatic distractors as semantic competitors, since both conditions show interference for targets with low imageability scores.
A usage-based approach to cognitive iconicity (Occhino, Reference Occhino2016; P. Wilcox, Reference Wilcox2000; S. Wilcox, Reference Wilcox2004) provides a framework that simultaneously explains both language-external and internal patterns of motivation as proficiency increases. Concepts with high imageability are more likely to have form/meaning mappings that are accessible to novices, but these mappings are still accessible to experts as well. Motivated signs are not independent of the entrenchment of language internal patterns of form and meaning. This perspective aligns nicely with the work by spoken linguists studying non-Western languages that recognize two groupings of lexical–semantic connections; one-to-one mappings between selected features of a referent and its form, called Absolute iconicity, and forms that are primarily linked to each other and related real-word referents called Relative iconicity (Dingemanse et al., Reference Dingemanse, Blasi, Lupyan, Christiansen and Monaghan2015). This study finds that, rather than absolute iconicity decreasing its impact on language processing in a signed language as proficiency increases, it grows in tandem with relative iconicity, building on the original heuristic aid given by iconic codification of visual stimuli to conceptual storage.
Discussions of cognates in unimodal bilingual studies have striking similarities to this perspective. The cognate status of a word depends on form-based similarity across translations. The Cognate effect is the observation that bilinguals have low latencies in picture naming tasks for items whose translations are phonologically similar across languages (as in the Spanish–English pair guitarra – ‘guitar’) (Christoffels, Firk, & Schiller, Reference Christoffels, Firk and Schiller2007; Costa, Caramazza, & Sebastian-Galles, Reference Costa, Caramazza and Sebastian-Galles2000; Costa, Santesteban, & Caño, Reference Costa, Santesteban and Caño2005). Van Hell and De Groot (Reference Van Hell and De Groot1998) argue that cognate status might influence conceptual processing, since cognates should have larger conceptual overlap compared with non-cognates. Iconicity in signed languages, then, seems to behave like a ‘universal cognate’ in the sense that it activates conceptual representations in contexts where lexical form-based processing would otherwise occur.
Many studies (Bosworth & Emmorey, Reference Bosworth and Emmorey2010; Caselli & Pyers, Reference Caselli and Pyers2017; Caselli et al., Reference Caselli, Sehyr, Cohen-Goldberg and Emmorey2017; Ortega & Morgan, Reference Ortega and Morgan2015; Thompson et al., Reference Thompson, Vinson and Vigliocco2009; Vinson, Cormier, Denmark, Schembri, & Vigliocco, Reference Vinson, Cormier, Denmark, Schembri and Vigliocco2008) have used non-signer judgements of iconicity to assess the effect of this property in language comprehension, production, and learning. Non-signers are insensitive to language-internal patterns that here show significant effects as language proficiency increases. Without accounting for both imagistic and diagrammatic motivations, investigations of iconicity effects in signed languages run the risk of skewing results toward behaviors that are primarily present early in the language learning process and overlook behaviors that surface only after more complete knowledge of a language’s patterns have been acquired.
Appendix A
Stimuli list (English translations)
