1.1. Introduction
Many recent studies have explored how adults acquire a second language (for review see Kroll, Dussias, Bice & Perrotti, Reference Kroll, Dussias, Bice and Perrotti2015). These studies typically include hearing adults learning whose first language is a spoken language and are subsequently acquiring another spoken language. In comparison, relatively fewer studies have investigated individuals who have one spoken language and one sign language (i.e., bimodal bilinguals or M2L2 learners). Furthermore, the study of the bimodal bilingual brain has been mostly restricted to native bimodal bilinguals who acquired their languages simultaneously (Emmorey & McCullough, Reference Emmorey and McCullough2009; Emmorey, Giezen & Gollan, Reference Emmorey, Giezen and Gollan2015; Emmorey, Grabowski, McCullough, Ponto, Hichwa & Damasio, Reference Emmorey, Grabowski, McCullough, Ponto, Hichwa and Damasio2005). Thus, it is important to expand our knowledge of sign language acquisition after a spoken language has already been established because there may need to be substantial neuroplastic reorganization and adaptation to a new visual language, which diverges so greatly from learners’ native spoken language (Li, Abutalebi, Zou, Yan, Liu, Feng, Guo & Ding, Reference Li, Abutalebi, Zou, Yan, Liu, Feng, Guo and Ding2015; Zou, Ding, Abutalebi, Shu & Peng, Reference Zou, Ding, Abutalebi, Shu and Peng2012). Given that learning sign languages as a second language (L2) requires processing in different sensorimotor modalities (M2), M2L2 learners of sign languages are the perfect test case to explore modality-independent and modality-dependent neural mechanisms for L2 acquisition.
There have been several previous neuroimaging studies that have investigated the neural substrates of L2 vocabulary processing as well as functional and structural changes due to increasing proficiency (Grant, Fang & Li, Reference Grant, Fang and Li2015; Perani & Abutalebi, Reference Perani and Abutalebi2005; Saidi, Perlbarg, Marrelec, Pélégrini-Issac, Benali & Ansaldo, Reference Saidi, Perlbarg, Marrelec, Pélégrini-Issac, Benali and Ansaldo2013; Stein, Dierks, Brandeis, Wirth, Strik & Koenig, Reference Stein, Dierks, Brandeis, Wirth, Strik and Koenig2006; Stein, Federspiel, Koenig, Wirth, Strik, Wiest & Dierks, Reference Stein, Federspiel, Koenig, Wirth, Strik, Wiest and Dierks2012). For instance, bilinguals demonstrate prefrontal engagement during L2 lexico-semantic processing, which diminishes with increased L2 proficiency (Grant et al., Reference Grant, Fang and Li2015). Similar structural plasticity has been observed in left prefrontal areas (e.g., inferior frontal gyrus, IFG), such that higher proficiency learners have greater grey matter volume in the left IFG than low proficiency learners. Decreased functional activity and increased grey matter volume are thought to represent enhanced automaticity of lexico-semantic processing, which is due to reduced neural load and more efficient pathways, respectively, and is also a function of proficiency (Grant et al., Reference Grant, Fang and Li2015; Ishikawa & Wei, Reference Ishikawa and Wei2009; Stein et al., Reference Stein, Federspiel, Koenig, Wirth, Strik, Wiest and Dierks2012). Also, there is evidence that greater activation of cognitive control regions is required due to parallel competition between lexical items at initial stages of lexical acquisition (Abutalebi, Reference Abutalebi2008; Grant et al., Reference Grant, Fang and Li2015; Van Hell & Tanner, Reference van Hell and Tanner2012). Despite our growing knowledge of the neural substrates of L2 (within-modality) lexical acquisition, we still know relatively little about the acquisition of lexical signs.
The study of second language acquisition of sign language in adulthood is important to both our understanding of neuroplasticity and second language theory. Sign languages differ substantially from spoken languages in their primary articulators. Sign languages rely on arbitrary manual-visual phonetic codes to convey messages. Additionally, sign languages exploit the use of spatial dependencies for grammatical and discourse purposes (Sandler & Lillo-Martin, Reference Sandler and Lillo-Martin2006). The phonology of sign languages, as such, includes sublexical features that relate to the hands in space. The primary sublexical features of manual signs are hand configuration, palm orientation, place of articulation, and movement (Baker, van den Bogaerde, Pfau & Schermer, Reference Baker, van den Bogaerde, Pfau and Schermer2016; Brentari, Reference Brentari1998). Therefore, tracking of hands through space is important for the lexico-semantic processing of sign languages. Given that bimodal bilinguals and M2L2 learners are hearing adults who have experience with both spoken language (oral-auditory modality) and sign language (manual-visual modality), it is of prime interest to understand both the modality-specific and amodal aspects of language acquisition and neuroplasticity.
Studies on simultaneous bimodal bilingualism have delineated differences and similarities in speech and sign processing. Bimodal bilinguals (who acquired their two languages simultaneously) show greater activation in the bilateral parietal cortex and bilateral occipitotemporal cortex during single sign comprehension (Emmorey, Luk, Pyers & Bialystok, Reference Emmorey, McCullough, Mehta and Grabowski2014; Söderfeldt, Ingvar, Rönnberg, Eriksson, Serrander & Stone-Elander, Reference Söderfeldt, Ingvar, Rönnberg, Eriksson, Serrander and Stone-Elander1997). These areas are thought to be unique to the processing of sign languages, and have been activated by spatial classifiers and verbs in American Sign Language (ASL) in addition to other demanding visual processing tasks. Sequential bimodal bilinguals show additional involvement of regions associated with language control, such as the anterior cingulate and basal ganglia (Li et al., Reference Li, Abutalebi, Zou, Yan, Liu, Feng, Guo and Ding2015; Zou et al., Reference Zou, Ding, Abutalebi, Shu and Peng2012). However, the time course for developing such activation patterns in late learners is unclear. Moreover, little is known about the changes in processing that occur in the bimodal bilingual or M2L2 learner brain between the initial, naïve state of learning and a more proficient level of language knowledge. Perhaps learners attempt to process lexical signs as gestures, but move towards lexico-semantic processing with time (see Williams, Darcy & Newman, Reference Williams, Darcy and Newman2016). Additionally, there is behavioral evidence that late M2L2 learners of sign language often have difficulty acquiring the visual-manual characteristics of the target sign language. Specifically, not only do learners have difficulty acquiring sign language movement, but they also show an overreliance on the handshape of the sign (Bochner, Christie, Hauser & Searls, Reference Bochner, Christie, Hauser and Searls2011; Pichler, Reference Pichler, Mathur and Napoli2011; Grosvald, Lachaud & Corina, Reference Grosvald, Lachaud and Corina2012; Morford, Grieve-Smith, MacFarlane, Staley & Waters, Reference Morford, Grieve-Smith, MacFarlane, Staley and Waters2008; Morford & Carlson, Reference Morford and Carlson2011; Rosen, Reference Rosen2012; Schlehofer & Tyler, Reference Schlehofer and Tyler2016; Williams & Newman, Reference Williams and Newman2015, Reference Williams and Newman2016). Therefore, it may be that the ability to acquire an L2 sign language is based on the ability to have automatized processing in regions involved in spatial processing, biological motion, and hand processing, which subserve visuomotor phonetic processing.
A previous longitudinal neuroimaging study showed that M2L2 learners of ASL progressed stepwise through stages of lexical processing during approximately one year of university-level ASL instruction (Williams, Darcy & Newman, Reference Williams, Darcy and Newman2016). Before exposure to sign language the learners processed ASL lexical signs using non-linguistic (or at least phonetic) regions implicated in visuospatial and motor processing, with significant activation in occipital (e.g., calcarine sulcus) and parietal (e.g., superior parietal lobule) lobes. After one semester of exposure, the learners show greater cortical recruitment in supramarginal gyrus and putamen, suggesting that they transitioned into a phonological processing stage (Booth, Wood, Lu, Houk & Britan, Reference Booth, Wood, Lu, Houk and Britan2007). Subsequently, the learners transitioned into a lexico-semantic processing stage after 10 months of exposure, where they recruited more activation in the left inferior frontal gyrus. It is possible that the progression of lexical processing is different for those who are poor learners, which was not investigated in the aforementioned study. It could be posited that poor vocabulary learners perseverate on the modality-specific aspects of sign language (e.g., visuospatial and visuomotoric features), or do not fully automatize such processing routines, which may block or delay successful sign language acquisition.
1.2. Aims
Given the gap in knowledge about M2L2 sign language acquisition, the present study aimed to characterize the neural substrates of vocabulary acquisition and lexical sign processing in hearing M2L2 learners of American Sign Language (ASL) at early stages of acquisition. This study used the same set of learners in Williams et al. (Reference Williams, Darcy and Newman2016) to investigate this question more directly. Specifically, there were three main aims of the present study:
The first aim was to characterize the pattern of vocabulary acquisition across the first year of instruction. Not only has there not been a prior study that has examined the lexical acquisition trajectory of M2L2 learners in foreign language classrooms, which is theoretically motivated, but also it is meaningful from an applied perspective so that we can better understand whether acquisition rates change during certain semesters. The expectation, of course, was that students would acquire new lexical knowledge in a relatively equal stepwise fashion.
Second, since learners have little knowledge of sign language at the start of their L2 instruction, we aimed to characterize how limited vocabulary knowledge affects M2L2 sign language processing. The individual variability in brain response prior to learning may be predictive of future attainment. For example, we hypothesized that students who later have a smaller sign vocabulary process lexical signs as holistic visual objects instead of decomposable linguistic objects which is how we hypothesize that those who later develop a larger sign vocabulary. This difference in initial processing, again, is expected to be observed in the brain activation patterns and will impact later learning.
Our third aim was to characterize how gains in vocabulary knowledge affect neural processing after 10 months of L2 instruction. Specifically, we examined whether the poor vocabulary learners, while controlling for gender differences, persisted with the holistic processing strategy after one year of instruction, or whether their strategy had shifted. We hypothesized that perhaps continued processing of signs using areas dedicated to biological motion (e.g., temporoparietal and occipitotemporal) and hand (e.g., intraparietal sulcus, putamen) processing would suggest the use of relatively less efficient strategies that focus on decoding the movement. Furthermore, given the year-long exposure to lexical signs, we might expect greater phonological and/or lexico-semantic processing (e.g., inferior frontal gyrus (IFG); “Broca's area”) at T2 because previous studies have shown lower proficiency learners recruit greater activation in IFG when viewing lexical items (Grant et al., Reference Grant, Fang and Li2015).
By examining these questions, the present study was able to elucidate how vocabulary knowledge affects the neural processing of ASL, especially in poor learners. Theoretically, it was aimed at corroborating behavioral studies that have characterized deficits in M2L2 acquisition. From a practical perspective, if the neurocognitive profiles of poor vocabulary learners can be identified, which may be indicative of a global sign language deficit, then potential classroom interventions that aim to improve M2L2 acquisition can be developed to address those deficiencies. We aimed to answer these questions using a longitudinal design. First, changes in vocabulary knowledge over 10 months, or two semesters, of ASL exposure were tracked, which was included to address our first aim and to determine whether our subset of learners was representational of a larger sample. The pattern of neural activation was examined in response to viewing ASL lexical signs before and during their L2 instruction in a subset of 12 hearing adult learners of ASL. The use of a longitudinal neuroimaging design affords us the opportunity to examine vocabulary acquisition without the confounding factors that plague cross-sectional designs (e.g., different brains, language and scholastic experiences, instructors, coursework, etc.).
2.1. Materials and Methods
2.1.1. Participants
Thirty-four (male = 10) hearing English-speaking college students participated. All participants were right-handed according to the Edinburgh Handedness scale (M = 85.6, SD = 19.1). All participants reported English as their first language. The participants in this study were recruited from introductory American Sign Language (ASL) courses at Indiana University. The participants had little to no exposure to ASL (or any other spoken second language) before enrollment in the ASL course. They were tested at three different time points: before ASL instruction (T0); at the end of one semester of instruction (T1); and at the end of a second semester of instruction (T2). All participants gave written consent to perform the experimental tasks, which was approved by the Indiana University Institutional Review Board.
At baseline (T0) the 34 participants had a mean age of 20.6 (2.5). Participants were recruited during their first week of Beginning ASL I enrollment. On average, they had 1.06 (range = 0 – 5, SD = 1.49) hours of instruction. According to course instructors, the instruction in the first week of classes included introduction to the course, the target language and culture, but little linguistic instruction. Furthermore, most to all instruction was conducted in English during the first week. That is, these participants had minimal to no linguistic knowledge of the target language. Additionally, the participants rated their proficiency in both English and ASL on a scale from 1 to 7 (1 = “Almost None”, 2 = “Very Poor”, 3 = “Fair”, 4 = “Functional”, 5 = “Good”, 6 = “Very Good”, 7 = “Like Native”) for their understanding and fluency abilities. Their average scores across both categories were 7 (0) for English and 1.15 (0.31) for ASL.
After one semester (approximately 13 weeks later), the participants were brought back. Twenty-five participants (male = 7) returned (i.e., attrition of nine participants) for their first post-exposure follow-up (T1). On average they had 43.98 (1.12) hours of instruction. They rated their ASL ability as a 3.32 (1.05) and their English ability as 7 (0). Course grades were also recorded and they received an average of 91.12% (4.87) in their ASL course.
After a second semester of ASL training, 12 participants (male = 5) returned (i.e., attrition of 13 participants) for their second post-exposure session (T2). On average they had 89.5 (1.95) hours of instruction and rated their ASL ability as a 3.92 (0.63). At T2, participants had an average of 91.7% (4.64) in their second ASL course.
2.1.2 Procedure
2.1.2.1 Vocabulary Test
Participants took a speed vocabulary-translation test to obtain a gross measure of their vocabulary knowledge over time. The test was constructed by taking the signs from the current ASL textbooks across all four semesters of the current ASL curriculum (Smith et al., Reference Smith, Lentz and Mikos1988a, Reference Smith, Lentz and Mikos1988b, Reference Smith, Lentz and Mikos2008). Based on data retrieved from ASL-LEX (Caselli et al., Reference Caselli, Sehyr Sevcikova, Cohen-Goldberg and Emmorey2016), the signs included in this test were 142 signs total and were relatively high frequency (M = 4.59, SD = 1.16; very frequent = 7, infrequent = 1) and non-iconic (M = 2.29, SD = 1.69; very iconic = 7, arbitrary = 1). During the speeded vocabulary-translation task, participants viewed video clips of the signed words produced by a native signer only once (Figure 1). Participants were required to type in the English translation (or a guess) within five seconds. An automated procedure was used to score their translations for correct answers, including any synonyms (e.g., bathroom or restroom would be accepted for bathroom). A total score, correct out of 142, was used as the participants proficiency score.
Vocabulary task design.
2.1.2.2. Phoneme Categorization Task
The impact of vocabulary knowledge on L2 sign language processing was also tracked using functional magnetic resonance imaging (fMRI). The participants performed a phoneme categorization task, which lasted nine minutes and included 30 categorization trials in total. Participants viewed a native signer signing words with the speaker's full face and torso shown in front of a blue-gray backdrop. All stimuli (see Appendix) were high frequency monomorphemic signs from various word classes (i.e., nouns, verbs, adjectives; Caselli et al., Reference Caselli, Sehyr Sevcikova, Cohen-Goldberg and Emmorey2016). Signs were split into two groups: signs with place of articulation (i.e., location) on the head or face and signs with the location on the body, non-dominant hand, or neutral space (i.e., not on the face). Recall that place of articulation (or location) is a sublexical feature of sign language and is considered similar to a phoneme (Baker et al., Reference Baker, van den Bogaerde, Pfau and Schermer2016; Brentari, Reference Brentari1998; Sandler & Lillo-Martin, Reference Sandler and Lillo-Martin2006). As such, this task is classified as a phoneme classification task. The selection of face and not-face locations was to match for the bilabial (visible on the face) and non-bilabial (not visible on the face) task that was implemented in Williams et al. (Reference Williams, Darcy and Newman2015, Reference Williams, Darcy and Newman2016), while still being phonologically valid in ASL. Although the task itself was meaningless, it required participants to phonetically process the signs by splitting the signs into these two conditions, which the viewing of signs was thought to engage automatic lexical processing (similarly for audiovisual speech, see Campbell et al., Reference Campbell, MacSweeney, Surguladze, Calvert, McGuire, Brammer, David and Suckling2001).
The functional task was presented in an event-related design. For each trial a 500-millisecond fixation point was presented before the video appeared. Each stimulus video varied in duration (M = 1593.33, SD = 2.53 ms) and was followed by a jittered interstimulus interval (ISI range = 4000 – 8000, M = 6000 ms). Participants were told to press the right index finger for signs that were produced on the face, and to press the left index finger for signs that were produced on the body. They were instructed to make their responses as quickly and accurately as possible. In addition to the ISI, a 30 second fixation was presented at the beginning of the task and was used as a baseline.
2.1.3. Imaging Parameters
Participants underwent two scans using a 32-channel head coil and a Siemens 3 Tesla TIM Trio MRI scanner. The first scan was an anatomical T1-weighted scan used to co-register functional images. An MPRAGE sequence (160 sagittal slices; FOV = 256 mm, matrix = 256x256, TR = 2300 ms, TE = 2.91 ms, TI = 900 ms, flip angle = 9°, slice thickness = 1 mm, resulting in 1-mm × 1-mm × 1-mm voxels) was used. The remaining scans were the experimental functional multiband EPI scans (59 axial slices using the following protocol: field of view = 220 mm, matrix = 128x128, iPAT factor = 2, TR = 2000 ms, TE = 30 ms, flip angle = 60°, slice thickness = 3.8 mm, 0 gap).
2.1.4. Analysis Approach
Two different analysis methods (R software) were used to analyze the changes in vocabulary knowledge across the three time points. Due to significant amount of attrition, only 12 subjects attended all three sessions.Footnote 1 Given that there was attrition in the number of subjects over time, the statistical methods needed to account for missing data. A typical analysis of variance would not be appropriate for missing data, or different sample sizes, across each time point. Therefore, two methods that are robust to missing data were used. First, a predictive mean matching method (k = 5) was used, in which missing values from the attrited participants were imputed and accessed using pooled data across multiple regressions (Landerman, Land & Pieper, Reference Landerman, Land and Pieper1997). Including T1 was not only advantageous because it improves the imputation approach by contributing more data, but also it provides clearer understanding of vocabulary growth over time. Given some downsides to imputation (see Landerman et al., Reference Landerman, Land and Pieper1997), a linear mixed-effect model was also performed (Verbeke & Molenberghs, Reference Verbeke and Molenberghs2009). Both methods have their respective downsides when it comes to missing data; thus, any convergence of results between the two models as taken as an indicator of confidence. A correlation was performed between T0 and T2 scores with the vocabulary growth (VocabT2 – VocabT0) for the 12 recurrent subjects to investigate whether the change was due to performance at baseline or after sign language exposure.
Functional images were analyzed using SPM8 (Wellcome Imaging Department, University College, London, UK; freely available at http://fil.ion.ucl.ac.uk/spm). As mentioned previously, only 12 learners participated in all three sessions; thus, only 12 subjects were included in the multiple linear regression analysis of the BOLD responses to ASL lexical signs. Given that vocabulary scores were relatively the same between T1 and T2, we decided to only include T0 and T2 scanning sessions in the current analysis; additionally, T2 was thought to maximize the potential of finding neuroplastic effects given that it was approximately 10 months post-baseline compared to 3 months at T1 and had the highest amount of variability in vocabulary scores. During preprocessing images were corrected for slice acquisition timing, and resampled to 2 mm3 isovoxels, spatially smoothed with a Gaussian filter with a 4 mm3 FWHM kernel. All data were high-pass filtered at 1/128 Hz to remove low-frequency signals (e.g., linear drifts). Six-parameter rigid body motion correction was performed and motion parameters incorporated into the design matrix as nuisance regressors in the General Linear Model (GLM). Each participant's anatomical scan was aligned and normalized to the standardized SPM8 T1 template and then fMRI data were co-registered to high-resolution anatomical images.
At the individual (first) level, statistical analysis was performed using the standard GLM with Gaussian random fields in SPM8. The ASL stimulus onsets and durations were entered as our main regressors in the GLM in order to model the hemodynamic response function with stimulus events (Friston et al., 1995). BOLD signal from a common fixation baseline was subtracted from BOLD related to viewing ASL signs and was used as our estimated contrast. For the second level analysis on group data, multiple regression analyses were performed at each time point (T0 and T2) using each subject's ASL-Fixation contrast with each learner's respective vocabulary score at that time point entered as a covariate. Vocabulary score was the only covariant added to the model. Both positive and negative correlations were carried out. To correct for multiple comparisons, the image dimensions and the smoothing parameter of the processed data were entered into AFNI's 3dClustSim program in order to determine cluster sizes that would be significant given our voxel-wise p-value. Given the results of 5000 Monte Carlo simulations, both positive and negative regressions were performed using a voxel-wise p < 0.005, which was corrected to alpha < 0.05 with a cluster extent threshold of 62 voxels or moreFootnote 2.
Since vocabulary score was the main predictor in our multiple regression model, we wanted to make sure that there were no confounding factors, like gender or task performance, that could explain our results. It is possible that task difficulty could explain the results; however, it is only intuitive that there will be a significant correlation between vocabulary score and the learner's ability to classify sublexical features of sign language. If vocabulary score and task performance are correlated then following common procedures for dealing with collinearity in predictor variables, one will be excluded. In this case, we will exclude task performance because it does not directly address the current aims. If they are not correlated, then vocabulary will remain within the model.
Previous studies have also shown that women are better at acquiring second languages than men (van der Slik et al., Reference van der Slik, van Hout and Schepens2015). Therefore, we also analyzed whether our vocabulary scores differed based on gender using a point-biserial correlation at each time point separatelyFootnote 3. If there is a significant correlation with gender, then parameter estimates of the BOLD signal will be extracted from the significant clusters in our whole brain analysis by defining a 5-mm3 sphere at the center-of-mass for each significant cluster using Marsbar, an SPM toolbox (Brett et al., Reference Brett, Anton, Valabregue and Poline2002). Parameter estimates will be correlated with vocabulary scores while controlling for gender in an ad-hoc partial correlation. Only the clusters that survive the gender-correction will be considered significant in our analyses. This method allows us to control for cofounding factors while maintaining the integrity of our whole brain analysis.
The results from the Checking Multicollinearity section (2.2.2) will determine how the multiple regression analysis will be carried out with respect to confounding predictors.
2.2. Results
2.2.1. Vocabulary Growth
Figure 2 illustrates the average vocabulary score for the ASL learners at each time point. The regression results showing the size of vocabulary increases over time are presented in Table 1. Results from the pooled regressions after missing data was estimated with ten imputations found that the intercept coefficient has a value of 7.03 (SE = 2.11), T1 has a value of 35.9 (SE = 4.78), and T2 has a value of 40.4 (SE = 12.80). This indicates that relative to the baseline, there was a significantly larger increase in vocabulary scores at T1 (t = 7.52, p < 0.001), while T2 only further increased the outcome by around five lexical signs (t = 3.16, p < 0.05). T-values were large (greater than 2.0) after both semesters and had p-values less than 0.05, meaning the differences were statistically significant when comparing them to the baseline. Results from the linear mixed-effects model corroborated these findings. There was a significant increase relative to baseline for both T1 (t = 17.58, p < 0.001) and T2 (t = 17.09, p < 0.001).
Mean vocabulary scores at each time point averaged across all subjects.
Vocabulary scores at each time point
No significant correlation was found between vocabulary growth and T0 gross vocabulary score [r = 0.218, p = 0.496], but there was a significant correlation between vocabulary growth and T2 gross vocabulary score [r = 0.918, p < 0.0001]. Therefore, those who had the largest increases in their score also had the largest T2 vocabulary, but the increase was not dependent on their baseline vocabulary score at T0.
2.2.2. Checking Multicollinearity
Task performance on the in-scanner phoneme categorization task was arcsine transformed and correlated with vocabulary score and gender at T0 and T2 separately. The analysis revealed trending or significant correlations between vocabulary score and performance at both T0 (r = 0.563, p = 0.057) and T2 (r = 0.989, p < 0.001). However, there was no significant effect of gender on task performance at either T0 (r = 0.083, p = 0.797) or T2 (r = −0.486, p = 0.055).
Point-biserial correlations between T0 vocabulary score and gender showed moderate negative correlation (r = −0.541) that was trending towards significance (p = 0.070), which means that those participants who reported as female tended to have higher vocabulary scores than those who reported as male. Similar correlations between T2 vocabulary and gender revealed a significant negative correlation (r = −0.714, p = 0.009).
These results revealed that phoneme categorization and vocabulary knowledge are significant correlated, especially after one year of sign language instruction; however, no correction for task performance will be used for the multiple linear regression analysis (in section 3.2.3) since they are adversely collinear and statistically represent the same amount of variability. On the other hand, gender did significantly influence vocabulary scores, where female learners outperformed male learners. As such, activation from significant clusters in our whole-brain multiple linear regression will be corrected for gender in our ad-hoc analysis.
3.2.3. Multiple Linear Regression
Figure 3 shows the increased activation in response to viewing ASL signs that is correlated with poor vocabulary knowledge. Results from the multiple regression analysis (see Table 2) indicated a negative correlation such that subjects with lower sign vocabulary knowledge at baseline (T0) had a significant increase in activation in bilateral inferior frontal gyrus, left insula, and left middle frontal gyrus. All of the clusters survived correction for gender differences in vocabulary knowledge. There was also increased activation in left anterior cingulate gyrus and the right anterior portion of the superior frontal gyrus; however, these clusters only trended towards significance once gender was controlled. There were no significant positive correlations with vocabulary score. In other words, there were not any regions that showed increased activation to viewing ASL signs that were positively correlated with vocabulary knowledge.
Activation shown here represents BOLD signal which is negatively correlated with vocabulary scores at baseline (T0; top) and at the second post-exposure scan (T2; bottom) for the subset (n = 12) of learners. Therefore, activation seen here is representative of increased processing in these regions in response to viewing ASL signs for those with poor vocabulary knowledge. Note: there were no significant correlations with vocabulary score at T1.
Multiple regression analysis (p-corrected < 0.05; k = 62)
Note: †p<0.1, *p < 0.05, **p < 0.01, ***p < 0.001
Results from the multiple regression analysis at T2 showed that there were significant negative correlations in bilateral prefrontal cortex (including superior frontal, middle frontal, and inferior frontal gyri), the right posterior temporoparietal junction (around the superior temporal cortex), the right middle temporal gyrus and the left temporal pole. All of these correlations survived correction for gender except for the right inferior frontal gyrus, only trended towards significance. There were no significant positive correlations with vocabulary score at T2.
4. General discussion
The overall objective of the present longitudinal study was to broadly capture how second language (L2) signed vocabulary knowledge affects neural activation during the processing of ASL. Specifically, there were three main research aims. First, we wanted to characterize vocabulary acquisition over two semesters of instruction. Second, given that these learners had little knowledge of sign language at the start of their L2 instruction, one aim was to characterize how limited vocabulary knowledge affects processing of a novel second language. The last aim was to characterize how vocabulary knowledge affects neural processing after 10 months of L2 instruction; specifically, we were interested in how vocabulary knowledge modulated the difference in ASL processing at T0 and after two semesters of instruction (T2) in order to capture potentially diagnostic neurocognitive profiles of learners who struggle with acquiring sign. The results from the present study indicated that those students who had poor ASL vocabulary knowledge were associated with increased activation in regions that are commonly involved in lexico-semantic and phonological processing and decision making as well as modality-specific processing of multimodal integration, salience and biological motion processing. This pattern fits into our overall theoretical framework that nonnative signers struggle with visuomotoric properties, namely movement, that impede their ability to acquire sign language.
In order to characterize how very limited knowledge may impact vocabulary acquisition and processing, ASL learners who had relatively little knowledge of ASL at the beginning of L2 instruction were studied. Despite their limited knowledge, these learners varied in their lexicon size. It was hypothesized that learners with poor incoming vocabulary knowledge would show greater activation in regions associated with modality-specific visuomotoric phonetic processing and, perhaps, lexico-semantic processing for known signs that are phonologically related to the target sign (neighbors). The pattern of results from baseline measurements (T0) confirmed our predictions, where increased activation in bilateral inferior frontal gyrus (IFG), left insula and anterior cingulate cortex (ACC), left middle frontal gyrus, and the anterior portion of the right superior frontal gyrus was observed. We will speculate on the role of each activated region in turn.
The left anterior insula has been implicated in language processing (Ardila, Bernal, & Rosselli, Reference Ardila, Bernal and Rosselli2014). More generally, however, the activation of the left anterior insula may be more indicative of increased difficulty in cross-modal multisensory integration (Allen et al., Reference Allen, Emmorey, Bruss and Damasio2008; Kurth et al., Reference Kurth, Zilles, Fox, Laird and Eickhoff2010). Allen and colleagues argue that the insula's connectivity to sensory cortices lends itself to multisensory integration, especially for sign language which requires visual-tactile-vestibular integration. Similarly, co-activation of the frontoinsular cortex and the ACC point to increased salience processing (Seeley et al., Reference Seeley, Menon, Schatzberg, Keller, Glover, Kenna, Reiss and Greicius2007; Uddin, Reference Uddin2015), where the insula is thought to be important for interoceptive and viceromotor body processing. The neural signal from frontoinsular cortex flows to the central executive network, including the ACC, which initiates decision-making (Uddin, Reference Uddin2015). High levels of activation in the anterior cingulate and other prefrontal regions suggests that these learners also required greater effort in areas of decision making and control (Allman et al., Reference Allman, Hakeem, Erwin, Nimchinsky and Hof2001; Cohen et al., Reference Cohen, Botvinick and Carter2000; Sohn et al., Reference Sohn, Albert, Jung, Carter and Anderson2007) and perhaps the need to cope with greater task demands (Burgess et al., Reference Burgess, Gilbert and Dumontheil2007). As such, it can be hypothesized that learners with smaller vocabulary sizes require more neural resources in order to integrate the visual and motoric salience of the signs so that the phonological content can be determined; therefore, poor vocabulary knowledge is likely linked to less efficient visuomotor phonetic perception, which requires greater neural resources.
Support for poor visuomotor phonetic perception is indicated by the distributed activation in the frontal lobe. The anterior portion of the superior frontal gyrus has been implicated in pseudosign recognition in hearing nonnative signers (Emmorey, Xu, & Braun, Reference Emmorey, Xu and Braun2011). Additionally, activation in the middle frontal gyrus corresponds to spatial judgements and perspective-taking (Kaiser et al., Reference Kaiser, Walther, Nennig, Kronmüller, Mundt, Weisbrod, Stippich and Vogeley2008; Smith et al., Reference Smith, Davis, Niu, Healy, Bonilha, Fridriksson and Rorden2010) and activation in bilateral inferior frontal gryi are implicated in movement imitation (Corina & Knapp, 2006; Newman-Norlund et al., Reference Newman-Norlund, van Schie, van Zuijlen and Bekkering2007). Particularly, left IFG has also been implicated in phonological processing of the hand in sign language (Corina, Reference Corina1999).
Together, these findings suggest that sign language learners with poorer vocabulary knowledge, and perhaps less exposure to sign language, require greater activation in regions involved in multimodal integration, salience, visuospatial and motor phonological processing, and decision making. This pattern of activation might be a predictor of M2L2 deficits in vocabulary acquisition later down the line. This predictive power must be tested in future experiments, but we can evaluate its potential validity by examining activation after one year of exposure.
After two semesters of L2 instruction, or about 10 months, there was a significant increase in the learners’ vocabulary scores. The study also aimed to investigate whether there was a change in the neural substrates recruited for those learners with poor vocabulary knowledge. In fact, the overall pattern of activation was similar at T2 insofar as we observed recruitment of prefrontal and frontal cortex. This means that learners with poor vocabulary scores, even after 10 months of exposure, required greater neural resources when viewing ASL signs compared to those with higher vocabulary scores. Particularly, the left IFG was activated. Left IFG has been implicated in the selection of semantic information, where increased activation is representative of more effortful processing and increased difficulty in selection among competing information (Fiez, Reference Fiez1997; Sakai, Reference Sakai2005; Thompson-Schill et al., Reference Thompson-Schill, D'Esposito, Aguirre and Farah1997; Vigneau et al., Reference Vigneau, Beaucousin, Herve, Duffau, Crivello, Houde, Mazoyer and Tzourio-Mazoyer2006). Right IFG has also been shown to be activated during word retrieval when more processing is needed, including bilinguals (Blasi et al., Reference Blasi, Young, Tansy, Petersen, Snyder and Corbetta2002; Marian et al., Reference Marian, Spivey and Hirsch2003, Reference Marian, Shildkrot, Blumenfeld, Kaushanskaya, Faroqi-Shah and Hirsch2007). Previous studies on L2 vocabulary acquisition have also shown increased grey matter volume in left IFG for high proficiency learners, suggesting more controlled automatic lexical processing (Stein et al., Reference Stein, Federspiel, Koenig, Wirth, Strik, Wiest and Dierks2012). Increased functional activation was seen in IFG for low-proficiency learners relative to high proficiency learners (Ishikawa & Wei, Reference Ishikawa and Wei2009). There was also differential activation at T2 in the temporal lobes. For instance, there was additional bilateral temporal lobe activation. Left anterior temporal pole activation has also been shown to be involved in the semantic network (Price, Reference Price2010). As such, it can be argued that there was greater recruitment needed for word retrieval for these learners with poor vocabulary knowledge. Greater word retrieval difficulties may be due to poor phonological processing more generally.
There is a consistent relationship between L2 phonological processing skills and vocabulary acquisition in spoken languages (Bundgaard-Nielsen, Best, Koors & Tyler, Reference Bundgaard-Nielsen, Best, Kroos and Tyler2012; Bundgaardd-Nielsen, Best & Tyler, Reference Bundgaard-Nielsen, Best and Tyler2011a,Reference Bundgaard-Nielsen, Best and Tylerb; Darcy, Park & Yang, Reference Darcy, Park and Yang2015). In other words, previous studies have shown that L2 learners’ vocabulary size expands as a function of being able to make phonological contrasts in their L2. Given this relationship, it can be hypothesized that the aforementioned deficits in the M2L2 population might be tied to poor phonological processing, especially given the activation in left IFG. Studies of both spoken and signed language have shown left IFG is related to phonological processing (Corina, Reference Corina1999; Corina & Knapp, 2006; Emmorey, Reference Emmorey and Arthur2015; Heim et al., Reference Heim, Eickhoff, Ischebeck, Friederici, Stephan and Amunts2009; Vigneau et al., Reference Vigneau, Beaucousin, Herve, Duffau, Crivello, Houde, Mazoyer and Tzourio-Mazoyer2006; Williams et al., Reference Williams, Darcy and Newman2015). For example, Corina (Reference Corina1999) found that left inferior frontal areas are important for recognition of bracheomanual articulation of signs. Moreover, Emmorey and colleagues (Reference Emmorey, Giezen and Gollan2015) similarly found that left IFG is important for complex manual movements. Activation in the supplementary motor area suggest potential motor simulation for bimanual coordination and movement sequence timing (Serrien et al., Reference Serrien, Strens, Oliviero and Brown2002; Shima & Tanji, Reference Shima and Tanji1998). Therefore, after more experience with signs, those learners who have poor vocabulary knowledge also required greater phonological activation when viewing ASL signs. This may be a result of poor modality-specific (i.e., hand and motion) processing that subserves sign language phonology contrasts.
Right hemispheric recruitment of the cortex surrounding temporoparietal junction has been implicated in hand processing and biological motion (see Puce & Perrett, Reference Puce and Perrett2003 for a review). Such an activation pattern may indicate difficulty in acquisition of a visual phonology. Difficulty to acquire visual phonology has been reported several times in L2 learners of sign language (Bochner et al., Reference Bochner, Christie, Hauser and Searls2011; Grosvald et al., Reference Grosvald, Lachaud and Corina2012; Morford et al., Reference Morford, Grieve-Smith, MacFarlane, Staley and Waters2008; Morford & Carlson, Reference Morford and Carlson2011; Pichler, Reference Pichler, Mathur and Napoli2011; Rosen, Reference Rosen2012; Schlehofer & Tyler, Reference Schlehofer and Tyler2016; Williams & Newman, Reference Williams and Newman2015, Reference Williams, Darcy and Newman2016). Behavioral data has shown that L2 learners of sign language often have more difficulty in processing the handshape and movement phonological parameters relative to other parameters (Bochner et al., Reference Bochner, Christie, Hauser and Searls2011; Morford & Carlson, Reference Morford and Carlson2011; Williams & Newman, Reference Williams and Newman2015, Reference Williams and Newman2016). Movement is important for sign acquisition because the phonological sequencing of the syllable (i.e., sonority) is directly related to movement, which is the syllable nucleus (e.g., Brentari, Reference Brentari1998). Therefore, poor inability to process the visuomotor phonetic properties of signs, particularly movement, hinder the construction of legal syllable structure, potentially preventing consolidation into or the retrieval from lexical memory. Here, we may have some of the first neural evidence that difficulty with biological motion processing, as indicated by increased activation, which underlies the phonetic and phonological foundation of a visual language, might contribute to poor sign language vocabulary acquisition.
Taken together, the results of the present study revealed that second language learners require activation of modality-independent neural substrates for lexico-semantic processing, such as the inferior frontal gyrus and temporal pole. Additionally, the results from the present study indicated that L2 learners of sign language require automatized processing in areas involved in multimodal integration, salience, biological motion processing and motor simulation. This is the first longitudinal neuroimaging study that has investigated modality–independent and –dependent mechanisms for second language acquisition of sign language in relation to measure of proficiency (see Williams et al., Reference Williams, Darcy and Newman2016 for measures related to length of exposure). Therefore, this study provides additional neural evidence that second language sign language proficiency (via lexical knowledge) rests on the ability to acquire and process visual phonology. It should be noted that the present study was conducted on a small sample of 12 learners, despite their relative representation of a larger group of learners. Future studies will need to be conducted to examine whether this is a reliable effect and whether these profiles are truly diagnostic of future, perhaps long-term, deficits in M2L2 acquisition.
Appendix
Words included in the fMRI task
