Reading in one language is an essential tool for life to all of us. In recent decades, the ability to read in more than one language has also become more and more important due to globalization. Nevertheless, it remains unclear whether people who read in two languages (i.e., bilinguals) process words in the same way as those read in only one language (i.e., monolinguals). Bilinguals differ from monolinguals in their language experience in a number of ways. For example, bilinguals generally have a larger vocabulary size (e.g., the number of orthographic forms acquired in one's language(s)) than monolinguals, as the same sets of objects or concepts are represented by different words in their two languages; for example, the same red fruit is named apple in English and manzana in Spanish. For languages that have similar alphabets, such as English and Spanish, bilinguals of these languages have to learn to recognize many more visual word forms that are composed of a serial order of letters from a restricted set than do monolinguals. Thus, words in these bilinguals’ vocabulary may share more common letters on average and look visually more similar to each other than those in a monolingual's vocabulary. Consequently, reading in these bilinguals may involve more complex processing of visual details than in monolinguals. In addition, bilinguals may have to cope with differences in the linguistic nature of the two languages, such as word-to-sound mapping, number of phonemes and graphemes/letters, and orthography/word forms. For instance, sound information can usually be obtained from the visual word form of alphabetic English by decomposing words into letters/graphemes, which map directly to phonemes. In contrast, this grapheme–phoneme correspondence does not apply to logographic Chinese. Thus, Chinese–English bilinguals have to learn to process visual words in their bilingual vocabulary differently due to this difference, and these visual processing skills may also influence each other. In short, compared with one language, mastering two languages may pose more visual processing demands, and the differences in visual experience between monolinguals and bilinguals may lead to different functional organizations of the brain in visual processing. Therefore, in the current study, we aim to investigate how visual word processing is modulated by bilingual language experience, using both behavioral and computational approaches.

In functional imaging studies, differences between monolinguals and bilinguals in brain activation for language processing have been observed. For example, in an fMRI study by Kovelman, Baker and Petitto (Reference Kovelman, Baker and Petitto2008), in which participants performed an English sentence semantic judgment task, greater activation in the classical language processing areas (i.e., left inferior frontal cortex) was observed in Spanish–English bilinguals than in English monolinguals. In a more recent magnetoencephalography (MEG) study by Wang, Xiang, Vannest, Holroyd, Narmoneva, Horn, Liu, Rose, deGrauw and Holland (Reference Wang, Xiang, Vannest, Holroyd, Narmoneva, Horn, Liu, Rose, deGrauw and Holland2011), brain activation was recorded during a single-word audio-visual matching task. When both English monolinguals and Mandarin–English bilinguals were processing English, in contrast to Kovelman et al.'s (Reference Kovelman, Baker and Petitto2008) study, monolinguals showed stronger activation in the classical language processing areas than bilinguals, whereas bilinguals showed activation in the right hemisphere (RH) (i.e., the right inferior frontal cortex and inferior parietal lobe) that were not observed in monolinguals. These results suggest that monolinguals and bilinguals may use different neural mechanisms for processing the same language, and that bilinguals with different language experiences may also process the same language differently.

In addition to language processing, it has been shown that bilinguals have reduced lateralization in some visual tasks that typically involves RH lateralization compared with monolinguals, such as visuospatial processing (e.g., Witelson, Reference Witelson1976) and face processing (e.g., Kanwisher, McDermott & Chun, Reference Kanwisher, McDermott and Chun1997; Le Grand, Mondloch, Maurer & Brent, Reference Le Grand, Mondloch, Maurer and Brent2003; Rhodes, Reference Rhodes1993). For example, in the 1980s, Sewell and Panou (Reference Sewell and Panou1983) observed the typical right visual field (RVF)/left hemisphere (LH) advantage in accuracy in an English word naming task in all English–German bilinguals, English–French bilinguals, and English monolinguals; however, the typical left visual field (LVF)/RH advantage in a spatial dot localization task was only found in the monolinguals but not in the bilinguals. In the dot localization task, a 4 × 5 grid with a dot in one of the boxes was shown unilaterally, and participants were required to report the location of the dot. Thus, Sewell and Panou's (Reference Sewell and Panou1983) results suggested that the processing of some visual tasks such as spatial dot localization may be influenced by participants’ language experiences. About 20 years later, Hausmann, Durmusoglu, Yazgan and Gunturkun (Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004) examined performance of Turkish–German bilinguals and German monolinguals in some visual tasks and found results consistent with Sewell and Panou's (Reference Sewell and Panou1983) study. They showed that in the accuracy data of both groups, a typical RVF/LH advantage was found in a sequential word-matching task whereas a typical LVF/RH advantage was found in a face detection task; however, the response time data revealed a significant LVF/RH advantage in the face detection task only in the monolinguals but not in the bilinguals. This result suggested again that language experience may influence processing in some visual tasks other than visual word recognition.

The above results suggest that bilingual experience could lead to changes in brain organization such as hemispheric asymmetry in some language processing and visual processing tasks, such as face perception and spatial localization. It remains unclear whether bilingual experience will also change brain organization for visual word processing. The results of the previous studies suggested that bilinguals and monolinguals might differ in how they process visual words. For example, in Sewell and Panou's (Reference Sewell and Panou1983) word naming task, words were presented unilaterally and participants were required to report the word they perceived; the display time used was 20 ms and 40 ms for monolinguals and bilinguals, respectively. The authors selected these display times so that the two groups made approximately the same number of errors. This display time difference suggested that bilinguals might process the words differently from monolinguals. In contrast, in the word sequential matching task used by Hausmann et al. (Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004), a centrally presented word was followed by a unilaterally displayed word, at the presentation time of 175 ms for both the bilinguals and monolinguals. Although the results did not suggest any differences in hemispheric asymmetry between the two groups in this task, it is possible that differences did not emerge because performance level was not controlled between the two groups. Therefore, in this study, we employ a paradigm that controls for performance level when measuring hemispheric asymmetry in visual word recognition.

In addition, as mentioned earlier, visual word processing in bilinguals may differ from that in monolinguals in at least two ways:

1. 1. Vocabulary size. Bilinguals who learned to read in two languages with similar alphabets such as English and Spanish have larger vocabulary than monolinguals, and the words in their vocabulary may share more common letters and look visually more similar to each other than those in monolinguals’ vocabulary. Consequently, visual word recognition in these bilinguals may involve more processing of visual details.

2. 2. Word-to-sound mapping experience. In bilinguals who learned to read in two languages with substantial differences in word-to-sound mapping such as Chinese (logographic) and English (alphabetic), different visual processing may be required and they may also influence each other.

Nevertheless, most of the previous results were based on bilinguals who read in two alphabetic languages (e.g., Hausmann et al., Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004; Sewell & Panou, Reference Sewell and Panou1983). It remains unclear how brain organization for visual word processing in bilinguals can be affected by logographic language reading experience (such as Chinese). Therefore, here we examine hemispheric asymmetry in visual word processing in the following three groups of participants with different language experiences: (i) English monolinguals; (ii) European–English bilinguals, who are proficient in two alphabetic languages with similar alphabets; and (iii) Chinese–English bilinguals, who have acquired both Chinese and English with high proficiency. The two bilinguals groups contrast in their first language experience (alphabetic vs. logographic language) but accord in their second language experience (i.e., English, an alphabetic language). Thus, we compare English monolinguals and European–English bilinguals to examine the effect of vocabulary size on visual word processing, and compare the two bilingual groups to examine the effect of logographic/alphabetic language experience. This investigation on the behavioral difference between European–English bilinguals and Chinese–English bilinguals will provide a broader view than previous studies on how different language experiences modulate hemispheric asymmetry in visual word recognition. We describe the differences between alphabetic and logographic languages below.

The processing of alphabetic languages such as English has been shown to be lateralized to the LH. For example, fMRI studies revealed a specific region in the LH (i.e., the visual word form area) that responds to words selectively (e.g., McCandliss, Cohen & Dehaene, Reference McCandliss, Cohen and Dehaene2003); Event-Related Potential (ERP) studies showed a larger N170 in the LH being elicited by words than strings of symbols (e.g., Maurer, Brandeis & McCandliss, Reference Maurer, Brandeis and McCandliss2005); some researchers (Maurer & McCandliss, Reference Maurer, McCandliss, Grigorenko and Naples2007) suggested that the observed LH lateralization in alphabetic language processing is related to the application of grapheme–phoneme conversion (GPC) rules during learning to read. Consistent with the brain imaging data, behavioral studies also typically found an RVF/LH advantage in reading words in alphabetic languages, in various tasks such as tachistoscopic recognition (e.g., Bryden & Rainey, Reference Bryden and Rainey1963), lexical decision (e.g., Faust, Babkoff & Kravetz, Reference Faust, Babkoff and Kravetz1995) and word naming tasks (e.g., Brysbaert & d'Ydewalle, Reference Brysbaert and d'Ydewalle1990). In short, the superiority of the LH in processing alphabetic languages has been consistently reported.

In contrast to alphabetic languages, the relationship between written and spoken logographic languages, such as Chinese, is more opaque due to its morphosyllabic nature. Specifically, the components of Chinese characters, or stroke patterns, do not map to phonemes in the pronunciation, and thus the GPC rules in alphabetic languages do not apply to Chinese reading. In contrast to English word processing, fMRI studies (e.g., Tan, Liu, Perfetti, Spinks, Fox & Gao, Reference Tan, Liu, Perfetti, Spinks, Fox and Gao2001; Tan, Spinks, Gao, Liu, Perfetti, Xiong, Stofer, Pu, Liu & Fox, Reference Tan, Spinks, Gao, Liu, Perfetti, Xiong, Stofer, Pu, Liu and Fox2000; see also a mata-analysis in Tan, Laird, Li & Fox, Reference Tan, Laird, Li and Fox2005, comparing Chinese character and alphabetic word processing) generally showed more activation in the visual areas in the RH than the LH in reading Chinese characters, as opposed to a left ventral occipitotemporal system for processing visual words in alphabetic languages. Consistent with the brain imaging data, in behavioral studies a LVF/RH advantage was observed in tachistoscopic recognition of Chinese characters, particularly in orthographic processing (e.g., Cheng & Yang, Reference Cheng and Yang1989; Leong, Wong, Wong & Hiscock, Reference Leong, Wong, Wong and Hiscock1985; Tzeng, Hung, Cotton & Wang, Reference Tzeng, Hung, Cotton and Wang1979; Yang & Cheng, Reference Yang and Cheng1999). This lateralization difference between English word and Chinese character processing has been argued to be due to more visual analysis required for processing spatial information and locations of strokes in Chinese characters (Tzeng et al., Reference Tzeng, Hung, Cotton and Wang1979), or due to difference in visual similarity among word stimuli and the nature of word-to-sound mapping between English and Chinese (Hsiao & Lam, Reference Hsiao and Lam2013). In addition, in a recent study, Hsiao and Cottrell (Reference Hsiao and Cottrell2009) showed a left side bias effect in Chinese readers but not in non-Chinese readers in a Chinese character perception task: Chinese readers judged a chimeric Chinese symmetric character made from two left half-characters (created in a similar fashion as chimeric faces, see e.g., Brady, Campbell & Flaherty, Reference Brady, Campbell and Flaherty2005) as being more similar to the original character compared with the one made from two right half-characters; thus it suggested RH involvement in Chinese characters processing. In sum, the superiority of the RH in processing the orthography of Chinese, a logographic language, has been consistently reported.

In addition, the Japanese writing system provides an opportunity for investigating alphabetic and logographic systems in a single language, because it consists of both non-phonetic logographic symbols (i.e., Kanji) and phonetic syllables (i.e., Hirakana and Katakana) which are similar to the upper and lower case letters in English. Hirata and Osaka (Reference Hirata and Osaka1967) examined naming performance of Japanese speakers when Hirakana or Katakana, the phonetic syllables, were presented unilaterally, and a RVF/LH advantage was observed. Hatta (Reference Hatta1977) examined hemispheric asymmetry of the processing of Kanji characters, the non-phonetic logographic symbols; in the experiment, the Kanji stimuli were presented briefly on one side of the screen and the participants were asked to report the stimuli. In contrast to the results observed in Hirata and Osaka's (Reference Hirata and Osaka1967) study, Hatta (Reference Hatta1977) found a LVF/RH advantage for processing the non-phonetic logographic Kanji. These results suggested a phonetic and non-phonetic distinction between the two hemispheres: LH superiority for phonetic characters and RH superiority for non-phonetic characters. Consistent with these findings, in an fMRI study, Nakamura, Oga, Okada, Sadato, Takayama, Wydell, Yonekura and Fukuyama (Reference Nakamura, Oga, Okada, Sadato, Takayama, Wydell, Yonekura and Fukuyama2005) investigated the hemispheric asymmetry during naming of Kanji, kana, and objects. The results showed that in the occipitotemporal cortex, Kanji (logograms) and object naming were RH lateralized whereas kana (phonograms) naming was LH lateralized.

Thus, here we examine whether difference in first language experience (logographic vs. alphabetic) will modulate hemispheric lateralization in visual English word processing. In the literature on hemispheric asymmetry in visual perception, it has been shown that there is a RH advantage in processing global features, and a LH advantage in processing local features (e.g., Sergent, Reference Sergent1982). Sergent (Reference Sergent1982) suggested a continuum of low and high sensory resolution (frequency) over the LH and RH; specifically the LH is relatively better tuned to high frequencies whereas the RH is relatively better adapted to dealing with low than high frequencies. Consistently, studies using sine-wave gratings (e.g., Christman, Kitterie & Hellige, Reference Christman, Kitterle and Hellige1991; Kitterie, Christman & Hellige, Reference Kitterle, Christman and Hellige1990) found a LVF/RH advantage in identifying gratings of relatively low spatial frequency and a RVF/LH advantage for gratings of high spatial frequency. Moreover, in these studies, hemispheric asymmetry was found in grating identification but not in grating detection tasks, suggesting that hemispheric asymmetry may take place relatively late in perceptual processing (see also Heinze, Hinrichs, Scholz, Burchert & Mangun, Reference Heinze, Hinrichs, Scholz, Burchert and Mangun1998; Hsiao, Cipollini & Cottrell, Reference Hsiao, Cipollini and Cottrell2013; Sergent, Reference Sergent1982). To account for this asymmetry in perception, Ivry and Robertson (Reference Ivry and Robertson1998) proposed the Double Filtering by Frequency theory (DFF); the theory posits that visual information coming into the brain is captured by frequency-based representation at multiple scales, and the frequency information is filtered at two stages; in the first stage, a task-relevant frequency range is selected through attention processes; and at the second stage, asymmetric filtering processing is applied in the two hemispheres: the LH amplifies high spatial frequency (HSF) information, while the RH amplifies low spatial frequency (LSF) information. Through implementing the DFF theory in a computational model of visual word recognition, Hsiao and Lam (Reference Hsiao and Lam2013) demonstrated the following two factors that could influence hemispheric lateralization in visual word recognition, without assuming language processes being LH lateralized:

1. 1. Visual similarity among words in the lexicon. More HSF information is required to distinguish words when the words in the lexicon look more alike, leading to stronger LH lateralization.

2. 2. The task requirement to decompose a visual word input into smaller parts/letters for performing grapheme–phoneme conversion in alphabetic reading but not in logographic reading. This decomposition requires HSF information, which leads to stronger LH lateralization in alphabetic reading than logographic reading (see also Cheung & Hsiao, Reference Cheung, Hsiao, Ohlsson and Catrambone2010).

According to these two factors, we have two predictions in the current study:

1. 1. The European–English bilinguals may have stronger LH lateralization than English monolinguals in English word processing. Alphabetic reading involves more LH visual processing. The European–English bilinguals have acquired one more alphabetic language than the English monolinguals, and their first languages usually have a similar alphabet to English (although words in some languages such as Spanish may have accent marks). Thus, in their vocabulary they may have more visually similar words than English monolinguals due to a larger vocabulary size, leading to stronger LH lateralization (factor 1).

2. 2. The European–English bilinguals may show stronger LH lateralization than the Chinese–English bilinguals in English word processing. Although both European–English bilinguals and Chinese–English bilinguals acquired English as their second language, logographic Chinese reading involves more RH visual processing than alphabetic reading (factor 2), and this may also influence how Chinese–English bilinguals process English words.

In the following, we will first present a behavioral study investigating the difference in hemispheric asymmetry between participants with different language backgrounds in English word processing; we will then present a corresponding computational modeling study based on Hsiao and Lam's (Reference Hsiao and Lam2013) model and examine whether it can account for the behavioral data.

Behavioral study

Here we examined hemispheric asymmetry in English word processing in three groups of participants with different language backgrounds, namely, (1) English monolinguals, (2) European–English bilinguals, and (3) Chinese–English bilinguals, using a divided visual field word sequential matching task modified from Hausmann et al. (Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004). The English word sequential matching task was employed in order to focus our examination on visual word form (orthographic) processing while minimizing potential influence from phonological or semantic processing of the words. Note that with minimal influence from phonological or semantic processing of words, English word sequential matching typically showed no visual field difference or an LVF advantage (e.g., Brand, van Bekkum, Stumpel & Kroeze, Reference Brand, van Bekkum, Stumpel and Kroeze1983; Gibson, Dimond & Gazzaniga, Reference Gibson, Dimond and Gazzaniga1972), in contrast to the RVF advantage observed in word naming and lexical decision tasks.

Method

Materials

The materials consisted of one hundred pairs of English words selected from the SUBTLEX_US corpus (Brysbaert & New, Reference Brysbaert and New2009; see Appendix for the list). In each pair, the two words had the same number of letters and the same initial and final letters (the words in a pair might differ in one or more letters), and were matched in word frequency. The length of the word stimuli ranged from four to seven letters and the average frequency of the word stimuli was 407.57 per million words in the SUBTLEX_US corpus. The mean and median frequencies of all the words in the SUBTLEX_US corpus were 31.41 and 0.51 per million words, respectively. Thus, the frequency of the word stimuli in the materials was considered to be high frequency. Some of the English words in the materials were cognates with corresponding words in some European languages. However, the spellings of the cognate words were different.

Participants

Seventy-two participants were recruited. All were right-handed according to the Edinburgh handedness inventory (Oldfield, Reference Oldfield1971), and had normal or corrected-to-normal vision (i.e., with glasses or contact lenses). All the participants received honorarium or course credit for their participation. Participants were undergraduate or postgraduate students at the University of Hong Kong and were divided into three groups according to their language background: 20 English monolinguals, 32 European–English bilinguals, and 20 Chinese–English bilinguals (mean age = 22.0 years). The English monolinguals spoke English as their first language (L1) and could not fluently use any other languages. The European–English bilinguals learnt a non-English, Western European alphabetic language as their L1, such as German, Spanish, Dutch, French, Norwegian, or Italian, and English as their second language (L2); they were proficient in both their alphabetic L1 and English. These languages were chosen because they have a similar alphabet to English. Both the English monolinguals and European–English bilinguals had none or very limited knowledge about logographic scripts such as Chinese characters. The Chinese–English bilinguals were local Hong Kong students who learnt to write traditional Chinese and read traditional Chinese in Cantonese as their L1 and English as an L2 since kindergarten in formal education; they were proficient in reading both logographic traditional Chinese and alphabetic English. Note that traditional Chinese characters can be pronounced in either Cantonese (e.g., in Hong Kong) or in Mandarin (e.g., in Taiwan). Data from 16 European–English bilinguals were removed from analysis because their age of acquisition of English did not match the Chinese–English bilinguals’. (However, similar effects were found when the data from all 32 participants were included.) After we removed the 16 European–English bilinguals, the average age of acquisition of English was 3.4 years (SD = 1.2) for the Chinese–English bilinguals and 4.3 years (SD = 2.0) for the European–English bilinguals, and there was no significant difference in age of acquisition of English between the two groups. The first languages of the 16 European–English bilinguals whose data were used in the analysis were: Spanish: 4, German: 2, French: 2, Polish: 2, Romanian: 2, Italian: 1, Dutch: 1, Norwegian: 1, and Portuguese: 1. In terms of orthographic depth (i.e., the deviation of an alphabetic language from having a one-to-one grapheme-to-phoneme correspondence), English and French have a deep orthography; Italian, Spanish, Polish, and Norwegian have a shallow orthography; the orthographic depth of German, Romanian, Dutch, and Portuguese are considered intermediate.

Design

The design consisted of one between-subject variable and one within-subject variable. The between-subject variable was language background: one alphabetic language (English monolingual) vs. two alphabetic languages (European–English bilingual) vs. one logographic and one alphabetic language (Chinese–English bilingual). The within-subject variable was visual field: left visual field (LVF) vs. right visual field (RVF). The dependent variable was the accuracy in performing the English word sequential matching task.

In order to avoid any priming effect, each participant saw each English word only once during the experiment, either in the LVF or RVF. The visual field condition for each English word and the presentation sequence of the English words were randomized. Participants were asked to sit in front of a computer screen, at a viewing distance of 50 cm; under this viewing distance, each English word subtended about 1.5 degree (four-letter words) to 2.5 degree (seven-letter words) of visual angle. In order to avoid presenting the English words in foveal vision, the English words were presented at 1.5 degree of visual angle away from the center of the screen.

In order to control for performance differences between participants with different language backgrounds, before the main test participants completed a perceptual threshold test, in which the perceptual threshold of display time for each participant in the English word matching task was determined. In the threshold test, the one-up three-down staircase rule was applied where the participants achieved reliably 80% accuracy (Hartmann, Reference Hartmann2004). That is, for every three consecutive correct responses, the display time was decreased by one refresh rate, and every single incorrect response made the display time increased by one refresh rate. Three staircases were run in the threshold test, and each run proceeded until eight turnarounds had occurred. Only the third to the eighth turnarounds were averaged and used as the estimate of the threshold. The display time for the English words in the subsequent test was calculated by averaging the estimated thresholds of the three runs.

Procedure

The experiments were conducted with a CRT monitor with 100 Hz refresh rate. In the threshold test, each trial started with a 1000 ms central fixation; participants were asked to look at it whenever it appeared throughout the whole experiment. The central fixation cross was followed by an English word at the center, presented for 120 ms. Another central fixation cross was presented for 1000 ms after the English word, and another English word was then presented at the center with the same duration as the first word. Participants were asked to judge whether the two words were the same by pressing keys on a keyboard as fast and as accurately as possible. The presentation duration of the English words in the next trial was determined by the performance of the current and previous trials, following the three-down one-up rules (Hartmann, Reference Hartmann2004).

For the main test, the experiment procedure is shown in Figure 1. It followed a similar procedure as the threshold test described above, in which a central fixation was presented before the first and the second English word. However, in the main test, the first English word was presented in the LVF or RVF instead of at the center, and the two English words in each trial were presented at the duration obtained from the threshold test; the duration did not vary with the performance. There were equal numbers of English words presented in the two visual fields. The keys representing “same” or “different” were counterbalanced among the participants. Before the experiment, the participants finished a practice session for both the threshold test and the main test. The word pairs used in the practice trials were not included in the experimental trials.

Figure 1.

Procedure of the English word sequential matching task. Each test trial started with a 1000 ms central fixation cross, which was followed by the first word presented in either the left or right visual field; the presentation duration was estimated in the threshold test. The second stimulus was then presented at the center with the same presentation duration as the first word after another 1000 ms central fixation cross.

Results

According to the threshold test, the mean presentation times (i.e., the measured perceptual threshold) used in the experiment for English monolinguals, European–English bilinguals, and Chinese–English bilinguals were 54 ms (S.D. = 28 ms, min = 19 ms, max = 130 ms), 47 ms (S.D. = 16 ms, min = 19 ms, max = 91 ms) and 57 ms (S.D. = 19 ms, min = 30 ms, max = 103 ms), respectively. No significant difference in presentation time between any two of the three groups was found (independent sample t-tests, n.s.). In the research on visual perception, the perceptual threshold test has been used to assess perceptual expertise in processing visual stimuli (e.g., Wong & Gauthier, Reference Wong and Gauthier2010a, b; see also Chung, Reference Chung2007; Green & Bavelier, Reference Green and Bavelier2007; Pelli, Palmares & Majaj, Reference Pelli, Palmares and Majaj2004), and thus could be used as a measure of proficiency in visual English word processing in the current study (alternatively, an objective assessment of L2 proficiency, LexTALE, has recently become available; see Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012). Here we found no significant difference in perceptual threshold among the three groups, suggesting that they had a similar level of expertise in visual English word processing.

In the accuracy data of the main test, there was no significant main effect of visual field (F(1,53) = 1.700, n.s.). This effect is consistent with the literature showing that English word matching tasks with brief word presentation time typically showed no VF advantage or a LVF advantage (e.g., Brand et al., Reference Brand, van Bekkum, Stumpel and Kroeze1983; Gibson et al., Reference Gibson, Dimond and Gazzaniga1972). In addition, there was an insignificant tendency of an interaction effect between language background and visual field (F(2,53) = 2.414, p = .099). To examine our two predictions regarding the influence of participants’ vocabulary size (i.e., English monolinguals vs. European–English bilinguals) and word-to-sound mapping of their first languages (i.e., European–English bilinguals vs. Chinese–English bilinguals) on lateralization of English word processing, we conducted post-hoc analysis comparing European–English bilinguals with the other two groups separately (the traditional significant level of .05 was adjusted to .017 according to Bonferroni correction). The results showed that (i) European–English bilinguals had marginally stronger LH lateralization than Chinese–English bilinguals in English word processing (F(1,34) = 4.340, p = .045), suggesting a modulation effect of word-to-sound mapping of their first languages; and (ii) European–English bilinguals had marginally stronger LH lateralization than English monolinguals in English word processing (F(1,34) = 2.990, p = .093). Although these marginal effects were insignificant, they were consistent with the hypothesized modulation effects of their vocabulary size and word-to-sound mapping of their first languages. Note that these effects were not observed in the response time data, in contrast to Hausmann et al. (Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004). This difference may be related to the brief stimulus presentation time used here. Future work will examine whether similar effects can be observed in response times with longer stimulus presentation time.

Since no significant main effect of visual field was observed, in order to examine whether the visual field effects in the current study were consistent with Hausmann et al. (Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004), paired t-tests were performed for individual groups. The results showed a significant LH lateralization effect only in European–English bilinguals (t(15) = 2.514, p = .023), but not in English monolinguals or Chinese–English bilinguals (t(19) = 0.186, n.s. and t(19) = –0.473, n.s., respectively; see Figure 2). Note that for the two groups learning only alphabetic language(s) (i.e., English monolinguals and European–English bilinguals), the results showed a marginally significant main effect of visual field (F(1,34) = 3.919, p = .056): participants were more accurate when the English words were presented in the RVF than in the LVF (i.e., LH lateralization). This result was consistent with Hausmann et al. (Reference Hausmann, Durmusoglu, Yazgan and Gunturkun2004), in which there was an LH lateralization main effect observed among their participants, who learned only alphabetic language(s) (i.e., Turkish–German bilinguals and German monolinguals).

Figure 2.

Results from the behavioral experiment: English monolinguals’, European–English bilinguals’ and Chinese–English bilinguals’ response accuracy when the first stimulus was presented in the left and the right visual field. The horizontal lines at the top indicate interactions between visual field and language group.

* p < .05; + p < .1. Error bars show one standard error.

In general, the results were consistent with our predictions that in the English word sequential matching task, European–English bilinguals have stronger LH lateralization than Chinese–English bilinguals due to the lateralization difference in orthographic processing between their first languages, and they also have stronger LH lateralization than English monolinguals due to higher visual similarity among words in their larger, bilingual vocabulary as compared with the smaller, monolingual vocabulary in English monolinguals. Thus, the results suggest that hemispheric lateralization in visual word processing may be affected by participants’ language experience, that is, whether the participants are bilinguals or monolinguals and whether the bilinguals’ first language is an alphabetic or a logographic language.

Computational modeling

Our behavioral results revealed stronger LH lateralization in European–English bilinguals than either English monolinguals or Chinese–English bilinguals in the English word sequential matching task. We hypothesized that these effects were because (i) European–English bilinguals had a larger vocabulary size than English monolinguals while their two languages had similar alphabets; thus, words in their vocabulary might share more common letters and require more processing of visual details, leading to stronger LH lateralization (i.e., factor 1 according to Hsiao & Lam, Reference Hsiao and Lam2013); and (ii) Chinese–English bilinguals’ first language was a logographic language (Chinese), in which word-to-sound mapping did not involve grapheme–phoneme conversion as that in English; this characteristic may have led to more RH visual processing in processing Chinese characters than words in alphabetic languages, and consequently influenced English word processing in Chinese–English bilinguals (i.e., factor 2 according to Hsiao & Lam, Reference Hsiao and Lam2013). However, the behavioral results alone could not demonstrate whether the underlying mechanism we hypothesized that led to these effects was plausible. Therefore, to verify our hypotheses, here we implemented computational models to examine two factors that may have contributed to the differences observed in the behavioral data: (i) the effect of vocabulary size and (ii) the effect of word-to-sound mapping on the lateralization in visual word recognition, while having other potential factors controlled, such as featural differences between different orthographies, and the LH-lateralized phonological processing in the human brain. An advantage of using a computational modeling approach is that it gives us better control over variables than human studies, and thus allows us to tease apart potential factors that may contribute to hemispheric lateralization effects in visual word processing.

We applied the intermediate convergence model proposed by Hsiao, Shieh and Cottrell (Reference Hsiao, Shieh and Cottrell2008) to model bilingual visual word recognition, as shown in Figure 3 (see also Cheung & Hsiao, Reference Cheung, Hsiao, Ohlsson and Catrambone2010; Hsiao & Lam, Reference Hsiao and Lam2013). Hsiao et al. (Reference Hsiao, Shieh and Cottrell2008) showed that this model was able to account for the left-side bias effect in face perception observed in human data (e.g., Brady et al., Reference Brady, Campbell and Flaherty2005; Gilbert & Bakan, Reference Gilbert and Bakan1973). The model incorporates several known observations about visual anatomy and neural computation: Gabor filters (Daugman, Reference Daugman1985) are used to simulate neural responses of complex cells in the early visual system (Lades, Vorbruggen, Buhmann, Lange, von der Malsburg, Wurtz & Konen, Reference Lades, Vorbruggen, Buhmann, Lange, von der Malsburg, Wurtz and Konen1993); the Principal Component Analysis (PCA), a biological plausible linear compression technique (Sanger, Reference Sanger and Touretzky1989), is applied to the Gabor representations to compress the representations. The DFF theory (Ivry & Robertson, Reference Ivry and Robertson1998) of hemispheric asymmetry in perception is also implemented in this model by selecting a task relevant frequency range and applying asymmetric filtering processing to the two hemispheres: the LH amplifies HSF information, while the RH amplifies LSF information. The PCA representation is used as the input to a two-layer neural network that can be trained to map an input representation to the corresponding output representation in the output layer by adjusting its connection weights (see Figure 3). Thus, the model is a generic image processing model that can be trained to perform recognition tasks of all types of visual stimuli. Here we applied this model to visual word recognition and trained the model to map visual word input to its pronunciation, aiming to test the hypothesis that the lateralization differences we observed in English word processing among the three participant groups can be accounted for by the difference in their word reading experience. Note that the model does not assume an LH-lateralized phonological processing network as that in the human brain. Thus, lateralization effects emerging from the model will reflect hemispheric asymmetry in spatial frequency content (i.e., the DFF theory) that is required for the visual word recognition task rather than lateralization of phonological processing.

Figure 3.

Architecture of Hsiao et al.'s (Reference Hsiao, Shieh and Cottrell2008) hemispheric processing model.

In order to test our hypotheses and compare with the behavioral data, we built three models of visual word recognition with different vocabulary sizes and different orthography-to-phonology mappings to account for the behavioral differences among our three groups of participants: (i) alphabetic mapping model, in which we simulated alphabetic reading by mapping each letter in a word systematically to a corresponding phoneme in the pronunciation; (ii) logographic mapping model, in which we simulated logographic reading by randomizing the mapping between each word and its pronunciationFootnote ¹ (i.e., no systematic letter-to-phoneme mapping; see also Hsiao & Lam, Reference Hsiao and Lam2013); and (iii) integrated mapping model, which was trained to perform alphabetic reading with half of the word stimuli and logographic reading with the other half in the lexicon. This model mainly served to compare with the behavioral data of the Chinese–English bilinguals who were proficient in both logographic Chinese and alphabetic English (more details will be discussed in the following part). Note that in all models, we used English pseudo-words as the stimuli, so that the behavioral differences between the models reflected differences in word-to-sound mapping instead of in low-level visual features of the word stimuli (Hsiao & Lam, Reference Hsiao and Lam2013). As both our English monolinguals and European–English bilinguals were experts in alphabetic reading, and the two languages acquired by our European–English bilinguals had similar alphabets (i.e., one was English and the other was a Western European language), the behavior of the alphabetic mapping model with a small and large vocabulary size could be compared with the behavioral data of the English monolinguals and the European–English bilinguals, respectively. Thus, to examine whether the difference between English monolinguals and European–English bilinguals in the behavioral data could be accounted for by their difference in vocabulary size, we can compare an alphabetic mapping model with another alphabetic mapping model with twice the vocabulary size and see whether the two models exhibit similar differences to the human data. In contrast, as our Chinese–English bilinguals were experts in both alphabetic and logographic reading, the behavior of the integrated mapping model could be compared to the Chinese–English bilinguals in the behavioral data. Thus, to investigate whether the difference between Chinese–English bilinguals and European–English bilinguals in the behavioral data could be accounted for by the difference in word-to-sound mapping between their first languages (i.e., Chinese vs. European languages), we can compare the integrated mapping model with an alphabetic mapping model with the same vocabulary size and examine whether the two models show similar differences to the human data. Note that in our logographic mapping models, we used the same English pseudo-words as those used in the alphabetic mapping models instead of Chinese characters; this is because we aimed to examine the effect of word-to-sound mapping with other variables controlled, such as the difference in low-level visual features between European and Chinese languages, so that we could test whether the behavioral difference between Chinese–English and European–English bilinguals could be accounted for simply by the difference in word-to-sound mapping between their first languages. Thus, our logographic mapping model was not meant to simulate precisely Chinese character processing, in contrast to models of Chinese character recognition in the literature (e.g., Xing, Shu & Li Reference Xing, Shu and Li2004; Yang, Shu, McCandliss & Zevin, Reference Yang, Shu, McCandliss and Zevin2013).

Accordingly, here we hypothesize that (i) The LH (HSF) lateralization of the alphabetic mapping model will increase with vocabulary size, since the similarity of words increase with vocabulary size; this hypothesis can account for our behavioral data showing that European–English bilinguals exhibited stronger LH lateralization compared with English monolinguals; (ii) the alphabetic mapping model will show more LH (HSF) lateralization than the logographic mapping model, since decomposition of words into letters is not required in the logographic mapping model; and (iii) when performing alphabetic reading, the integrated mapping model will show less LH (HSF) lateralization than the alphabetic mapping model, since during learning, decomposition of words is required in only half of the times, and thus less HSF information is required overall. We describe our modeling details below.

Method

Input and output representations

We created artificial lexicons of three-letter words, whose pronunciations had a consonant–vowel–consonant (CVC) structure for the current examination. Each lexicon had an alphabet of size 26; half of the letters in the alphabet were randomly assigned as consonants and the other half as vowels in the pronunciation in the alphabetic mapping model. In the simulations, two sets of alphabet were used: one was lowercase English letters and the other was uppercase English letters. To counterbalance the information available in the left and right side of the input images towards word identities, palindrome words of six letters long were created as input images with the three-letter words from the artificial lexicon (Hsiao & Lam, Reference Hsiao and Lam2013); for example, the palindrome word cattac (Figure 4a) was created by repeating each letter of the word cat from the artificial lexicon from the end. For each palindrome word, images of eight different fonts were used as input images; four of them were used as the training set and the other four as the testing set. Fonts used in the training and testing sets were swapped in half of the simulation runs. In addition, mirror images were used in half of the simulation runs.

Figure 4.

Images used in the modeling: (a) palindrome pseudo English words; (b) and (c) right and left lateralized input of the palindrome pseudo English words.

Modeling details

As shown in Figure 3 above, following Hsiao et al. (Reference Hsiao, Shieh and Cottrell2008), each input image (35 × 100 pixels) was first filtered with a rigid grid (5 × 18) of overlapping 2-D Gabor filters (Daugman, Reference Daugman1985) at four scales and eight orientations. The frequency range represented the task-relevant frequency range in the DFF theory, as the four scales corresponded to 2 to 16 (i.e., from 2¹ to 2⁴) cycles per word whereas our image height was 35 pixels (the 5th scale, 2⁵ = 32 cycles per word image exceeded the Niquest frequency of the images). After the Gabor filters, each input image was transformed into a vector of size 2880 (5 × 18 sample points × 4 scales × 8 orientations). A base-line (control) condition and a biased condition were then created. The second stage of the DFF theory was only applied to the biased condition, in which the Gabor responses of the left and right half of the word were biased to low and high spatial frequencies, respectively, by applying a sigmoidal weighting function to give more weights to LSF scales in the Gabor responses of the left half-word and more weights to HSF scales in the Gabor responses of the right half-word. In contrast, in the base-line condition, equal weights were given to the Gabor responses of different scales. The PCA was then applied to the Gabor representations of the left and right half-words separately to compress each representation into a 50-element representation (i.e., 100 elements in total). This PCA representation was then used as the input to a two-layer neural network (see Hsiao et al., Reference Hsiao, Shieh and Cottrell2008; Hsiao & Lam, Reference Hsiao and Lam2013, for more simulation details).

The task of the model was to map each word input image to its pronunciation with a consonant–vowel–consonant (CVC) structure. In the alphabetic mapping model, each letter in a word was systematically mapped to each phoneme in the pronunciation (Figure 5a). In the logographic mapping model, the mappings between each word and its pronunciation were randomized (i.e., no systematic letter-to-phoneme mapping, Figure 5b, Hsiao & Lam, Reference Hsiao and Lam2013). In the integrated mapping model, two alphabets of size 26 were used in each simulation run (one set was lowercase English letters, whereas the other was uppercase English letters); letters in one of the alphabets were systematically mapped to phonemes in the pronunciation, whereas in the other alphabet there was no systematic mapping (i.e., random mapping) between letters and phonemes in the pronunciation.

Figure 5.

Examples of the output layer of (a) systematic mapping in the alphabetic mapping model, and (b) random mapping in the logographic mapping model, between the input word and the output pronunciation.

Our model was trained to map each input image to its output pronunciation until the performance on the training set reached 100% accuracy; the model generated a correct answer if the most active nodes in the output layer matched the correct output. The training algorithm used was gradient descent with an adaptive learning rate (implemented by the Matlab Neural Network Toolbox): initial weights were randomly assigned and the initial learning rate used was 0.01; the learning rate was increased by 5% if the performance improvement of the latest two runs was smaller than the previous one, and it was decreased by 30% if the performance improvement was larger than the previous one. Each of the alphabetic, logographic, and integrated mapping models was trained and tested with the lexicon size ranging from 26 to 60, with a stepwise increase of 2, and each simulation with a different lexicon size was run 16 times with different weight initializations in both base-line and biased conditions.

To examine hemispheric asymmetry effects, after training, we tested the model with left or right lateralized inputs (see Figures 4b and Figures 4c above), which were generated by setting one half of the PCA representation to zero (Cheung & Hsiao, Reference Cheung, Hsiao, Ohlsson and Catrambone2010; Hsiao & Lam, Reference Hsiao and Lam2013; see also Hsiao et al., Reference Hsiao, Shieh and Cottrell2008). When mapping these lateralized inputs to their corresponding outputs, only the representation from one of the visual hemifields was available in recognition. Thus, in the biased condition, a left-lateralized word carried only LSF/RH information and a right-lateralized word carried only HSF/LH information. The LH (HSF) lateralization effect was then measured as the accuracy difference between recognizing a right-lateralized word and a left-lateralized word as the original word.

Results

In the data analysis, LH lateralization was defined as the performance difference between recognizing a right-lateralized input (i.e., only LH information was available) and a left-lateralized input (i.e., only RH information was available) as the original one in the biased condition, minus this difference in the base-line (control) condition. Figure 6 shows the LH lateralization data of the three models (i.e., alphabetic mapping model, logographic mapping model, and integrated mapping model) and their correlations with vocabulary size. Both the alphabetic mapping model and the logographic mapping model showed an increase in LH (HSF) lateralization with increasing lexicon size; a significant positive correlation were observed between LH (HSF) lateralization and lexicon size (alphabetic mapping model: R² = .105, p < .001; logographic mapping method: R² = .033, p = .002). However, the integrated mapping model did not exhibit a similar correlation; rather, it showed a significant, weak negative correlation between LH lateralization and lexicon size (R² = .014, p = .047). These results showed that LH (HSF) lateralization increased with vocabulary size in the alphabetic mapping model and the logographic mapping model (see also Hsiao & Lam, Reference Hsiao and Lam2013), but the effect was not apparent in the integrated mapping model. These results supported hypothesis (i): the LH (HSF) lateralization of the alphabetic mapping model increased with vocabulary size, since the similarity of words increased with vocabulary size. This result suggests that the stronger LH lateralization in English word processing in the European–English bilinguals than the English monolinguals in our behavioral data may be because European–English bilinguals had a larger vocabulary size overall.

Figure 6.

Results from computational modeling. Both the alphabetic mapping model and the logographic mapping model showed a significant positive correlation between LH lateralization and vocabulary size.

In order to examine the difference in LH lateralization between the models, we used ANOVA with vocabulary size and model as independent variables. There was a main effect of model (F(2,540) = 139.317, p < .001) and an interaction effect between model and vocabulary size (F(34,540) = 2.401, p < .001). Figure 7a shows the LH lateralization data of the alphabetic mapping model and the logographic mapping model across vocabulary size. Post-hoc analysis showed a main effect of model between the alphabetic mapping model and the logographic mapping model (F(1,540) = 320.914, p < .001; the traditional significant level of .05 was adjusted to .017 according to Bonferroni correction); the interaction between vocabulary size and model was not significant (F(17,540) = 0.912, n.s.). Thus, the results showed that the logographic mapping model had a weaker LH (HSF) lateralization than the alphabetic mapping model, suggesting that decomposition of words into letters in alphabetic mapping led to increase in LH lateralization when compared with logographic mapping, where no decomposition of words into letters was required. These results supported hypothesis (ii): the alphabetic mapping model showed more LH (HSF) lateralization than the logographic mapping model (see also Hsiao & Lam, Reference Hsiao and Lam2013).

Figure 7.

Results from computational modeling. Differences in LH lateralization between (a) alphabetic mapping model and logographic mapping model and (b) alphabetic mapping model and integrated mapping model.

Independent t-tests: * p < .05; ** p < .01; *** p < .001. Error bars show one standard error.

Post-hoc analysis also revealed a main effect of model between the alphabetic mapping model and the integrated mapping model in LH lateralization (F(1,540) = 45.030, p < .001), which suggested that in general the integrated mapping model had weaker LH (HSF) lateralization than the alphabetic mapping model. There was also a significant interaction between these two models and vocabulary size (F(17,540) = 3.069, p < .001), which suggested that the integrated mapping model did not show weaker LH lateralization than the alphabetic mapping model in all the vocabulary sizes. As shown in Figure 7b, the LH lateralization of the alphabetic mapping model was in general larger than that of the integrated mapping model with a vocabulary size bigger than 40 (this trend can also be seen in Figure 6). These results supported hypothesis (iii): since during learning, decomposition of words was required in only half of the times in the integrated mapping model when compared with the alphabetic mapping model, when performing alphabetic reading, the integrated mapping model showed less LH (HSF) lateralization than the alphabetic mapping model. This result suggests that in the behavioral data, the stronger LH lateralization in English word processing observed in European–English bilinguals as compared with Chinese–English bilinguals may be due to the difference in word-to-sound mapping between their first languages.

We also tried to account for the behavioral data using the modeling data with three samples of different vocabulary sizes. Figure 8a shows the LH lateralization data of the three simulated language groups (English monolinguals, European–English bilinguals, and Chinese–English bilinguals) using three different samples (Sample A, B, and C). In the simulations, the vocabulary size of the European–English bilinguals was assumed to be twice of the English monolinguals’ vocabulary size;Footnote ² thus, in the three simulations, simulated English monolinguals’ data was derived from the alphabetic mapping model with a vocabulary size of 26 (Sample A), 28 (Sample B) and 30 (Sample C); data from simulated European–English bilinguals and simulated Chinese–English bilinguals were obtained from the alphabetic mapping model and the integrated mapping model of vocabulary size of 52 (Sample A), 56 (Sample B) and 60 (Sample C), respectively. Note that the integrated mapping model was trained to perform both alphabetic and logographic reading; here, in order to compare with the human data, we presented the performance on alphabetic reading only. In each simulation, we used ANOVA with simulated language group as the between-subject variable and LH lateralization as the dependent variable. In all three simulations, the results showed a main effect of simulated language group (Sample A: F(2,45) = 9.178, p < .001; B: F(2,45) = 4.767, p = .013; C: F(2,45) = 4.374, p = .018); post-hoc analyses showed stronger LH (HSF) lateralization in the simulated European–English bilinguals than the simulated English monolinguals (Sample A: t(30) = –2.750, p = .01; B: t(30) = –2.849, p = .008; C: t(30) = –4.433, p < .001), and the simulated Chinese–English bilinguals (Sample A: t(30) = –3.685, p = .001; B: t(30) = –2.547, p = .016; C: t(30) = –1.909, p = .066). The modeling data suggested that the simulated European–English bilinguals had a stronger LH lateralization than the simulated English monolinguals and the simulated Chinese–English bilinguals, which is consistent with the behavioral data of the three language groups shown in Figure 8b.

Figure 8.

(a) Results from computational modeling. Simulations of hemispheric lateralization in alphabetic reading in English monolinguals, European–English bilinguals, and Chinese–English bilinguals, with three different samples from the modeling. (b) Results from the behavioral data, where LH lateralization was calculated by subtracting the accuracy level in the LVF/RH condition from the accuracy level of the RVF/LH condition.

+ p < .1; * p < .05; ** p < .01; *** p < .001. Error bars show one standard error.