The phonological level

The phonological level of the BLINCS model was constructed using a modified version of PatPho (Li & MacWhinney, Reference Li and MacWhinney2002), which quantifies phonemic items by virtue of their underlying attributes (e.g., voicedness, place of articulation, etc.). In this system, each phoneme is represented as a three-element vector, with each element capturing a different aspect of the phoneme. Thus, a three-dimensional vector was created for each phoneme from the International Phonetic Association alphabet (International Phonetics Association, 1999), with two notable additions. First, as with PatPho, the phoneme /H/ was added as a voiceless, glottal approximant (first sound in hospital). Second, a category of affricates was added to include the sounds /ʧ/ (first phoneme in church) and /ʤ/ (first phoneme in jail), which were defined as unvoiced/voiced (respectively), alveolar affricates. A full list of the phonological forms and the quantified three-dimensional vectors is available as online Supplementary Material.

The phono-lexical level

Phono-lexical items in the model were constructed by inserting the three-element phonological vectors into a multi-syllabic template. In the current model, each word in the model is placed in a three-syllable template of the structure CCVVCC/CCVVCC/CCVVCC. Each C or V slot contains a three-element vector, resulting in a total length of 54-elements per word (three-elements multiplied by 18 C or V slots). For example, the two-syllable word rabbit would be represented as [r_Cæ_VCC/b_CI_Vt_C/_CCVVCC]; each empty C or V position contains a vector of zeros. Thus, information about the syllable structure of words is retained at the level of the lexicon.

Embedding the phoneme vectors into syllabic phrases allows the model to draw connections between items that a simple ordered structure might not. For example, using an ordered structure, the English words tap and trap would be formed as [tæp. . .] and [træp. . .], respectively – the model would then compare each phoneme from one word to the phoneme that occurs in the same position in the second word. In other words, the /r/ in trap would be compared to the /æ/ in ‘tap,’ and the model would fail to recognize the significant overlap between the two words. By embedding the items into syllable phrases, word input is instead formulated as [t_Cæ_Vp_C. . .] and [træ_Vp_C. . .], so the vowel in tap occupies the same slot as the vowel in trap, allowing the model to recognize their phonological similarity.

For the present model, a list of 480 words ranging from one to three syllables was chosen, consisting of 240 English words and 240 Spanish words. The list contained 142 English–Spanish translation equivalents (totaling 286 words), 88 cognates, 34 false-cognates, and 72 single-language words (split evenly between English and Spanish). Each word was written with broad phonetic transcription in accordance with the IPA. The phonological information was then transformed into the modified PatPho vectors and concatenated into the three-syllable carrier. A list of words is available as online Supplementary Material.

The ortho-lexical level

In addition to the phono-lexical level, we included a level that contained orthographic representations for the lexical items. Though our primary interest in developing the model was speech comprehension, research has indicated an interaction between phonological and orthographic systems (Bitan, Burman, Chou, Lu, Cone, Cao, Bigio & Booth, Reference Bitan, Burman, Chou, Lu, Cone, Cao, Bigio and Booth2007; Kaushanskaya & Marian, Reference Kaushanskaya and Marian2007; Kramer & Donchin, Reference Kramer and Donchin1987; Schwartz, Kroll & Diaz, Reference Schwartz, Kroll and Diaz2007). Because orthographic information is known to be co-activated during phonological processing (Rastle, McCormick, Bayliss & Davis, Reference Schwartz, Kroll and Diaz2011; Ziegler & Ferrand, Reference Ziegler and Ferrand1998), and orthography has been shown to activate phonological representations (see Rastle & Brysbaert, Reference Rastle and Brysbaert2006, for a review), BLINCS includes an orthographic system that interacts with the phono-lexical level during processing. Each letter in the English and Spanish alphabets (the 26 traditional characters, as well as the Spanish characters ñ, á, é, í, ó, and ú) was quantified using a method similar to that of Miikkulainen (Reference Miikkulainen1997). Each letter was typed in 12 point, Times New Roman font in black on a white background measuring 50 × 50 pixels, where 1 represented a black pixel and 0 represent a white pixel. The proportion of black pixels in each of the four corners was then calculated for each image (i.e., number of black pixels/total number of pixels) and used to create a four-element vector for each letter. The letters were then concatenated into a 16-slot carrier of the form CCVVCCVVCCVVCCVV (resulting in a 64-element vector). For example, the Spanish word fotó “photo” and the English words carpet and telescope were coded as [f₀o₀t₀ó_000000000], [c₀a₀rpe₀t_0000000], and [t₀e₀l₀e₀sco₀p₀e₀], respectively.

The semantic level

Semantic representations of the words were obtained using the Hyperspace Analogue to Language (HAL; Burgess & Lund, Reference Burgess and Lund1997; Lund & Burgess, Reference Lund and Burgess1996), which provides quantified measures of word co-occurrence from large text corpora. In essence, HAL captures lexical meaning through the frequency with which words co-occur with other words and is able to automatically derive semantic information from co-occurrence information. For the present model, the HAL tool from the S-Space Package (Jurgens & Stevens, Reference Jurgens and Stevens2010) was used to derive semantic representations for our lexical items from a portion of the UseNet Corpus (Shaoul & Westbury, Reference Shaoul and Westbury2011) totaling approximately 330 million words. Each semantic entry consisted of a 200-dimensional vector. Because evidence suggests that the semantic space is shared in bilinguals (Kroll & De Groot, Reference Kroll, De Groot, De Groot and Kroll1997; Salamoura & Williams, Reference Salamoura and Williams2007; Schoonbaert, Hartsuiker & Pickering, Reference Schoonbaert, Hartsuiker and Pickering2007), the values for the English lexical items matched the values of the Spanish translation equivalents. The semantic vectors for Spanish words were obtained using their English translations – thus, the English word duck and its Spanish translation pato had the same semantic vector.

Visual information

Listening to spoken language in the real world naturally involves the integration of auditory and visual information. Visual information can influence the process of speech recognition at the level of perception, where visual and auditory input are integrated to impact phoneme perception (e.g., the McGurk effect; Gentilucci & Cattaneo, Reference Gentilucci and Cattaneo2005; McGurk & MacDonald, Reference McGurk and MacDonald1976), or can provide context that shapes how the linguistic message is interpreted by limiting processing to objects in the visual scene (Knoeferle, Crocker, Scheepers & Pickering, Reference Knoeferle, Crocker, Scheepers and Pickering2005; Spivey, Tanenhaus, Eberhard & Sedivy, Reference Spivey, Tanenhaus, Eberhard and Sedivy2002). With this in mind, the BLINCS model was designed to accommodate the potential influence of visual information on language processing. Specifically, the BLINCS model was not designed to perform visual recognition (i.e., tracking of lip movements or recognition of specific images), but rather to focus on the effects that information gained by the visual system might have on language processing. Thus, direct connections from a visual input module to the phonological level were used to simulate the effect of additional identifying phonemic information from lip or mouth movements, as with the McGurk effect, by averaging the phonological values associated with the visually-represented phoneme and those corresponding to the spoken language input. Likewise, direct connections from a visual-input module to the semantic level served to simulate non-linguistic constraint effects, where the presence or absence of objects in a visual scene can affect language processing. More specifically, the model increases the resting activation of semantic representations for items that the visual-input module indicates are currently visible.

Training the BLINCS model

The four levels (phonological, phono-lexical, ortho-lexical and semantic) and the Hebbian connections between phono-lexical/ortho-lexical maps and phono-lexical/semantic maps were trained concurrently over 1000 epochs. Training within each level was performed using the SOM algorithm. The learning rate (which determines the strength of learning) was initially set at 0.2 and decreased linearly to 0 from epochs 1 to 1000. The learning radius (which determines the nodes that are trained based on their radial proximity to the best-match-unit) was initially set at 10, decreased linearly to 4 in the first 100 epochs, and then decreased linearly to 0 from epochs 101 to 1000. Furthermore, the learning radius function was Gaussian in nature, so the learning-strength of a given node decreased relative to its distance from the best-match-unit. The items were presented in random order and an equal number of times. Spanish and English lexical items were intermixed during training, thereby approximating simultaneous acquisition of the two lexical systems. The phono-lexical, ortho-lexical, and semantic representations for each single word were presented concurrently; this allowed the model to strengthen the inter-level connections between the best-match-units at each level, thereby enhancing the links between a given word's phono-lexical, ortho-lexical, and semantic representations during training. To train the inter-level connections, we applied a Hebbian learning algorithm, defined as:

\begin{equation} \Delta w_{i,j} = \lambda x_i x_j \end{equation}

In addition to the between level connections, the Hebbian weights were used as a basis for drawing lateral links between translation equivalents. For example, the lexical items duck and pato mapped to the same semantic unit, meaning that both words were activated at the phono-lexical level, along with the representation of “duck” at the semantic level. A network of lateral connections at the phono-lexical level was developed by increasing the weights between two lexical nodes that were accessed by the same semantic node using the Hebbian learning rule.

Training at each level was able to capture similarities between phonological, lexical, ortho-lexical, and semantic items. Figure 2 shows the phonological map after 1000 epochs of training. The most notable distinction on the map is the separation of vowels from consonants – a finding that echoes recent neurological evidence for distinct neural correlates of vowels and stop consonants (Obleser, Leaver, VanMeter & Rauschecker, 2011). Within these two major categories, distinctions based on phonemic class also emerge. For example, items such as /p/ and /t/ mapped together in the phonemic space based on their shared membership to the category of voiceless plosives, suggesting that they should co-activate based on their similarity.

Figure 2.

The Phonological SOM after 1000 epochs of training. Vowels and consonants are separated by virtue of their underlying phonological features (e.g., voicedness, manner, or place of articulation). White-shaded areas represent consonants, and gray-shaded areas represent vowels.

The phono-lexical SOM, shown in Figure 3, successfully captures similarities between lexical items driven by phonological overlap. For example, words like bone, boat, and road are mapped near one another. However, we are most interested in the way in which the map organizes the two lexicons into somewhat distinct space. We can see from the map that Spanish and English words are primarily separated based upon the phonotactic probabilities inherent to the input. There are some notable exceptions, primarily in the mapping of cognates and false cognates, which often map to the boundaries of the distinct language spaces within the SOM. For example, the cognate words tobacco (English) and tabaco (Spanish) are mapped near one another, but directly below the Spanish tabaco the map contains primarily Spanish words, and directly above the English tobacco is a primarily English neighborhood. The organization of the phono-lexical SOM represents a separated but integrated system, where the BLINCS model classifies words by language membership according to phonotactic rules, while also allowing words from both languages to interact within a single lexical space.

Figure 3.

The phono-lexical SOM after 1000 epochs of training. The model automatically separates English and Spanish in the two-dimensional vector space according to phonotactic principles. White-shaded areas represent English, and gray-shaded areas represent Spanish.

In order to capture the interaction between the phonological and orthographic representations of lexical items, each word on the phono-lexical SOM mapped directly to its written equivalent on the orthographic map (Figure 4) via trained Hebbian connections. Items in the ortho-lexical SOM are mapped together based on their spelling-similarity (e.g., hint and pint; cerco “fence” and carta “letter”). The ortho-lexical level allows for items that do not share phonology, but that share orthography, to be accessed or activated at the same time. For example, the English words beard and heart share a significant orthographic overlap (three of five letters exact, with a high degree of similarity between ‘b’ and ‘h’), but overlap very little in phonology. However, because these words do map closely in ortho-lexical space, orthographic information is able to influence lexical processing. As with the phono-lexical level, the ortho-lexical level also displays a separated but integrated structure.

Figure 4.

The ortho-lexical SOM after 1000 epochs of training. The model automatically separates English and Spanish in the two-dimensional vector space according to orthotactic principles. White-shaded areas represent English, and gray-shaded areas represent Spanish. (Because Spanish and English overlap orthographically more than they overlap phonologically, the ortho-lexical map is more integrated than the phono-lexical map, across languages.)

The BLINCS also includes a level dedicated to semantic/conceptual information (Figure 5). Here, we see that semantic relationships between items are successfully represented in the SOM space. Words such as car and road, which are associated concepts, map near one another. Likewise, the SOM can capture category relationships; duck, lamb, and rabbit – all types of animals – map together, and musical instruments like saxophone and flute map together as well. In cases of translation equivalents, single nodes encapsulate the meaning for both lexical items, based on the notion of a shared semantic space across languages. Thus, single nodes in the semantic space can map to multiple nodes at the lexical level. Through these connections, words that do not share semantic similarity may be co-activated by virtue of phonological similarity. In addition, words that are phonologically distinct but are closely related in meaning may be co-activated via top–down connectivity from the semantic system.

Figure 5.

The semantic SOM after 1000 epochs of training. Translation equivalents are mapped to a single node, which reflects a semantic system that is shared across languages. The inset shows a subsection of the SOM onto which related concepts were mapped.

Language co-activation in the BLINCS model

When listening to speech, bilinguals display an impressive degree of cross-linguistic interaction. A growing body of evidence suggests that a bilingual's two languages communicate and influence one another at the levels of phonological (Ju & Luce, Reference Ju and Luce2004; Marian & Spivey, Reference Marian and Spivey2003a, Reference Marian and Spiveyb), orthographic (Kaushanskaya & Marian, Reference Kaushanskaya and Marian2007; Thierry & Wu, Reference Thierry and Wu2007), lexical (Finkbeiner, Forster, Nicol & Nakamura, Reference Finkbeiner, Forster, Nicol and Nakamura2004; Schoonbaert, Duyck, Brysbaert & Hartsuiker, Reference Schoonbaert, Duyck, Brysbaert and Hartsuiker2009), syntactic (Hartsuiker, Pickering & Veltkamp, Reference Hartsuiker, Pickering and Veltkamp2004; Loebell & Bock, Reference Loebell and Bock2003), and semantic processing (FitzPatrick & Indefrey, Reference FitzPatrick and Indefrey2010). Therefore, one important goal for the BLINCS model was to accurately simulate the way in which bilinguals process speech by capturing this interactivity. We tested the model by providing it with sequential phonological information and then measuring the overall activation of all the items within the phono-lexical level as a function of phonological, orthographic and semantic activation. This allowed us to rank the items that were “most active” during the entire trial. The results provided strong support for the model's ability to capture effects of cross-linguistic activation. To illustrate these effects, Table 1 shows several target words that were provided to the model, and words that consistently ranked in the top 15% of co-activated items (at least 80 occurrences in 100 model simulations; there was a degree of variability within this cohort due to the noise in the model).

These examples highlight the interactivity inherent to the BLINCS model. For example, tenedor “fork” activated a within-language onset competitor, tiburón “shark”, as well as cross-language onset competitors, tunnel and tent. This is consistent with research indicating that during speech processing, multiple candidates are active early in the listening process for both monolinguals (Marslen-Wilson, Reference Marslen-Wilson1987; Tanenhaus, Spivey-Knowlton, Eberhard & Sedivy, Reference Tanenhaus, Spivey-Knowlton, Eberhard and Sedivy1995), and bilinguals (Marian & Spivey, Reference Marian and Spivey2003a, Reference Marian and Spiveyb). The model is also capable of activating rhyme cohorts during listening. For instance, pear activates chair in English, and arena “sand” activates ballena “whale” in Spanish via input from the phonological level. However, activation in BLINCS is not driven entirely by the phonological input. Consider the co-activated word vaso “glass” that accompanies the target word pie. In this instance, vaso is only active because the target word, pie, activates the word pato “duck” via shared onset, which in turn activates vaso due to their close proximity in the phono-lexical map (note that vaso is a near-rhyme to pato). Thus, activation of vaso from pie depends upon lateral mapping within the phono-lexical system. Both the case of pear to chair, and pie to vaso, illustrate the BLINCS model's ability to capture rhyme effects (Allopenna, Magnuson & Tanenhaus, Reference Allopenna, Magnuson and Tanenhaus1998), either through direct phonological match to the input, or via lateral connections between rhyming items. The co-activated item analyses also indicate the influence of semantic feedback during processing in the BLINCS model. Consider the target/co-activated pairs, road/car, and hielo “ice”/sol “sun”. Co-activation of these items occurred by virtue of their proximity in the semantic map, since roads and cars are often associated, and ice and sun both represent weather phenomena. In other words, the target word road activated the semantic representation of road, which increased the activation of nearby related concepts (e.g., car), and both semantic representations passed their activation values down to their corresponding phono-lexical representations, resulting in car co-activating during presentation of road. These effects of semantic knowledge on lexical processing in the model are supported by empirical work highlighting the influence of semantic relatedness on language comprehension (Huettig & Altmann, Reference Huettig and Altmann2005; Yee & Sedivy, Reference Yee and Sedivy2006).

The time-course of activation in the BLINCS model

While measuring overall activation (i.e., collapsed across time) in BLINCS is informative for exploring the process of speech comprehension, the model also allows one to trace the activation of lexical items as speech unfolds. For a sequential and incremental process like spoken comprehension, looking at the relative activation of words across time provides a more nuanced measure of language co-activation. In each example, the model is given the same target word for 10 trials. Those trials are then averaged together to obtain activation curves for the target item and for items that are phonologically, orthographically, or semantically related. Each subsequent graph contains the activation of nodes at the phono-lexical level, but reflects the integrated activation of phonological, orthographic, and semantic processing.

The first example (Figure 6) shows the activation curve of the target word tenedor “fork”, a within-language onset competitor, tortuga “turtle”, a cross-linguistic onset competitor, tent, the translation equivalent fork (English), and an unrelated item, mailbox. We can see an increase in activation for both within- and cross-language onset competitors until the point at which the model no longer considers those items to match the input, resulting in tent reaching a higher peak of activation than tortuga because it overlaps more with the target. Both items remain overall more active than the unrelated item, mailbox. The activation of the translation equivalent in this example is driven by feedback from the semantic level to the lexical level. First, as tenedor becomes active, it activates its semantic representation (as FitzPatrick & Indefrey, Reference FitzPatrick and Indefrey2010, suggest, this process can begin with presentation of a word's initial phoneme), which feeds back to the nodes for tenedor and its translation equivalent fork. Through this mechanism of feedback, the presentation of a word in one language can result in the rapid activation of its translation equivalent.

Figure 6.

Activation of the BLINCS model with the Spanish target word tenedor “fork”. The curves show simultaneous activation of a within-language competitor (tortuga/turtle), a cross-language competitors (tent), and the English translation equivalent of the target, fork. In contrast, there was no activation of the unrelated word mailbox.

The orthographic relationships found in the proximal structure of the BLINCS model are further reflected in the activation patterns across time. Figure 7 shows an example of a model simulation with the target word beard. The activation curves again show increased activation of an item that does not share phonology with the target, but overlaps substantially in orthography (i.e., heart). The reason for this heightened activation is that beard and heart map closely in ortho-lexical space, so that when the phono-lexical representation of beard is activated, it spreads to its ortho-lexical form, subsequently activating nearby items that feed back to their phono-lexical representations. Through this pathway, ortho-lexical information is able to influence phono-lexical processing, consistent with previous research (Rastle et al., Reference Rastle, McCormick, Bayliss and Davis2011; Ziegler & Ferrand, Reference Ziegler and Ferrand1998).

Figure 7.

Activation of the BLINCS model with the target word beard. The curves show greater activation of an orthographic competitor, heart, relative to an unrelated item, mushroom.

The BLINCS model also makes predictions regarding the extent to which semantic knowledge can drive activation of lexical items. Previous research using eye-tracking has suggested that semantic relatedness plays a role in language comprehension (Shook & Marian, Reference Shook and Marian2012; Yee & Sedivy, Reference Yee and Sedivy2006; Yee & Thompson-Schill, Reference Yee and Thompson-Schill2007). However, the extent to which semantic information can impact lexical processing is unclear. Yee and Sedivy found that when participants heard the word log, they looked more to a picture of a key, because log partially activated the word lock, which activated key due to their semantic relatedness. In other words, bottom–up phonological information activated multiple candidates (e.g., log and lock), which spread activation upward to their corresponding semantic representations. Then, at the semantic level, the representation for lock activated the representation of key via lateral connections. Feedback from the conceptual representation of key to its lexical counterpart is not necessary for participants to look more at the image of the key. While this result provides evidence for semantic processing of multiple candidates, and for lateral connections between semantically related items, the BLINCS model predicts that the semantically related information can cause the lexical forms of semantically-related items to become active as well. Consider Figure 8, which contains the activation curves for the target word, face, its translation equivalent, cara, an object that is phonologically related to the translation equivalent, cama “bed”, the translation equivalent of the phonologically related item, bed, and an unrelated item, windmill. The activation curves suggest that the phono-lexical representation of the English word bed, which is not directly related to face through semantics, orthography, or phonology, may nevertheless show greater activation than a word like windmill, given the target word face (see Li & Farkas, Reference Li, Farkas, Heredia and Altarriba2002, for a discussion of similar cross-language activation effects for semantically-unrelated words). Thus, the BLINCS makes a testable prediction regarding the degree of impact that semantic knowledge can have on language co-activation in bilinguals that is consistent with monolingual research (Yee & Sedivy, Reference Yee and Sedivy2006; Yee & Thompson-Schill, Reference Yee and Thompson-Schill2007).

Figure 8.

Activation of the BLINCS model with the target word face. The curves show greater activation of the Spanish translation equivalent, cara, a word that is phonologically related to the translation, cama “bed”, and the English translation of that phonological competitor, bed, relative to an unrelated word, windmill.

In addition to examining the sets of words that are co-activated during speech comprehension as a product of phonological, ortho-lexical, or semantic information, we were also interested in how the model activated different types of words. The structure of the BLINCS model often maps cognates and false-cognates close together in the phono-lexical space, with cognates further benefitting from overlap at the semantic level. This proximity suggests an advantage for cognate activation, which is consistent with empirical findings that indicate faster or increased activation for cognates (Blumenfeld & Marian, Reference Blumenfeld and Marian2007; Costa, Caramazza, Sebastián-Gallés, Reference Costa, Caramazza and Sebastián-Gallés2000; Dijkstra, Grainger & van Heuven, Reference Dijkstra, Grainger and van Heuven1999). To determine how cognates are processed in the BLINCS, we compared the overall activation of all cognate-words (e.g., doctor (English) and doctor (Spanish)) to false-cognates (e.g., arena and arena “sand”), translatable words (e.g., pato “duck” and duck, and party and fiesta “party”), and words for which the model did not contain a translation (e.g., árbol “tree”). Figure 9 reveals that cognates show higher activation than false-cognates, translatable words or single-language words. The graph also reveals a trend for false-cognates to have slightly lower overall activation than translatable or single-language words. This finding is consistent with research from priming studies indicating that false-cognates show no priming advantages relative to non-cognate words (Lalor & Kirsner, Reference Lalor and Kirsner2001; Sánchez-Casas & García-Albea, Reference Sánchez-Casas, García-Albea, Kroll and De Groot2005), and research showing that naming latencies are slower for false-cognates relative to non-cognate words (Kroll, Dijkstra, Janssen & Schriefers, Reference Kroll, Dijkstra, Janssen and Schriefers2000), perhaps due to reduced false-cognate activation as a function of increased competition at the lexical level.

Figure 9.

Activation of the BLINCS model with different types of target words. Cognate words showed greater activation than false-cognate words, translatable words (i.e., the model contained both Spanish and English translations of a single concept), and single-language words (i.e., the model contained only the Spanish or English word).

Integration of visual information in the BLINCS model

At the phonological level, visual information is able to influence the model's ability to select a target phoneme by providing visual articulatory information in the form of additional phonemic input. Along with the auditory input provided to the model, a secondary input, meant to represent the end-state of recognition of lip-movement, is simultaneously integrated with the phonological vector. For example, the model may be given the word bill, but simultaneously be provided with visual input that is consistent with the initial phoneme /g/, as with the word gill. When the model processes the word bill, the initial phoneme is therefore the average of the two quantified vectors /b/ and /g/, which results in activation of the phoneme /d/ as the initial phoneme. In this way, the visual information results in a change in percept, causing the model to select the phoneme that best fits the quantified integration of the two inputs (see Figure 10). Visual input at the phonological level can also help perception – a classic study by Sumby and Pollack (Reference Sumby and Pollack1954) showed that corroborating visual information improves word recognition in noise. When a large amount of noise is added to the phonological level in the BLINCS model (by randomly shifting the value of three elements in the phonological vector), the system is less able to select a particular phoneme. The addition of corroborative visual input (e.g., visual information for /b/ during noisy presentation of bill) reduces the noise by making the phonological vector more like its originally-intended target. This process is especially important for bilingual speakers, as evidence suggests that bilinguals may rely more on this sort of multisensory integration than monolinguals (Kanekama & Downs, Reference Kanekama and Downs2009; Marian, Reference Marian and Pavlenko2009; Navarra & Soto-Faraco, Reference Navarra and Soto-Faraco2007).

Figure 10.

Activation of the words bill and dill in the BLINCS model. Panel A shows auditory presentation of the word bill when accompanied by consistent visual information (the initial phoneme /b/ as in bill). Panel B shows auditory presentation of the word bill when accompanied by inconsistent visual information (the initial phoneme /g/ as in gill). The curves show that in the inconsistent case (B), dill is activated more than bill, which reflects the integration of the auditory phoneme /b/ with the visual phoneme /g/ resulting in perception of the phoneme /d/, as in dill (i.e., the McGurk effect; McGurk & MacDonald, Reference McGurk and MacDonald1976).

The BLINCS model also contains a mechanism for constraining the activation of words based on the presence of items in the visual scene. Compelling evidence for a relationship between linguistic and visual input during language processing comes from the visual world paradigm (Marian & Spivey, Reference Marian and Spivey2003a, Reference Marian and Spiveyb; Shook & Marian, Reference Shook and Marian2012; Tanenhaus et al., Reference Tanenhaus, Spivey-Knowlton, Eberhard and Sedivy1995), which measures eye-movements as an index of language activation in visual contexts. For example, visual context can constrain listeners’ syntactic interpretation of an utterance (Knoeferle et al., Reference Knoeferle, Crocker, Scheepers and Pickering2005; Spivey et al., Reference Spivey, Tanenhaus, Eberhard and Sedivy2002), and may play an important role in cross-linguistic lexical activation (Bartolotti & Marian, Reference Bartolotti and Marian2012; Shook & Marian, Reference Shook and Marian2012). Furthermore, given sufficient viewing time, visual scenes themselves appear capable of activating the labels for the items they contain (Huettig & McQueen, Reference Huettig and McQueen2007; Mani & Plunkett, Reference Mani and Plunkett2010; Meyer, Belke, Telling & Humphreys, Reference Meyer, Belke, Telling and Humphreys2007), suggesting that the visual context can boost activation of lexical items. The BLINCS model allows for the inclusion of visual activation in order to simulate the effects found in the visual world paradigm. BLINCS assumes that this visual activation is a product of connections between a visual recognition system and the semantic level; therefore, BLINCS can constrain lexical access by providing additional activation directly to nodes at the semantic level that correspond to items that the visual-input module indicates as currently visible. Here, the visual input module is meant to simulate the activation constraint born from presenting a limited set of visual stimuli during language processing and does not reflect the sensory or perceptual processes involved in visual recognition (i.e., shape or color recognition).

The model is given a list of “visually presented” objects, as well as phonological input, and activation simultaneously begins at the stages of semantic access and phonological processing. Thus, as the phonological input enters the system and feeds upward to the phono-lexical level, feedback from the semantic level down to the phono-lexical level increases activation to those items that were present in the visual display. For example, Figure 11 shows how presentation of the English word pear with an image of a pear results in activation of both pear and perro “dog”, but not volcano (panel A), but that the addition of the image of a dog results in greater activation of perro “dog” than when only the pear is present (panel B). Finally, including an image of a phonologically unrelated item (a volcano given the target word pear) results in the activation of the lexical entry volcano. This mechanism can explain how linguistic activation may be constrained by objects in a visual scene, while also providing a mechanism for explaining how visual context can potentially access the lexical labels for those objects before, or without, linguistic input. The current framework also assumes that the visually presented objects will likely activate lexical items in both languages – since the additional activation occurs at the semantic level, the semantic representation will be able to feed back to both English and Spanish phono-lexical items. Further empirical and computational research will be necessary to determine the extent and the exact manner in which visual information affects bilingual language activation.

Figure 11.

Activation of the words pear, perro “dog”, and volcano in the BLINCS model during auditory presentation of the target word pear, accompanied by visual presentation of (A) pear alone, (B) pear and perro “dog”, (C) pear and volcano, or (D), pear, perro “dog”, and volcano.

Language identification and control

The structure of the BLINCS model, like that of the SOMBIP (Li & Farkas, Reference Li, Farkas, Heredia and Altarriba2002), provides a means of organizing the lexicons of Spanish and English without explicitly tagging or labeling the items during training. Specifically, the phono-lexical and ortho-lexical levels showed separation of the two languages within the structure of the self-organizing maps, such that same-language words tended to cluster together, with the notable exception of cognates and false-cognates. This separation has implications for the process of language selection and control in bilinguals. One crucial function that may involve the use of language tags is inhibitory control of one of a bilingual's two languages during processing. According to Green's (Reference Green1998) Inhibitory Control model (or, IC Model), the activation of an entire lexicon can be dampened during processing by an inhibitory mechanism, which identifies the language for suppression through the use of language tags. In contrast, Li (Reference Li1998) suggests that the language system may develop so that lexical items that belong to the same language are grouped together by a learning mechanism (much like in the SOMBIP, and the present BLINCS model), where the language system can draw associations between patterns of localized activation and the form of the input. In other words, if a given word consistently activates a set of other words, the language system may associate those items together, resulting in language-specific patterns of activation.

Although BLINCS is able to utilize language tags as a mechanism for language selection, the present implementation of the BLINCS model is also equipped to capture associations between words within a single language and can use these associations to guide language selection. The model received each of the 480 words that it was given during training and we measured the overall activation of both English and Spanish words. By subtracting the average activation of English words from that of Spanish words and dividing by the average activation of all nodes, we calculated a language activation score. A positive language activation score would represent a system that was tilted towards English activation, i.e., those nodes representing English words were more active on average, as a result of the target word. Figure 12 shows the histogram of language activation scores for each word, separated by target language. Statistical analyses on the model-generated data revealed a significant difference between English words (N = 240, M = 0.147, SD = 0.1) and Spanish words (N = 240, M = –0.135, SD = 0.08), t(478) = 34.13, p < .001. When English words are presented as targets, the model is more likely to activate other English words, and when Spanish words are presented, BLINCS is more likely to activate Spanish words. We also measured the top 10 co-activated words for each word in the model; when the model was given an English word, the average proportion of Spanish words in the cohort was 21.1% (SD = 0.19), and when given a Spanish word, the cohort was 69.5% (SD = 0.22) Spanish, t(478) = –24.8, p < .001. These accuracy levels differ from those found in natural language, where bilinguals recognize the language to which a word belongs with high accuracy. This difference is likely due to the fact that in natural language situations, listeners can draw upon their previous experience and both the linguistic and environmental context to provide clues about language membership. Despite these differences, the model is successful in its ability to utilize localized activation patterns to determine which lexical items to suppress (e.g., a portion of the words with relatively low activation), resulting in a generalized pattern of specific-language inhibition (consistent with Li, Reference Li1998).

Figure 12.

Histogram of language activation scores for each word. Points above zero reflect greater overall activation of English lexical items, while points below zero reflect greater overall activation of Spanish lexical items. The mean activation for English words was 0.147 (SD = 0.1) and the mean activation for Spanish words was –0.135 (SD = 0.08), indicating that English and Spanish words activated items in their corresponding languages to a similar degree.

The language system may retain this localized activation from word-to-word, so that if a word in one language immediately follows a word from the other language, as with a code-switch, the second word will be more difficult to access. This is consistent with evidence for a processing cost associated with switching languages (Costa & Santesteban, Reference Costa and Santesteban2004; Costa, Santesteban & Ivanova, Reference Costa, Santesteban and Ivanova2006; Gollan & Ferreira, Reference Gollan and Ferreira2009; Meuter & Allport, Reference Meuter and Allport1999). This process could occur either through a mechanism of general inhibition, in which the lexical nodes of one language are suppressed, or due to the lexical items of the presently-in-use language being more highly active on average.

One way to accomplish the goal of a general inhibitory mechanism is to mathematically reduce the relative activation of one language in order to promote activation or selection of the other. Individual items can be inhibited in an attempt to normalize the language activation score of the to-be-suppressed language to zero (equivalent English/Spanish activation). In this way, if the model considers one language to be dominant relative to the other, more inhibition would be required to normalize the relative activation score, which would effectively simulate the switch-cost asymmetry such as the one found in Meuter and Allport (Reference Meuter and Allport1999). Conversely, a balanced model may not show such an asymmetry, as the amount of suppression required would be equal across languages (see Costa & Santesteban, Reference Costa and Santesteban2004).

Alternatively, the model may not require a general suppression mechanism for lexical selection at this level. Rather than depending on suppression of lexical items, the language system may be represented by a threshold model, where lexical items race to reach a selection threshold. In BLINCS, a switch-cost might occur in the following way: Prior input to the model may result in localized activation of one language. When a new word from the unused language is presented to the model, that new word is forced to compete with items from the previously used language that are already active, resulting in delayed processing. A switch-cost asymmetry could arise as a consequence of the localized activation for each language, and reactive lateral inhibition at the level of the lexicon. Given sustained L1 input, activation in the model would be more localized (though not completely) to other L1 words, which would inhibit each other through lateral connections. Upon switching to L2, the L2 word would compete with L1 words, resulting in a switch-cost. In contrast, given sustained input in L2, relative activation in the model would be more balanced for L1 and L2 candidates than in the previous case where the target word was in L1. The co-activated L1 words would again reactively inhibit one another, so that upon an L2-to-L1 switch, the L1 word would compete with L2 candidates, and encounter a language system where its L1 neighbors are potentially primed for lateral inhibition, or are less able to facilitate the target (by virtue of proximity in the map) due to being previously inhibited.

A third possibility is that a language switch-cost is not necessarily inherent to the language system. Finkbeiner, Almeida, Janssen and Caramazza (Reference Finkbeiner, Almeida, Janssen and Caramazza2006) had bilinguals name pictures in their L1, preceded by a digit-naming trial (similar to Meuter & Allport, Reference Meuter and Allport1999) in either their L1 or L2. They found that bilinguals did not show a switch cost; they replied equally fast to switch trials and non-switch trials (though the switch-cost was found when participants switched between L1 and L2 for naming digits). In addition, when participants are allowed to voluntarily switch, they do not show a switch-cost and sometimes even show facilitation of the response (Gollan & Ferreira, Reference Gollan and Ferreira2009). It is therefore possible that the switch-costs found in previous studies are not intrinsic to language processing, but reflect the participants' expectation of language change. Under this scenario, the BLINCS model would show switch-costs only when primed to do so by a context or task-dependent module apart from the language system (which would be compatible with either a suppression-based or activation/threshold-based account of switch-costs). Future computational and empirical work will need to determine the exact nature of these effects.

Irrespective of the underlying processes responsible for language-specific activation or suppression, the BLINCS model is capable of distinguishing between languages (a necessary feat for both accounts) without the need for explicit tags or nodes by learning the characteristics of the input and using that information to define the layout of the language system.