3.1 Identifying the Appropriate Conceptual Framework

The focus in this chapter is on methodological issues in research with particular attention to the bilingualism and the childhood language impairment literatures. In the past few decades, the number of studies examining the cognitive control skills (common term: executive functions) of children with different language abilities and proficiencies has shown significant growth in both bodies of literature, largely because most research questions target complex problems that cut across traditional disciplines. Oftentimes, however, the questions of these studies are not conceptually but educationally or clinically driven. Having no or an unclear conceptual framework appears to be related to issues such as the lack of comprehensive theoretical models integrating cognitive control concepts and language processing theories; the mixed use of terminology across disciplinary literatures; and limited familiarity with other disciplines’ literature and conceptual frameworks among scholars who work at the intersections of different fields. Moreover, even those who claim to perform interdisciplinary research, frequently just borrow theoretical constructs rather than integrate concepts and theories from two or more disciplines (CohenMiller & Pate, Reference CohenMiller and Pate2019).

A further issue is that the term “conceptual framework” itself is used with at least three different meanings. It may refer to major ideas and arguments of a study; it may be used as a synonym for “theoretical model”; or it may be the guiding principle for the entire research process, from the general statement through the literature review, research questions, methodology, data analysis, and theoretical, as well as practical implications (Ravitch & Riggan, Reference Ravitch and Riggan2017). Thus, depending on one’s scientific approach, the same conceptual framework may play different roles in the research process and may be used with different meanings.

To provide perspectives and constructs for multiple disciplines, integrative models are needed. Recent shifts in focus to cultural and linguistic diversity within both the bilingualism and language impairment literatures has prompted scholars to develop more integrated conceptual frameworks (e.g., Chen & Padilla, Reference Chen and Padilla2019; Jiménez-Castellanos & García, Reference Jiménez-Castellanos and García2017; Mistry et al., Reference Mistry, Li, Yoshikawa, Tseng, Tirrell, Kiang and Wang2016). A number of these studies included researchers and participants from ethnic and linguistic minority groups, and they added new social constructs to the existing models to integrate such cultural, social-economic, linguistic, and life-experience factors that may help to interrupt educational and social inequality. Traditionally, however, the bilingualism literature focused on children’s language background (i.e., level of proficiency, age of acquisition, and frequency of language use), whereas the sociolinguistic literature examined aspects of biculturalism. As of today, when studying cognitive control in bilingual children, demographic factors, such as immigration status, community background, and educational opportunities (e.g., school district), are not typically considered. Thus, specific social constructs are not included in cognitive-linguistic conceptual frameworks, despite having results that link children’s cognitive, language, and academic performances to their socioeconomic status (Merz et al., Reference Merz, Wiltshire and Noble2019; Sheridan et al., Reference Sheridan, Peverill, Finn and McLaughlin2017) and cultural background (Grossmann & Varnum, Reference Grossmann and Varnum2011). In future studies, a number of these constructs should be included in conceptual frameworks that are used in the child language impairment literature, especially in studies with bilingual children with developmental language disorder.

3.2 Matching Conceptual Frameworks and Research Designs

Following the selection or development of a comprehensive or integrative conceptual framework, it is important to ensure that it is in line with the study’s design. Both the bilingualism and the child language impairment literatures are full of contradictory findings related to methodological issues. One problem is a mismatch between the conceptual framework and the methods. The approaches that researchers select to design their studies reflect the ways they conceptualize the problems they examine. When the methods do not match the conceptual framework, the outcomes are often misinterpreted and explained without any compelling arguments. The most common methodological issues are related to the selection of participants, tasks, and target functions. There is a complex relationship among these variables, and any meaningful research outcome will depend on how they interact within the given research design.

3.2.3 Target Function Selection

As discussed in various chapters in this book, cognitive control represents a broad multifaceted construct comprising a wide range of processes and components. There have been attempts in the literature to determine the functional organization of the control processes by identifying higher order functions (e.g., attention) and core functions (e.g., inhibition). Despite the differences in concepts, all models nevertheless operationalize the control system and the underlying neural structures as componential (Baggetta & Alexander, Reference Baggetta and Alexander2016). While the components are separable, they also interact with each other, which makes the assessment of the cognitive control functions challenging. Friedman and Miyake (Reference Friedman and Miyake2017) described this dynamic relationship as the “unity and diversity” of cognitive control components.

When selecting the target functions for a study on cognitive control, it is important to examine the development and functioning level of the individual components. It is equally critical to understand the interactions among the different components to help describe complex human behaviors. Unfortunately, matching tasks with varying demands to specific control processes and components is quite difficult (Badre, Reference Badre2011; Luna et al., Reference Luna, Marek, Larsen, Tervo-Clemmens and Chahal2015). This fractionation of the components appears to contribute to the low correlations among the cognitive control tasks even within the same construct. Studies using latent variable analyses of cognitive control functions across age groups revealed differences in the rate of component development, with more unity in younger children and more differentiation in older ones (Brydges et al., Reference Brydges, Fox, Reid and Anderson2014; Huizinga et al., Reference Huizinga, Dolan and van der Molen2006; Wiebe et al., Reference Wiebe, Sheffield, Nelson, Clark, Chevalier and Espy2011). Karr and colleagues (Reference Karr, Areshenkoff, Rast, Hofer, Iverson and Garcia-Barrera2018) performed a systematic review and a reanalysis of an extensive body of research, including data from 46 samples (N = 9,756) across the human lifespan. Their results showed one- or two-factor models in preschool-age children, three-factor models in school-age children and adolescents, three-factor or nested-factor models in adults, and two-factor models in elderly people. The results across studies were fairly consistent, with some exceptions, such as the findings of a four-factor model in adolescents, including working memory, shifting, inhibition, and planning (Laureys et al., Reference Laureys, De Waelle, Barendse, Lenoir and Deconinck2022). Furthermore, Karr and colleagues noted a certain degree of publication bias based on their systematic review. Many studies replicated previously highly cited models and accepted those as standards. It is not clear how much this practice has contributed to the consistencies in findings across studies.

In addition to age, task complexity also affects the clustering of the different cognitive control components. The most general latent variables (i.e., inhibition/common executive function, updating, shifting) have been observed only with a medium level of task complexity. In more complex tasks, different combinations and separations of the components were apparent (Friedman & Miyake, Reference Friedman and Miyake2017). Further, when evaluating the cognitive control abilities of participants, individual differences in behavioral performance, brain activation, and connectivity must be considered in addition to the age-related and task-complexity factors.

3.2.3.1 Mismatch between Tasks and Targeted Cognitive Control Functions

Mismatches between tasks and targeted functions are quite frequent in both the bilingualism and the language impairment literatures. There are in-depth discussions of this problem with respect to language abilities and levels of proficiency in Chapter 1, “The Language Continuum” and in Chapter 5, “Associations among Language Ability, Language Proficiency, and Cognitive Control.” Therefore, here, we focus on the measures of cognitive control.

Issues related to task impurity and to the complexity of different cognitive control constructs may contribute to the mismatches between both task selection and identification of target functions (see Figure 3.1). Mismatches have been reflected by findings from studies using different tasks (that may not correlate with one another) to test the same cognitive control function (e.g., resisting distractor interference assessed with the Simon, flanker, and matching-to-sample tasks) or by outcomes from studies using the same task to assess different cognitive functions (e.g., the Wisconsin card sorting task to assess task switching, implicit learning, response inhibition, working memory, and conceptual thinking). Variations in age and task complexity further confound the ability to match between task and target function. For instance, in a developmental study using complex linguistic span tasks to measure verbal working memory, Marton and colleagues (Reference Marton, Schwartz, Braun, Bara, Barsalou and Bucciarelli2006) found that younger children (7–8 years old) relied more on storage resources and exhibited difficulty in attention switching and flexible adaptation to changing task requirements compared to older children (10–11 years old) and young adults. The latter two groups relied more on their monitoring and working memory updating skills while performing the same tasks. These differences were also reflected in variations in error types. While younger children produced many interference errors, older participants not only made fewer errors, but their errors were simple omissions rather than interference. Thus, to measure the same cognitive control functions, different tasks should be used with different age groups.

Figure 3.1

Methodological issues and suggestions

The effect of task complexity was clearly demonstrated in a study on processing speed by Cepeda and colleagues (Reference Cepeda, Blackwell and Munakata2013). A large variety of tasks were used to test processing speed across the lifespan, including box completions, digit copying, color naming, and others. Additionally, a series of cognitive control tasks were administered to target participants’ working memory, inhibitory control, and task switching abilities. This study revealed that the more complex processing speed tasks also target cognitive control functions, particularly in children and older adults. Thus, a number of tasks used in the literature as pure measures of processing speed assess complex cognitive control skills as well. Therefore, these processing speed tasks should not be used as baseline measures in complex cognitive control tasks, which is the current practice, because of their overlapping constructs. Instead, Friedman and Robbins (Reference Friedman and Robbins2022) suggest assessing cognitive control skills across multiple contexts and statistically extracting the common variables to receive purer measures with no or minimal random measurement errors.

3.3 Data Analysis

The first question we address in this section is how much the selected scoring method matters. Beyond the basic measures of overall accuracy and reaction time, many behavioral studies have involved more detailed scoring to better describe their groups’ performance profiles, including the examination of the effect of set size, item type, trial order, the application of Z-scores, and partial credit.

Considering the complexity and the popularity of working memory span tasks, we use these tasks to demonstrate how the findings may differ, depending on the scoring method. In an earlier study, Gillam and colleagues (Reference Gillam, Cowan and Day1995) scored each participant’s responses with three different methods. With the conservative method, answers were counted as correct only when they were also recalled in the correct serial position (first place, second place, etc.). With the liberal scoring method, credit was given for each correctly recalled item in the proper order, even if they were not recalled in the correct serial position. For example, if the first two items were omitted but the following three items were in the correct order, then the serial positions were all incorrect, but the order of the three words was correct. Therefore, the child received credit for those three items. With the free scoring method, credit was given for each correct item, regardless of its serial position or the order of recall. Although all participants received the highest scores with the liberal scoring method, the presence of the primacy effect (i.e., recalling the first few items with higher accuracy than the following items) was not impacted by the different scoring styles. In contrast, when examining the recency effect (i.e., recalling the last few items with higher accuracy than the prior items), significant interactions with group × task × serial position emerged. Some researchers have incorporated this serial position idea into their working memory model, so they could examine both the content and the binding between the content and its context (e.g., serial or temporal position) when evaluating the outcomes (Cowan, Reference Cowan, Izawa and Ohta2005; Oberauer, Reference Oberauer2009).

Other modifications to overall scoring were implemented by Leeser et al. (Reference Leeser, Sunderman, Granena, Jackson, Yilmaz, Granena, Jackson and Yilmaz2016), who explored how recall correlated with accuracy, reaction time, and set size using five scoring methods: (1) recall, (2) recall + accuracy, (3) recall + accuracy + reaction time, (4) set size, and (5) composite Z-score. The five scoring methods were correlated within and across working memory tasks with set size showing the weakest correlation, which is an interesting finding, given that set-size manipulations are common in the working memory literature. Set size manipulations are often used to reveal limitations in working memory or attentional capacity (Baddeley et al., Reference Baddeley, Thomson and Buchanan1975; Oberauer et al., Reference Oberauer, Farrell, Jarrold and Lewandowsky2016; van den Berg et al., Reference van den Berg, Awh and Ma2014). Moreover, when Leeser et al. (Reference Leeser, Sunderman, Granena, Jackson, Yilmaz, Granena, Jackson and Yilmaz2016) compared the working memory outcomes with sentence interpretation, they found interactions across working memory task type × scoring method × type of analysis. Their conclusions included two important points: (1) even if the scoring methods show high correlation, there may still be relevant differences among them, and (2) depending on the manner tasks are scored and how the variables are modelled, completely different results may emerge. Thus, in answer to our first question, we may conclude that scoring methods do matter.

In addition to questions about scoring, a number of concerns have been raised regarding manipulations with statistical analyses. Here, we highlight only a few problems from the literature, specifically cognitive biases and issues related to replicability. Two cognitive biases, “dichotomania” and “nullism” (Greenland, Reference Greenland2017), are common issues in the bilingualism and language impairment literatures. Dichotomania refers to the perception of quantities as dichotomous, even when they are not. We have touched upon this problem in our discussion of the measures of language proficiency. When individuals are grouped as monolingual or bilingual speakers, regardless of their level of language proficiency, then a continuous variable is treated as dichotomous. Greenland (Reference Greenland2017) demonstrated this issue with regards to the p value and its categorization as “significant” or “nonsignificant.” Significancy, as a dichotomous indicator is determined by the use of a fixed cut-off point, instead of reporting the real value of the p. Another cognitive bias, nullism, is about the assumption of “no difference,” even when there is no theoretical or conceptual ground for it. This bias is seen in both frequentist and Bayesian analyses. To avoid this issue, Amrhein and Greenland (Reference Amrhein and Greenland2022) suggest that the p value should be calculated for multiple values for the effect size and not just for the null hypothesis that is posited with the assumption of no difference between the inspected values.

Replicability is a further problem in the bilingualism and language impairment literatures. According to Baker (Reference Baker2016), more than 70 percent of active researchers have tried and failed to replicate someone else’s experiments. The question is whether replicability is really important and whether it can tell us anything about the quality of research. Given the number of issues related to the selection of participants, tasks, and target functions, it seems inevitable that two studies would differ in a number of variables. Baker (Reference Baker2016), however, found some other answers based on a survey. More than 60 percent of the scientists surveyed reported pressure to publish and selective reporting in the literature as the two main reasons for the low replicability rate. With current funding and promotion systems, the pressure of publishing has not vanished, despite reports of scientific misconduct linked to such pressure (e.g., John et al., Reference John, Loewenstein and Prelec2012). Selective reporting or publication bias was identified as one of the factors behind the inconsistent findings on the bilingual cognitive advantage. De Bruin and colleagues (Reference De Bruin, Treccani and Della Sala2015) found that the chances for having one’s paper published was much higher (68 percent) if the results supported superior performance in the bilingual group compared to less than half that publication rate (29 percent) if the outcomes showed no group difference in cognitive control between monolingual and bilingual speakers.

These few examples demonstrate that there are many issues that we have been unable to solve yet. Advancements in methodology and research techniques also come with new challenges. It is important, however, to be aware of these issues and incorporate these limitations into the interpretations of the results.

3.4 Bridging the Gap between Scientific Research and Clinical and Educational Approaches

Even though the majority of papers published in applied research journals devote a section to the educational or clinical implications of their research, or to the description of their study’s broader impact on the society, there is a critical need for more systematic translational and clinical research, and for greater collaboration among educators, clinicians, and scientists. Cognitive control is a favored topic, not only in research but in education and clinical practice as well (in the latter two fields, the term “executive function” is used). Multiple hypotheses have been suggested to explain the existence of this gap between research and practice, including the differences between the kind of knowledge that educators and clinicians need during their daily practice and the kind that researchers produce and share. Researchers and educators or clinicians also seem to speak “different languages”; they use different terminology; create and present arguments within different conceptual frameworks; and prioritize different ideas and goals (Garivaldo & Fabiano-Smith, Reference Garivaldo and Fabiano-Smith2023; McIntyre, Reference McIntyre2005; Nye, Jr., Reference Nye2008; Snyder et al., Reference Snyder, Miyake and Hankin2015). This, of course, is not an exhaustive list of issues that contribute to the largely independent paths that researchers and educators, as well as clinicians follow.

When examining the ways this gap could be narrowed, one of the first steps should undoubtably be the expression of mutual respect toward each other’s expertise and the acknowledgement of the value of complementarity. It is crucial that professionals from different disciplines and subject fields demonstrate interest and willingness to work toward intellectual coherence, scholarly merit, and true commitment to teamwork (Hruby, Reference Hruby2012; McIntyre, Reference McIntyre2005). An openness toward real dialogues and the expansion of everyone’s own knowledge base may further strengthen collaborations and interprofessional practice. These examples are general steps that provide a foundation for more specific strategies related to individual areas of research and practice, such as the investigation and clinical application of cognitive control. Snyder and colleagues (Reference Snyder, Miyake and Hankin2015) recommended the utilization of different cognitive control models in clinical research and practice. Their suggestions included the careful selection of specific cognitive control components, the use of multiple tasks for each component, and the extraction of the most sensitive and reliable measures from each task. To further strengthen the partnership and enhance practical relevance, these goals and strategies may be complemented by specific, clinically oriented research questions and approaches, as well as with insider perspectives. A recent exploratory study from the bilingualism literature provides an excellent example of how the quality of research may benefit from the involvement of an insider’s perspective (Garivaldo & Fabiano-Smith, Reference Garivaldo and Fabiano-Smith2023). The data from three studies were reanalyzed and reinterpreted by a researcher from within the target community. The combination of a novel theoretical framework and the addition of an insider’s perspective resulted in the reinterpretation of the language data that reflected some previously hidden sociolinguistic aspects of the participants’ natural language practice.

An in-depth review of the literature shows that the gap between researchers and educators and clinicians is a long-standing one, despite continuous efforts to identify the problems and find solutions. It is a complex problem that involves conceptual, methodological, and ethical issues (Hruby, Reference Hruby2012). Surely in the current digital era, however, we can aspire to find new ways to initiate lively dialogues more easily among professionals from different disciplines.

3.5 Conclusions

In this chapter, we reviewed a few examples from a burgeoning number of methodological problems. Without trying to solve all of these issues, the goal was to demonstrate the complexity of the problems and to call attention to the importance of researchers’, educators’, and clinicians’ awareness of the variety of intricate questions. It is essential that we allow researchers and practitioners to report freely even unexpected and diverse results and findings without having to worry about not getting published or funded because of interpretations that may diverge from the mainstream literature. This is particularly important in research, education, and clinical practice involving disadvantaged communities, groups of speakers of minority languages, and participants from ethnically and culturally diverse backgrounds. Both research and education have to respond more sensitively using individual-difference approaches to the dynamic and distinct practices of the communities in the twenty-first century.