MIXED METHODS RESEARCH IN LANGUAGE TESTING AND ASSESSMENT
The field of language testing and assessment (LTA) has been traditionally dominated by a quantitative paradigmFootnote 1; however, it has experienced a significant shift in the past two decades in the wake of expansion of both the theoretical definitions of language competence and test validity. LTA practice has faced prolific demand to serve various purposes from informing individual students’ learning to leveraging educational policy and curricular reforms and accountability. As a consequence, researchers in LTA are increasingly turning to mixed methods research (MMR) to address the complexity of phenomena underlying LTA practices.
Simply speaking, MMR is an inquiry approach that includes both qualitative and quantitative methods within a single study or program of inquiry (Johnson, Onwuegbuzie, & Turner, Reference Johnson, Onwuegbuzie and Turner2007). The legitimacy of MMR as an inquiry approach in applied linguistics has been acknowledged (Dörnyei, Reference Dörnyei2007), with an increasing number of publications appearing in academic journals. For example, Magnan (Reference Magnan2006) reported that 6.8% of research articles published in The Modern Language Journal from 1995 to 2005 were MMR.
There are two additional reasons that contribute to this turn toward MMR research in LTA. First, the definition of the key construct of interest in LTA, language ability, has embraced multicomponential communicative competence models, prompting researchers to seek to understand the linguistic, psychosocial, political, and cultural dimensions of language competence and its discursive interactions among the dimensions (Abbuhl & Mackey, Reference Abbuhl, Mackey, King and Hornberger2008). Accordingly, MMR researchers seek to understand the complex nature of language acquisition and interaction among various language users in specific sociopolitical and cultural contexts.
Second, these research endeavors indicate LTA and educational researchers’ recognition of MMR not only as a legitimate inquiry approach, but also as a necessity for establishing strong validity frameworks to warrant inquiry claims. Historically, the notion of validity was conceptualized as a set of scientific and objective criteria used to determine the degree to which the findings of scientific investigations could approximate the truth with certainty. Therefore, the notion of test validity was equated with objectivity and accuracy in measuring unobservable latent abilities, such as language proficiency. This early view was clearly postulated in Guilford's (Reference Guilford1946) definition of validity that a test is valid for anything with which it correlates. A more common definition of validity at that time was that a test is valid to the extent that it measures what it is supposed to measure.
Such views were largely rejected by contemporary scholars partly because these conceptualizations failed to recognize testing and assessment as social practices (Cronbach, Reference Cronbach, Wainer and Braun1988). As noted by Cronbach:
As psychological science generates new concepts, test interpretation will have to be reconsidered. Also, because psychological and educational tests influence who gets what in society, fresh challenges follow shifts in social power or social philosophy. (p. 5)
Furthermore, with the rise of the interpretative and constructive paradigms in the 1980s (Lincoln & Guba, Reference Lincoln and Guba1985), scholars came to question the adequacy of value-detached test validity and the existence of any foundation to be used as absolute criteria for legitimizing knowledge claims. Messick's (Reference Messick and Linn1989) seminal article on test validity was reflective of the paradigmatic shift and expansion in that era and succinctly summed up the evolving notion of validity:
What is to be validated is not the test or observation device as such but the inferences derived from test scores or other indicators—inferences about score meaning or interpretation and about the implications for action that the interpretation entails. . . . Validity always refers to the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores. (p. 13)
As the quoted text illustrates, Messick (Reference Messick and Linn1989) emphasized validity as evaluative judgments about the quality of interpretations and use based on test scores. Subsequently, scholars explored how human experiences and interpretations are context-bound, intersubjective, and co-constructed through interactions with other constituents (Schwandt & Jang, Reference Schwandt and Jang2004). The rise of this interpretive camp led to the historic paradigm wars in the early 1990s (Gage, Reference Gage1989), which prompted researchers to recognize MMR as a third wave of the methodological movement. Table 1 provides a historical overview of the expanded notion of validity organized across main inquiry traditions.
Evolving Notion of Validity
The issue and conceptualization of validity in MMR is a topic of recent interest and discussion among researchers (e.g., Creswell & Plano Clark, Reference Creswell and Plano Clark2007; Dellinger & Leech, Reference Dellinger and Leech2007; Onwuegbuzie & Johnson, Reference Onwuegbuzie and Johnson2006). Publications in this area have emerged in response to the concern that evaluating validity claims in mixed methods investigations are particularly complex because of the combination of the “complementary strengths and non-overlapping weaknesses of quantitative and qualitative research” (Onwuegbuzie & Johnson, Reference Onwuegbuzie and Johnson2006, p. 48). Essentially, there is recognition that validity is conceptualized differently in quantitative and qualitative paradigms, as such when these methods are mixed, regardless of the purpose of mixing, a complementary notion of validity needs to be articulated. Teddlie and Tashakkori (Reference Teddlie, Tashakkori, Tashakkori and Teddlie2003) emphasized inference quality as a key validity concept that includes conceptual consistency, interpretive agreement, and interpretive distinctiveness. Greene (Reference Greene, Chen, Donaldson and Mark2011) argued that inference quality should “embrace argument alongside evidence, divergence alongside agreement, contextual understanding alongside causal explanations, and inclusion of the uses and action consequences of the inferences rendered” (p. 89). MMR researchers seek to gain ecological and authentic understandings of cultural contexts (Kirkhart, Reference Kirkhart, Hood, Hopson and Frierson2005) beyond causal claims based on the observation of samples drawn from an unknown population.
Specifically in the field of LTA, researchers seek evidence supporting the validity of assessment practice by integrating the nuanced understanding of human experiences and relations in LTA contexts into the traditional validity discourse. LTA researchers raised moral and ethical issues arising from the impact of testing on the lives of people and society and questioned if intention-based construct validity claims obscure social, consequential responsibilities of test developers and researchers (see the special issue Ethics in Language Testing; Davies, Reference Davies1997).
It is from this historical vantage point that we intend to examine MMR in LTA by analyzing empirical research in the field published in the last 5 years. Accordingly, we begin with a general discussion of the conceptualizations of MMR by focusing on some of its key tenets as well as its various purposes and designs. We continue with an examination of the MMR research in LTA. Based on an evaluation of emerging trends of MMR in LTA, we then offer some general suggestions for how to advance MMR in LTA. These suggestions are not solely based on our belief in the value of MMR, but our contention that the contribution of MMR is integral to researching the complexities of the language construct, which necessitates integration of multiple approaches through multiple perspectives. Our discussion now continues with an overview of MMR not simply as a method, but as an alternative paradigm to traditionally qualitative and quantitative inquiry.
MMR AS AN ALTERNATIVE PARADIGM
As we noted earlier, a common definition of MMR is an inquiry approach that mixes both qualitative and quantitative methods within a single study (Greene, Reference Greene2007; Johnson et al., Reference Johnson, Onwuegbuzie and Turner2007; Tashakkori & Teddlie, Reference Tashakkori and Teddlie2003). This relatively simple definition of MMR lends itself to varying interpretations. As such, there are inconsistencies in the way that MMR is conceptualized despite its increasing applications in LTA and education with part of the difference arising from its literal meaning (i.e., mixing methods). In particular, a main conceptual inconsistency is reflected in the view that MMR constitutes a methodical mixing; that is, it combines qualitative with quantitative methods. While this view contributed to expanding research techniques, it fails to address intricate issues associated with whether or not different methods are tied to particular research paradigms. Patton (Reference Patton2002) notes:
My practical (and controversial) view is that one can learn to be a good interviewer or observer, and learn to make sense of the resulting data, without first engaging in deep epistemological reflection and philosophical study. . . .[One can] simply conduct interviews and gather observational data to answer concrete program and organizational questions without working explicitly with a particular theoretical, paradigmatic, or philosophical perspectives. . .without making a paradigmatic or philosophical pledge of allegiance. (pp. 69, 145)
Such a methodical view is criticized for its emphasis on technical mixing, resulting from a lack of attention to the logic of justification that can help researchers question the nature of social realities, clarify the relationship between the researcher and the researched, and bring forward the epistemological assumptions made when generating knowledgeFootnote 2 claims.
Conversely, the view of MMR as a methodology posits it as an alternative paradigm that is intended to contribute to overcoming the existing paradigmatic divide. This perspective posits that paradigmatic differences are reconcilable through a new emergent paradigm that actively promotes the mixing of methods across research traditions as Greene (Reference Greene2007) notes:
Mixed method inquiry is an approach to investigate the social world that ideally involves more than one methodological tradition and thus more than one way of knowing, along with more than one kind of technique for gathering, analyzing, and representing human phenomena, all for the purpose of better understanding. (p. 119)
This view encourages MMR researchers to engage in philosophical discussions about the nature of being (ontological), different forms of knowledge (epistemological), and particular ways of knowing (methodological). At the methodological level, mixing can take place throughout the entire inquiry process including one or more of the following aspects:
• Inquiry problems with qualitative and quantitative inquiry approaches
• Sampling involving probability and purposeful procedures
• Data with numerical and textual types
• Data analysis involving inferential statistics and interpretive phenomenological analysis
• Inductive and deductive reasoning for theory building and testing
• Interpretations based on emic and etic representations
• Focus on particularity and generality of an inquiry phenomenon
In this article, we conceptualize MMR as a methodology situated within a specific paradigm of inquiry rather than methodical techniques and seek to understand the trends of MMR in the field of LTA.
KEY TENETS OF MMR
As an alternative paradigm, MMR, endorses pluralism in that the complex nature of a social world is best understood from multiple perspectives and multiple methodological lenses each offering partial, yet valuable, insights. MMR researchers may adopt various stances including: pragmatism, dialectical and transformative. Pragmatists take the view that philosophical assumptions are logically independent and can be mixed and matched to achieve the most appropriate answer to a given research problem (Johnson et al., Reference Johnson, Onwuegbuzie and Turner2007; Morgan, Reference Morgan2007; Reichardt & Cook, Reference Reichardt, Cook, Cook and Reichardt1979). Rejecting a traditional dualistic divide, they prioritize practical problems crossing paradigmatic boundaries. The pragmatists choose methods that are most likely to provide data useful for answering research questions given a research context and available resources and informing the decision-making processes that are practically useful and contextually relevant (Datta, Reference Datta, Chelimsky and Shadish1997; Johnson & Onwuegbuzie, Reference Johnson and Onwuegbuzie2004).
Researchers adopting a dialectical stance (Greene, Reference Greene2007; Greene & Caracelli, Reference Greene, Caracelli, Greene and Caracelli1997) recognize paradigms as social constructions that are historically bound in cultural contexts. MMR is viewed as a way of intellectually engaging contradictory or hidden views, values, and assumptions by juxtaposing them with contesting arguments. In this way, the dialectical researchers seek to gain critical knowledge claims from different paradigms or different value positions.
The transformative stance (Mertens, Reference Mertens2007, Reference Mertens2010) acknowledges inequalities and power differentials that exist in society and views MMR as a way to bring forward the voices of socially marginalized people. Transformative researchers (Mertens, Reference Mertens2007) believe that MMR can provide a basis for social change by seeking input from those who were traditionally excluded from power positions.
Although there are differences among these three stances, there are also commonalities. Namely, all the MMR stances posit that social inquiries can have multiple purposes, such as theory building and testing for a better understanding, improving programs and informing decision-making processes, and promoting social justice and equity. They further agree that human and social phenomena are too complicated and multilayered to be known through a single inquiry lens. By adopting an MMR methodological stance, researchers are interested in actively engaging both particular local details and underlying patterns and generality. They strive to gain more comprehensive, insightful, and richer understandings that are more defensible and credible (Cook, Reference Cook, Shortland and Mark1985). Understanding these stances provides a lens through which we can make explicit the underlying purposes and designs of MMR.
TYPES OF MMR BY PURPOSE AND DESIGN
MMR cannot be wholly understood through a discussion of its underlying stances; as with any other approach, MMR demands a clear rationale for its purpose and design (Maxwell & Loomis, Reference Maxwell, Loomis, Tashakkori and Teddlie2003). There are four broad purposes of MMR, including triangulation, complementarity, development, and initiation. Each of these purposes has identifying characteristics at various stages of inquiry, from formulating research questions and collecting and analyzing data to interpreting and synthesizing findings.
Triangulation is one of the most common purposes of MMR and is used to confirm and cross-validate findings while offsetting biases associated with different methods (Denzin, Reference Denzin1978; Mathison, Reference Mathison1988). Originating from Campbell and Fiske's multitrait-multimethod matrix (1959), the notion of triangulation has been expanded to entail triangulations of data, investigators, theory, and methodology (Denzin, Reference Denzin1978). MMR mainly concerns the methodological triangulation in that it involves independent data collection and analysis and joint interpretations at a synthesis stage. Parallel comparison for triangulation requires that different methods elicit the same phenomenon, samples used for different methods are independent, and each method should be given an equal weight. Although convergence is the primary purpose of triangulation, researchers acknowledge that MMR cannot only provide converging but also inconsistent and even contradictory results, which prompt researchers to seek alternative explanations for the findings (Mathison, Reference Mathison1988).
Another common purpose of MMR is complementarity in that it seeks to elaborate, clarify, and explain the results from one method with the results from another method. It may address overlapping or different yet interrelated aspects of the same phenomenon to seek the breadth of the inquiry (e.g., teacher beliefs and practice or program process and outcomes). This latter purpose of complementarity is referred to as an expansion model of MMR (Greene, Caracelli, & Graham, Reference Greene, Caracelli and Graham1989). Data from different methods may be collected concurrently and analyzed independently (Teddlie & Tashakkori, Reference Teddlie and Tashakkori2006). Alternatively, the complementary MMR can be used in other MMR designs, such as triangulation, developmental, and nested designs.
The nested MMR design typically places a lower priority method within a dominant one. Data from the lower priority method are used to elaborate, illustrate, and enrich the findings from the dominant method. For example, an experimental study may be conducted in which qualitative methods are embedded, or it may be a largely qualitative inquiry with a complementary quantitative method.
A third, and equally common, purpose of MMR investigations is developmental MMR. In a developmental MMR design, data from one method are used to guide subsequent data collection and interpretations or to complement and explain results from the preceding method. The full cycle of the developmental MMR features abductive reasoning (Fulcher, Reference Fulcher1996; Mislevy, Reference Mislevy1995) involving (a) inductive case observations used to develop a working theory; (b) hypothetico-deductive testing used to test the theory for its generality; and (c) the application of theory back to cases to confirm or disconfirm its generality. The decision of how to sequence methods may depend on the availability of relevant theoretical frameworks. MMR can start with an exploratory qualitative inquiry for inductive case observations and test the emerging theoretical framework. Creswell (Reference Creswell2007) and others call this MMR design an exploratory sequential.
Alternatively, an explanatory sequential design is used to test an existing theory for its construct congruence and generalizability to the given target population, and subsequently, the core characteristics of the theory, with specific cases in hand, are examined. Note that the developmental MMR does not need to be linear nor complete in cycle. The process is iterative involving both inductive and deductive reasoning processes (Jang, Reference Jang and Robinson2013).
A fourth purpose of MMR, initiation, seeks to discover any potential paradox and contradictions underlying social inquiry problems (Greene et al., Reference Greene, Caracelli and Graham1989). Researchers may mix methods purposefully to recast questions based on results from one method and analyze both consistencies and discrepancies between qualitative and quantitative findings for fresh insights.
The various MMR designs we have reviewed so far are useful for clarifying its purposes; however, in practice, MMR is too complex to be adequately represented by any single design (Maxwell & Loomis, Reference Maxwell, Loomis, Tashakkori and Teddlie2003; Teddlie & Tashakkori, Reference Teddlie and Tashakkori2006). MMR designs should not be applied rigidly or exclusively (Tashakkori & Teddlie, Reference Tashakkori and Teddlie2010); they can be further mixed or modified to create a new component of the design.
One important area for growth in MMR is data integration and synthesis. Parallel presentations of findings by method or question type limit the synergistic effects of MMR, prompting the reader to question its intent. This lack of integration in MMR can be attributable to logistical constraints, such as limited space in most journals to publishing the entire MMR investigation or pressure to amass large numbers of publications in academia. Fully integrated designs mix qualitative and quantitative approaches in an interactive and iterative manner throughout a study. Preliminary results from one method may be transformed to a form compatible with the other for further analysis or be consolidated to create blended variables to further examine new insights. Therefore, mixing methods does not automatically guarantee stronger claims; however, fully integrated MMR has the potential to provide a holistic understanding. We now turn to a discussion of our review of MMR in LTA. We begin with a discussion of our process for identifying these publications.
THE REVIEW OF MMR IN LTA
Our review process of MMR in LTA involved several collaborative stages among the authors in order to address three main purposes: (a) identifying all of the peer-reviewed journal articles employing MMR in LTA for the past 5 years; (b) reading and analyzing each article to understand the rationale for and the approaches to MMR; and (c) synthesizing the analyses in order to highlight some key trends of MMR. We now discuss each of these stages in detail.
Identifying MMR Studies
The process of reviewing the literature involved identifying and applying specific inclusion and exclusion criteria for studies that employed MMR published between January 2007 and July 2013. These criteria included the fact that the research needed to be peer reviewed and published in an English-language journal addressing LTA for learners of any age. Additionally, we identified key terms to use to search research databases and specific journals. We excluded any study that either did not meet our specified criteria or did not report on mixed methods data even if it was mentioned as a method of data collection. We also excluded studies that exclusively used multiple quantitative or qualitative data, but included studies that used both qualitative and quantitative data. See Appendix A for the summary of criteria used for identifying MMR studies. We identified 32 empirical studies that met the criteria for MMR that we subsequently analyzed.
Analyzing MMR Examples
Upon identifying an initial set of research studies, we worked collaboratively to determine specific areas of foci to use for the analysis of each study. Initially, each researcher analyzed two studies independently, and then we met to discuss the findings as well as the categories that we used to analyze the articles. Through a collaborative process we revised these categories and agreed on a final list of the areas of foci to use to analyze the language assessment research employing mixed methods. Specifically, we used the following categories: population of interest, study participants, language modality, research context, research questions and/or purpose(s), rationale for the use of mixed methods, timing of mixing, evidence of collaboration, data sources, data analysis approaches, presentation of results, and synergetic effects (see Appendix B). We further examined ways in which the studies addressed validity issues. Additionally, we each recorded additional comments highlighting important facets of each article. We each analyzed a fraction of the articles. Upon completion of this work, we met to discuss the results in depth and assess if all of the questions had been fully answered. In some cases, we revisited the research articles to ensure that our analyses were fully completed.
Synthesizing the Analyses
In the final stage of the review process, we met once again to synthesize the findings by analyzing trends and discussing our observations and emergent issues. We synthesized the findings by discussing each of the categories of analysis independently and also by examining overall trends of MMR.
TRENDS OF MMR IN LTA
Of 27 journals that were reviewed, 32 studies from 15 journals satisfied the specific inclusion criteria. Overall, 21 studies across various educational sectors focused mainly on learners as study participants. Eleven studies involved more than one participant type (e.g., raters). Teachers were recruited as raters in nine studies. There were two studies that involved administrators as well as students and teachers.
Figures 1–3 summarize the distribution of the 32 MMR studies according to publication year, language component, and study population.
Distribution of MMR studies by publication year.
Distribution of MMR studies by language modality.
Distribution of MMR studies by study population.
As shown in the figures, MMR publications have increased over the past 10 years (note that the data for 2013 are incomplete as we reviewed articles published up to July 2013). These MMR investigations most frequently focused on writing assessment as compared to other language skills. The study contexts varied from elementary to postsecondary school sectors while the latter tended to be dominant, which is consistent with the fact that large-scale language tests are predominantly used to assess adult learners rather than young learners.
Purposes of Mixing
Before we delve into our analysis of the purposes of mixing among the empirical studies we identified, it is necessary to highlight the fact that not all of these studies explicitly acknowledged or defined their investigation as MMR, but rather, it was through our reading and analysis of the articles that we identified them as MMR. The interested reader might wonder how we uncovered these studies when there was no explicit mention of MMR; the credit is owed to powerful search engines that search for key terms not only in the body of published articles, but also in reference lists. Oftentimes, the studies that did not explicitly address MMR in the body of their articles cited other publications with MMR explicitly stated in their titles (and hence searchable through the reference lists). It is also possible that additional MMR investigations have been published in the last five years that we were not able to uncover, and hence did not include in this review, because they did not explicitly acknowledge mixing.
Among the 32 MMR studies we reviewed, we identified several studies that mixed methods for multiple purposes, which made it difficult for us to categorize them using the MMR typologies (Creswell & Plano Clark, Reference Creswell and Plano Clark2007; Greene et al., Reference Greene, Caracelli and Graham1989) discussed earlier. We acknowledged this multiplicity in a summary of our analyses presented in Table 2. Note that Table 2 also includes two additional MMR typologies (initiation and transformative) that reflect researchers’ paradigmatic orientations to MMR.
Distribution of MMR According to Purposes of Mixing
Note: *Did not explicitly state the purpose for employing a mixed methods study. The classification is based on our analysis.
**Study is placed in multiple categories.
Overall, our review showed that complementary mixing was the most frequently identified as the purpose of mixing. We identified 18 studies that used mixed methods to elaborate, clarify, and illustrate the results. There were 12 studies that combined methods for triangulation purposes to determine if there was convergence or corroboration of the results emerging from the different methods employed. The prevalent use of MMR for the triangulation and complementarity purposes in LTA is consistent with the application of MMR to applied linguistics and second language (L2) acquisition (Hashemi, Reference Hashemi2012).
One common concern regarding the application of MMR for triangulation arises when mixed data point to divergence and inconsistency. Researchers express concerns when they encounter inconsistent or even contradictory findings from mixed methods data. Unlike the traditional notion of triangulation used for mixing multiple methods without crossing the methodological border (e.g., qualitative interviews with open-ended observations), the purpose of triangulation used in MMR should not solely focus on producing converging evidence to support a singularly true proposition (Mathison, Reference Mathison1988). MMR scholars may choose to welcome not only converging but also inconsistent or even contradictory findings as plausible explanations that need to be thoroughly validated. This welcoming of contradiction was exemplified in Yu's (Reference Yu2010) MMR investigation.
In Yu's study (Reference Yu2010), 157 students read extended English texts, presented on computer or in print, and then wrote summaries on paper in English and Chinese. The study also used questionnaires and interviews to gather data about the students’ perceptions about presentation mode and computer familiarity. The students’ summary task performance data were analyzed using univariate analyses to determine the effect of presentation mode and computer familiarity. The questionnaire data were factor analyzed to identify factors associated with computer familiarity. The interview data were thematically analyzed to understand psychological differences in students’ experience with different presentation modes. Although the analysis of each data source was performed and presented separately, Yu discussed both converging and diverging results in the discussion. He stated that although the presentation modes did not affect students’ summarization task performance, except for a main effect on the length of Chinese summaries (converging evidence), the results from the qualitative data suggested physical and psychological differences in students’ experience with the two presentation modes and their potential effect on the task performance.
Five studies were identified that mixed methods for developmental purposes. In these studies, findings from earlier phases of research were used to inform subsequent phases. The developmental purpose of MMR is often confusing because its sequential order of data collection is necessitated by the interdependency of mixed methods. Simply collecting one data type, say using a questionnaire, before another type (e.g., interviews) does not make a study developmental. It is considered developmental if the results from the first data source are used to guide a facet of the second. For example, questionnaire data could be used to inform interview protocols or determine criteria for interview sampling.
Ghanbari, Barati, and Moinzadeh's (Reference Ghanbari, Barati and Moinzadeh2012) examination of Iranian English as a foreign language (EFL) raters’ use and perceptions of a writing rating scale exemplifies a developmental MMR design. They interviewed 10 expert raters and used the results from the interviews to develop a questionnaire. They interviewed 7 (out of 30 questionnaire participants) to seek further information about their responses to the questionnaire. Another developmental MMR example is illustrated by Isaacs and Trofimovich's (Reference Isaacs and Trofimovich2012) investigation. They designed a sequential MMR over three phases. In phase 1, 60 listeners rated 40 Francophone English language learners’ speech samples. In phase 2, the same speech samples were presented to teachers to seek their written reports on the aspects of speech they attended to. In the final phase, the researchers identified core speech measures that distinguished among L2 comprehensibility levels to develop a guideline for L2 comprehensibility scales.
We identified only one study (DiPardo, Storms, & Selland, Reference DiPardo, Storms and Selland2011) that mixed methods for initiation purposes in order to uncover contradictions or reveal new perspectives. There was also only one investigation by Hamid, Sussex, and Khan (Reference Hamid, Sussex and Khan2009) among the 32 we analyzed that mixed methods for transformative purposes to address inequities and advocate social change. Specifically, Hamid et al. examined the private tutoring in English (PT-E) practices in a disadvantaged rural area of Bangladesh. They insisted that although PT-E is considered as a necessary condition for academic success in Bangladesh, the quantity and quality of the participation are heavily mediated by the family affordability in the studied area, where mainstream schooling and the self-study resources are shown to be inadequate. We now turn our attention to the study contexts.
Context
The majority of the studies took place predominantly in higher education learning contexts with adult immigrant or university populations. For example, Hyland (Reference Hyland2009) involved first-year college students who took a school-mandated academic writing assessment to examine referencing practices of students who speak English as a first language (EL1) and learners who speak English as a second language (EL2). Nine studies focused on the K–12 context in general, six of which addressed issues related to secondary adolescent learners. It is notable that there were only a few studies that focused primarily on young language learners. Uchikoshi and Maniates's (Reference Uchikoshi and Maniates2010) MMR research examined bilingual literacy development involving Spanish-English and Cantonese-English bilingual children to investigate the bilingual classroom practices and home–school connections. Nakatsuhara's (Reference Nakatsuhara2011) investigation also placed a focus on younger learners, specifically on upper secondary EFL learners in Japan. She investigated the nature of co-constructed interactions in group-speaking tests.
Collaboration
To what extent are the MMR investigations in these contexts a result of collaborative endeavors? We chose collaboration as a category of analysis in this review because of its importance to MMR. Most researchers have specialized in their academic disciplines and professional training in either a predominantly qualitative or quantitative tradition, the so-called trained incapacities (Reiss, Reference Reiss, Becker, Geer, Riesman and Weiss1968, cited in Bryman, Reference Bryman2007). Dörnyei (Reference Dörnyei2007) posits that this localized expertise may be a barrier to conducting MMR. One common solution to this challenge is for researchers to bring together teams with a range of expertise. Interestingly, there was little mention of collaboration in the MMR studies we analyzed beyond the fact that the reviewed manuscripts were sometimes authored by multiple researchers or made references to research assistants coding data mainly to establish reliability estimates. For example, Perrone's (Reference Perrone2011) research engaged additional researchers to videotape and also code data to enhance reliability.
No explicit mention of collaborative teamwork in MMR may suggest that LTA, compared to other disciplines, is equipped with researchers who specialize in both research traditions. If findings from mixed data are well integrated to generate insights that data from one method alone cannot achieve, then, the trained capacity in MMR is much more desirable than developing a team of researchers, each of whom has a strong inclination or training in either tradition. However, Hesse-Biber and Leavy (Reference Hesse-Biber, Leavy, Hesse-Biber and Leavy2006) question the value of MMR done by researchers inadequately trained in both inquiry approaches. Bryman (Reference Bryman2007) warns that “the presence of skill specialisms may lead to compartmentalization of roles and responsibilities that can hinder the integration of findings” (p. 16). Our observation that there was no explicit mention of collaborative teamwork for MMR is tentative and needs to be further probed by looking into the current trend of training in LTA in graduate programs. As Dörnyei (Reference Dörnyei2007) asks, “Apart from a relatively small number of unique, methodologically ambidextrous specimen, can we assume the vision of a multimethodologically savvy new breed of researchers is realistic?” (p. 46). We now turn our attention to the sources of data used in the MMR studies.
Data Sources
Together, the MMR studies incorporated a range of data sources that primarily included test data, ratings (e.g., scores from oral interviews, writing scripts), written accounts (e.g., reflective essays), interviews, questionnaires, and think-aloud protocols. To a lesser extent, classroom observations and field notes were also used as data sources. Among the 32 studies reviewed, 13 focused on writing assessment and another 8 focused on issues related to speaking assessment. Three studies examined reading, while there were four studies that examined all four modalities (see Figure 2).
Quantitative data sources predominantly originated from ratings of students’ written or spoken responses, questionnaires, and scored responses to proficiency or achievement tests. Considering that 13 out of the 32 reviewed studies focused on writing assessment, it is not surprising that the most frequently observed quantitative data source was derived from students’ written responses, which were subsequently treated to generate mixed data, rating scores, and written accounts. This treatment exemplifies the case of mixing at the stage of analysis by quantizing qualitative data. Some studies (e.g., Poonpon, Reference Poonpon2011) also analyzed students’ written and spoken responses qualitatively and quantitatively simultaneously in order to triangulate the findings to offer confirmatory evidence supporting the accuracy of rating in speaking and writing assessment.
Other notable mixing of data was observed between (a) students’ test responses and their verbal reports from either think-alouds or interviews, (b) raters’ ratings and their verbal/written accounts, or (c) quantitative data from closed or Likert-scale responses to questionnaires and written responses from open-ended questions included in the same questionnaire. These pairs of mixing were often used for triangulation and were also predominantly used to complement each other while qualitative data played a complementary role by illustrating proficiency levels inferred from the scores with specific examples. For example, Winke (Reference Winke2011) investigated the “perceived effectiveness” (p. 628) of the English Language Proficiency Assessment (ELPA) by educators in order to investigate validity claims. The primary data source was a set of responses to a survey that comprised (a) items gathering participants’ demographic information; (b) belief statements with Likert-scale response options, as well as space for written comments; and (c) five open-ended questions. Winke reported that the quantitative data were used to identify response patterns, and that the qualitative data were used for illustrative purposes.
As we discussed earlier, MMR cannot be reduced to the inclusion of both qualitative and quantitative data included in a single study; there is a need for integration. The next section of our analysis addresses the MMR studies’ approaches to data analysis and integration.
Data Analysis and Integration
Integrative data analysis and interpretation of mixed methods are critical in MMR. Our review showed that some studies did not provide sufficient information about their data analysis, and that the majority of the studies (n = 23) reviewed kept both analysis and interpretation of the two data types separate. There were nine studies that analyzed and interpreted integratively.
Studies that kept the two types of data separate presented results according to data type. For example, Baker (Reference Baker2010) compared raters’ scores between high- and low-stake testing conditions in order to examine differences in grading in the raters’ perceptions. Results from each data source and phase were reported separately and integrated in the discussion of all the study findings in the later sections of the article.
Similarly, some studies analyzed mixed methods data separately to answer different research questions. In such cases, results were presented by questions (not through a quantitative or qualitative distinction), and integration took place at the discussion stage. For example, Kim (Reference Kim2009) analyzed 80 speech samples using the Rasch model and independently analyzed teachers’ written comments thematically to answer two separate questions respectively. She brought the results together in the discussion by highlighting that “the inclusion of a qualitative analysis provided insight into the different ways in which NS and NNS teachers assessed students’ oral language performance, above and beyond findings from the quantitative analysis alone” (p. 210).
We also examined the MMR data analysis and integration of the studies by examining them according to established MMR data analysis strategies. Caracelli and Greene (Reference Caracelli and Greene1993) discuss four integrative mixed methods data analytic strategies: data transformation, typology development, extreme case analysis, and data consolidation. Data transformation involves the conversion of one data type into the other (e.g., quantitizing qualitative data by assigning numerical or categorical codes or qualitizing quantitative data into a narrative to be compared with thematically analyzed qualitative data). Typology development is used when one type of data yields typology (a set of distinct categories that explain dimensions and patterns) and the resulting typology is used as an explanatory variable for subsequent analysis. Extreme case analysis is used to confirm or disconfirm findings from one data type by using cases from the other data type. Data consolidation is used through a joint analysis of both data types to create consolidated data sets for further analyses.
None of the studies we reviewed used typology development or data consolidation strategies. A few studies we reviewed analyzed data through integrative mixed methods data analytic strategies including data transformation and extreme case analysis. The most frequent data transformation was done by coding students’ writing and speech samples both inductively and deductively. This integrative analysis allowed for a direct comparison of the results and highlighted both overlapping and non-overlapping aspects of the phenomenon under investigation. For example, Nakatsuhara (Reference Nakatsuhara2011) analyzed group oral tests from 269 Japanese upper secondary school students in order to investigate the influences of students’ extraversion levels and oral proficiency levels on their conversational styles in different-sized groups. The data sources included several questionnaires, students’ oral proficiency levels as measured using a rating scale, and students’ performances on group oral tasks. Students’ oral performance data were used to quantify conversational styles in multiple ways, and these results were used to conduct several multiple regression analyses. However, Nakatsuhara also analyzed students’ performances qualitatively “to illuminate the quantitative results” (p. 491). Nakatsuhara explained the integrated data analyses approach:
. . .the coded data were analysed through multiple regression to provide a generalizable picture of the data. However, CA [conversational analysis] is generally hostile towards explicit coding and quanti-fication (Schegloff, Reference Schegloff1993). Therefore, when applying CA approaches in the second part of the results, the analysis focuses on interpretation, avoiding the use of explicit quantita-tive language. The two perspectives are seen as complementary, the CA providing insights that help to explain the statistical results. (p. 492)
Extreme case analysis is an integrative analytic strategy used to confirm or disconfirm findings from one data source using cases from the other type (Caracelli & Greene, Reference Caracelli and Greene1993). There were two studies that used additional case analyses (Jang, Reference Jang2009; Lee & Greene, Reference Lee and Greene2007). In one of these studies, Lee and Greene (Reference Lee and Greene2007) investigated the relationships between graduate English as a second language (ESL) students’ placement test scores and their academic performance measures (i.e., grade point average [GPA], faculty evaluations, and student self-assessment). Using correlational analyses, they triangulated students’ perspectives with that of the faculty by mixing data from questionnaires with interviews. They further analyzed mixed data using extreme case analysis and joint display strategies to shed light on inconsistencies and contradictions across the data sets. They identified three lower-level cases and one higher-level case based on statistical analysis that detected outliers (two standard deviations below and above the placement and GPA means). Through in-depth case analyses, Lee and Greene sought to explain the case students’ perspectives about academic success. The joint display of the quantitative and qualitative results by participant type (students vs. faculty) supported the predictive validity of the placement measure and further contributed to revealing the nuanced nature of academic learning and achievement.
Synergy refers to the added value of MMR as a result of integration of the findings from mixed methods data analysis. This synergistic effect is greater than if only qualitative or only quantitative approaches are taken (Nastasi, Hitchcock, & Brown, Reference Nastasi, Hitchcock, Brown, Tashakkori and Teddlie2010). Overall, the degree of integration in the studies we reviewed is limited because the majority of the studies kept the qualitative and quantitative components separate rather than integrative. A few studies that we regarded as having synergism are among those using the most complex MMR designs (e.g., Jang, Reference Jang2009; Nastasi et al., Reference Nastasi, Hitchcock, Brown, Tashakkori and Teddlie2010).
One of the core principles of synergism is the dialectical stance that we introduced earlier and can be exemplified in Jang's (Reference Jang2009) research, which took this stance to provoke potential issues with cognitive diagnostic modeling (CDM) by using case analyses. Jang was interested in the extent to which reading skills on an assessment could be used for diagnostic purposes. Her study generated diagnostic feedback using CDM and sought out students’ reaction to it. Students’ responses to the usefulness of the (quantitative) diagnostic feedback contradicted Jang's observations of the teachers’ use of the feedback. She observed that teachers did not have time to address all the skills, prompting her to question the validity of students’ reports. Consequently, she sought evidence for counterarguments using additional methods. The attempts to reconcile these differences highlight the tenets of the dialectical stance and further demonstrate the synergistic effects prompted by the use of multiple methods.
Clearly, MMR does not guarantee a sum greater than its parts (Dörnyei, Reference Dörnyei2007; Hesse-Biber & Leavy, Reference Hesse-Biber, Leavy, Hesse-Biber and Leavy2006). As we have discussed, MMR necessitates the integration of quantitative and qualitative data at some stage of the research investigation. Many examples of MMR demonstrate integration of data at the discussion stage after analyzing and presenting each type of data separately (e.g., Barkaoui, Reference Barkaoui2011; Kim, Reference Kim2009). Our review suggests that the application of MMR in LTA has not reached its fullest potential due to a lack of integration of mixed methods data throughout the inquiry procedure, which seems to be a common trend of MMR across disciplines.
Dörnyei (Reference Dörnyei2007) discusses three reasons why people do not mix more actively and pinpoints practical constraints (such as page limits in journal publication) or pressures for more publications:
Finally, there may be a third reason for the paucity of mixed methods studies. The publication pressures of the current research climate promote piecemeal publications in general and therefore even when a project involves methods mixing, the authors might choose to try and publish the results of the different phases separately. (p. 175)
More examples of successful practice employing integrative data analysis strategies in mixed method contexts are needed.
Role of MMR in Expanding the Validity Framework
We introduced this article with a discussion of the need for MMR being partly prompted by LTA researchers’ desire to expand validity frameworks. It is fitting that we now revisit this issue following our analysis of 32 MMR examples. As illustrated in Table 1, the notion of validity has historically evolved throughout the paradigm shifts in the fields of education and assessment. We carefully examined the MMR studies by focusing on if and how the authors of the investigations addressed or conceptualized validity.
The results of this analysis indicate that of the 32 studies included in this review, 13 did not focus on validity issues in their studies. The remaining investigations all address various facets of validity including inference quality, consequential validity, and construct validity. Due to the (relatively) high number of studies within this review that focused on raters and ratings, it is not surprising that that there were numerous instances of scoring validity (Weir, Reference Weir2005) observed. For example, Harsch and Martin's (Reference Harsch and Martin2012) mixed methods investigation discussed the development and validation process of a local rating scale based on the Common European Framework of Reference for Languages proficiency scales. As they discuss, part of the process of scale validation is piloting it for a specific purpose in its context of use. Accordingly, they involved 13 raters to use the scale to assess writing scripts (following a rigorous series of training workshops and practice sessions). Harsch and Martin developed an index for rater agreement, which they used as an indicator of scoring validity.
Table 3 presents the aspects of validity addressed by each MMR study in this review, and draws attention to the validity frameworks spanning the various studies across contexts, participants, and research foci. One example of the use of mixed methods to address this expanding framework is illustrated in Anthony's (Reference Anthony2009) validity inquiry into the use of reflective, time-essay responses to “elicit memories of meaningful experiences” (p. 195) in undergraduate students’ education. He further wished to investigate whether these experiences would reflect students’ computer-mediated classroom experiences. Anthony clearly articulates the need for MMR in this study:
Discussions of Validity in MMR Examples
The first lens I used in looking at the responses to the test was a quantitative one designed to uncover and to attempt to discount plausible rival hypotheses for my interpretations of those responses. But when the epistemological limitations of a quantitative-only perspective became evident, I applied a different, more qualitative lens. (p. 195)
As illustrated in Table 3, Anthony's investigation addressed three primary facets of validity: construct, consequential, and inference quality. This latter category, inference quality, was another facet of validity that we observed across multiple studies.
Inference quality refers to the legitimacy of the results. Such legitimacy or trust is a function of the design quality that necessitates best practices and interpretative rigor (Teddlie & Tashakkori, Reference Teddlie, Tashakkori, Tashakkori and Teddlie2003). For example, Lee and Greene (Reference Lee and Greene2007) investigated the predictive validity of graduate students’ English language test scores through an MMR inquiry. In their attempt to achieve such interpretive rigor through their research design, Lee and Greene explain:
Different methods are needed precisely to assess various facets, dimensions, and even definitions of academic performance (rather than the same facets or the same definitions, as is required in a triangulation design for convergence). This use of mixed methods to generate a more complete and elaborated understanding of the various dimensions or facets of a complex phenomenon is called complementarity. (p. 369)
The authors argue that through the use of a complementarity design they were able to incorporate a correlational design, which is typically employed in quantitative studies. They situated this design within a mixed methods investigation by gathering perspectives of multiple stakeholders and by analyzing data not only to identify consistencies but also variations and contradictions across different data sets.
Overall, our analyses of these 32 empirical investigations revealed an increasing trend toward the use of MMR in language testing and assessment. At the same time, the turn toward this alternative paradigm is not always explicitly acknowledged. The research studies reviewed clearly evidence the complexities in the purposes and uses of MMR, as well as its power to address multiple facets of LTA. At the same time, our analyses also revealed that MMR was primarily being used in higher education contexts with adult participants, and for complementary and triangulation purposes. Furthermore, the writing skill was most often the focus of the studies. It is worth noting that only a few studies that we reviewed explicitly articulate validity issues as the rationale for MMR; however, the validity issues still appear to be focused mainly on warranting claims about the test score interpretations, necessitating further investigation and development.
CONCLUSION
Our review of recent studies in LTA shows that the field has welcomed MMR as a legitimate inquiry approach. Because most of the studies we reviewed were not explicit about what was being mixed for what purpose and claims, it was difficult to judge what researchers had intended to achieve through mixing. For MMR to be a third wave of inquiry paradigms in LTA, we suggest that there should be more explicit discussions about the logic of justification, that is, a rationale that guides methodological decisions. Research purposes and questions should guide methodological decisions as Greene (Reference Greene2007) states “methodology is ever the servant of purpose, never the master” (p. 97).
We would like to see a wider range of purposes for which MMR is used beyond triangulation and complementarity. Researchers using MMR for triangulation should expect and welcome potential inconsistencies and contradictions rather than treating them as undesirable (Mathison, Reference Mathison1988). Ultimately, the goal of MMR is to contribute to broader, deeper, and more inclusive understandings of multilayered social worlds and human behavior. In doing so, differences should be welcomed as a potential pathway to a new understanding. To achieve these goals, it is necessary for researchers to not apply MMR designs rigidly (Tashakkori & Teddlie, Reference Tashakkori and Teddlie2010), but to (creatively) design methods that will wholly respond to research questions. The majority of the studies avoided mixing at the data analysis stage by keeping and reporting results separately. Although many studies integrated findings in the discussion, we encourage researchers to experiment with integrative analytic strategies for new insights. Our traditional ways of handling data and representing knowledge may need to be revisited (Greene, Reference Greene2007). We observe that digital knowledge mobilization (i.e., electronic publications) allows researchers to embed links to make implicit assumptions explicit without interfering with a main flow or that researchers add a hyperlink to a video clip of participants (talking or acting) instead of translating data into numbers and text. For MMR to be a truly new paradigm, researchers shift away from quasi-mixing of quantitative and qualitative methods to abductive mixing of reasoning moving from the particular to the general, and vice versa.
We considered the validity frameworks as a guiding principle of our review of MMR research in LTA. While methodological repertoires have expanded with the increased capacity to handle various data thanks to rapid advances in computer-assisted data analytic technologies, researchers need to embrace MMR as a way of reasoning, involving making implicit assumptions explicit and seeking for greater coherence than the sum of the parts. In doing so, it is critical for researchers to consider recovering flexibility in human reasoning beyond either hypothetico-deductive or inductive reasoning alone. MMR researchers should make explicit the logic of reasoning based on practically reasoned dialectical argumentations, including not only confirming claims but also rebutting potential counterarguments. MMR should be fully grounded in specific contexts where LTA practices impact people's lives and encourage the participation of LTA users, practitioners, and policymakers. Participatory MMR practice may bring forward unheard voices in LTA practices. We conclude by welcoming further MMR that examines the complexity of power relations among various stakeholders, which will significantly contribute to strengthening and expanding validity issues in LTA.
APPENDIX A
Summary of Criteria Used for Identifying MMR Studies
APPENDIX B
Description of Categories of Analysis
