VARIETIES OF EXPERIMENT

The papers in this issue represent three separate studies—a study of Spanish learners (Collentine; Diaz-Campos; Lafford; Segalowitz & Freed), a study of French learners (Freed, Segalowitz, & Dewey), and a study of Japanese learners (Dewey). These studies provide excellent examples of the crucial distinction between randomized experiments and observational studies already alluded to. This distinction is particularly essential in the study of causality, as previously noted. Recall that, in a randomized experiment, subjects are assigned at random by the researcher to each of the treatment groups. The process of randomization is supposed to average out any differences among the subjects in (possibly relevant but unmeasured as well as observed) variables that might have an effect on the outcome of interest. The different treatment groups are assumed, via the randomization, to be equal (in a broad distributional sense). In observational studies, subjects assign themselves to treatment groups, and the researcher has no control—direct or indirect—over the distribution of unmeasured variables in each group.

Notice that ethnographic studies, case studies, and the like cannot be used to infer causality under the traditional assumptions of the randomized experimental approach or even within the framework of observational studies, which is the main focus of this paper, because they generally do not result in data that are amenable to statistical analysis. However, they are useful in providing the theories that will drive the process of causal inference; detailed ethnographic and qualitative research generates questions and theories about the underlying causal mechanisms, which can then be tested via a statistical hypothesis testing approach. I will also return to the idea of this important interaction between qualitative and quantitative studies in the section “Implications for Context of Learning Research.” I use the terms qualitative and quantitative to roughly dichotomize SLA studies into case and ethnographic studies, which produce mostly qualitative data, on the one hand, and observational or quasi-experimental studies, which can—but do not necessarily—produce quantitative data, on the other (quantitative studies also include, of course, controlled experimental studies). Both types of study are prevalent in the social sciences, and, as I explain in more detail, both have important roles to play in the study of causality in SLA research.

Typically, SLA studies on learning context that are designed to adhere to the statistical approach tend to be observational in nature, and this places serious constraints on the ability to draw conclusions regarding causality. All the studies reported in this issue are observational—students decided on their own which of several contexts they wished to learn in for the semesters in question—classroom at home (AH), study abroad (SA), or intensive domestic immersion (IM). The researchers did not have the ability, nor will they typically, to randomly assign students to different learning settings. Correspondingly, the authors of the papers in this issue have been suitably cautious in making causal statements regarding context of learning, of the type “Study abroad causes improvements in students' abilities to converse in the L2.”

In an important meta-analysis of the efficacy of various instructional approaches, Norris and Ortega (2001) examined 77 reports from the published literature. Sixty-eight percent of those reports were quasi-experimental (in their description, intact classes of learners were observed in order to investigate treatment and control conditions). This suggests that, in the published SLA literature to date, many unobserved, but possibly important, variables, have remained unaccounted for. Other characteristics of the studies worth noting include that most of them did not have true control groups (only 17%) and that almost half (44%) made direct comparisons among the treatment groups. Interestingly, the results of the meta-analysis indicate that even the most novel teaching approaches are not (statistically) more effective than more traditional approaches, although 91% of the studies reported statistically significant findings. This is striking and again suggests that many important variables are not being accounted for. It is therefore apparent that, to be able to attribute causal effects to context of learning, a more specific statistical theory is needed.

RUBIN'S CAUSAL MODEL

One of the most useful frameworks for studying causal inference for observational studies, and therefore of pertinence to the studies in this issue, is Rubin's causal model (Holland, 1986; Rubin, 1974, 1977, 1978). In this discussion I use the terms causes and treatments interchangeably, as in Holland. A treatment, in Holland's terminology, is any variable that can be manipulated, so that it makes sense to think about “what if?” questions—for example, “What if this subject had received a different treatment—SA instead of IM?” Thus context of learning is a treatment in this usage because a student who studied abroad could have chosen to enter an IM program instead. Attributes, on the other hand, are considered to be intrinsic to the subjects and hence cannot be causes (but see Pratt & Schlaifer, 1984, for a very different perspective on this crucial point). Race or gender in this view are attributes, not causes (treatments), because they cannot be manipulated for the purposes of the study. We could not, therefore, make a statement such as “Being female causes an increase in gains in L2 ability.” This is not to deny the possibility of sex differences, which might indeed occur. However, attributes can, at best, be associated with outcomes; they cannot cause them. Only with treatments is there the possibility of claiming a causal relationship.

To achieve this, there are two simplifying assumptions that must first be made: there are only two treatments (the methods can be generalized to more than two treatments, as in the studies in this issue, where there can be three treatments—AH, SA, and IM), and the study is a randomized experiment. Both of these can be relaxed, and both, particularly the latter, will typically need to be for SLA studies. Denote the two causes by t (for treatment) and c (for control) and let I be a variable that denotes the cause to which an individual unit u was exposed—that is, I = t or I = c for units exposed to the treatment or to the control, respectively. In a controlled or randomized experiment, the researcher fixes the value of I for each subject. In an observational study, the researcher does not have the ability to determine I, given that this is fixed by the subject (who chooses to participate in one or another setting of the observational study). Notice, however, that in either setup, the value of I(u)—the treatment that subject u received—could have been different. This is the key notion in Rubin's causal model. By contrast, an attribute—call it A(u)—of an individual unit could not have been different. Thinking about what might have been is called a counterfactual argument in the statistical literature on causal inference. In the paper by Dewey (this issue), the two treatments are IM and SA. Each student in the study can then be given a value of the variable I, which is either IM or SA, appropriately. It makes sense to ask the counterfactual, to think about the question, “How would this IM student have performed had he or she been an SA student, all else being equal?” Among the attributes in the Dewey study are gender, age, and length of time spent learning Japanese prior to the semester in question. All of these are intrinsic to the subject in that the researcher did not have the ability to manipulate them for the purposes of the experiment. We cannot ask the counterfactual question, “How would this female student have performed had she been a male, all else being equal?”

Now, introduce the response variable—call it Y. As pointed out by Holland (1986), the value of Y, and indeed any variable measured posttreatment, is (perhaps) affected by the cause to which a unit was exposed. In other words, “causes have effects, which is the very heart of the notion of causation” (Holland, p. 946). Therefore, it is not sufficient to describe the outcome in terms of Y alone; rather, we need notation that alludes to the fact that the outcome depends on the treatment assignment. Note that this is not different, in essence, from a typical statistical analysis of a randomized experiment, where we also need to distinguish between subject groups. What does differ, however, is the type of question we are trying to answer—“Are the two groups different?” in the typical analysis—and “Does this variable cause the difference between the groups?” in the causal analysis. The change in perspective is derived from introducing the “what if?” or counterfactual thinking alluded to previously. In much of the literature on counterfactual analysis (also known as potential outcome analysis), the possible outcomes are denoted by Y_t(u) and Y_c(u)—that is, the value of the response that would be observed if unit u were exposed to cause t and the value of the response that would be observed if unit u were exposed to cause c, respectively. Consider the difference: Y_t(u) − Y_c(u). Holland calls this difference the “causal effect of t (relative to c) on u (as measured by Y)” (p. 947). It is a simple causal statement—namely, that the treatment, t, causes the effect Y_t(u) − Y_c(u), relative to the control, c.

There is an obvious problem with this statement, however. It is impossible to observe this effect because, on any given unit, we can only observe one of Y_t(u) − Y_c(u). For instance, in the studies reported in this issue, each student was exposed to one, and only one, of the possible contexts of learning. Potentially, at the start of each of the studies, any student could have chosen to be in any of the groups, hence these are treatments in Holland's sense. But the gains in L2 ability of a given student could only be observed for a single treatment—if a student participated in the AH group of the Spanish studies, for example, we can never know how he or she would have performed on the acoustic analysis reported by Díaz-Campos (this issue) or the sociolinguistic analysis of Lafford (this issue) had he or she spent the semester abroad in Spain.

Fortunately, there is a way around this difficulty, and that is to consider, instead of the individual effect, the average causal effect. Because we cannot directly observe the causal effect of the treatment for any given individual, we look at averages over groups of subjects, some of whom were exposed to treatment t and some of whom were exposed to control c. In this way, we avoid the implementational difficulty imposed by the fact that each individual can see only one experimental condition. Formally, the average causal effect is defined as: ACE = E(Y_t − Y_c), where E(Y_t − Y_c) is the expected value of the difference between responses in the treatment and control contexts. Using linearity of expectations, ACE = E(Y_t) − E(Y_c), where E(Y_t) and E(Y_c) are the expected values of responses in the treatment and control conditions, respectively. Now, we can use the units exposed to the treatment to gain information about E(Y_t) and the units exposed to the control to gain information about E(Y_c) and, hence, learn something about the value of ACE. Again, on the surface, this is quite similar to a typical statistical analysis, but the type of question we are asking has changed.

The way in which units are assigned to the causes is of extreme importance in the estimation of ACE. In controlled experiments, this is usually accomplished by the process of randomization, whereby units are assigned at random to the two treatments. To gain information about the value of ACE, we need E(Y_t) and E(Y_c). The data we observe do not quite give us those quantities, however; in fact, what we observe is information about E(Y_t|I = t)—the expected value of the response in the treatment context, Y_t, given that the context assigned was the treatment context, I = t—and E(Y_c|I = c)—the expected value of the response in the control context, Y_c, given that the context assigned was the control context, I = c—which is not the same thing. What is the distinction? E(Y_t) is the expected value of the response Y_t(u) over all units in the population; E(Y_t|I = t) is the expected value of Y_t(u) over those elements of the population that were exposed to the treatment context t. The two need not be the same and, in general, are not. One case in which they are the same is under proper randomization. When individuals are randomized appropriately, their assignments are statistically independent of all other variables, including Y_t and Y_c. If this independence holds, then E(Y_t) = E(Y_t|I = t) and similarly, E(Y_c) = E(Y_c|I = c). The observed data can then be used to estimate ACE as follows: take the difference between the averages of the responses observed on units with I = t and those with I = c. Randomized experiments have the advantage that causal statements, of the type delineated by Holland (1986), can be made.

To this point, I have set out the basics of the theory available for randomized experiments. When studies are observational in nature, as with much SLA research, matters are more complicated, but not without resolution. A major stumbling block is the fact that, without random assignment, treatment and control groups may not be directly comparable. This is a significant challenge in SLA context-of-learning studies.

What can be done? One possibility, and one that I believe will prove to be relevant and useful for SLA research, is the accumulation of converging evidence. Although one observational study may not be sufficient to imply causality, many observational studies, on different groups, pointing in the same direction, are much more persuasive. This idea is not without its own controversial aspect. Research on the relationship between smoking and cancer is a prototypical example. Although there has been a preponderance of evidence accumulated over decades that smoking is among the causes of lung cancer, R. A. Fisher, one of the most important statisticians of the twentieth century, refused to believe this research because it was observational (Zabell, 2001) although (or perhaps because) he was, himself, a smoker. The correlation between smoking and lung cancer, Fisher argued, could be due to a third, unmeasured factor, such as a genetic predisposition to both smoke and contract lung cancer. Within SLA research, a similar argument has been called the “Contradiction Principle” by Pienemann and Johnston (1987, p. 48), as cited in Mellow (2002, p. 14): “For every proposition about the influence of an external factor [on language acquisition] and evidence for that proposition there is a corresponding counter-proposition and corresponding counter-evidence.” Today, however, most scientists and statisticians would probably agree that the preponderance of evidence in the case of smoking lies in the direction of a causal mechanism. We are not yet at that point in SLA research on context of learning and might never be.

Let us now consider how, beyond the accumulation of evidence, it is possible within a single observational study to assess the causal effects of a treatment, under certain circumstances. An important technique in this process is to adjust for observed covariates (Cochran, 1965). The studies in this issue provide examples of the usefulness of this idea. Lafford and Segalowitz and Freed, for instance, consider differences in performance of SA and AH students of Spanish in their use of communication strategies and acquisition of oral fluency, respectively, after controlling for certain background variables pertaining to prior experience with the L2 and for variables that measured time-on-task (using Spanish outside of the classroom for various purposes and with different people). Among the variables that were measured prior to the start of the Spanish study were number of years spent studying Spanish prior to the semester in question, self-rated ability in Spanish (reading, writing, speaking, and listening), using Spanish to communicate prior to the semester in question, and conducting different activities in Spanish (reading books, watching movies, or listening to music). Time-on-task variables were similar—measuring the amount of time per week, on average, that students spent carrying out various activities in Spanish—and refer to the semester in question. One hypothesis considered by these authors, and others in this issue, is that students who spend more time using the L2 will show greater gains in proficiency. The SA and IM contexts offer students more possibilities (albeit in different ways and for different reasons) to use the L2 than the AH setting, and so it can, for instance, be postulated that time on task is one of the factors that contributes to the differences in performance observed at the end of the semester, in students who studied in the different contexts. That is, it is not context of learning alone, but rather specific characteristics of the contexts, that causes differences in gains. Another interesting use of covariate adjustment is in Dewey's study of learners of Japanese. Citing the theory that first language (L1) reading skill might transfer to the development of L2 reading skill, Dewey examines the SAT Verbal scores of the IM and SA students and finds that they are not significantly different. This simple check enables Dewey to eliminate one possible confounding factor in his analysis—namely, that any differences found in the development of the Japanese reading skills of his students would be attributable to differences in their L1 (English) reading skills. By considering or adjusting for relevant covariates, these authors succeed in narrowing the possible explanations for any observed differences between the groups, making a causal interpretation for the role of context of learning more plausible.

More generally, researchers can match subjects in different groups, according to their covariate values. Exact matching may not be possible, particularly if the samples are small; in this case, as is explained more fully in the next section, it is often possible to match subjects on a function of the covariates, called the propensity score.

Under the assumption of strongly ignorable treatment assignment (Rosenbaum, 1984; Rosenbaum & Rubin, 1983a), covariate adjustment can produce unbiased estimates of treatment effects in observational studies, which are needed to gain information on ACE. Recall from the previous discussion that randomization is one mechanism by which the counterfactual “what if?” thinking can be used to learn about the ACE. Strongly ignorable treatment assignment plays a similar role for observational studies as randomization does for randomized experiments. For treatment assignment to be strongly ignorable given observed (possibly multiple) covariates x, two conditions need to be met. First, the responses (outcome variables) are conditionally independent of the treatment assignment given the observed covariates. This means that the (statistical) distribution of the value of the outcome is the same for all treatments, when the values of the covariates are the same. Second, at each value of x, there is positive probability of receiving each treatment. Generally, we can say that treatment assignment is strongly ignorable if the treatments are randomly assigned to subjects with probabilities that are functions of the covariates and perhaps other unknown parameters, but these probabilities are not zero or one. As noted by Rosenbaum, most analyses of observational studies implicitly make the assumption that treatment assignments are strongly ignorable. In fact, we could be overlooking covariates that are needed to make the strong ignorability assumption applicable. It is thus crucial that researchers think very carefully about any variables that might possibly have an explanatory effect and strive to include them in the study.

Consider a simple example where the covariates are age and gender, the treatments are the learning contexts of AH and SA, and the outcome is an Oral Proficiency Interview (OPI) score (assigned on the basis of the ACTFL Oral Proficiency Interview given at the end of the semester). The two conditions for strongly ignorable treatment assignment imply the following. First, it implies that the distribution of OPI scores at the end of the semester, as a function of context of study, does not depend on age and gender. If the (distributions of the) OPI scores differ for AH and SA students according to their values on the observed covariates (age, gender) this condition would not be met. This is often the case when so-called lurking variables have not been measured; in randomized experiments, randomization is supposed to eliminate this problem. In observational studies, however, we need to be more careful; although matching on covariates plays much the same role as randomization, we cannot be certain that important unmeasured variables are not affecting the analysis in critical ways. Second, for any values of the covariates measured by the researcher, the subject can potentially wind up in either of the contexts of learning, SA or AH, being studied. If female students below a certain age or male students above a certain age are not allowed to participate in the SA group, for instance, this condition would not be met. Rather, for any values of age and gender, students must have a nonzero chance of being in either the AH or the SA context.

TESTING THE ASSUMPTIONS OF RUBIN'S CAUSAL MODEL

When treatment assignment is strongly ignorable given x, Rosenbaum (1984) showed that it is possible to obtain an unbiased estimate of ACE, the average causal effect of interest. He also proposed several tests that can be used in observational studies to determine if the assumption of strong ignorability holds. In randomized experiments, treatment assignment is always strongly ignorable; in observational studies, it is an assumption that should be checked if causal inferences are to be drawn. A simple example of such a test is to use the fact that a causal theory may imply constraints (equalities or inequalities) on the potential outcomes that an individual may exhibit under the treatments in question. For instance, a researcher may postulate that students who participate in IM programs and students who study abroad will not differ in their writing skills in the L2 at the end of the semester but that they will differ in their speaking ability. Theories such as these, which derive from an assumed causal mechanism (in this case, the context of learning) imply constraints on the responses. Another example of such a constraint is if the researcher believes that certain students will have the same response no matter what context they are put into. In other words, some pairs of treatments have the same effects on particular aspects of the response. Mellow (2002) referred to this idea as well in his discussion of difficulties in determining outcomes. His claim that the “optimal unit” for investigation of causality is not an entire course, but rather a single lesson or perhaps a small set of lessons, fits in with this line of thinking. Simple, discrete, specific outcomes can be tested because they imply particular constraints that are amenable to statistical manipulation. Using Rosenbaum's notation, we may write the statement of constraints as f_j = f_k, where f_i = f_i(Y_i, x) represents a function of the outcome and the covariates. As before, Y_i is the outcome of treatment i (which has multiple components if the researcher is interested in more than one response) and x is the vector of covariates. In many cases, f_i will simply be Y_i itself, as when we postulate that certain individuals will be unaffected by the treatment. But the notation is general enough to include functions of the outcome variable(s) as well.

The following examples should clarify both the meaning and the flexibility of the notation.

1. Unaffected individuals. A causal theory might predict that certain individuals will not be affected by the treatments—that is, they display a lack of susceptibility to any treatment. In SLA research, an example is students who are expected, according to some theory of SLA, to learn the L2 equally well, or equally poorly, no matter what context of learning they happened to be exposed to. If the causal theory is correct, then the unaffected individuals would produce the same response to each treatment—that is, Y_AH = Y_SA = Y_IM along all dimensions (outcomes) that could be measured by the researcher.
2. Essentially equivalent treatments. Another possibility is that a causal theory might predict that certain of the treatments differ in ways that are irrelevant to the response. For instance, a context-of-learning theory might specify that the SA and IM contexts are essentially the same; hence, students would be expected to show the same response whether they studied abroad or in an IM program: Y_SA = Y_IM.
3. Unaffected responses. Finally, according to a certain causal theory, particular elements of the response, Y, might not be susceptible to the effects of a treatment. In SLA research, an example is a specification that students who stay at home and students who study in an IM program will not exhibit changes in pronunciation of the L2, or Y_p,AH = Y_p,IM, where p is the outcome “changes in pronunciation.”

In all these settings, f_i is Y_i, the outcome or vector of outcomes. By framing it in the more general notation, it is possible to develop a theory for the different situations (and potentially many others) all at once. Note too that the ability to derive these constraints depends on having rich enough causal theories. A cornerstone of causal inference in observational studies is elaborate theories (Fisher, as quoted by Cochran, 1965). In other words, when a researcher is interested in evaluating a causal effect from an observational study, the implications of the causal mechanism should be specified as completely as the current scientific knowledge of the field allows (a good guideline in any research project). This is what frees the investigator to hypothesize equivalency of treatments or constraints on the responses of certain subjects, as described previously. Without elaborate theories, researchers may find it difficult to formulate their predictions in a verifiable manner. Here again, we see the potential importance of ethnographic studies in SLA research, in addition to more formally structured statistical studies, as they can be a valuable source of elaborate theories. In the study of SLA, these theories should describe, for example, which context-specific variables lead to differential gains in L2 ability among students exposed to a variety of learning experiences.

Strongly ignorable treatment assignment together with equality constraints implies that: P(f_j|I = j, x) = P(f_k|I = k, x). If treatment assignment is not ignorable, this will not hold in general. Thus, under the postulated causal theory, any test of the equality of the conditional distributions of f_j and f_k will be a test of the strong ignorability assumption. The form of this test will depend on the data and the models being used in a given study.

As mentioned previously, matching on observed covariates may not always be possible (i.e., in the studies reported on in this thematic issue, relatively small initial sample sizes precluded matching, as it would have eliminated too many potential subjects), although it is certainly helpful, playing as it does a similar role to randomization in randomized studies—that is, to identify and control for variables that may have an effect on the outcome. In situations where it is not possible to match or stratify on covariates, it is sufficient to match on the propensity score, as shown by Rosenbaum and Rubin (1983b). In our setting of two learning contexts, one designated treatment (e.g., study abroad) and one designated control (e.g., AH study in the subject's native country), the propensity score is the probability of being treated—that is, being in the SA learning context and given the covariates x: e(x) = p(I = t|x). Matching on the propensity score means that the researcher needs to create separate subclasses for each distinct value of the known propensity score, instead of stratifying on the distinct values of all of the covariates. Continuing the example, the matching now involves ensuring that, for every value of e(x), there is at least one SA student and one AH student. The hope is that it will be easier to work with this coarser scale—matching a function of the covariates and not the covariates themselves.

There is a potential difficulty here also, because, as noted by Rosenbaum and Rubin (1984), “e(x) is not known, and it is not feasible to form subclasses that are exactly homogeneous in e(x) and contain both a treated and a control unit” (p. 517). They therefore suggested estimating the propensity score instead, using a statistical model such as the logit: log[e(x) / (1 − e(x))] = a + b^Tg(x), where a and b are unknown parameters and g(x) is a specified (hence known) function of the covariates.²

Often in the social sciences, there is the additional concern that not all relevant and important covariates have in fact been observed (Sobel, 2000). Not observing relevant variables leads to bias in the estimates of the causal effect, even after adjusting or subclassifying. Rosenbaum and Rubin (1983a) addressed this issue, suggesting methods for evaluating the sensitivity of a causal conclusion in an observational study to unobserved (binary) covariates. If estimates of causal effects are relatively insensitive to variations in assumptions about the unobserved covariates deemed plausible by the researcher, a causal interpretation is more defensible. By contrast, if these estimates are highly variable, depending on the assumptions that are made, the postulated causal model becomes more unlikely. The method involves assuming strong ignorability given a set of observed covariates and a set of unobserved covariates, all binary for simplicity. This allows the use of well-understood models for contingency table data. By changing the value of the unobserved covariate from 0 to 1, it is possible to evaluate the changes in parameter estimates of the model or in the log odds of the binary response of interest. Insofar as the log odds of the response do not vary much with the value of the unobserved covariate, the causal mechanism tends to be supported by the data. More complicated data settings obviously require more intricate analysis.

IMPLICATIONS FOR CONTEXT-OF-LEARNING RESEARCH

It is, I hope, apparent that doing a thorough causal analysis via the propensity score approach and the Rubin causal model is not an easy undertaking. Yet having come this far, a natural question to ask is: “If the causal approach is so good and so powerful for analyzing observational studies, why was it not implemented for the studies presented in this issue?” One reason has already been alluded to—namely, the small sample sizes, which precluded any serious attempt at matching retrospectively. It is important to emphasize that, even with very large studies, comprising hundreds or even thousands of subjects, the stratification of the sample that results from matching on covariates usually means that the stratified subsamples—the level at which causal analysis is then carried out—can be small. If we start with small samples, we can only stratify on one or two covariates, and this is not sufficient to establish causal relations. As more covariates are included in the matching process, alternative explanations of the observed results are ruled out. This is an essential part of the process of isolating the causal effect. Financial and other logistic constraints limited the researchers represented in this issue from collecting samples large enough to allow even an initial causal analysis. Another related factor that should not be overlooked is the choice of covariates on which to match. What are the relevant covariates that need to be controlled to arrive at a conclusion of probable causality for context of learning? Prior exposure to the target language? Gender? Personality factors (and if so, which ones)? Age? Should we consider only variables that can be measured before exposure to the treatment (i.e., at the start of the semester)? It would have been difficult, if not impossible, to formulate these questions precisely enough a priori to work them in at the design stage of the studies reported in this issue.

Is the situation for deducing a causal role for context of learning in SLA therefore hopeless? The task in general is a difficult one. It can take years, if not decades, to formulate causal theories that are precise enough and rich enough to permit for the design of experiments that will test those hypotheses. As discussed, much of the theoretical work already completed regarding causal inference makes certain strong assumptions that may or may not be met by context-of-learning studies of SLA. It should be clear that, typically, SLA studies, no matter what the learning context, will not be randomized experiments. Researchers normally do not have the ability to randomly assign students to one learning context or another, and it is very likely that the choices made by the students are influenced by individual difference variables that may not be observed. To the extent that these variables also affect a student's ability to learn the L2 in a particular context, researchers need to worry about confounding and bias in the estimates of the causal effects. It is not hard to imagine, as one possible scenario, that students who choose to study abroad have certain character traits (e.g., outgoing behavior, openness to new experiences, and curiosity about other cultures) that also make it easier for them to take advantage of the opportunities offered to improve their language skills by a semester overseas. It is also evident from the studies reported in this issue, however, that even students within the same context of learning react differently to the situation in which they find themselves. The interconnections between the possible explanatory and outcome variables still need to be investigated, and more detailed causal theories need to be put forward by SLA researchers.

Because context-of-learning research deals with observational data, the ability of the investigator to draw causal inferences is limited but not, of course, entirely curtailed. The advantage of the work by Rubin and his colleagues is that it provides a framework, via the notion of strong ignorability of treatment assignment, for inferring causality from observational studies. SLA researchers might incorporate the results of Rosenbaum and Rubin (1983b) into the design of future studies. As I discussed in the previous sections, matching on covariates in general, and on the propensity score more specifically, plays a central role. The idea is that the distributions of covariates in the control and treatment groups should be similar or at least that the study groups will be similar in terms of the distribution of a function of the covariates. If the pool of potential controls is large enough, it should be possible to attain this goal although, as discussed previously, it will not necessarily be easy, for reasons ranging from limited resources (financial and other) to difficulties in recruiting students. Whether matching is done on the covariates directly or on the propensity score, the researcher needs to have some discretion in deciding who is assigned to the control group. For studies of adult SLA, one way in which to design this is to consider the formal language classroom in the country of origin as the control setting. This greatly increases the pool of control students on which researchers could draw for purposes of matching. Students who spend a semester abroad or enroll in an IM program or who take advantage of any other possible learning contexts that the researchers might devise are the treatment group(s). Researchers could then identify relevant covariates, define and estimate a propensity score, and pick students for the control group from among those continuing to study at home, who match the treatment students on the propensity score. By working with the propensity score, SLA researchers can gain more flexibility and partially offset sample size limitations, which probably make it impossible to do exact matching. Logistically, however, the approach will still present some difficulties. For instance, it will probably no longer be possible to simply take entire AH classrooms to compare with the treatment group students; rather, only students who match will be included. This might restrict studies to larger universities, or even consortia of universities, where there is a big enough pool of control students to draw from. SLA researchers need to decide if the conclusions that can be drawn from the potential outcomes approach make this extra effort worthwhile.

Finally, there is the concept of accumulation of evidence, which is where I see the major contribution of the studies in this issue (see also, the meta-analysis of Norris & Ortega, 2001, in which the authors encourage replication of results). Each study in this issue looks at the issue of context of learning from a different perspective. A variety of outcomes is examined, for students who studied in three of the common contexts (AH, SA, and IM), and several languages—French (Freed, Segalowitz, et al.), Spanish (Collentine; Diaz-Campos; Lafford; and Segalowitz & Freed), and Japanese (Dewey). Additionally, there is, of course, a preexisting literature on context of learning (e.g., Brecht & Davidson, 1991; Brecht, Davidson, & Ginsberg, 1990; DeKeyser, 1991; Freed, 1995; Guntermann, 1995; Huebner, 1995; Lafford, 1995; Meara, 1994; Möhle, 1984; Regan, 1995). Taken individually, no one of these studies can demonstrate that a particular context, such as SA, is the cause of the observed results. Taken as a group, on the other hand, they offer some intriguing insights into this question. I take the paper by Collentine (this issue) as a representative of the Spanish study. Note that simply looking at different outcomes on the same group of students is much less informative than looking at different outcomes on different students; hence, for ease of discourse I restrict myself to only one aspect of the Spanish study. Together with the French study of Freed, Segalowitz, et al. and the Japanese study of Dewey, this survey covers the three groups of students in the issue (Spanish, French, and Japanese) and the three major contexts of L2 learning (AH, SA, and IM). The three studies are based on different samples of students, and this, together with the general consistency of the results, strengthens the argument for context as an underlying causal mechanism.

Collentine reports that neither AH nor SA students in the Spanish project progressed appreciably more than the other in their ability to conjugate verbs in the past tense. Use of the past tense is, of course, important for narrating events and is often emphasized in language classes, so at first blush this result might seem puzzling. What he did find, however, is that the SA students did much better when he looked at a group of features that are connected to good narration. That is, improvement not on a single variable, but on a cluster of variables, was apparent for the SA but not the AH students. The study by Freed, Segalowitz, et al. demonstrates gains in fluency for students of French in the SA (to some extent) and IM (to a greater extent) contexts as compared to the AH context. Here, we have a very different response from the one in Collentine. Yet, as in his study, what Freed, Segalowitz, et al. find is actually improvement of a cluster of behaviors, which taken together may correspond to something they recognize as fluency. Finally, in his work on students of Japanese, Dewey investigates changes in the reading processes (development of literacy) of the SA and IM groups. He finds significant differences between the groups on dimensions of monitoring understanding (the IM group monitors less over time than the SA group) and affective reaction to text (the IM group displays more affective reaction over time than the SA group). Dewey also correlates changes in these two measures with context features such as exposure to culture, interaction with others, and teacher contact, in an attempt to explain the differences in the data. Contact with teachers outside of the classroom was higher for the IM students than for the SA students, and the amount of contact correlates highly with changes in monitoring of understanding and in demonstrating affective reactions to texts. Although correlation most certainly does not equal causation, Dewey's findings on differences in the SA and IM contexts, what effect these have on the learning experience, and how this is all manifested in changes in the reading process of Japanese learners are intriguing and indicate what form a causal theory of context might take.

When we look at the studies together (although the outcomes and the languages are different), the findings are similar enough that we can cautiously say that they begin to point to the unique experiences of the SA or IM context, as compared to the AH context, as a cause and further indicate that differences in the SA and IM experiences may also be significant in a causal sense. If I hedge my words in making these statements, it is only to emphasize yet again the subtle nature of the arguments surrounding causal reasoning in the (social) sciences. One may also speculate as to the reasons for this apparent causal relationship—differences in the expectations that teachers have of the students in different learning contexts, differences in the amount of time spent on task (i.e., actively using the L2), and so forth. Such questions are the first stage of creating more elaborate theories on the role of context of learning as a causal mechanism in SLA. Norris and Ortega (2001) related this notion to the idea of having a database of information on social, cultural, and individual conditions that together might be more influential than the instructional strategy that a researcher or teacher might use with a student. Their meta-analysis provides the beginning of just such a repository of knowledge. See also Collentine and Freed's Introduction to this issue for further discussion of this important piece of the causal puzzle. More research is needed to confirm (or refute) the role of context of learning as a cause of gains in L2 ability and to pull out the differential causal effects of the SA versus the IM context. The studies in this issue represent an important step in understanding the role that learning context plays in SLA and present interesting directions for future investigation.