1. Background
1.1 Foreign Language Reading Anxiety Scale
Psychological affect is one of the individual differences that determine success in second language (L2) learning. Among the various affective variables, research on foreign language anxiety has grown by demonstrating that ‘it can interfere with the acquisition, retention, and production of the new language’ (MacIntyre & Gardner, Reference MacIntyre and Gardner1991, p. 86; see also Horwitz, Reference Horwitz2010; De Costa, Reference De Costa2015; Dörnyei & Ryan, Reference Dörnyei and Ryan2015). An initial interest of foreign language anxiety research was to operationalize it as a measurable variable, in which Horwitz, Horwitz, and Cope (Reference Horwitz, Horwitz and Cope1986) developed a psychometric inventory titled the Foreign Language Classroom Anxiety Scale (FLCAS). It is composed of 33 items with a 5-point Likert scale, focusing on anxiety of L2 listening and speaking in the classroom. Furthermore, recent studies have examined the constructs of language-skill-specific anxieties to examine their associations with L2 learning, including reading, writing, listening, and speaking (e.g. Cheng, Horwitz, & Schallert, Reference Cheng, Horwitz and Schallert1999; Pae, Reference Pae2013; Cheng, Reference Cheng2017).
Foreign language reading anxiety is the worry and negative affectivity that is aroused during reading comprehension in a foreign language (Saito, Horwitz, & Garza, Reference Saito, Horwitz and Garza1999). Since print exposure is an essential source of language input in L2 learning (Grabe, Reference Grabe2009), a better understanding of learners' anxiety in reading should support L2 learning as well as foreign language reading pedagogy. For example, previous studies revealed the negative effects of foreign language reading anxiety on reading comprehension (Sellers, Reference Sellers2000; Rai, Loschky, Harris, Peck, & Cook, Reference Rai, Loschky, Harris, Peck and Cook2011; Zhao, Guo, & Dynia, Reference Zhao, Guo and Dynia2013), reading attitude (Yamashita, Reference Yamashita2007; Bahmani & Farvardin, Reference Bahmani and Farvardin2017), reading strategy use (Lien, Reference Lien2016), and incidental L2 vocabulary learning (Zhao, Guo, Biales, & Olszewski, Reference Zhao, Guo, Biales and Olszewski2016). To quantify this construct as a researchable variable, these studies used the FLRAS developed by Saito et al. (Reference Saito, Horwitz and Garza1999). Presently, their study has been cited more than 800 times in Google Scholar; thus, L2 researchers agree that the FLRAS is a standard inventory of foreign language reading anxiety.
When using a scale to examine learners' affects, it is important to assure its psychometric validity, namely, the psychometric soundness of an inventory concerning to what extent it measures what it is designed to measure (Dörnyei, Reference Dörnyei2003, Reference Dörnyei2007). The FLRAS has been used in a unidimensional way; thus, a single construct of reading anxiety is measured by its total score. However, Matsuda and Gobel (Reference Matsuda and Gobel2004) showed that the constructs measured by the FLRAS were composed of three different causes of foreign language reading anxiety. Their results suggest that it is not appropriate to add up individual response scores to obtain a total score, but rather to divide them to obtain average scores for the three subcategories. Therefore, the present study aimed to resolve the discrepancy in the administration, scoring, and interpretation methods of the FLRAS among previous studies by determining whether Matsuda and Gobel's results would be reproduced. Given the vast number of studies that have referred to the FLRAS when researching foreign language reading anxiety, a replication study will contribute to supporting the validity of the previous studies as well as the original.
1.2 The primary studies
1.2.1 Saito et al. (Reference Saito, Horwitz and Garza1999)
Research on foreign language reading anxiety started by exploring the phenomenon of foreign language reading anxiety as distinct from general foreign language anxiety. Saito et al. (Reference Saito, Horwitz and Garza1999) assumed that the construct of the FLRAS was different from that of the FLCAS. They hypothesized that foreign language reading anxiety would be caused by two subcomponents: unfamiliar scripts and writing systems, and unfamiliar cultural material. To measure these constructs, the FLRAS included 20 items with a 5-point Likert scale which asked learners to manifest the following psychological affects (Saito et al., Reference Saito, Horwitz and Garza1999, pp. 205–207):
• Anxiety over various aspects of reading (e.g. ‘I get upset when I'm not sure whether I understand what I am reading in English’).
• Perceptions of reading difficulties in a target language (e.g. ‘When reading English, I often understand the words but still can't quite understand what the author is saying’).
• Perceptions of the relative difficulties of reading as compared to those of other language skills (e.g. ‘The hardest part of learning English is learning to read’).
There were two research questions (RQs) related to the psychometric validation of the FLRAS. One was to examine whether foreign language reading anxiety existed as a phenomenon distinguishable from general foreign language anxiety. The other was to test construct validity by exploring whether learners' difficulty in foreign language reading could predict their levels of foreign language reading anxiety. English-speaking students who learned French (n = 192), Japanese (n = 114), and Russian (n = 77) at university answered each statement in both the FLRAS and FLCAS.
As for the first question, a Pearson product-moment correlation analysis was used to test discriminant validity between the FLRAS and FLCAS. Since the two measures shared only 41% of the variance (i.e. r = .64, R 2 = .41), they concluded that ‘there is at least preliminary support for the existence of [foreign language] reading anxiety as a phenomenon related to, but distinct from, general [foreign language] anxiety’ (Saito et al., Reference Saito, Horwitz and Garza1999, p. 211). Regarding the second question, the participants were classified into three groups based on their perceptions of the difficulty in their target language. The result of an analysis of variance indicated that learners who perceived reading in their target language as relatively difficult had significantly higher levels of reading anxiety. They also showed that students with higher final grades in a reading course had significantly lower reading anxiety. Thus, foreign language reading anxiety had a negative effect on learners' classroom performance as measured by their final grades in a reading course. These results supported a part of the psychometric validity of the FLRAS.
Some further studies successfully reproduced this negative relationship between reading anxiety and reading comprehension (e.g. Sellers, Reference Sellers2000; Rai et al., Reference Rai, Loschky, Harris, Peck and Cook2011; Zhao et al., Reference Zhao, Guo and Dynia2013), while others could not (e.g. Mills, Pajares, & Herron, Reference Mills, Pajares and Herron2006; Wu, Reference Wu2011). The most critical problem of Saito et al. (Reference Saito, Horwitz and Garza1999) was that they did not consider the factorial dimensionality of the FLRAS into its scoring and interpretation methods (Hsiao, Reference Hsiao2002; Matsuda & Gobel, Reference Matsuda and Gobel2004; Zoghi & Alivandivafa, Reference Zoghi and Alivandivafa2014; see also Park, Reference Park2014; Horwitz, Reference Horwitz2016). They theoretically supposed that the FLRAS consisted of three subcategories by distinguishing the questionnaire statements. Thus, it should be inappropriate to add up individual response scores to obtain a total score for the entire inventory; it is necessary to confirm the factor structure of the FLRAS to validly measure learners' foreign language reading anxiety. Development and validation studies on psychometrics have emphasized the importance of considering a substructure of instruments when examining the relationships between learner characteristics and their language skills (Dörnyei, Reference Dörnyei2003, Reference Dörnyei2007; Horwitz, Reference Horwitz2016; see also Pae, Reference Pae2013; Baghaei, Hohensinn, & Kubinger, Reference Baghaei, Hohensinn and Kubinger2014; Park, Reference Park2014; Zoghi & Alivandivafa, Reference Zoghi and Alivandivafa2014; Cheng, Reference Cheng2017).
1.2.2 Hsiao (Reference Hsiao2002)
Hsiao (Reference Hsiao2002) examined the factorial validity of the Chinese version of the FLRAS because foreign language reading anxiety tends to vary with target L2 languages. This study is not a replication study of Saito et al. (Reference Saito, Horwitz and Garza1999); however, it employed a rigorous methodology to investigate the FLRAS's factor structure. In the survey, Hsiao presented the FLRAS to 1,252 Chinese learners of English as a foreign language (EFL) and conducted a factor analysis (FA) with a maximum likelihood method. As the result of an exploratory factor analysis (EFA) with a subsample, one-factor and two-factor models were constructed. Three-factor or four-factor models were not established because at least one latent factor was loaded by only one item. A confirmatory factor analysis (CFA) with the other subsample was conducted to examine which of the one-factor and two-factor models were statistically valid. After eliminating nine problematic items, the two models were compared from the perspective of goodness of fit statistics (see details in Tabachnick & Fidell, Reference Tabachnick and Fidell2014). The result showed that the one-factor model produced the most acceptable fit statistics. Table 1 shows Hsiao's one-factor model of the FLRAS.
Comparisons of the factor loadings obtained in Hsiao (Reference Hsiao2002), Matsuda and Gobel (Reference Matsuda and Gobel2004), and the present study (PCA and EFA)
Note. F1 = familiarity with vocabulary and grammar (reading difficulty), F2 = reading confidence and enjoyment, F3 = language distance. The statements with an asterisk were inverted items. The eigenvalues of more than .30 (bolded) were regarded as the corresponding statements contributed to each factor. The eigenvalues with a circle showed that they were composed of the same factor as Matsuda and Gobel (Reference Matsuda and Gobel2004).
This study's limitation is that it examined only the factorial validity of the FLRAS. To ensure that the scores obtained from the inventory predict learners' levels of foreign language reading anxiety, further studies need to collect as much evidence as possible to support its construct validity. Specifically, it is necessary to provide evidence of negative associations between foreign language reading anxiety and proficiency in a similar manner as in Saito et al. (Reference Saito, Horwitz and Garza1999). The other concern is that the FLRAS was treated as a 6-point Likert scale instead of a 5-point Likert scale. Given that a procedural difference is one of the variables affecting further replications (Porte, Reference Porte2012), replicability of the factorial structure of the FLRAS has not yet been examined.
1.2.3 Matsuda and Gobel (Reference Matsuda and Gobel2004)
Matsuda and Gobel (Reference Matsuda and Gobel2004) used the FLRAS to examine reading anxiety factors that could accurately predict classroom performance with 252 Japanese EFL learners. This study submitted the observed data into a principle component analysis (PCA) to extract the factor structure of the FLRAS in an exploratory way. The results, summarized in Table 1, showed that the FLRAS consisted of three subcomponents: (a) familiarity with vocabulary and grammar, (b) reading confidence and enjoyment, and (c) familiarity with different cultures, ideas, and writing systems, labeled as language distance. After deleting three items due to problematic factor loadings, the three factors accounted for 41% of the total variance. The most influential factor was familiarity with vocabulary and grammar (21%), followed by reading confidence and enjoyment (12%), and language distance (8%). Thus, this study suggested that the factor structure of the FLRAS was a three-factor model, not a unidimensional or two-factor model.
The other analysis was conducted to predict classroom performance by foreign language reading anxiety measured by the FLRAS. A multiple regression analysis was conducted where the dependent variables were two types of final class grades evaluated in content-based (n = 89) and four-skill classes (n = 89). The independent variables were two FLCAS subcomponents, three FLRAS subcomponents, gender, and general EFL proficiency measured by the Test of English for International Communication (TOEIC®). The results were inconsistent with Saito et al. (Reference Saito, Horwitz and Garza1999); only one component of the FLCAS (self-confidence in speaking) determined classroom performance. None of the three subcomponents of the FLRAS were associated with Japanese EFL learners' classroom performance.
There are two concerns regarding Matsuda and Gobel (Reference Matsuda and Gobel2004) from the viewpoint of psychometric validity. First, their interpretation of the PCA results was not robust because some statements cannot obviously be categorized into an intended factor. For example, Table 1 shows that although Items 4 and 9 belong to Factor 3 (i.e. language distance), these statements should manifest learners' anxiety in language processing in foreign language reading. This kind of discrepancy might be caused by the PCA because it does not require any rationale underlying which questionnaire statements should be associated with which factors (Tabachnick & Fidell, Reference Tabachnick and Fidell2014). Second, they did not examine whether the multicomponent of foreign language reading anxiety was related to L2 learners' reading proficiency. They only showed that it was not related to classroom performance. Thus, a negative effect of foreign language reading anxiety on L2 reading ability has not been clarified.
1.3 Approaches to replication and psychometric validation
A first concern in the present replication relates to the different assumptions and results on the factor structure of the FLRAS. Saito et al. (Reference Saito, Horwitz and Garza1999) hypothesized two kinds of causes of foreign language reading anxiety and three kinds of questionnaire statements to be provided for learners. However, previous studies have used a total or average score of 20 items in the FLRAS because Saito et al. stated that the theoretical range of a score is from 20 to 100. While Hsiao (Reference Hsiao2002) supported this one-factor model by using EFA with item reductions and CFA, Matsuda and Gobel (Reference Matsuda and Gobel2004) agreed to the three-factor model as a result of the PCA.
The significant differences in research methodologies between Hsiao (Reference Hsiao2002) and Matsuda and Gobel (Reference Matsuda and Gobel2004) were the sampled populations (Chinese vs Japanese EFL learners) and statistical approach (FA vs PCA). As for the sample populations, Horwitz (Reference Horwitz2016, p. 74) agreed that ‘a different factor structure might be found in a different population of learners.’ Although Rodríguez and Abreu (Reference Rodríguez and Abreu2003) indicated that foreign language anxiety was stable for a group of highly motivated learners, Kim (Reference Kim2009) and Park (Reference Park2014) showed that the components of foreign language anxiety varied in sampled populations with respect to instructional, cultural, and proficiency differences. These findings require a replication study to use the same population as far as possible to assess the replicability of the original study.
Methodological differences influence research replicability (e.g. Porte, Reference Porte2012; Marsden, Morgan-Short, Thompson, & Abugaber, Reference Marsden, Morgan-Short, Thompson and Abugaber2018). Regarding the statistical approaches, the PCA with orthogonal rotation applied by Matsuda and Gobel (Reference Matsuda and Gobel2004) is based on the assumption that there are no correlations between latent factor variables. The use of FA with oblique rotation indicates that Hsiao (Reference Hsiao2002) and others (e.g. Pae, Reference Pae2013; Baghaei et al., Reference Baghaei, Hohensinn and Kubinger2014; Zoghi & Alivandivafa, Reference Zoghi and Alivandivafa2014; Cheng, Reference Cheng2017) have considered the significant correlations between the latent factors of foreign language reading anxiety. Given that the practical purposes of EFA and CFA are theory development and testing, respectively (Tabachnick & Fidell, Reference Tabachnick and Fidell2014), the present study will retain Matsuda and Gobel's methodology as far as possible (i.e. PCA) and additionally use other statistical approaches (i.e. EFA and CFA).
A second goal of this replication is to reexamine the relationship between reading anxiety measured by the FLRAS and classroom performance and reading proficiency. Theoretically, the FLRAS scores have to be associated with reading proficiency to support its psychometric validity. While this rationale has been supported by previous studies (e.g. Sellers, Reference Sellers2000; Rai et al., Reference Rai, Loschky, Harris, Peck and Cook2011; Zhao et al., Reference Zhao, Guo and Dynia2013) as well as Saito et al. (Reference Saito, Horwitz and Garza1999), it has also been rejected (Mills et al., Reference Mills, Pajares and Herron2006; Wu, Reference Wu2011). If the three-factor model is replicated, one aspect to be raised through replication is to examine what causes of foreign language reading anxiety determine learners' classroom performance and reading proficiency.
Thus, the present replication addressed Matsuda and Gobel's (Reference Matsuda and Gobel2004, p. 24) RQ: ‘What variables and anxiety factors can be accurate predictors of overall classroom performance?’ Since they did not posit an RQ to explicitly state that they examined the number of latent variables the FLRAS included, we investigated their original RQ with respect to (a) the factorial structure of the FLRAS (RQ1) and (b) the relationship between foreign language reading anxiety and classroom performance (RQ2). Unlike the original study, we further examined the negative associations between foreign language reading anxiety and proficiency to confirm the psychometric validity of the FLRAS (RQ3). This allows us to support the potential reliability and generalizability of the conclusions provided by the original studies (Porte, Reference Porte2012).
-
RQ1. What kinds of factorial structure of the FLRAS can be reproduced with approximately the same population as in Matsuda and Gobel (Reference Matsuda and Gobel2004)?
-
RQ2. What variables and anxiety factors can be accurate predictors of overall classroom performance?
-
RQ3. What variables and anxiety factors can be accurate predictors of overall reading performance?
2. Method
2.1 Research design
Table 2 displays the research design of Matsuda and Gobel (Reference Matsuda and Gobel2004) and that of the present study. Based on Marsden et al.’s (Reference Marsden, Morgan-Short, Thompson and Abugaber2018) meta-analysis, this study specified two types of changes for replication as follows: (a) motivated changes and (b) acknowledged but not motivated changes. A motivated change involved conducting EFA and CFA to reproduce Matsuda and Gobel's results.
Differences between Matsuda and Gobel (Reference Matsuda and Gobel2004) and the present study
Note.
a Motivated changes.
b Acknowledged but not motivated changes.
One of the acknowledged but not motivated changes was the measurements of reading proficiency evaluated by TOEIC® vs TOEIC Bridge® and classroom performance evaluated by end-of-term tests. This study used TOEIC Bridge® for accurate measurements because the participants were not familiar with TOEIC® but were familiar with TOEIC Bridge®. Different course work between the original and present studies caused differences in the specifications of classroom assessments; however, both studies were based on skill-based activities in a classroom including listening, speaking, reading, and writing. While these might affect the reproduced results, the results of this study could be compared with those of the original study because the constructs quantified by those measurements did not differ theoretically from each other. It was also possible to strengthen the potential generalizability of the original study. Because the present study introduced these principled changes from the original study, we labeled it as an approximate (or partial) replication (Porte, Reference Porte2012).
Marsden et al. (Reference Marsden, Morgan-Short, Thompson and Abugaber2018) showed that principled changes to participant demographics in L2 replication studies included participants' first language (L1), targeted L2, proficiency, and age. The participants' L1, targeted L2, and age range of the present study were identical to that of participants in Matsuda and Gobel (Reference Matsuda and Gobel2004). Although they did administer TOEIC® Listening & Reading to measure their participants' EFL proficiency, they did not report any descriptive statistics. Instead, they labeled participants as high beginner to high intermediate (p. 25).Footnote 1 Nevertheless, we did not regard the participants' proficiency levels as substantially different because the level of textbooks used in the original (Interchange Book 2: A2-level, Cambridge University Press) and present (Four Corners Book 2: A2-level, Cambridge University Press) studies was the same as defined in the Common European Framework for Reference of Languages. Other variables related to the participants' characteristics, such as the sex ratio of the group and their majors, were inevitably different, but were not motivated changes for this replication.
2.2 Participants
This replication survey's participants were 165 Japanese EFL learners (female = 23, male = 142, age = 18–21), who were first-year undergraduates majoring in engineering. They had studied English for at least six years in formal education in Japanese secondary schools. Their proficiency ranged from high beginner to high intermediate according to their TOEIC Bridge® scores (M = 108.45, SD = 16.14, Min = 74, Max = 156). They were notified of how the data would be used and provided informed consent before answering the questionnaires. They were enrolled in two obligatory 90-minute English courses at university. As mentioned in Table 2, it should be noted that the participants' sex ratio and major differed between the present study and Matsuda and Gobel (Reference Matsuda and Gobel2004).
2.3 Materials
Following Matsuda and Gobel's (Reference Matsuda and Gobel2004) procedure, the questionnaire statements, adopted from Saito et al. (Reference Saito, Horwitz and Garza1999: pp. 205–207), were translated into Japanese and discussions were held to check whether the original meanings were retained. The FLRAS (see Table 1) consisted of 20 self-report items as follows: (1) strongly disagree, (2) disagree, (3) neither agree nor disagree, (4) agree, and (5) strongly agree. The sequence of the statements was rearranged using a random-number method.
2.4 Procedure
The present study followed the data collection procedures used in Matsuda and Gobel (Reference Matsuda and Gobel2004). The questionnaire survey was conducted during regular English classes. The participants were asked not to answer the questions based on the specific class in which the questionnaire was administered. Although we did not set any time limit, the administration time was approximately 10 minutes.
The English reading proficiency test (TOEIC Bridge®) was implemented at the beginning of the academic year, and measured participants' English listening and reading skills ranging from 10 to 90, respectively (Maximum possible score = 180, Cronbach's α = .86). Classroom performance was evaluated using the average scores of four end-of-quarter tests (Cronbach's α = .74). The tests consisted of integrated reading-to-write task performance (50%), independent listening skills (15%), independent reading skills (15%), and spoken interaction (20%).
3. Results and discussion
3.1 RQ1: The factor structure of the FLRAS
Table 3 reports descriptive statistics of the FLRAS responses. The final sample size was 158 after removing missing values by list-wise deletion (n = 7). The means, 95% confidence intervals, and standard deviations showed no ceiling or floor effects. The full range of possible values were observed except for Item 18. Normality among the observed variables was assessed in terms of skewness and kurtosis, and univariate and multivariate normality were violated (Zs > 1.96). This result required the use of a robust maximum likelihood method in EFA and CFA (Schmitt, Reference Schmitt2011). The Kaiser-Meyer-Olkin measure of sampling adequacyFootnote 2 was satisfied (.75). Bartlett's test of sphericity rejected a null hypothesis of no correlations between the observed variables, χ2(190) = 665.64, p < .001; no multicollinearities were found (r range = −.36 to .47). These preliminary analyses assured the appropriateness of using PCA, EFA, and CFA. All analyses were performed on R with the packages psych (Revelle, Reference Revelle2018) and lavaan (Rosseel, Reference Rosseel2012).
Descriptive statistics of each item response
Note. n = 158.
To replicate the factor structure of the FLRAS, we conducted PCA with a varimax rotation method following Matsuda and Gobel (Reference Matsuda and Gobel2004). Table 1 shows the factor-loading result of a replicated three-factor model. The model explained 14.4%, 9.4%, and 9.9% of the variances of the observed scores for Factors 1, 2, and 3, respectively. However, the individual statements corresponding to each factor were not identical to Matsuda and Gobel's results. Only 11 out of 17 statements were accurately categorized into the same factors suggested in the original study (agreement ratio = 64.7%).
An alternative statistical approach was used to reexamine the factor structure of the FLRAS. The present study applied an EFA with the robust maximum likelihood and Oblimin rotation methods to the sample (n = 158). A parallel analysisFootnote 3 indicated that the three-factor solution was best. The EFA result showed that each statement was almost appropriately classified into the intended factors (see Table 1); Factors 1, 2, and 3 stood for familiarity with vocabulary and grammar, reading confidence and enjoyment, and language distance, respectively, as suggested in Matsuda and Gobel (Reference Matsuda and Gobel2004). According to the grouped items, Factor 1 did not represent only familiarity with vocabulary and grammar, but should be labeled as learners' perception of reading difficulty because it included both lower-level (e.g. Item 7: ‘I get nervous and confused when I don't understand every word’) and higher-level (e.g. Item 5: ‘I am nervous when I am reading a passage in English when I am not familiar with the topic’) cognitive processing (Grabe, Reference Grabe2009). This is consistent with Saito et al. (Reference Saito, Horwitz and Garza1999), who defined it as anxiety over various aspects of reading. Given that some statements were not properly classified into the intended factors in Matsuda and Gobel, it is necessary to use FA to duplicate the factor structure of the FLRAS.
CFAs were conducted to reexamine the goodness of fit statistics for Hsiao's (Reference Hsiao2002) one-factor model, Matsuda and Gobel's (Reference Matsuda and Gobel2004) three-factor model, and the present three-factor model. First, comparing the goodness of fit statistics in Table 4, the one-factor model showed the second-best fit to the observed data when deleting Items 12 to 20 in the same way as in Hsiao's study (see Figure 1). The item statements comprising the one-factor model reflected learners' perceptions of reading difficulty, because eight out of 11 items are shared with Factor 1 which was elicited by the present study.Footnote 4 Thus, when applying the one-factor model in researching foreign language reading anxiety by the FLRAS, it cannot measure reading confidence and enjoyment (Factor 2) and familiarity with different cultures, ideas, and writing systems (Factor 3).
The replicated one-factor model of the FLRAS suggested by Hsiao (Reference Hsiao2002). All the standardized path coefficients except Item 11 were significant at the level of p < .05.
Goodness of fit statistics and model selection indices among three models
Matsuda and Gobel's (Reference Matsuda and Gobel2004) model showed the least best fit to the data. Their model was established using PCA on the assumption that any factors would not be correlated with each other. However, Figure 2 shows that reading difficulty (Factor 1), reading confidence and enjoyment (Factor 2), and language distance (Factor 3) were strongly associated. Although this appears to be empirically (Yamashita, Reference Yamashita2007) and theoretically (Grabe, Reference Grabe2009) consistent with past studies, the most critical problem is positive associations between Factor 2 and Factors 1 and 3. Whereas Factor 2 reflects self-confidence in reading, Factors 1 and 3 reflect anxiety in reading and different cultures; therefore, the two different factors have to be correlated negatively. This may be one of the reasons for the weak association between foreign language reading anxiety and classroom performance in their study.
The replicated three-factor model of the FLRAS suggested in Matsuda and Gobel (Reference Matsuda and Gobel2004). All the standardized path coefficients except Items 19 and 20 were significant at the level of p < .05.
As discussed above, the other perspective for Matsuda and Gobel's (Reference Matsuda and Gobel2004) problematic factor model is the differences in the item statements comprising each factor. The present study reached the same conclusion as Matsuda and Gobel in that the three-factor solution was best for the FLRAS; however, our model produced the best fit to the data. The model selection indices (Akaike's Information Criterion [AIC] and Bayesian Information Criterion [BIC]) showed that the revised three-factor model was the most statistically plausible. As Figure 3 shows, the associations between each factor are acceptable because of a positive relationship between Factors 1 and 3 and a negative relationship between Factor 2 and Factors 1 and 3. This suggests that the FLRAS can measure three different constructs of foreign language reading anxiety.
The three-factor model of the FLRAS suggested by the present study. All the standardized path coefficients were significant at the level of p < .05.
3.2 RQ2: Predictors of classroom performance
Table 5 shows means with 95% confidence intervals, standard deviations, and intercorrelations for dependent and independent variables. A multiple regression analysis examined what causes of foreign language reading anxiety affected classroom performance. Following Matsuda and Gobel (Reference Matsuda and Gobel2004), the independent variables included participants' overall proficiency estimated by TOEIC Bridge® and gender, and three factors of reading anxiety.
Means, standard deviations, and intercorrelations for the measurements
Note. Factor 1 = reading difficulty, Factor 2 = reading confidence and enjoyment, Factor 3 = language distance.
* p < .05.
Table 6 showed that Factor 3 predicted class grades, indicating that anxiety in language distance suppressed classroom performance (β = −.22, p = .003). This was inconsistent with Matsuda and Gobel (Reference Matsuda and Gobel2004), who showed that no factors were related to Japanese EFL learners' classroom performance. The measurements for classroom performance differed between the original and present studies. While the differences may produce inconsistent outcomes, the statements consisting of the three anxiety factors differed between the two studies. For example, Matsuda and Gobel's Factor 3 included Items 4 (‘I feel intimidated whenever I see a whole page of English in front of me’), 9 (‘I usually end up translating word by word when I'm reading English’), and 17 (‘I don't mind reading to myself, but I feel very uncomfortable when I have to read English aloud’), which were categorized into either Factors 1 or 2 in the present study. Since it was unlikely for these statements to reflect learners' anxiety in learning unfamiliar scripts, writing systems, and cultural material (i.e. Factor 3), revised Factor 3 (Items 10, 11, and 19) was a significant predictor of classroom performance. The present result also supports Saito et al. (Reference Saito, Horwitz and Garza1999) and Zhao et al. (Reference Zhao, Guo and Dynia2013) who showed the negative effect of language distance on classroom reading performance.
Results of multiple regression analysis with the FLRAS for classroom performance
Note. Factor 1 = reading difficulty, Factor 2 = reading confidence and enjoyment, Factor 3 = language distance. F(5, 152) = 10.92, p < .001. R 2 = .27.
The present study partly reproduced Matsuda and Gobel's (Reference Matsuda and Gobel2004) result by finding no associations between learners' classroom performance and their reading difficulty (β = .01, p = .850) and reading confidence and enjoyment (β = −.06, p = .472). Saito et al. (Reference Saito, Horwitz and Garza1999) and previous studies (e.g. Sellers, Reference Sellers2000; Rai et al., Reference Rai, Loschky, Harris, Peck and Cook2011; Zhao et al., Reference Zhao, Guo and Dynia2013) showed that students who received higher grades manifested significantly lower levels of overall reading anxiety than those with lower grades. According to Matsuda and Gobel, these inconsistent outcomes can be attributed to differences in the participants' course work. For example, Saito et al. provided students with instructions specific to reading skills such as word reading, strategy use, and text comprehension.Footnote 5 However, Matsuda and Gobel and the present study implemented listening, speaking, and writing in addition to reading. Matsuda and Gobel evaluated their students' classroom performance by writing/reading assignments and grammar-based tests (50%), class participation (40%), and attendance (10%) in the four-skills class. They also used writing assignments and grammar-based tests (20%) and oral presentation (80%) in the content-based class. Similarly, the percentage of independent reading measures in the present study was relatively small (15%). Thus, whether the FLRAS is able to predict learners' classroom performance depends on the degree of engagement in reading in a class. Assuming that Saito et al.'s class grades mainly reflected their students' reading performance achieved through the course work, these discrepant results should be resolved if the FLRAS scores were associated with learners' reading proficiency.
3.3 RQ3: Predictors of reading proficiency
Descriptive statistics of the TOEIC Bridge® reading score were as follows: M = 52.78 (95% CI [51.19, 54.37]), SD = 10.10. This score was weakly correlated with Factor 1 (r = −.27, p = .001), Factor 2 (r = .37, p < .001), and Factor 3 (r = −.22, p = .006). Table 7 shows the results of a multiple regression analysis; the dependent variable was the reading proficiency score and the independent variables were three causes of foreign language reading anxiety and gender.
Results of multiple regression analysis with the FLRAS for reading proficiency
Note. Factor 1 = reading difficulty, Factor 2 = reading confidence and enjoyment, Factor 3 = language distance. F(4, 153) = 9.16, p < .001. R 2 = .19.
Unlike the regression model for classroom performance, all three factors were significant predictors of reading proficiency. The three factors explained 19% of the variance in reading proficiency. Reading comprehension was inhibited by a higher difficulty in reading processing (β = −.17, p = .032), less confidence in reading (β = .30, p < .001), and higher anxiety in language distance (β = −.15, p = .040). These results demonstrate that the psychological affects measured by the FLRAS are related to foreign language reading proficiency, but not to general classroom performance. Matsuda and Gobel (Reference Matsuda and Gobel2004) showed that the FLCAS better predicts classroom performance than the FLRAS. They did not examine the relationships between the three causes of reading anxiety and reading proficiency; however, the present findings provide evidence for the psychometric validity of the FLRAS.
Although the percentage of variance explained by Hsiao's (Reference Hsiao2002) one-factor model (R 2 = .14) was lower than the present three-factor model (R 2 = .19), the reading difficulty was also an independent predictor of reading performance (B = −6.54, 95% CI [−9.14, −3.94], SE B = 1.32, β = −.37, t = 4.96, p < .001). Previous studies produced contradictory findings by adopting a one-factor model by summing up almost all the statements of the FLRAS to examine the associations between reading anxiety and proficiency. Since the one-factor model (i.e. Items 1 to 11) could predict participants’ reading performance, we support those negative relationships (Sellers, Reference Sellers2000; Rai et al., Reference Rai, Loschky, Harris, Peck and Cook2011; Zhao et al., Reference Zhao, Guo and Dynia2013) and reject the null results (Mills et al., Reference Mills, Pajares and Herron2006; Wu, Reference Wu2011) as long as the FLRAS is appropriately employed.
4. Conclusions
4.1 Summary and limitations
This approximate replication sought to assure the psychometric validity of the FLRAS with respect to its factorial structure and relationships between reading anxiety and proficiency. Matsuda and Gobel's (Reference Matsuda and Gobel2004) results could not be replicated well when using the same statistical approaches. Instead, our study successfully reproduced the three-factor model using EFA and CFA (RQ1) and the weak relationship between reading anxiety and classroom performance (RQ2). Although the original study did not explore the relationships between the three factors of reading anxiety and reading proficiency, the present study supported the theoretical constructs of the FLRAS by demonstrating the inhibition effects of reading anxieties on reading comprehension (RQ3). Thus, when the use of the FLRAS is appropriate, this inventory can safely be used in a classroom and in foreign language reading anxiety research.
Nevertheless, this study's results should be viewed in the context of research limitations because they affect the administration, scoring, and interpretation of the FLRAS. First, the questionnaire responses were collected from Japanese EFL university students with the aim of replicating the results of Matsuda and Gobel (Reference Matsuda and Gobel2004). However, the restricted range in terms of L1 background, targeted L2, proficiency, and age may be considered for further psychometric validation to generalize our conclusions and those of Matsuda and Gobel (Marsden et al., Reference Marsden, Morgan-Short, Thompson and Abugaber2018). Psychological affects including foreign language anxiety are sometimes context-dependent (e.g. Kim, Reference Kim2009; Zhao et al., Reference Zhao, Guo and Dynia2013; De Costa, Reference De Costa2015; Dörnyei & Ryan, Reference Dörnyei and Ryan2015) and some researchers have developed their own instruments targeting a specific population (e.g. Baghaei et al., Reference Baghaei, Hohensinn and Kubinger2014; Zoghi & Alivandivafa, Reference Zoghi and Alivandivafa2014). If the results of this study cannot be reproduced with different populations, it is necessary to conduct similar replication studies using context-specific instruments with corresponding populations.
In addition, the present study could not exactly replicate Matsuda and Gobel (Reference Matsuda and Gobel2004) with respect to the measurements of classroom performance and reading proficiency. Because it is difficult to use the exact same measurements in different intact classes, research synthesis is necessary focusing on the extent to which foreign language reading anxiety is associated with classroom performance. For example, Teimouri, Goetze, and Plonsky's (Reference Teimouri, Goetze and Plonsky2019) meta-analysis has indicated that foreign language anxiety is moderately associated with course grade (k = 33, r = −.34, 95% CI [−.39, −.29]). To this end, they have emphasized the importance of reporting the grade specifications such as measurement source, descriptive statistics, and reliability indices.
4.2 Administration
The FLRAS can be administered individually (Rai et al., Reference Rai, Loschky, Harris, Peck and Cook2011) and to groups of L2 learners (Saito et al., Reference Saito, Horwitz and Garza1999). Although there is no time limit, the estimated administration time ranges from five to ten minutes. After explaining the purpose of the questionnaire, teachers need to ask students to read each statement and choose the closest scale to their thoughts and perceptions. Teachers should check that students understand all aspects of the inventory. It is important to tell students that their responses do not refer to the specific class where the questionnaire is administered (Matsuda & Gobel, Reference Matsuda and Gobel2004) and do not contribute to their class grade (Dörnyei, Reference Dörnyei2003).
4.3 Scoring
The valid way to calculate the anxiety scores is to average the scores provided by the three subscales, such as reading difficulty (Items 1, 2, 3, 4, 5, 6, 7, and 8), reading confidence and enjoyment (Items 12, 13, 14, 16, and 17), and language distance (Items 10, 11, and 19). The reverse code scale items (Items 12, 13, 14, and 16) must be corrected before summing and averaging the scores so that a high value manifests the same type of response on the other items. While the present study employed the factor scores in the psychometric validation, the correlations between the factor scores and the averaged scores were extremely high (r range = .98–1.00). This indicates that both scores can be used alternatively. Using a total score for the entire inventory is not recommended unless adopting Hsiao's (Reference Hsiao2002) one-factor model (Items 1 to 11) for the measurements.
4.4 Interpretation
The average scores for each subscale in the FLRAS indicate the causes of foreign language reading anxiety in L2 learners. Currently, there are no cut-off points to interpret the magnitude of their anxiety, unlike Oxford's (Reference Oxford1990) Strategy Inventory for Language Learning.Footnote 6 The average scores should be used to determine whether learners have relatively higher anxiety than is caused by any other source. It is necessary to note relatively high scores on any of the subscales or specific statements because specific anxieties may interfere with learning to read in a foreign language (Saito et al., Reference Saito, Horwitz and Garza1999). However, dealing with foreign language anxiety will depend on the situational contexts of a classroom, such as individual differences in L2 learners (e.g. Kim, Reference Kim2009; Horwitz, Reference Horwitz2010; De Costa, Reference De Costa2015; Lien, Reference Lien2016), their reading proficiency (e.g. Saito et al., Reference Saito, Horwitz and Garza1999; Yamashita, Reference Yamashita2007; Rai et al., Reference Rai, Loschky, Harris, Peck and Cook2011; Zhao et al., Reference Zhao, Guo and Dynia2013), text difficulty (Bahmani & Farvardin, Reference Bahmani and Farvardin2017), and other potentially related factors (see Dörnyei & Ryan, Reference Dörnyei and Ryan2015). Great care should be taken when using the present results in score interpretation for different populations from Japanese EFL university students.
Acknowledgements
This research was supported by Japan Society for the Promotion of Science Grants-in-Aid for Young Scientists (B) No. 17K13512 and 18K12443. We thank Professor Sae Matsuda, the author of the original study, for her cooperation. We would also like to thank Dr Graeme Porte and the Language Teaching reviewers for their insightful comments on earlier versions of this article.
Akira Hamada (Ph.D., University of Tsukuba) is a lecturer in the Department of English, Faculty of Languages and Cultures at Meikai University, Japan. His research interests include second language reading, instructed second language acquisition, and educational assessment.
Shuichi Takaki (Ph.D., University of Tsukuba) is an associate professor in the Faculty of Human Development and Culture at Fukushima University, Japan. His research interests include second language reading, language assessment, and research methodology.
