Improving academic achievement during mandatory secondary schooling is a top priority in countries like Spain where issues like academic failure and drop-out are especially prominent in the educational system. Recent research findings attest to the importance and efficacy of programs promoting self-regulated learning strategies at improving students’ academic achievement (Dignath, Buettner, & Langfeldt, Reference Dignath, Büttner and Langfeldt2008; Rosário, González-Pienda et al., Reference Rosário, González-Pienda, Pinto, Ferreira, Lourenço and Paiva2010). However, estimations of such programs’ efficacy have not accounted for potential within-group variability prior to intervention, except for very few studies that have randomly assigned students to experimental conditions (e. g., Rosário, Núñez et al., Reference Rosário, González-Pienda, Pinto, Ferreira, Lourenço and Paiva2010). In fact, we are aware of no study to date that has analyzed such a program’s differential efficacy as a function of pretest levels of the dependent variables. Toward that end, the present study aims to determine the efficacy of a program promoting self-regulated learning competency that targets Compulsory Secondary Education (ESO from the acronym in Spanish) students, as a function of students’ pretest levels of the study’s dependent variables (knowledge of self-regulated learning strategies, perceived use of said strategies, and study time).
Self-regulated Learning and Academic Achievement
Several factors precipitate students’ low or high achievement (Miñano, Castejón, & Gilar, Reference Miñano, Castejón and Gilar2012; Rosário et al., Reference Rosário, Costa, Núñez, González-Pienda, Solano and Valle2009), but self-regulated learning ability has gained special prominence within education research (Zimmerman, Reference Zimmerman2008; Zimmerman & Schunk, Reference Zimmerman, Schunk, Schunk and Zimmerman2008), especially during adolescence. This is because adolescence is a period of constant change, and during this transition to the upper levels of the educational framework, students are expected to take on a larger homework load, complete more tasks, and manage various subjects. It truly requires autonomy and taking personal responsibility for the learning process.
To meet the demands of ESO, it becomes necessary to spend more hours studying, yet that alone is not enough (Ramdass & Zimmerman, Reference Ramdass and Zimmerman2011); time management is linked to more or less efficient optimization of study time. Often, students respond to the higher complexity and quantity of educational materials by adopting efficient, self-regulated learning strategies, which have consequences in terms of learning and academic outcomes (Duckwort, Akerman, MacGregor, Salter, & Vorhau, 2009; Rosário et al., Reference Rosário, Núñez, Ferrando, Paiva, Lourenço, Cerezo and Valle2013). Other times, though, students are unaware of self-regulated learning strategies or manage their schoolwork by inefficient methods, the consequences of which may include academic results that are negative or disproportionate to the amount of effort put forth (Zimmerman, Bonner, & Kovach, Reference Zimmerman, Bonner and Kovach1996).
Fortunately, self-regulated learning processes respond to training in academic contexts (Duckworth et al., Reference Duckworth, Akerman, MacGregor, Salter and Vorhau2009; Núñez, Rosário, Vallejo, & González-Pienda, Reference Núñez, Rosário, Vallejo and González-Pienda2013) and ESO seems an optimal time to intervene in the classroom (Camahalan, Reference Camahalan2006). However, school curriculum seldom calls for instruction to develop these competencies (Kistner et al., Reference Kistner, Rackoczy, Otto, Dignath, Büttner and Klieme2010) despite the growing body of evidence for their usefulness at promoting meaningful learning and improving academic results in students with deficient study skills or learning ability. Few interventions have been designed specifically to train these competencies. Interventions often develop only one self-regulated learning phase or strategy (Weinstein, Husman, & Dierking, Reference Weinstein, Husman, Dierking, Boekaerts, Pintrich and Zeidner2000), are tied to specific areas of academic content (Perels, Dignath, & Schmitz, Reference Perels, Dignath and Schmitz2009), or are geared toward students with unique characteristics, for example, students with learning difficulties or, conversely, high-achieving or “gifted” students (Perels, Gürtler, & Schmitz, Reference Perels, Gürtler and Schmitz2005).
In order to provide the educational community with useful tools that can be implemented in ordinary classrooms, the “Testas’s (Mis)adventures” program was developed for ESO first-years (Rosário et al., Reference Rosário, González-Pienda, Pinto, Ferreira, Lourenço and Paiva2010). It aims to improve motivation and strategy conditions among first-year ESO students, and uses a cross-sectional method to work primarily on the self-regulated learning process. The present research studied this program’s differential efficacy at promoting self-regulation strategies as a function of students’ pretest levels of the study’s dependent variables. Thus, it is expected that following intervention: a) students will have more knowledge of effective study strategies (declarative knowledge), b) will report greater use of self-regulated learning strategies, and c) will spend more time studying each day. Furthermore, by grouping students according to their pretest levels of knowledge and use of self-regulated learning strategies, we predict that students will differ significantly in how much their means change from pre to posttest.
Method
Participants
This study’s sample included 277 students in their first year of ESO (12 to 14 years-old). All schools offering ESO within a northern Spanish region were invited to participate, of which 67% accepted. Of those that accepted, four schools were chosen at random, one from each area of the region. All first-year ESO classes at each of the four schools took part (eleven classes). 52.3% of the sample were boys and 47.7% were girls.
Variables and Instruments
To analyze the intervention’s efficacy, measures were taken before and after intervention of declarative knowledge of self-regulated learning strategies, perceived use of said strategies in academic contexts, and weekly study time.
Knowledge of Self-regulated Learning Strategies
Knowledge of self-regulated learning strategies was assessed using the Cuestionario de Conocimiento de estrategias de autorregulación “CEA” (Knowledge of Self-regulated Learning Strategies Questionnaire) (Rosário, González-Pienda et al., Reference Rosário, González-Pienda, Pinto, Ferreira, Lourenço and Paiva2010). It consists of ten items with three response choices. Students are asked to select the option they deem most correct (only one is true) in terms of the self-regulated learning strategies (cognitive strategies, metacognitive strategies, resource management strategies, and motivational strategies) covered by the intervention (e.g. “Subrayar es una estrategia de estudio cuya función principal es: a) Señalar las partes de los contenidos que después se deberán estudiar, b) Seleccionar la información más importante después de leer y comprender el texto, c) Decorar los apuntes para hacerlos más amenos y motivadores a la hora de estudiar;” “Highlighting is a study strategy whose main function is: a)Noting which parts of the content should later be studied, b) Selecting the most important information after reading and understanding the text, c) Memorizing notes to make them more fun and motivating to study). The scale’s Cronbach’s alpha value is .89.
Use of Self-regulated Learning Strategies
Self-regulated learning was measured using the Inventario de Estratégias de Autorregulación del Aprendizaje (IEAA) (Self-regulated Learning Strategies Inventory). It is comprised of nine items representing the three stages of the self-regulated learning process: planning (e.g., “I make a plan before beginning a writing assignment. I think about what I am going to do and what I need to succeed”), execution (e.g., “While I’m in class or studying, if I get distracted or lose the thread of the discussion, I usually do something to return to the task at hand and achieve my goals”), and assessment (e.g., “I compare my grades against the goals I set for this class”). Items appear in a Likert-type response format with 5 choices ranging from 1 (never) to 5 (always). Its reliability indices were (α = .80) for the planning factor, (α = .85) for the execution factor, and (α = .87) for the assessment factor (Rosário, Lourenço, Paiva, Núñez, & González-Pienda, Reference Rosário, Lourenço, Paiva, Núñez and González-Pienda2012) (See Appendix ).
Weekly Study Time
Time students dedicate to studying during the school week and on weekends was captured by an item asking them to indicate how many hours per week (including Saturday and Sunday) they spend doing schoolwork.
Intervention: “TESTAS’s (Mis)adventures” for ESO First-years
This intervention is a tool to teach self-regulated learning strategies to 12 to 14 year-old students. It was designed according to the PLEJA (planning, execution, assessment) model of self-regulated learning (Rosário et al., Reference Rosário, Mourão, Núñez, González-Pienda, Solano and Valle2007), which is based on Zimmerman’s social cognitive model (2008).
It consists mainly of a set of narrative texts that give students an opportunity to work with the fourteen self-regulated learning strategies posited by Zimmerman and Martínez-Pons (Reference Zimmerman and Martínez-Pons1986) (self-assessment, organization and transformation, planning and goal-setting, information-seeking, etc.). In the stories, the main character, Testas, describes his day as a student, the personal and academic problems arising in his way, and how he and his classmates gradually enact cognitive, motivational, and behavioral strategies to help overcome these same problems (see examples in Table 1).
Story Excerpts Illustrating 1 of 14 Categories of Self-regulated Learning Strategies Covered by the Program
The intervention lasted 12 sessions held over the course of an academic term (see Table 2). It was designed to take the form of a narrative, giving students the opportunity to think about themselves, their experiences, and their strategies based on what happens to a student like them, who serves as a model. From a social cognitive standpoint, it is understood that students sometimes learn vicariously, observing how other people act and analyzing the positive or negative outcomes of certain behavior. Thus, we can presume that not all learning stems from direct practice (Pintrich & Schunk, Reference Pintrich and Schunk2002), and that in academic contexts, observing a model can guide instruction in self-regulated competencies, attitudes, beliefs, and behaviors, especially when the model is a student, too. On another note, the tool was designed so that tasks would develop through methods that are unconventional for this type of program; the students themselves would analyze texts, extracting their underlying self-regulated learning strategies. This inductive methodology encourages students to work independently and deeply with the texts provided. It invites them to dive into the stories, extract the information they deem relevant, and relate that information in some way to their own experiences as students. The purpose of this approach is for students to reflect on the strategies covered in the texts, and use them to “construct their own learning stories.”
Strategies and Activities in Each Intervention Session
SRL = Self-regulated Learning.
Research materials included: a) a booklet containing the “Testas” stories in five chapters, each consisting of one, two, or three different passages; b) an activities bank designed to elicit reflection on the topics covered in each chapter; and c) activities to practice the self-regulated learning strategies embedded in each text.
Procedure
The program was implemented over the course of 14 classes, of which twelve were dedicated to instruction (see Table 2) and two to assessment (pretest and posttest). Since some earlier research results showed that interventions can be just as effective, if not more so, when implemented by the researchers themselves (Dignath & Büttner, Reference Dignath and Büttner2008), four educational psychologists (tutors from here on) were specifically trained to conduct the intervention at the four schools. One tutor was randomly assigned to each school. Throughout the intervention, the four tutors held weekly meetings to review the progress made in the previous weekly session, and to oversee the criteria for implementing the next. That way, sessions were as similar as possible in all eleven classrooms.
The program was imparted to each group of students one day per week (approximately one hour), usually during with their group tutoring hour. Since it was part of the school’s curriculum, we were able to ensure the regular attendance of all the student participants.
The same overall structure was followed in all eleven classrooms when conducting each session. Students were assigned a chapter to read at home and were instructed to fill out note cards relating to it. In class, the tutor and students together briefly summarized the material covered in the program thus far. That is, they reflected as a group on the topics addressed in each chapter, class comprehension of learning strategies, and the academic issues “Testas” faced in the stories. Next, as a group, they solved the tasks assigned as homework. Then, classroom activities were carried out and during the final minutes of each session, students reflected on and wrote down what they had learned that day of the program. To conclude, the tutor briefly summarized what they had worked on during the session.
Data Analysis
Pre and posttest differences were analyzed by means of Student’s t-test for related samples. Cohen’s d was utilized to estimate effect size. After the sample was divided into three groups (low, moderate, high) according to percentile scores, the analysis of posttest differences was conducted using ANOVAs, utilizing Scheffé’s method as a post-hoc comparison test.
The procedure used to create the low, moderate, and high groups (on each of the three variables) was to first determine what scores corresponded to the 33rd and 66th percentile on each of the three variables at pretest, and then use those to define the groups’ limits (low: scores below or at the 33rd percentile; moderate: scores between the 33rd and 66th percentiles; high: scores above the 66th percentile). As expected, the differences between the three groups (low, moderate, high) turned out to be statistically significant on all three variables: knowledge of self-regulated learning strategies F(2, 274) = 702.96; p < .001, ηp 2 = .84; use of self-regulated learning strategies F( 2, 274) = 374.61; p < .001, ηp 2 = .73; and weekly study time F( 2, 274) = 275.44; p < .001, ηp 2 = .67. Post-hoc analyses revealed statistically significant differences (p < .001) between the three groups on all three variables.
Results
Differences between Pre and Posttest Scores for the Total Sample
The means, standard deviations, skewness, and kurtosis of the variables involved in the present study appear in Table 3. According to the criteria established by Finney and DiStefano (Reference Finney, DiStefano, Hancock and Mueller2006), these values of skewness and kurtosis are within the recommended limits.
Mean, Standard Deviation, Skewness, and Kurtosis of Each Dependent Variable (Knowledge and Use of Self-regulated Learning, and Weekly Study Time Outside of Class)
SRL = Self-regulated Learning; Knowledge of Self-regulated Learning Strategies (Min = 1; Max = 10); Use of Self-regulated Learning Strategies (Min =1; Max = 5); Weekly Study Time Pretest (Min = 0; Max = 40.5); Weekly Study Time Posttest (Min = 2.5; Max = 38.25).
The first step of data analysis was to find out whether following intervention, students had increased knowledge of self-regulated learning strategies (K-SRL), perceived greater use of those strategies (U-SRL), and invested more time per week in their studies (ST).
The results show that statistically significant differences occurred between pre and posttest on K-SRL (DM pre-post = −.47; t(276) = −3.50; p < .001; d = .30) and ST (DM pre-post = −1.11; t(276) = −2.07; p = .039; d = .18), though for both variables, the effect size was small. As for U-SRL, though the posttest mean was higher, the difference between pre and posttest was not statistically significant (DM pre-post = −.05; t(276) = −1.57; p = .118; d = .13).
Difference between Pretest and Posttest Means in the Three Sub-samples (High, Moderate, and Low)
Phase two of data analysis consisted of analyzing whether the intervention’s effectiveness was altered by pretest levels of each variable. Table 4 displays each group’s mean, standard deviation, and sample size for each of the three dependent variables.
Sub-sample Size, Mean, and Standard Deviation by Level of Each Variable (N = 277)
SRL = Self-regulated Learning; N = Number of Participants at Each Level of the Variable (Low, Moderate, and High).
In the group of students with low baseline levels, pretest-posttest differences were found to be statistically significant for all three dependent variables: K-SRL (DM pre-post = −1.68; t(92) = −7.89; p < .001; d = 1.16; U-SRL (DM pre-post = −.34; t(96) = −5.25; p < .001; d = .76); and ST (DM pre-post = −6.25; t(91) = −7.21; p < .001; d = 1.07). The effect size was very large in the case of K-SRL, and large in the case of U-SRL and ST. As for the group with moderate pretest levels, pretest-posttest differences were not found to be statistically significant for any of the three variables: K-SRL (DM pre-post = −.28; t(102) = −1.31; p = .192; d = .18); U-SRL (DM pre-post = .00; t(92) = −.04; p = .972; d = .01); and ST (DM pre-post = −1.01; t(91) = −1.78; p = .079; d = .27). The results of the group with high baseline levels indicated that pretest-posttest differences in means were statistically significant in all cases: K-SRL (DM pre-post = .69; t(80) = 3.61; p < .001; d = .57); U-SRL (DM pre-post = .20; t(86) = 4.31; p < .001; d = .66); ST (DM pre-post = 3.88; t(92) = 3.99; p < .001; d = .59). All three variables exhibited a medium effect size. Please note, however, that in this group, the pretest-posttest differences were negative; in other words, these variables actually dropped in level after intervention.
Finally, an analysis of between-groups differences after intervention (posttest) revealed statistically significant differences between groups on all three variables: K-SRL, F(2, 274) = 33.81; p < .001, ηp 2 = .20; U-SRL, F(2, 274) = 59.49; p < .001, ηp 2= .30; and ST, F(2, 274) = 13.91; p < .001, ηp 2= .09. Post-hoc analyses showed significant differences at all levels (high, moderate, and low) of K-SRL and U-SRL (p < .01), but not the moderate and low levels of ST (p = n.s.). However, the magnitude of the differences observed between the three groups was noticeably smaller post-intervention than pre-intervention (K-SRL: .84 vs. .20; U-SRL: .73 vs. .30; ST: .67 vs. .09). Hence, the data suggest that intervention served to narrow the gap between students on the three variables examined.
Discussion
This study’s purpose was to compare the differential efficacy of an intervention designed for first-year ESO students at increasing knowledge and use of self-regulation strategies in the process of studying and learning. We worked for three months (one session per week) with eleven classes at four high schools in a region in northern Spain. The overarching hypothesis was that after intervention, students would have more knowledge of self-regulated learning strategies, report greater use of said strategies, and spend more time each week studying.
In analyzing the full set of student data (not taking baseline levels into account), we observed that after intervention, students reported greater knowledge of self-regulation strategies, greater use of said strategies (though in this case, differences did not reach the level of statistical significance), and more time spent studying. However, when students’ pretest levels of the three dependent variables were taken into consideration, the results showed that students with lower baseline levels benefited tremendously, but those with moderate and high levels of the three variables did not improve noticeably. Ergo, the slight improvement reflected in the full sample of students really only captured considerable improvement in students with marked deficits in self-regulation strategies at pretest. These data would suggest the program is especially beneficial for students at-risk of academic failure due to limited knowledge and use of study and learning strategies.
Viewing the group as a whole, this program has specifically proven effective at boosting declarative knowledge of self-regulation strategies, as earlier research on similar tools also reported (Rosário et al., Reference Rosário, Mourão, Núñez, González-Pienda, Solano and Valle2007; Rosário, González-Pienda et al., Reference Rosário, González-Pienda, Pinto, Ferreira, Lourenço and Paiva2010). Designing this program to include an inductive learning structure, conveyed through narrative, seems to have been a useful method to introduce ESO students to self-regulated learning strategies. This was especially true of students with a particularly low baseline level of these strategies at their disposal.
Since we were working with previously formed class groups, it seemed apt to analyze the program’s efficacy in groups of students with different baseline levels of the variables (Kistner et al., Reference Kistner, Rackoczy, Otto, Dignath, Büttner and Klieme2010). Dividing the sample into low, moderate, and high-level groups revealed that the program was highly effective for students starting at lower levels of declarative knowledge and use of self-regulated learning strategies, and who spent less time on schoolwork. That group’s results are quite promising, considering it is probably exactly those students for whom academic demands are the hardest to meet. These data reinforce the notion that self-regulated learning competencies are susceptible to improvement through suitable training, and that training is especially effective for students who are novices when it comes to this type of strategy.
Meanwhile, students with moderate baseline levels of the variables did not improve significantly by participating in the program. Perhaps that is because from the outset, they exhibited optimal levels of knowledge and use of self-regulated learning strategies, and time dedicated to schoolwork. While they did tend to improve, perhaps for them, not enough sessions were imparted and decisive change might have occurred if a more prolonged intervention were conducted. It would be sensible to analyze a longer intervention’s impact using repeated measures, and to determine how much time is needed to bring about favorable, significant change. Furthermore, as mentioned previously, it may be that the academic context (e.g. the homework or system of evaluation) does not require such far-reaching use of self-regulated learning strategies (Núñez et al., Reference Núñez, Cerezo, González-Pienda, Rosário, Valle, Fernández and Suárez2011). In other words, their current behavior might be optimal enough to get by in their current academic grade (Kistner et al., Reference Kistner, Rackoczy, Otto, Dignath, Büttner and Klieme2010; Rosário et al., Reference Rosário, Lourenço, Paiva, Núñez and González-Pienda2012).
The group of students with high pretest levels did not improve significantly by participating in the program either. This students, since they already possess a great deal of declarative knowledge of self-regulated learning strategies and use them with some frequency, could fall into boredom and amotivation during an intervention of this kind. For them, the program might be improved by focusing primarily on transfer of learning and skill-perfecting tasks, as other authors have done in past studies of academically high-achieving students (Cleary, Platten, & Nelson, Reference Cleary, Platten and Nelson2008; Perels et al., Reference Perels, Gürtler and Schmitz2005). In fact, this group’s average scores were actually lower at posttest. These results may seem disconcerting; one would expect scores to either improve or remain the same at posttest, even if just by reactivating self-observations. However, it is important to consider that these students started off with very high self-report levels of the three variables. Maybe in their case, the intervention caused them to answer the questionnaires with more stringent response criteria, adapting their responses about their true learning processes to be more rigorous and precise (Núñez, Solano, González-Pienda, & Rosário, Reference Núñez, Solano, González-Pienda and Rosário2006).
The results of this research should be interpreted without losing sight of its limitations. First of all, the intervention was assessed through a quasi-experimental design with pre and posttest measures, but no control group was used. It is widely known that when this type of design is employed, even though a measure of change may register, multiple hypotheses can pose valid alternatives to the central one, in this case that the intervention was responsible for the change participants experienced. It is entirely possible that the program did indeed elicit the change, but even so, there are various potential threats to internal validity (e.g. the stories themselves, other participant characteristics interacting with the intervention, statistical regression), so we highly recommend conducting more rigorously designed studies in the future (e.g., experimental designs). Second, the three variables used to determine the intervention’s efficacy were evaluated through self-report, posing another important limitation (Zimmerman, Reference Zimmerman2008). Along those lines, it would be interesting to compare results obtained through self-report measures, as in the present research, with others that measure self-regulation as an event in and of itself and attempt to capture the natural process of self-regulated learning (Boekaerts & Corno, Reference Boekaerts and Corno2005). For example, it would be worthwhile in future research to use diaries to access study time and use of self-regulated learning strategies. That might provide information that is more reliable and closer to specific learning situations.
This article was completed with the help of funding from the Universidad de Oviedo and Banco de Santander (Ref: UNOV-09-BECDOC), and the Spanish Ministerio de Ciencia e Innovación (EDU2010-19798 and EDU2010-16231).
Appendix: IEAA
Inventario de estrategias de autorregulación del aprendizaje
(Self-regulated Learning Strategies Inventory)
