%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
The role of task encoding when answering questions from a text
Answering questions is an activity which has been used to help the reader comprehend a text better (Cerdán, Vidal-Abarca, Martinez, Gilabert, & Gil, Reference Cerdán, Vidal-Abarca, Martínez, Gilabert and Gil2009). According to recent models, reading a text with the purpose of answering questions can be considered a task-oriented reading situation and differs from traditional reading (Kintsch, Reference Kintsch1998) as Vidal-Abarca, Mañá, and Gil (Reference Vidal-Abarca, Mañá and Gil2010) have demonstrated.
Reading to answer questions makes the student display specific strategic processes to focus on relevant pieces of information which may be crucial for a learning goal (McCrudden & Schraw, Reference McCrudden and Schraw2007). In relation to this, differences in the text inspection processes and learning outcomes have been found depending on the type of question and readers’ individual differences (Cerdán et al., Reference Cerdán, Vidal-Abarca, Martínez, Gilabert and Gil2009). When students answer questions, they have to activate question-answering strategies and to engage in a search for the information, either in memory or in the text, as recent models of document use have highlighted (Rouet, Reference Rouet2006). The processes that unfold may in part depend on the task representation that students build when reading the question. A common assumption when providing students with questions is that they will be able to fully understand what they are asked to do. Failures in the task may mostly be attributable to the process and not to the initial starting point, which is understanding what they are asked for and selecting the most appropriate strategy to fulfill the task goals.
To our knowledge the existing research has not addressed this issue with sufficient detail, with few exceptions (Vidal-Abarca et al., Reference Vidal-Abarca, Mañá and Gil2010). Nevertheless, task encoding is an essential component in task-oriented reading, which determines successive actions to solve a task (Graesser & Franklin, Reference Graesser and Franklin1990; Rouet, Reference Rouet2006). In relation to this, Rouet (Reference Rouet2006) has proposed the TRACE model (Task-based Relevance and Content Extraction), which accounts for the processes involved in a broad range of task-oriented reading situations, such as answering questions with availability of the text. According to this model, when a student is asked to answer questions from a text, several steps may unfold iteratively and conditionally. Firstly, the student has to understand the task demands (step 1) and build a task model (step 2), which is a mental representation of the actions to perform and outcomes to reach. The task model will also determine the need of an external search (step 3).
If this is the case, the student would inspect the text and select the relevant content (step 4). This selected content would be examined in light of its relevance (steps 5 & 6). When the relevant content has been selected and processed, an internal response model would be updated (step 7) and the student would check if this model fits the specifications of the task demands (step 8), reinstating the whole answering process again in case answer to step 8 is negative. Thus, according to Rouet’s model (2006) the selection and processing of the relevant pieces of information are performed in consideration to the task model. Depending on the task representation, the student will decide to trust more superficial or deeper semantic cues in order to find the answer to the question.
In sum, the TRACE model clearly emphasizes the task encoding process component and mentions its relevance in relation to the strategies that unfold afterwards. Moreover, decisions to consider a specific piece of information as relevant for the task or an answer as sufficient for a question are examined in light of the task model (Rouet, Reference Rouet2006). Individual differences in task-oriented reading, such as students’ reading comprehension skills, may determine how a task is processed and the subsequent behaviors that unfold to solve it. This is an issue we examine next.
Individual differences in task-oriented reading
Differences in how students represent task demands have consequences in which information is selected and given as sufficient. Research has demonstrated that less-skilled comprehenders superficially represent task goals and the selection of information process and performance may be affected. Vidal-Abarca et al. (Reference Vidal-Abarca, Mañá and Gil2010) investigated high school students’ individual differences in task-oriented-reading. They specially focused on three main components that capture the interaction between a reader and a text when answering questions: a) monitoring the comprehension of the question, b) self-regulating the search process, and c) monitoring the decision to search. Using the error detection paradigm (Otero, Reference Otero, Otero, León and Graesser2002) applied to the detection of inconsistencies within questions they found that less-skilled comprehenders were less able to detect them than more skilled comprehenders.
Results were interpreted as showing less-skilled comprehenders’ tendency to superficially process information, according to the partial semantic processing interpretation (Hannon & Daneman, Reference Hannon and Daneman2004). Results were also related to less-skilled comprehenders’ propensity to apply a low standard of coherence (van den Broek, Risdem, & Husebye-Hatmann, Reference van den Broek, Risdem, Husebye-Hatmann, Lorch and O’Brien1995), due to which they may have considered a partial or even mistaken representation of the question as sufficient. In this study, less-skilled comprehenders were not only inaccurate in their task representation, but they also tended to wrongly decide not to search when they did not know the answer and selected fewer relevant pieces of information. Thus, less-skilled comprehenders produced a less accurate representation of the question, and this had consequences on subsequent task-oriented behaviors in a question-answering situation.
Other research has also found less-skilled comprehenders’ poor processing of task demands, with consequences on the search process. For instance, Cerdán, Gilabert, and Vidal-Abarca (Reference Cerdán, Gilabert and Vidal-Abarca2011), made secondary school students read two texts and answer questions. Half of the questions had been manipulated to induce a misleading matching between the wording of the question and distracting pieces of information in the text. They analyzed how skilled and less-skilled comprehenders were attracted to the distracting pieces of information and how this affected reading patterns and task outcomes.
Whereas skilled comprehenders scored higher and were able to discard the distracting information, less-skilled comprehenders copied and processed this information with more detail, with negative consequences on task outcomes. These results suggested that skilled comprehenders had probably built a task model representation based on deep comprehension, whereas less-skilled comprehenders tended to consider superficial cues in their task model as valid, with this leading to a greater selection of distracting pieces of information from the text and consequently a reduced number of relevant pieces of information.
In general, a significant number of studies have investigated the differences between skilled and less-skilled comprehenders when processing textual information. For instance, less-skilled comprehenders have difficulties in integrating propositions (Rubman & Waters, Reference Rubman and Waters2000). Consequently, they tend to produce fewer inferences while reading in contrast to skilled comprehenders (Magliano & Millis, Reference Magliano and Millis2003; Oakhill, Yuill, & Donaldson, Reference Oakhill, Yuill and Donaldson1990).
This picture makes less-skilled comprehenders prone to unsuccessfully perform on questions that require them to select, analyze and combine information from a text (Cerdán, et al., Reference Cerdán, Vidal-Abarca, Martínez, Gilabert and Gil2009). Because of their tendency to superficially process textual information (Hannon & Daneman, Reference Hannon and Daneman2004), less-skilled comprehenders may more frequently misinterpret what they are asked to answer for a given question. That is, they may build a mistaken or incomplete task representation with consequences on the number of task-relevant pieces of information selected from a text to answer and with poorer performance on the task.
Even though research has demonstrated less-skilled comprehenders’ more superficial representation of task demands, little is known on how less-skilled comprehenders build a task model. For instance it may be too literal, incomplete or it may also include mistaken ideas, which would explain their poorer performance. One of the purposes of the present paper is precisely to deepen into which kind of task representation skilled and less-skilled comprehenders build while answering questions, and how it influences task-oriented behavior.
The effects of self-explanations on the comprehension of task demands
On-line methodologies that register processing while reading and answering such as Read&Answer (Cerdán & Vidal-Abarca, Reference Cerdán and Vidal-Abarca2008; Vidal-Abarca, et al., Reference Vidal-Abarca, Martinez, Salmerón, Cerdán, Gilabert, Gil and Ferris2011) and self-generated-explanations (McNamara & Magliano, Reference McNamara, Magliano, Hacker, Dunlosky and Graesser2009) could be used to capture readers’ construction of task model.
More precisely, making students self-explain while reading, as the process of explaining text to oneself, has allowed researchers capture skilled and less-skilled comprehenders strategic individual differences (Magliano & Millis, Reference Magliano and Millis2003) and it has been demonstrated even to improve comprehension of textual information (Chi, de Leew, Chiu, & LaVancher, Reference Chi, de Leeuw, Chiu and LaVancher1994), as it increases awareness of reading goals (McNamara & Magliano, Reference McNamara, Magliano, Hacker, Dunlosky and Graesser2009).
Self-explanation is the process of explaining text to oneself, either orally or in writing. It can occur spontaneously, as a result of a metacognitive awareness of a need to explain what is being read (Chi, Bassok, Lewis, Reimann, & Glasser, Reference Chi, Bassok, Lewis, Reimann and Glaser1989), it may derive from a search for coherence (Graesser, Singer, & Trabasso, Reference Graesser, Singer and Trabasso1994) or it may be associated to an explicit instruction to explain while reading (Chi et al., Reference Chi, de Leeuw, Chiu and LaVancher1994).
Whereas similar methodologies such as think-aloud (Cerdán & Vidal-Abarca, Reference Cerdán and Vidal-Abarca2008) aim to capture natural processing that would occur regardless of the think-aloud process, self-explaining is generally assumed to modify comprehension and learning processes (Chi et al., Reference Chi, de Leeuw, Chiu and LaVancher1994). Self-explanation instructions aim to activate explanatory processes which are known to contribute to the generation of inferences, which in turn are clearly linked to better comprehension and learning (McNamara & Magliano, Reference McNamara, Magliano, Hacker, Dunlosky and Graesser2009).
This may be especially true for skilled comprehenders. In fact, research demonstrates that self-explanations generated by skilled readers (Best, Ozuru, & McNamara, Reference Best, Ozuru, McNamara, Kafai, Sandoval, Enyedy, Nixon and Herrera2004; Ozuru, Best, & McNamara, Reference Ozuru, Best, McNamara, Forbus, Gentner and Regier2004) include more paraphrases and a higher number of bridging inferences and knowledge-based elaborations. This could be due to the process of self-explaining, to their higher reading skills or an interaction of the two. Thus, in general students who self-explain text are more successful at solving problems and more likely to generate inferences. However these advantages seem only to emerge for students who are skilled self-explainers, which more frequently occurs in more skilled readers (Ozuru et al., Reference Ozuru, Best, McNamara, Forbus, Gentner and Regier2004).
In contrast, less-skilled comprehenders are in general less likely to engage in inference processes while reading (Magliano & Millis, Reference Magliano and Millis2003). If externally prompted to self-explain, they should also especially benefit from this additional processing on textual information, and even in a greater extent than more skilled comprehenders, as they would be prompted to engage in additional processing activity that would otherwise not have been activated without the instruction to self-explain (Chi et al., Reference Chi, de Leeuw, Chiu and LaVancher1994).
In fact, findings across several studies have demonstrated that the effects of self-explanations are most evident for students showing the poorest comprehension, either because of low domain knowledge or low reading skills (McNamara, O’Reilly, Best, & Ozuru, Reference McNamara, O’Reilly, Best and Ozuru2006; O’Reilly, Best, & McNamara, Reference O’Reilly, Best, McNamara, Forbus, Gentner and Regier2004).
Results from the self-explanation research could be exported to the study of task demands, in task-oriented reading. More precisely, it would be interesting to test if making students create self-explanations only on the comprehension of the question would produce differential effects or not for skilled and less-skilled comprehenders. Thus, in addition to measuring students’ task model representation when answering questions, a second goal of this study is to test the effects of self-explanations on the comprehension of the question and performance. These would be aimed at improving students’ understanding of the specific demands of a task, thus facilitating task question encoding (Rouet, Reference Rouet2006), especially for less-skilled comprehenders, provided that these explanations actively contribute to the task understanding by eliciting appropriate text comprehension processes.
In sum, the present study has two main goals. First, to measure how skilled and less-skilled readers understand task demands when answering questions. Second, test the impact of an instruction to self-explain on readers’ task model construction and task-oriented behavior. To achieve these two goals, we designed a simple learning situation. High-school more and less-skilled comprehenders (as determined by TEC, Vidal-Abarca et al., 2007) were asked to read two texts and answer five open-ended questions per text. The experiment was conducted using the software Read&Answer (Vidal-Abarca et al., Reference Vidal-Abarca, Martinez, Salmerón, Cerdán, Gilabert, Gil and Ferris2011), which presents the texts and tasks on the computer screen, and students can answer the questions while having the textual information available to search for the answer. One of the main advantages of Read&Answer is that it allows the registration of task-oriented on-line behavior.
Students were asked to read the first text, answer its corresponding questions and then read the second text and answer its questions, having the opportunity to search the text to find the relevant information for the answers. Randomly, in one of the texts students were additionally asked to write down with their own words what the five questions were asking them for, in an attempt to explain to themselves what the questions were saying. They typed their explanations on the question demands in an available space just before searching the text and answering the question.
We raised four specific predictions. First, the analysis of the explanations’ products should provide evidence for an explanation effect. That is, less-skilled comprehenders’ explanations should include indicators that deep comprehension processes are taking place, as a consequence of the prompts to explain the questions. These indicators should also be present in skilled comprehenders (Best et al., Reference Best, Ozuru, McNamara, Kafai, Sandoval, Enyedy, Nixon and Herrera2004; Ozuru et al., Reference Ozuru, Best, McNamara, Forbus, Gentner and Regier2004). Nevertheless, they might be more due to their higher reading skills than to the process of explaining in itself. Second, the analysis of on-line data on task-oriented reading behavior (i.e., selection of relevant pieces of information in the search process) should contribute to explaining the effects of the experimental manipulation for skilled and less-skilled comprehenders.
As regards performance, skilled comprehenders should outperform less-skilled comprehenders in the question-answering task. However, when asked to explain questions, less-skilled comprehenders should specially benefit from this instruction to explain task demands, given that it would elicit explanatory processes that contribute to deep understanding and are not spontaneously activated in this kind of students (Chi et al., Reference Chi, de Leeuw, Chiu and LaVancher1994; McNamara et al., Reference McNamara, O’Reilly, Best and Ozuru2006). These processes should have been reflected in the analysis of protocols and on-line measures, providing convergent evidence to product data.
Method
Participants
They were 40 secondary school students from a Spanish middle-class urban area; with a mean age of 14 years old (fourth grade) and basic computer skills, as reported by their teachers. Twenty-five of them were female and the rest fifteen male. They were measured on reading comprehension skills using a standardized comprehension test (TEC, Vidal-Abarca et al., 2007). The TEC test (α = .79) presents students with two texts and 20 multiple-choice questions scored one point each. These cover basic comprehension processes that is, the formation of text ideas, anaphoric inferences, knowledge-based inferences and macro-idea formation.
In this experiment, the overall mean score was 15.5 and the standard deviation 2.67. We split the sample in two blocks according to the sample mean and median TEC value and classified students as less-skilled (TEC scores ranging 8 to 15; n = 17) and skilled comprehenders (TEC ranging 16 to 20; n = 23). As expected, there were significant differences between students classified as less-skilled (M = 13.05, SD = 2.10) and those classified as skilled (M = 17.52; SD = 1.12), t = 8.23, df = 16, p < .01)
Materials
Texts
We used two texts (i.e., Flu and Runners) and five questions per text. They were adapted from the Program for International Students Assessment´s (PISA) reading materials. The Flu text, in its original Spanish version, is 439 words long and explains the details of a Voluntary Flu Immunization Program. The Runners text, in its original Spanish version, is 406 words long and describes the features good runners should have to avoid foot injuries.
Questions
PISA-2000 reading assessment (OECD, 2009) classifies questions into three main levels. First, questions prompting the students to retrieve specific units of information; second, questions that require that the students interpret textual information. Finally, questions that make students reflect beyond the text. We selected a pool of original retrieve and interpret PISA questions from the PISA Flu and Runners texts and adapted them for the specific purposes of the present study, making them all open-ended. Some new questions were also created following the Pisa guidelines. Please note that interpret questions included in this study required the location of a few units of information and simple inferences, so they were quite similar in the processes activated to retrieve questions. In all, we had 10 questions, five per text, with a higher representation of retrieve (i.e., 6, three per text) in contrast to interpret questions (i.e., 4, two per text).
Here is an example of one of the retrieve questions in the Flu text. Which negative symptoms can the vaccine provoke? The correct location of information for this question was the following: Doctors inform that Flu vaccine does not induce Flu infection and it is an effective means to prevent it. Nevertheless, it may create some secondary undesirable effects such as tiredness, light fever and muscle pain.
Please note that we carefully consider both the location of the answer (i.e., relevant part of the text for each question) as well as the processes to answer correctly (i.e., retrieve information).
The Read&Answer software
Students were given two texts and five questions per text. In one of the texts, students were asked to explain with their own words what the questions were asking them for just before answering. Texts and questions were presented to the students using the Read&Answer software. This software presents the text and their corresponding related questions on the computer screen and allows the registration of on-line behavior. Read&Answer has been successfully used in previous studies and has provided valuable data on students’ question-answering behavior and task-oriented reading patterns (Cerdán & Vidal-Abarca, Reference Cerdán and Vidal-Abarca2008; Cerdán et al., Reference Cerdán, Vidal-Abarca, Martínez, Gilabert and Gil2009; Vidal-Abarca et al., Reference Vidal-Abarca, Martinez, Salmerón, Cerdán, Gilabert, Gil and Ferris2011).
Read&Answer presents readers with a screen showing the full text. All text except the unit currently selected by the reader is masked. Readers unmask a unit by clicking on it; when they unmask another unit, the first one is remasked. In the present experiment information was divided and presented to students paragraph by paragraph. Read&Answer allows the presentation of textual information at the word, sentence or paragraph level. In this case, the paragraph level was chosen as we are focusing on reading at the discourse level (Vidal-Abarca et al., Reference Vidal-Abarca, Martinez, Salmerón, Cerdán, Gilabert, Gil and Ferris2011). Moreover, relevant information for each different question was located in the different paragraphs that were defined as units for research purposes.
Consequently, a textual unit would be equivalent to one textual paragraph. Readers can access the question screen from the text screen. The question screen is divided into two parts, the upper part for the question and the lower part for the answer. A simple interface allows the reader to move from one question to another and from the question screen to the text screen, and vice versa.
Procedure
The experiment lasted two sessions of approximately fifty minutes each. In session one, participants were measured on the reading comprehension skills test (TEC). In session two, students first received training on the Read&Answer software and especially on the explanation procedure. They were presented with one simple text and two questions. In one of these, they were asked to type with their own words what the question was asking them for. We told them that they should explain to themselves what the question said in order to better understand it for answering. These instructions were aimed at emphasizing the explanatory component of self-explanation instructions, in contrast to a more neutral instruction to think-aloud (McNamara & Magliano, Reference McNamara, Magliano, Hacker, Dunlosky and Graesser2009).
After training, they were randomly distributed across four conditions in which they were asked to read two texts and answer five questions per text on the computer. In only one of the texts, students were asked to explain what the questions were asking them for just before answering.
Both the order of the texts and the corresponding questions to be explained were counterbalanced across participants, to neutralize possible contamination and ordering effects. That is, whereas student 1 would read text A first, and then text B; student 2 would receive the opposite order. Moreover, while student 1 would explain the five questions in text A; the opposite would occur for student 2. Reversed combinations of texts and questions to be explained or not were applied to students 3 and 4, and so forth. In all, we had four different orders by combining the presentation of the texts (two) and the questions (explained vs. non-explained).
Students were told to read the first text, answer their corresponding five questions and then read the second text, and answer the other five questions. When required to explain the questions, they typed their explanation in an answer box just before accessing a second screen where they could re-read the question and type the answer. Only in this answering screen, they could inspect the text at will to look for their answers.
Measures
Explanation protocols
We first analyzed students’ self-generated explanations on the questions. For that purpose we a priori analyzed the number of idea units that each original question contained, using a propositional analysis framework (Kintsch, Reference Kintsch1998). We then created the following categories, mostly following other research analyzing students’ self-generated explanations, and how they reflect a more superficial or deeper approach to comprehension (McNamara & Magliano, Reference McNamara, Magliano, Hacker, Dunlosky and Graesser2009). First, presence of literal ideas, any time the students would just copy what the question said without any modification on the contents. Second, number of paraphrased ideas, indicating that students were performing some transformation on the question wording that reflected an effort to understand. Third, we analyzed the number of incomplete ideas. They were those explanations which were inaccurate but partially overlapped with the ideas in the questions. Fourth, we counted the number of wrong ideas. We considered them as those ideas which implied a misinterpretation of the idea units in the questions. Finally, we computed the number of bridging inferences, as those ideas which reflected an effort to connect more than one idea unit within the question in order to achieve global comprehension of what the question says. Coding was performed by two experimenters. They also scored one third of the responses together until more than 90% agreement was reached. Discrepancies would be solved through discussion and the rest of the self-explanations were scored separately, although any doubt regarding coding at any time of the analysis would be discussed and solved in cooperation.
On-line data
Using the on-line data provided by Read&Answer, we especially focused on how the processing of task demands could have influenced the search for information process, as indicated by the number of visits to specific paragraphs when answering the questions. According to the TRACE model (Rouet, Reference Rouet2006), task encoding strongly influences which information is selected for the task. For this reason, we computed the mean number of visits to relevant and irrelevant units of information, and the mean number of visits in explained and non-explained questions. By visits we mean every time a student clicked on a specific textual unit and unmasked it when searching for information to answer the questions. Relevant units were those paragraphs where the ideas needed for answering a specific question were located. In contrast, irrelevant units were those paragraphs that contained non-related information for the questions.
Percentage of success
Each of the questions was scored up to a maximum of 10 points (100% success). Percentage of success was calculated for explained and non-explained questions separately. It would be a rough indicator of performance on the task. Two experimenters participated in the scoring. Criteria would be established to score each of the questions, up to a maximum of 1 point. A zero score would be assigned to a wrong or non-related answer. To guarantee interrater reliability, one third of the responses were first scored by the two raters until more than 90% agreement was reached. Discrepancies would be solved and the rest of the question sets were scored separately. This two-stage system has been successfully used in previous research (Cerdán & Vidal-Abarca, Reference Cerdán and Vidal-Abarca2008) to code comprehension categorical data, which provides reliability to the procedure.
Results
Analysis of the explanation protocols
According to the first prediction of this study, analyzing students’ self-explanations would help to clarify whether they benefited or not from the instruction to explain the task demands, as well as it would shed light on the characteristics of participants’ task model construction under the instruction to explain, in relation to the first goal of the study. To accomplish these purposes, we calculated one multivariate analysis of variance (i.e., MANOVA) with independent measure reading comprehension skills (less-skilled and skilled comprehenders) and with dependent variables, the explanation categories explained in the measures section. That is, number of literal, paraphrased, incomplete and wrong ideas, and number of bridging inferences. These dependent measures should reflect a processing effort on the part of the student to more deeply understand the task demands under the instruction to explain and would thus contribute to understanding how students built a task model from the question and why skilled and less-skilled comprehenders should have benefited or not from the instruction to explain.
A first MANOVA was conducted that also included the four different orders with which students were presented the materials (i.e. between-subjects variable order) as independent variable, to control for possible contamination or ordering effects. Neither the main effect for order nor the interaction with reading comprehension skills were significant, F(3, 32) = .94, p = .51, η p 2 = .13 and F(3, 32) = .97, p = .49, η p 2 = .14, respectively. The analysis was then performed without order as an independent variable.
Results were significant for the variables number of incomplete and wrong ideas, and number of bridging inferences (Table 1). On the one hand less-skilled comprehenders included a higher number of incomplete and wrong ideas in their explanations, F(1, 38) = 8.9, p = .01, η p 2 = .14 and F(1, 38) = 7.8, p < .05, η p 2 = .10, respectively. On the other hand, skilled comprehenders produced a higher number of bridging inferences, F(1, 38) = 15.79, p < .01, η p 2 = .19. It should be noted that there were no significant differences for the more text-based measures, that is the number of literal and paraphrased ideas, F(1, 38) = .29, p = .59, η p 2 = .00 and F(1, 38) = 2.2, p = .13, η p 2 = .05, respectively. Despite this, skilled comprehenders tended to include a higher number of literal and especially paraphrased ideas in their explanations (Table 1).
Mean and standard deviation of the explanation protocols measures as a function of reading comprehension skills
Thus, we seemed to replicate the result that the advantages of self-explaining only seem to appear for students who are skilled self-explainers, which is in turn more frequent in more skilled readers (Ozuru et al., Reference Ozuru, Best, McNamara, Forbus, Gentner and Regier2004). The analysis of on-line data reflecting the search for information process should provide evidence of how the instruction to explain modified skilled and less-skilled behavior while reading and its consequences on performance.
Analysis of on-line data
The analysis of students’ search for information process should shed light on how the explanation task affected the selection of textual units of information, and if there was a differential effect for skilled and less-skilled comprehenders. For this purpose, we conducted one mixed ANOVA, with between-subjects variable reading comprehension skills (less-skilled and skilled comprehenders) and repeated measures the mean number of visits to relevant and irrelevant units of information (i.e., relevance) and mean number of visits according to type of question (explained-non explained), see table 2.
Mean and standard deviation of search for information measures, as a function of reading comprehension skills
Similarly to the previous analysis, we conducted a first mixed ANOVA including order as between-subjects variable. As expected, there was effect neither for order nor for the interaction, F(3, 32) = 1.27, p = .30, η p 2 = .10 and F(3, 32) = .44, p = .72, η p 2 = .04, respectively. The analysis was then performed without the variable order.
We found significant differences for the number of visits to relevant and irrelevant units, with a higher number of visits to irrelevant than relevant units, F(1, 38) = 105.32, p < .01, η p 2 = .73. Relevance did not interact with comprehension level, F(1, 38) =.28, p = .60, η p 2 = .00.
We also found significant differences for the number of visits according to type of question. A higher number of visits was performed when not explaining in contrast to explaining, F(1, 38) = 6.15, p < .05, η p 2 = .14. Moreover, only when explaining, less-skilled comprehenders performed a fewer number of visits to textual segments, F(1, 38) = 4.25, p < .05, η p 2 = .10. This result especially indicates that explaining questions refrained less-skilled comprehenders from engaging in an active search for information process, with consequences in lower question outcomes.
Finally, the interaction between relevance and type of question was significant, F(1, 38) = 46.40, p < .01, η p 2 = .55. In general, whereas in explained questions a higher number of irrelevant textual units were visited, in non explained questions students visited more relevant units. The triple interaction relevance x type of question x comprehension level was non-significant, F(1, 38) = 3.66, p = .07, η p 2 = .08.
Effects of reading comprehension skills on success in explained and non-explained questions
We first calculated one repeated measures ANOVA, with between-subjects measure reading comprehension skills (less-skilled and skilled comprehenders) and as dependent measure percentage of success in explained and non-explained questions (Table 3). This analysis was aimed at testing the possible relationships between reading comprehension skills, success in questions and the effects of explaining the questions, as formulated in predictions 3 and 4. The analysis also including order as between-subjects variable yielded significant effects neither for order nor for the interaction with comprehension skills, F(3, 32) = .43, p = .73, η p 2 = .03 and F(3, 32) = .28, p = .83, η p 2 = .02, respectively. Thus, the analysis was then performed without the variable order.
Mean and standard deviation of percentage of success in questions as a function of type of question and reading comprehension skills
We only found a significant effect for the interaction between reading comprehension skills and success, F(1, 38) = 15.23, p < .01, η p 2 = .29. Mean comparisons indicated the following specific differences. First, considering reading comprehension skills, skilled comprehenders significantly outperformed poor comprehenders only when explaining the questions, t = 4.03, df = 16, p < .01; however, no significant differences between skilled and less-skilled comprehenders were apparent when not explaining, t = -1.13, df = 16, p > .05. Second, considering the explanation of questions, skilled comprehenders performed similarly when having to explain or not. However, less-skilled comprehenders performed worse when having to explain the questions, t = -4.02, df = 16, p < .01.
In sum, less-skilled comprehenders were significantly hindered by the instruction to explain. In addition, skilled comprehenders outperformed less-skilled comprehenders only when explaining. This could be due to their better ability to produce explanations that significantly contribute to understanding task demands (Ozuru et al., Reference Ozuru, Best, McNamara, Forbus, Gentner and Regier2004), as we have already shown in previous sections.
Relations between processing and performance measurements
Finally, to better understand how the instruction to explain the questions was related to students’ reading comprehension skills, the nature of the explanations produced, the search processes displayed to answer and performance on the questions, we obtained the correlation between the following measures, only when students were asked to explain (Table 4): reading comprehension skills; success in explained questions; explanation categories (i.e., number of literal, paraphrased, incomplete, wrong ideas and bridging inferences) and number of visits to relevant and irrelevant units of the text.
Correlations between processing and performance measurements
Note:
*p ≤ .05. **p ≤ .01.
Reading comprehension skills were related to the kind of explanations students produced: they were negatively related to the number of incomplete, r = −.53, p <.01, and wrong ideas, r = −.36, p <.05, but positively related to the number of bridging inferences, r = .39, p < .05.
Success in explained questions was positively related to reading comprehension, r = .40, p < .01. It was also positively related to the number of visits to relevant units, r = .41, p < .01, and negatively related to the inclusion of incomplete ideas in the explanation protocols, r = −.31, p < .05.
The search for information behavior was also related to reading comprehension, to the kind of explanations produced and to success in questions. The number of visits to relevant units was positively associated to reading comprehension, r = .42, p < .01. On the other hand, the number of visits to relevant units was positively related to the production of literal ideas in the explanations, r = −.37, p < .05, and to the number of inferences,r = .32, p < .05.
Consequently, the quality of the task model that students built, as reflected in the explanation protocols, seemed to determine the selection of pieces of information in the text, as predicted by the TRACE model (Rouet, Reference Rouet2006). In addition, the selection of relevant pieces of information was associated to success in the explained questions.
We submitted all these measurements to a stepwise multiple regression analysis to determine the best model for predicting performance scores in explained questions (i.e., success in explained questions). Only the number of visits to relevant units accounted for 17% of the variance in success in explained questions (R 2 = .17, β = .42, t = 2.79, p < .01), and no other measure accounted for a significant amount of variance above that mentioned. Considering that relevant visits were significantly associated to the production of literal and inferential ideas in the protocols that reflected an understanding of the questions, we could conclude that success in questions was highly linked to the selection of the relevant parts of the text, which was in turn highly associated to an appropriate understanding of the task demands. These results provide empirical support to the main theoretical assumptions in the TRACE model, as we will discuss next.
Discussion
In this study we had two goals. First, to measure high-school students’ task model representation under the instruction to self-explain questions; and second, to test the effects of self-generated explanations on task-demands understanding and performance on questions. To accomplish these goals, we designed a simple experimental situation where students were asked to read two texts and answer questions. Only in half of the questions students were required to self-explain with their own words what the question was asking them for before answering.
The analysis of the explanation protocols showed a different pattern of results as that stated in prediction1. Whereas skilled comprehenders included a higher number of bridging inferences, which indicated an effort to understand the questions, less-skilled comprehenders’ explanations contained a greater number of incomplete and wrong ideas. This indicated that they had built an incorrect and incomplete representation of task demands, with consequences on poorer performance, as predicted by the TRACE model (Rouet, Reference Rouet2006). The analysis of the explanation protocols reflected skilled comprehenders’ advantage in a self-explanation task, with this increasing their performance on questions, as shown by performance data.
On-line data confirmed that less-skilled comprehenders had difficulties when searching for information to answer the questions, this being more apparent under the instruction to explain. As indicated by the data, explaining questions limited less skilled comprehenders’ search for information in the text. Thus, the explanation task seemed to inhibit less-skilled comprehenders’ search for information, in contrast to a non-explain situation.
According to predictions 3 and 4, we expected that skilled comprehenders would outperform less-skilled comprehenders when answering the questions. However, when asked to explain the questions, less-skilled comprehenders would be candidate to benefit in a greater extent from the positive effects that the literature has found for self-explanation instructions (Chi et al., Reference Chi, de Leeuw, Chiu and LaVancher1994), given that it would encourage them to engage in explanatory processes that would otherwise not have been activated spontaneously in students of this kind. Surprisingly, results were quite the opposite.
On the one hand, we did not find a significant effect for the overall difference between skilled and less-skilled comprehenders. Despite this, skilled comprehenders outperformed less-skilled comprehenders in the situation of explaining questions. On the other hand, less-skilled comprehenders not only did not benefit from the instruction to explain, but were clearly hindered by it. Thus, they performed poorly when asked to explain the questions with their own words, in contrast to a non-explain situation.
Correlational analyses indicated that students’ comprehension level was positively associated to the quality of the explanations produced (i.e., number of inferences) and the selection of relevant parts of the text, which was in turn the strongest and single predictor of success in explained questions. Moreover, the quality of the explanations was associated to searching relevant parts of the text, which is in agreement with the main predictions raised in the TRACE model (Rouet, Reference Rouet2006), in which a strong link is established between constructing an appropriate task representation, searching the relevant pieces of information and successfully solving a task. This path is what seemed to fail in less-skilled comprehenders.
To sum up, on-line data converged with product data (i.e., performance and explanation protocols) to offer a poor image of less-skilled comprehenders when demanded to explain with their own words what a question was asking them for. What was meant to be an instructional support to aid in the task model construction and thus facilitate the successive steps in question-answering (Rouet, Reference Rouet2006), turned out to be a burden for less-skilled comprehenders. We will try to provide a coherent explanation for this unexpected effect. We have at least two explanations for these results.
One possible explanation could be related to the nature of less-skilled comprehenders’ self-generated explanations. These were erroneous, as indicated by the analysis of the explanation protocols, and could have diverted them from the true demands of the task, as correlational data in this study evidence. In fact, according to Rouet’s Trace Model (2006), the way students encode the task demands (i.e., either more superficially or in depth) has direct consequences on which information is searched and considered for an answer. This is also in accordance with the partial semantic processing hypothesis (Hannon & Daneman, Reference Hannon and Daneman2004), which indicates that less-skilled comprehenders superficially process information, and Vidal-Abarca et al. results (2010), who found that less-skilled comprehenders were inaccurate when they had to detect inconsistencies in a question, with negative consequences on the search process and task outcomes.
A second and complementary possible interpretation may come from an illusion of knowing effect (Glenberg, Wilkinson, & Epstein, Reference Glenberg, Wilkinson and Epstein1982), due to which less-skilled comprehenders, when explaining, could have gained confidence in their own capability of providing a good answer without the need to search for information, as shown by their fewer textual searches under the instruction to explain. Searching the relevant parts of the text was indeed the strongest and single predictor of success in the explained questions.
Despite we have converging evidence from off-line and on-line measures, the present design is unable to disentangle which of the above explanations or a combined effect of them is responsible for less-skilled comprehenders’ poorer performance on questions under the instruction to explain. It should be noted that less-skilled comprehenders’ difficulties in question-answering go beyond the comprehension of task demands. This is only a first step in task-oriented behavior. As Vidal-Abarca et al. noted (2010), less-skilled comprehenders not only have difficulties in understanding the questions, but also in deciding whether to search the text or not and in using the appropriate textual information to answer their questions.
This research has some limitations. First, we found no overall differences between skilled and less-skilled comprehenders for success in questions, even though good comprehenders did outperform less-skilled comprehenders in the most demanding situation, which is when asked to explain the questions. Second, the effects of explaining questions may depend on the type of question to be explained as well as on the characteristics of the statements contained in the questions. This should be more systematically tested in future studies. Third, learning has only been assessed with ten questions, five per text. Results would need to be replicated with a higher number of questions. Fourth, this research has failed to find a way to facilitate less-skilled comprehenders’ task encoding. Using this data as a starting point, future research should find ways to help poor comprehenders understand task demands.
One possible way to do this could be to facilitate simple textual aids after the reading of questions and just before answering, instead of making students explain. This would have the advantage of providing students with accurate representations of what the question says, thus preventing them from trusting their erroneous self-representations, and consequently avoiding the illusion of knowing effect (Glenberg et al., Reference Glenberg, Wilkinson and Epstein1982). This is precisely a follow-up experiment we plan to conduct next.
