1. Introduction
Website analysis is an interdisciplinary field of inquiry that focuses on both digital literacy and language competence. This interdisciplinarity is at the core of the tasks that are the focus of this study. Various existing studies have focused on L2 digital literacy issues involving the information & communication technology (ICT) competence of students and the effects of digital literacy on task performance. Examples of these include a Korean study on student perception of English video guides made on cell-phone video cams and uploaded to an L2 English social networking site (Meurant, Reference Meurant2008), a Taiwanese study on student perception of web-resource integrated language learning (Chuo, Reference Chuo2007), and a Singaporean study on digital storytelling and digital literacy learning (Churchill et al., Reference Churchill, Ping, Oakley and Churchill2008).
It is, however, surprising to find that little of the literature in CALL has focused on how website analysis and resultant design studies might help widen our thoughts on approaching language education and ICT competence through critical thinking (Roy, Reference Roy2012). One of the few existing studies (Hughes, McAvinia & King, Reference Hughes, McAvinia and King2004) examined what makes students like a website, and the implications of this for designing web-based language learning sites. The findings of this study showed that students liked websites for their visual attributes, usability, interactivity, support for schoolwork and for their cultural and heritage associations, as well as their content and functionality.
One possible reason as to why this field of inquiry is under-researched is because of scant knowledge about users’ ability to provide open-ended feedback on website content (Elling, Lentz & de Jong, Reference Elling, Lentz and de Jong2012). Moreover, usability experts suggest that users’ variable language proficiency and individual differences related to cognition and behaviour, design knowledge and background, etc., play a role in defining the productivity of a website.
This article is a follow-up study based on research undertaken by Roy (Reference Roy2012) and Roy and Brine (Reference Roy and Brine2013) into EFL readers’ ability to comprehend website information as part of a website analysis exercise. Roy (Reference Roy2012) and Roy and Brine (Reference Roy and Brine2013) undertook a case study analysis with computer science majors at a Japanese technical university which suggested that web design analysis situates language acquisition in the target language through content-based learning and higher order thinking. A design-based language learning approach has the potential to promote grammatical understanding through increased writing practice, systematic thinking, schematization, presentation and structured content authoring. As part of the analysis, participants authored open-ended responses to a variety of design and inference-based queries, based on their analysis of a website every week (for six weeks). The results provided enough indication to suggest that readers largely understood most of the questions related to interface design, navigation design, information design, audience analysis and product goals, besides other design-based queries, but lacked a definite structure and pattern for analysis and thinking, as is often seen in heuristic evaluations. The results further suggested that the use of structured design rubrics, examples, regular feedback, and practice with web design analysis might pave the way for more systematic and higher-order thinking in the long run. This area is an important domain to explore because EFL readers in their daily life might be exposed to different English language websites, and their ability to deal with them might influence actual work efficiency.
Website analysis in an EFL educational context could facilitate logical thinking and language development when the students have some basic understanding of “good” design and the instructors understand how English websites could be accessed by non-native EFL learners, and how the design of the websites could potentially promote or impede language learning in a non-native context (Roy, Reference Roy2012).
Non-native users’ understanding of website design, and also predicting how any reader is likely to access a website, becomes more straightforward when a website is designed using a comprehensive framework that considers both usability and functionality (Lu & Yeung, Reference Lu and Yeung1998). Usability and functionality studies measuring website text-graphics design, accuracy, speed, and perceived comprehension suggest that cognitively designed home pages lead to better comprehension than non-cognitively designed home pages, regardless of whether the home pages are primarily graphics-based or text-based (Dalal, Quible & Wyatt, Reference Dalal, Quible and Wyatt2000). This can be linked to the fact that reading and extracting knowledge from a home page requires connecting different elements of the display in temporal sequence using internal and external representations (Scaife & Rogers, Reference Scaife and Rogers1996).
There are existing studies that have investigated the determinants of an effective website. A literature survey by Gehrke and Turban (Reference Gehrke and Turban1999) indicated that the major categories of determinants are page loading speed, business content, navigation efficiency, security, and marketing/customer focus. A further study by Lindegaard and Dudek (Reference Lindegaard and Dudek2003) suggested a correlation between the aesthetic quality of an interface, its perceived usability, and overall user satisfaction.
A web development company called Abeleto (2002) (http://www.abeleto.nl/) developed a comprehensive evaluation checklist for identifying usability hazards based on criteria proposed by Nielson (Reference Nielsen2000) and Rosenfeld and Morville (Reference Rosenfeld and Morville2002). In this checklist, the usability criteria were divided into five dimensions, each representing an area where usability hazards are common. These dimensions are as follows:
1. Language: refers to the choice of words used to present information on the website.
2. Layout & graphics: pertains to how elements are visually rendered on the website.
3. Information architecture: pertains to how the website content and features are arranged.
4. User interface: pertains to how the user interface of the website determines the ease of navigating through its content.
5. General: refers to the general practice of website design and maintenance.
It should, however, be pointed out that even when a website design is based on usability heuristics, that in itself is not a guarantee that non-native English speakers will be able to use and analyze an English language website successfully. The website designer also needs an understanding of the process of L2 reading in a hypermedia-learning environment. However, understanding L2 reading patterns is complex, in part because the literature remains inconclusive as to when and how often learners access online support resources to facilitate comprehension of L2 online texts (Chun, Reference Chun2001). Nevertheless, there is a growing body of literature emphasizing vocabulary glossing and metacognitive reading strategies (awareness of how a written text is understood in a given context) in a web-based environment (Lomicka, Reference Lomicka1998; Yeung Jin & Sweller, Reference Yeung, Jin and Sweller1997; Yu, Chen-Hsiang & Miller, Reference Yu and Miller2010).
In the study reported in this paper, a key methodological decision was the choice of the Belize tourism website. This website was chosen in order to explore EFL readers’ ability to gather information from a travel website with which they were not familiar, either culturally or linguistically. In other words, could the readers successfully navigate through the website without having background knowledge about its subject area; specifically the country Belize, its culture, and its tourist attractions? In this regard, it is important to point out that research into non-native English as a second language (ESL) reading comprehension by Carrell (Reference Carrell1983) found that background knowledge (specifically, knowledge of the subject area of a text) did not have a significant effect on non-native readers’ ability to comprehend a text.
2. Literature review
The literature review will explore the following questions:
∙ How to design an effective information environment for affordance-based (consumer-environment interactions) website usability? (Turvey & Shaw, Reference Turvey and Shaw1995);
∙ Which models could technically assist us in measuring analytical thinking for website analysis? (Ennis & Millman, Reference Ennis and Millman1985; Watson & Glaser, Reference Watson and Glaser2002; Kitchener & King, Reference Kitchener and King1981);
∙ Is website analysis a beneficial activity? (Atherton, Reference Atherton2002; Murray & McPherson, Reference Murray and McPherson2004; Reference Murray and McPherson2006);
∙ What are some of the cognitive and metacognitive strategies used towards successfully navigating in an unknown information environment? (Sheorey & Mokhtari, Reference Sheorey and Mokhtari2001).
2.1. Designing an information environment: designers’ perspective
Purpose: The following literature explores the design of an information-gathering website in a non-native language learning context
People are being confronted with increasingly rich information environments during website interaction (Davern, Te’eni & Moon, Reference Davern, Te'eni and Moon2000). Outside of the basic interface (e.g., the point-and-click aspects of a specific web browser), an information environment can be characterized by two key dimensions: its content and structure. Content is the information that the consumer seeks to use directly for his/her functional purposes. Structure is the meta-information about the content that facilitates the accessing, processing, and sharing of information (it provides value to the consumer through easier, more effective, and more efficient access to the content) (Norman, Reference Norman2013).
These tasks can be thought of as residing in what Rasmussen (Reference Rasmussen1988) terms as a means-end abstraction hierarchy. Tasks at lower levels of the hierarchy are a means of achieving higher-level ends. In the context of website usage, three broad levels can be identified in the hierarchy:
1. Information gathering
2. Navigation of the specific website
3. Browser operation
These levels are equivalent to the abstract, generalized, and physical functions in Rasmussen’s abstraction hierarchy. Taken together, they can be used to describe a task for accomplishing some higher-level goal or functional purpose. However, to explain how a typical EFL speaker with low-intermediate English language skills could process information in terms of Rasmussen’s abstraction hierarchy, it is first necessary to understand EFL readers’ critical thinking ability.
2.2. Measuring critical thinking:
Purpose: The following literature explores how EFL readers’ critical thinking ability with website analysis was targeted
Two of the most popular measures of critical thinking are the Cornell Critical Thinking Test (CCTT), which measures ability to solve well-structured problems (Ennis & Millman, Reference Ennis and Millman1985) and the Watson-Glaser Critical Thinking Appraisal (WGCTA), which measures ability to solve both well- and ill-structured problems (Watson & Glaser, 1964). In addition, a more recent measure of critical thinking, the Reflective Judgment Interview (RJI), measures ability to solve ill-structured problems. The RJI model is a model of post-adolescent intellectual development and describes changes in the assumptions that people have about knowledge. In particular, this model focuses on people's assumptions about, first, the certainty of a knowledge claim, second, how knowledge is gained, and, third, how a belief or knowledge claim can be justified. These three measures (CCTT, WGCTA, and RJI) were used as a methodological starting point for the design of the questions used in this study.
2.2.1. Operationalizing the following tools:
(a) Cornell Critical Thinking Test (CCTT) – The choice of test items (specific questions) in the multiple-choice Website Information Comprehension Questionnaire was based on the CCTT.
(b) Watson-Glaser Critical Thinking Appraisal (WGCTA) – The choice of test items in the open-ended website design questionnaire was based on the WGCTA and previously used in the Roy study (Reference Roy2012).
(c) The Reflective Judgment Model (RJM) – The overall choice of test items was based on the RJM to examine the certainty of a knowledge claim (the extent to which both the website analysis questions and the website content were understood), how the knowledge was gained (personal observation, personal assessment of arguments and data), and how the knowledge claims were justified (observation, specific or generalized rules of inquiry).
It should be made clear that the CCTT, WGCTA, and RJI were not used directly in this study. Rather, they acted as a methodological framework or starting point for the design of the study questions.
2.3. Website design and analysis and developing analytical thinking
Purpose: The following literature explores why website analysis could be an effective task towards language acquisition
Website analysis has the potential to be a powerful tool for developing analytical thinking, by focusing on creating, evaluating and analyzing (Atherton, Reference Atherton2002). Website analysis could bridge the gap between language learning and analytical thinking (Murray & McPherson, Reference Murray and McPherson2004; Reference Murray and McPherson2006), thereby countering the claim that the institutional practice of language has never really been supported as a thinking and learning tool (Liaw, Reference Liaw2007).
Lee (Reference Lee2000) argues that having students create and publish websites in ESL/EFL classes is interesting and energizing for them. Furthermore, Spyridakis (Reference Spyridakis2000) and Bunz (Reference Bunz2001) argue that having students analyze websites is potentially useful for developing user-centered website models.
Van Hoosier-Carey (Reference Van Hoosier-Carey1997) points out that website assignments as a rhetorical and persuasive exercise in the technical communication classroom can lead to the understanding of a message embedded in the design of a website. Furthermore, Warschauer (Reference Warschauer1997) draws attention to the fact that web design skills incorporate ‘situated learning’, which allows students to plan meaningful tasks, explore, design and solve meaningful problems in an environment that suits their own personal interests, and opens up multiple channels that might facilitate the application of their knowledge in the future (Collins, Brown & Newman, Reference Collins, Brown and Newman1989).
2.4. Cognitive and metacognitive reading strategies
Purpose: The following literature explores how EFL readers use different strategies to comprehend website content
Reading strategies are an indicator of how a task is conceived, the textual cues attended during reading, how a text is perceived and sensed, and how a text is understood (Block, Reference Block1986).
These strategies include cognitive strategies (deliberate actions taken by readers when faced with comprehension problems), metacognitive strategies (advanced planning and comprehension monitoring techniques used by readers), and support strategies (tools sought out by readers to aid comprehension).
The work of Sheorey and Mokhtari (Reference Sheorey and Mokhtari2001) suggests that both native-English speaking students and ESL students display awareness of all the aforementioned strategies. However, higher-reading ability native-English speaking and ESL students use cognitive and metacognitive strategies more than lower-reading ability native-English speaking and ESL students. Furthermore, ESL students view support strategies as being very valuable irrespective of their reading ability.
Sheorey and Mokhtari focused on ESL rather than EFL environments. Nevertheless, understanding cognitive and metacognitive reading strategies in a web-based EFL reading context is important for judging how readers might approach the task of website reading and analysis. Brown (Reference Brown1978) defines cognitive strategies (e.g., bottom-up and top-down) as specific learning tasks used to help an individual achieve a particular goal (for example, understanding a text). Metacognition refers to our deliberate and conscious control of our cognitive actions (Brown, Reference Brown1978). Research by O’Malley and Chamot (Reference O’Malley and Chamot1990) identified metacognitive strategies (declarative and procedural knowledge) like skimming and scanning for successful reading. In this regard, research in an EFL context (e.g., Dhieb-Henia, Reference Dhieb-Henia2003) has shown that strategies like skimming and scanning are often used for extracting required information without a deeper-level analysis. Further research on the use of metacognitive reading strategies has also shown that those participants who received training and/or read a text with headings remembered the text topics and their organization better than participants who received no training and read the text without headings (Sanchez, Lorch & Lorch, Reference Sanchez, Lorch and Lorch2001). Other research projects have focused on the effect of prior domain knowledge on the processing of informative texts (Surber & Schroeder, Reference Surber and Schroeder2007); importance of signaling effects, previews, and logical connectives (Spyridakis, Reference Spyridakis1989; Spyridakis & Standal, Reference Spyridakis and Standal1986).
2.5. Propositions
This paper covers five broad areas of analysis and exploration.
∙ P1: The overall study as an indicator as to whether EFL learners could write responses to open-ended questions with reasonable success, and advanced EFL learners (the peer reviewers) could peer-review the merit of such responses with reasonable consistency and agreement (based on studies conducted by Hruschka et al., Reference Hruschka, Cummings, Cobb St. John, Moore, Khumalo-Sakutukwa and Carey2004).
∙ P2: Consistency in grading as an indicator as to whether advanced EFL learners understood the coding rubric with reasonable success (based on the study by Kondo-Brown, Reference Kondo-Brown2002, on variability in raters’ bias when grading writing in a Japanese L2 writing context).
∙ P3: Accuracy scores as an indicator as to whether and to what extent EFL learners could search for and comprehend information in a website with which they were not familiar (Atherton, Reference Atherton2002; Murray & McPherson, Reference Murray and McPherson2004; Reference Murray and McPherson2006).
∙ P4: Response scores for the usability questionnaires as an indicator as to whether EFL learners could meaningfully respond to design questions, asking them to self-report their feelings/understanding about the website design, features, navigation, usability, and content. The purpose of this was to reveal the level of parity, if any, between what the participants and principal investigator thought about the website.
∙ P5: The overall study as an indicator as to whether EFL learners use cognitive and metacognitive reading strategies for comprehending English website information (Brown, Reference Brown1978, Reference Brown1994; O’Malley & Chamot, Reference O’Malley and Chamot1990).
3. Methodology
3.1. Sample and context
The participants in the first week (N=16), and the second and third weeks (N=59), of the study were junior level students (age group: 18-20 years with a pre-intermediate level of English language proficiency) taking the elective course Writing and Design for World Wide Web as part of an undergraduate program specializing in computer science at a Japanese technical university. This course focuses on the process of writing, designing, and analysing websites. The participants selected for the first week of the study by the principal investigator were sixteen students (out of the 59 students on the course) who had performed best during in-class website analysis activities during the first seven weeks of the course.
All the participants had previously taken English language courses (which focus mainly on language production and reception skills) and content-based courses taught in English at the university.
3.2. Study procedure
The study as reported here was conducted as part of a graded assignment for the aforementioned elective course. The students analyzed websites regularly in class during the first seven weeks of the ten-week course. The study took place during the final three weeks of the ten-week course.
3.2.1. Actual experiment
First week (towards answering propositions 1 and 2): During the first week of the actual experiment, a group of sixteen students selected by the by the principal investigator analysed the English Belize tourism website (http://www.travelbelize.org/) using eight open-ended questions that they had previously used to analyse other websites as part of the course.
The participants had one week to complete the analysis. This included 90 minutes in-class time during which they were able to consult with other students on the course. Students entered their responses in the e-learning platform Moodle as in-line text in open-ended format. To encourage writing and proper explanation, the minimum total word limit was set by the principal investigator at 300 words. The students had to write the responses in English in their own words. The instructions given to students were also in English and were the same for each student.
Second week (towards answering propositions 3 and 4): During the second week, the entire group of 59 students on the course were given a set of close-ended usability questionnaires to answer, using the same Belize tourism website. The questionnaires focused on the following:
∙ Website Information Comprehension Questionnaire: The ability of the students to navigate through and look for information from the website.
∙ CSUQ, QUIS and MPRC Usability Questionnaires:Footnote 1 The ability of the students to meaningfully self-report on the usability of the website in terms of navigation, content, and organization.
Third week (towards answering proposition 5): During the third week, the entire group of 59 students on the course self-reported on their reading strategies.
∙ MARCI 1.0 (Metacognitive Awareness of Reading Strategies Questionnaire): The ability of the students to meaningfully self-report on their reading strategies when analysing the Belize tourism website.
The activities carried out in the second and third weeks of the experiment were completed in class. In addition, the instructions given to students for these activities were in both Japanese and English to ensure that all the students understood what they had to do.
3.3. Instruments
3.3.1. Open-ended design questions: (used to answer proposition 1)
Figure 1 shows the open-ended questions used during the first seven weeks of the course and the first week of the actual experiment. As reported earlier, the sixteen participants in the first week of the actual experiment had one week to complete the open-ended questions and submit them in Moodle.
Open-ended design questions, design categories and response cues
3.3.2. Instruments used to answer propositions 3~5:
The design-based criteria used for selecting the Belize tourism website are as follows.
∙ The content of the website is not text heavy and clear navigation is possible.
∙ Information can be searched directly from the home page.
∙ Attractive pictures are available to keep the reader engaged in the task of finding information.
∙ A Japanese version of the webpage is not available.
Figure 2 shows a screenshot from the Belize tourism website.
Screenshot of the Belize tourism website (used for propositions 3~5)
3.3.3. Independent and dependent variables
Dependent variables: The accuracy score for website information comprehension is the only dependent variable in the study.
Independent variables:
∙ The ability to search for key words from the website based on questionnaire instructions (reflected in the results for propositions 3~4).
∙ The ability to identify the key drop-down menus and links when navigating through the webpage (reflected in the results for propositions 3~4).
∙ The use of appropriate cognitive and metacognitive reading strategies when analysing the webpage (reflected in the results for proposition 5).
Background variables: The ability to understand design issues related to the website (including information organization, screen interface design, and technology used) requires ICT competence that is a background variable when judging performance. The study was undertaken at a Japanese technical university, thus the study participants can be assumed to have reasonable ICT competence.
3.3.4. Use of advanced participants for the first week of the analysis (peer review): towards answering propositions 1 and 2
Peer review has an active role in promoting collaboration, written expression, critical thinking, and professional responsibility (Kern, Saraiva & Pacheco, Reference Kern, Saraiva and Pacheco2003). Three undergraduate students (named A, B, and C for the purposes of this paper) who had previously taken the course were selected by the principal investigator to peer review the answers to the open-ended questions given by the sixteen participants during the first week of the study. The three peer reviewers were advanced undergraduate students with an intermediate-advanced level of English language proficiency and greater experience of website design and analysis than the sixteen participants. The lead investigator provided guidance to the peer reviewers on how to grade the responses from the participants prior to them undertaking this task. This guidance included providing them with a set of criteria (shown in Figure 2) for grading each open-ended response. This set of criteria was based on the design lectures that the students had had on the Writing and Design for World Wide Web course. Each open-ended response from each participant was graded by each peer reviewer. Each of the eight open-ended questions could be allotted a maximum of six points, one point for each of the six criteria. The inter-coder reliability of the three coders is. 83 (Cohen’s Kappa). Figure 3 shows an example of the coding schema used.
The coding schema used by the peer reviewers
3.3.5. Questionnaire items/categories used during the second week (propositions 3~5)
First task (proposition 3): To ascertain EFL readers’ ability to navigate through and look for information from the Belize tourism website, a website information comprehension questionnaire was designed specifically for the website. The questionnaire focused on several search categories (primarily using skimming and scanning techniques):
∙ For a specific search, sequencing the navigation process in the correct order;
∙ Identifying the correct links to follow;
∙ Identifying the correct drop-down menus;
∙ Using the search function successfully;
∙ Locating the page on the website containing the information searched for.
Second task (proposition 4): To ascertain EFL readers’ impressions about the website, a set of three usability questionnaires was used. These questionnaires were the Questionnaire for User Interaction Satisfaction (QUIS), Computer System Usability Questionnaire (CSUQ), and Microsoft Product Reaction Card (MPRC), described in detail in the panel below.
QUIS Usability Questionnaire:
*QUIS (Questionnaire for User Interface Satisfaction)—the original questionnaire, developed at the University of Maryland, was composed of 27 questions. Three items were then dropped that did not seem to be appropriate to websites (e.g., “Remembering names and use of commands”). The term “system” was replaced by “website”, and the term “screen” was generally replaced by “web page”. Each question is a rating on a ten-point scale with appropriate anchors at each end (e.g., “Overall Reaction to the Website: Terrible … Wonderful”) ( Tullis & Stetson, Reference Tullis and Stetson2004).
CSUQ Usability Questionnaire:
*CSUQ (Computer System Usability Questionnaire)—this questionnaire, developed at IBM, is composed of 19 questions. The term “system” or “computer system” was replaced by “website”. Each question is a statement and a rating on a seven-point scale of “Strongly Disagree” to “Strongly Agree” ( Tullis & Stetson, Reference Tullis and Stetson2004).
MPRC Usability Questionnaire:
Words (adapted from Microsoft’s Product Reaction Cards)—this questionnaire is based on the 118 words used by Microsoft on their Product Reaction Cards. Each word was presented with a check-box and the user was asked to choose the words that best describe their interaction with the website. They were free to choose as many or as few words as they wished. ( Benedec and Miner, Reference Benedek and Miner2002).
Third task (proposition 5): To ascertain EFL readers’ awareness of metacognitive reading strategies on the web, a Metacognitive Awareness of Reading Strategies Inventory (MARSI, Version 1.0) questionnaire developed by Mokhtari and Reichard (Reference Mokhtari and Reichard2002) was used. This questionnaire is designed to assess adolescent and adult readers' metacognitive awareness and perceived use of reading strategies while reading academic or school-related materials.
Grading the responses (propositions 3~5): This rating system was specific to the three usability questionnaires used (QUIS, CSUQ and MPRC). For each participant, an overall score for the website was calculated by averaging all the ratings for the questionnaires. The scales were coded internally so that the “better” end corresponded to higher numbers. The MPRC had to be treated slightly differently as it does not involve rating scales. Before the study, each word in the MPRC was classified as “positive” or “negative”. For each participant, an overall score was calculated by dividing the total number of positive words selected by the total number of words in the list. By way of illustration, if participants selected 7 positive words and 12 words in total, their score would be around 58%.
Before looking at the findings, it is worth noting that for all the questionnaires used in this study, research suggests that a sample size of at least 12-14 participants is needed to assure reasonably reliable findings (Tullis & Stetson, Reference Tullis and Stetson2004).
4. Findings
4.1. Analysis for the first week with open-ended questions – data towards answering propositions 1 and 2
The first key finding concerns the eight open-ended design questions used during the first week of the actual experiment. Each of the three peer-reviewers graded each of the eight questions separately, based on six criteria, as described earlier. This means that the mean and SD values are for the sum of the mean grades given by the peer reviewers, with the maximum sum of the mean grades for each question being no more than 18 (6 * 3) (6 being the maximum score for each question).
The data show that the highest mean value was for Q1 (whether the organization of the information is user-friendly) at 14.56, with the mean value for Q2 (whether the presentation of the content is appealing) slightly less at 12.56. There were relatively lower mean values for Q3~Q7 in the range of 10.33~11.67. A particularly low value of 10.33 was observed for Q7 (whether the resources use real-world situations). However, a further analysis of the responses to Q7 by the principal investigator suggested that the participants were not sure about how to answer this question.This may be because the question requires a deeper level of analysis of the website content than the other questions.
Figures 4, 5 and 6 show the total criteria score for the sixteen participants on each of the eight questions for the three coders.
Total criteria score for the 16 participants on each of the eight questions for Coder A
Total criteria score for the 16 participants on each of the eight questions for Coder B
Total criteria score for the 16 participants on each of the eight questions for Coder C
4.2. Results on website information comprehension and usability questionnaires (second week) - data towards answering propositions 3 and 4
4.2.1. Website information comprehension questionnaire
The website information comprehension questionnaire showed a Chronbach’s alpha value of.086 (from Q2~Q8). The questions in this questionnaire were designed to test whether readers were able to search through the information on the website. Each question tested the readers’ ability to search for a different type of information on the website. The questions were not ordered in terms of complexity.
On average, for questions 2~8, the data reveal that accuracy in searching for information from the website was around 83%. For the first question (which asked readers to sequence steps when searching for specific information from the website), accuracy was as high as 86.6%. Table 1 shows the mean value for each question individually.
Descriptive statistics for the eight website information comprehension questions
The results of the non-parametric Friedman test for Q2~Q8 (to test whether the accuracy scores are significantly similar) suggests that responses are significantly different (χ2 (2)=78.120, P=0.000). Since p-value=0.00 ≤ 0.01=α, we can state that there is a statistically significant difference between the mean ranks for the accuracy scores.
4.2.2. CSUQ usability questionnaire
The study participants self-reported their impressions of the website in different categories using the CSUQ usability questionnaire. The purpose of this questionnaire was to explore the self-reported ease-of-use of the Belize tourism website. The descriptive statistics show mean values for all 19 questions in the range 2.92 (Q10)~4.63 (Q15). Furthermore, the results overwhelmingly show a significant correlation between all the questions.
Next, a Friedman non-parametric statistical analysis was performed to find out if there was an overall statistically significant difference between the mean ranks of the self-reported scores in the different questions asked. The results suggest that there was indeed an overall statistically significant difference in self-reported scores for a combination of all the questions asked in the questionnaire, with χ2 (2)=126.626, P=0.000 (see Table 2).
Friedman Test Values for the 19 Questions in CSUQ Questionnaire
4.2.3. QUIS usability questionnaire
The QUIS usability questionnaire had five different categories: (1) overall reaction to the website, (2) website design, (3) terminology and website information, (4) learning, and (5) website capabilities. Figure 7 shows the percentage agreement for the respondents with the 5 different categories in the QUIS usability questionnaire. Reliability testing showed a Chronbach’s alpha of .687 for category 1, .809 for category 2, .637 for category 3, .785 for category 4, and .731 for category 5. Furthermore, for Q1~Q5 (overall reaction to the website), the mean values were in the range 5.25~6.12; for Q6~Q8 (website design), the mean values were in the range 5.03~6.25; for Q9~Q14 (terminology and website information), the mean values were in the range 3.81~5.73; for Q15~Q19 (learning), the mean values were in the range 4.14~4.73, and for Q20~Q24 (website capabilities), the mean values were in the range 2.12~5.51.
Percentage agreement with the different categories in QUIS usability questionnaire
A mean value of 5 on a 1~5 Likert scale shows 100% agreement and a mean value of 1 shows 0% agreement.
The data in Table 3 show the results of the Friedman analysis for each of the five categories in the QUIS usability questionnaire considered separately.
Friedman Test Statistics for the 5 Different Categories in the QUIS Questionnaire
The results suggest that there was a statistically significant difference in self-reported scores for four of the five categories. However, there was no significant difference between responses with regards to the “learning” category (χ2 (2)=5.769, P =.217).
4.2.4. Microsoft Product Reaction Card questionnaire
The MPRC usability questionnaire was used in the final stage following the QUIS and CSUQ questionnaire. In total, 118 words were selected by the study participants out of a possible 121 words. The study participants chose 77 positive words and 41 negative words. Of particular interest is the fact that the total frequency of the positive words chosen was much higher than the total frequency of the negative words chosen. Table 4 shows the data obtained.
Words chosen with Maximum Frequency
4.3. Results from metacognitive reading strategies questionnaire - data towards answering proposition 5
The following section provides an in-depth summary of how different metacognitive reading strategies were used when analyzing the website. The results indicate that the mean varied between 2.5 and 3.8 across the 13 global reading strategies, with most scores between 2.5 and 3.5.
The “GLOB” reading strategy mean score for the 13 questions is 3.024. The data provided the self-reported scores on the use of seven “problem-solving strategies” whilst using the website. For question # 13 (I adjust my reading speed according to what I am reading), we see a high self-reported mean score of 4.08 suggesting that it was a popular strategy among the study participants.
The “PROB” reading strategy mean score for the eight questions is 3.574. For the data on the use of the five “support reading strategies” during website analysis, we see a self-reported score of less than or around 2.5.
The “SUP” reading strategy mean score for nine questions is 2.762. Figure 8 shows a line diagram with comparative mean scores on the three subscales. These data show a relatively wider use and acceptability of the PROB metacognitive reading strategy with an overall mean value of more than 3.5.
Line diagram showing comparative mean scores on all 3 subscales
Figure 9 shows relatively wide variability among the self-reported use and acceptability of the GLOB reading strategies, with question 1 (I have a purpose in mind when I read) and 19 (I use context cues to help me better understand what I am reading) showing relatively higher values.
Line diagram showing comparative mean scores on global reading strategies
Figure 10 for problem-solving strategies shows relatively less variability in self-reported scores with all the scores above 3.0. This suggests relatively more acceptability and use of problem-solving strategies.
Line diagram showing comparative mean scores on problem solving strategies
Figure 11 shows that all the mean scores on support reading strategies are around 3.0. However one strategy is only 2.05 (I ask myself questions I liked to have answered in the text). Pearson’s r-value shows a strong pattern between the reported use of different strategies in the GLOB, PROB, and SUP categories. Furthermore, a significant correlation is visible across the GLOB, PROB, and SUP categories suggesting similar use of different strategies.
Line diagram showing comparative mean scores on support reading strategies
The Friedman non-parametric statistical analysis was performed to find out if there was an overall statistically significant difference between the mean ranks of the related reading strategies. Table 5 shows the Friedman analysis for all three sub-scales (GLOB, PROB, and SUP) separately and combined.
Friedman Test Values for Reading Strategies
Null Hypothesis: The mean ranks for all the reading strategies are equal; Alternative Hypothesis: Not all the mean ranks are equal.
The results of the Friedman analysis for all three sub-scales combined (Table 5) suggests that responses are significantly different (χ2 (2)=382.491, P=0.000). Since p-value=0.00 ≤ 0.01=α, we can state that there is statistically significant difference between the mean ranks for the scores in the three sub-scales (GLOB, PROB, and SUP).
5. Discussion
5.1. Performance with open-ended questions and coding consistency (propositions 1 and 2)
The results (in section 4.1) show some variability between the scores given by the coders (peer reviewers), especially between coders A and B, and A and C. This can be attributed to different coders interpreting the same open-ended responses differently even though the interpretations were based on the same set of criteria (the coding schema in Figure 3).
Based on experience in coding data from several previous studies, researchers have concluded that (1) a team of coders initially produce very different codings, but (2) it is possible, through a process of codebook revision and recoding, to establish strong levels of intercoder reliability (e.g., most codes with kappa 0.8). Furthermore, steps can be taken to improve initially poor intercoder reliability and to reduce the number of iterations required to generate stronger intercoder reliability (Hruschka et al., Reference Hruschka, Cummings, Cobb St. John, Moore, Khumalo-Sakutukwa and Carey2004).
Prior to the start of the present study, the lead investigator had several meetings with the coders. Over the course of the meetings, the coders showed clear improvement in their ability to interpret website design based on a given set of criteria. This may help to explain why the coders did not all produce very different codes during the study. However, this suggestion should be taken with caution as the coders practised with actual websites during the meetings rather than with open-ended responses to questions about websites.
5.2. Website information comprehension and usability perceptions (propositions 3 and 4)
The website information comprehension questions focused on understanding EFL readers’ ability to search through information (e.g., to search for hotels, shopping areas, places to visit, food, and festivals) available on the website. However, the information was not available via a single direct click and link from the home page. Rather, multiple clicks and links were required. Question 1 tested the efficiency with which readers were able to follow the sequence of activities necessary to find certain information. The score of 8.66 on a scale of 10 for all correct answers suggests high levels of efficiency among the study participants in following the steps correctly on the webpage. In addition, the scores on the other questions related to searching for information on the website demonstrated moderate mean scores.
These scores suggest a different kind of language proficiency among EFL readers, not necessitating complex in-depth readings and content comprehension. However, Friedman test scores (section 4.2.1) reveal that there are significant variations between accuracy scores. This suggests that readers’ success in finding information depended on the complexity of the search process.
The CSUQ usability questionnaire suggested a mean score in the range of 3~4.3 for all the 19 questions combined, in a possible range of 1~7. This indicates that readers were not highly comfortable with the website with regard to its features, design, etc. Furthermore, there is a very high number of significant correlations between the self-reported scores in the CSUQ usability questionnaire, suggesting a similar range of responses for most questions.
The results are more varied for the self-reported scores on the QUIS usability questionnaire. For the bivariate correlation results, we see cases both within and between the five categories where the scores are not significantly correlated. The Friedman test scores further reveal that there are significant differences in the self-reported scores for all categories including overall website evaluation, webpage design, terminology and website information, and website capabilities. However, we see no significant difference between the self-reported scores for questions in the learning sub-category. This suggests that the study participants had similar views with regard to learning from the Belize tourism website. However, for other categories there was a difference in the way people perceived the website and self-reported about its features, capabilities etc.
Finally, we could clearly see that the number of positive words chosen to describe the Belize tourism website was significantly more than the number of negative words chosen. In addition, the positive words were chosen more frequently. This suggests a positive overall impression of the website. However, it should not be assumed that this translates to better overall comprehension of the website information.
5.3. Metacognitive reading strategies (proposition 5)
The data show that most strategies in the GLOB and PROB categories were used consistently at a higher level when compared to SUP strategies. These findings are consistent with the literature (e.g., Sheorey & Mokhtari, Reference Sheorey and Mokhtari2001) which suggests that readers with low reading ability attach lower importance to support reading strategies when compared to L1 readers. In fact, a comparison between L1 and L2 reading strategies could help us interpret the data further. According to Day and Bamford (Reference Day and Bamford1998), one of the factors influencing L2 reading attitude is first language (L1) reading attitude. Furthermore, two aspects of reading, product and process, are important considerations here. The product of reading refers to the level of understanding that readers achieve from a text. The process of reading refers to the different strategies that readers use when reading a text (Yamashita, Reference Yamashita2004). So, taking into consideration product and process, the reading attitude of the study participants is another area that could beneficially be considered in another study.
Clearly, the results indicate a high awareness for strategies such as, “I have a purpose in mind when I read”, and “I think about whether the content of the text fits my reading purpose”. These global strategies were clearly beneficial for the study participants when reading a website in English. While reading the website content, and during overall navigation of the website, the study participants also adopted problem-solving strategies such as, “I adjust my reading speed”, “paying closer attention to what the person is reading” and “re-reading text when comprehension becomes difficult”. There is a lack of recognition for reading strategies in the SUB subscale, except for when readers acknowledge use of reference materials such as dictionaries to help understand reading. However, support-reading strategies appear to have been as important as any other strategy type for the study participants.
In conclusion, this study has drawn on previous literature on website design principles and analysis (focusing on content and structure), and L2 metacognitive awareness of reading strategies, and bridged the literature from these areas in an attempt to understand how readers in a typical Japanese context perform in a L2 web reading and analysis context, their perceptions of web usability, and their awareness of reading strategies.
5.4. Implications for language teaching
The following is a list of suggestions for instructors wishing to use website analysis as a tool for thinking about design and promoting analytical ability in an EFL educational context.
∙ Choose websites from different domains to arouse student interest;
∙ Ensure wide variability yet structured progression in the way that content complexity is chosen for website analysis;
∙ Provide formative and summative feedback during website analysis;
∙ Develop structured and compartmentalized questions for broad design queries to make thinking easier and channelized;
∙ Ensure that the design analysis is done in iterations with language production and critical thinking embedded during each stage of the analysis;
∙ Teach technical writing skills (information organization, formatting, typography, headings, layout, etc) alongside critical thinking and design analysis;
∙ Promote self-reflection on how students use different content learning strategies.
6. Conclusion and future research
The current study has enabled a preliminary evaluation of, first, how EFL learners approach English website analysis and, second, how EFL learners with better language proficiency and understanding of website design interpret responses using rubrics. In doing this, it has provided a first step in using web-based design education for improving critical thinking and language proficiency. Future studies might, for example, contribute towards CALL research by looking at website analysis in an EFL educational context with more structured design questions, and with each design question having specific sub-questions to channel EFL learners’ thought processes and reading strategies. It is, of course, essential that such studies, when undertaken in a language learning context, have clear language learning outcomes, processes and expectations.
The current study has also explored limited feedback and self-assessment as a tool for developing and testing participant sense for web design and language proficiency. However, it is possible that systematic continuous feedback from the course instructor, or systematic continuous self-assessment by the students, might enrich the teaching and learning process in an EFL educational context.
Furthermore, the study has provided strong preliminary evidence that there is wide variability in the efficiency with which EFL learners are able to analyze an English language website. Future studies with a focus on information scanning might, for example, focus on testing EFL learner accuracy in searching for and understanding information in English language websites. The accuracy scores could then be matched to self-reported scores on the CSUQ and QUIS questionnaires.
Further future studies could be designed to map metacognitive awareness of reading strategies with actual use of reading strategies using, for example, screen capture software, eye-tracking data, and formal usability testing methods to provide direct evidence of how each reading strategy is used by EFL learners.
Finally, other possible future studies might focus on the exact way websites are accessed by EFL learners, using, for example, reading patterns, navigation styles, and time spent on pages to provide direct evidence of different ways of access. In conclusion, the breadth of these possible future research projects suggests that there remains much to be explored with regard to website analysis in an EFL learning context.
