2.1 Emerging AI chatbots and their application for language learning

The first versions of intelligent chatbots began with ELIZA, created by Joseph Weizenbaum in the 1960s, and Parry, developed by Kenneth Colby in the 1970s, which were programs that simulated human-like conversation in the form of texts (Janarthanam, Reference Janarthanam2017; Shum, He & Li, Reference Shum, He and Li2018). These early chatbots recognized keywords or phrases in users’ input and provided pre-programmed, and thus limited, responses corresponding to those key expressions (Weizenbaum, Reference Weizenbaum1966). In the 1990s, ALICE, developed by Richard Wallace, was able to maintain a more sophisticated conversation. This chatbot adopted Artificial Intelligence Markup Language (AIML), which identified topics, categories, and key patterns in users’ input and utilized saved information to provide corresponding responses (Wang & Petrina, Reference Wang and Petrina2013).

Chatbots can be classified into text-based chatbots and voice-enabled chatbots by their mode of communication. Text-based chatbots interact with users via texts, and the aforementioned chatbots are representative text-based chatbots. In recent years, Cleverbot, created by Rollo Carpenter, is one of the most well-known chatbot applications. Cleverbot is unique in its ability to learn from users’ input and to provide human-like responses that are not pre-programmed (Torrey, Johnson, Sondergard, Ponce & Desmond, Reference Torrey, Johnson, Sondergard, Ponce, Desmond and Leake2016). It automatically saves users’ entire input and uses it to provide adequate responses to other users by determining how a user responds to the chatbot’s input. Kuki, previously known as Mitsuku, developed by Steve Worswick, is another representative text chatbot adopting AIML files. Kuki simulates the most human-like conversation in the world, shown by its being a five-time Loebner Prize winner from 2013 to 2019. It is capable of providing human-like responses to users’ input and even understanding mood in users’ typed language.

Voice chatbots mainly interact with users via voice commands and responses. They have been widely implemented in personal devices like laptops and IoT devices that are connected with appliances and smart home systems. Voice chatbots are equipped in smartphone devices as stand-alone programs like Siri by Apple or applications like Google Assistant by Google, and Echo by Amazon. In response to users’ commands or queries, voice chatbots in smartphones simply perform phone actions, check a user’s schedule, or search the internet for news. Voice chatbots are also available in the form of mobile applications. For example, Lyra Virtual Assistant sets alarms and searches users’ contacts or information on the web as a personal AI assistant. It allows users to chat with her by answering simple queries or even presenting her opinions on certain topics. Talk to Eve simulates natural human-like conversation by exchanging voice queries. However, it is restricted in its capability to process lengthy input from users (Kim, Cha & Kim, Reference Kim, Cha and Kim2019). The aforementioned chatbots have served as general conversation chatbots that would be more suitable for native speakers of English to carry out daily life inquiries. Such limitations call for the necessity of further developing chatbots, especially for L2 learning.

With the exponential growth of chatbots, researchers have integrated them into language classrooms. In Fryer and Carpenter’s study (Reference Fryer and Carpenter2006), 211 university students using ALICE and Jabberwacky for learning English reported that the students felt comfortable and enjoyed using the chatbots. Coniam (Reference Coniam2008) evaluated the ability of five chatbots, including Cybelle, Dave, George, Jenny and Lucy. Their capabilities as language learning tools, especially focusing on their linguistic accuracy, were evaluated based upon the following criteria: word-, sentence-, and text-level language structures. He found that these chatbots, chosen as currently the most capable chatbots, held the most human-like conversation and could be suitable for advanced language learners. However, he concluded that the chatbots were still insufficient to serve as ESL conversation partners from a language learning perspective.

Kim (Reference Kim2017) examined the effects of a commercial voice-based chatbot, called Indigo, on EFL students’ negotiation of meaning according to their proficiency levels. This study compared student–student communication to student–voice chatbot communication in spoken interactions for 16 weeks. It examined different communication strategies used by the students depending upon their language proficiency. The results showed that the students who interacted with a voice chatbot exhibited more negotiation strategies than those in student–student communication groups.

Kim, Shin, Yang and Lee (Reference Kim, Shin, Yang and Lee2019) explored the potential use of commercial AI chatbots as conversation partners in EFL English classrooms. The participants interacted with Google Assistant and Amazon Alexa to perform three different tasks, namely exchanging small talk, asking for information, and solving problems. They found that both chatbots could serve as effective L2 conversation partners if well-designed language learning tasks are supplied. These studies pointed out that the chatbots provided students with language learning experiences to some extent, but this activity would result in meaningful learning only if well-designed tasks were integrated together with them. Coniam (Reference Coniam2008) also highlighted the necessity of developing L2 learner-centered chatbots for English learners.

2.2 Recent development of AI chatbots for L2 learning

Concerns about implementing AI chatbots into L2 learning have consistently arisen and relevant chatbots have been studied. A virtual talking chatbot, called Computer Simulation in Educational Communication (CSIEC), was developed and used for English learners (Jia & Chen, Reference Jia, Chen, Pan, Zhang, El Rhalibi, Woo and Li2008; Jia & Ruan, Reference Jia, Ruan, Woolf, Aïmeur, Nkambou and Lajoie2008). After a six-month deployment of the chatbot in a high school, the students reported that the chatbot helped them feel more confident in using English and improved their listening skills, subsequently increasing their interest in language learning. Although the system allowed English learners to converse with the chatbot to some extent, the system focused more on providing automatic scores on restricted activities like gap-filling exercises, listening practices, or pattern drills in given scenarios.

Wang and Petrina (Reference Wang and Petrina2013) presented a commercial chatbot called Lucy for English learners that offered language learners the chance to practice over 1,000 sentences on certain topics in terms of travel, hotels, and restaurants, to name a few. The chatbot provided learners with feedback on the pronunciation or grammatical accuracy of their utterances. Although this enabled learners to practice the target language to some extent, it was very close to pre-scripted pattern drills on given topics, which was far from being free conversation.

Chang, Lee, Chao, Wang and Chen (Reference Chang, Lee, Chao, Wang and Chen2010) also investigated the perspectives of instructors and students in an elementary school toward a robot equipped with five different modes, namely a storytelling mode, an oral reading mode, a cheerleader mode, an action command mode, and a question-and-answer mode. The findings emphasized the necessity of teacher training and well-prepared content that is suitable for the target learners’ needs.

GenieTutor, developed by the Electronics and Telecommunications Research Institute in South Korea, is another AI chatbot developed for English learners (Choi, Kwon, Lee, Roh, Huang & Kim, Reference Choi, Kwon, Lee, Roh, Huang and Kim2017). The chatbot enabled English learners to practice diverse expressions on given topics at a restaurant or a shopping mall, among others. Although it provided feedback on English learners’ grammar and pronunciation, the user–chatbot conversation was close to form-focused dialogues.

The implementation of AI chatbots into language learning has consistently arisen in research, but previous studies have revealed significant research gaps that need to be further investigated. Above all, the existing chatbots designed for language learning, like CSIEC (Jia & Chen, Reference Jia, Chen, Pan, Zhang, El Rhalibi, Woo and Li2008; Jia & Ruan, Reference Jia, Ruan, Woolf, Aïmeur, Nkambou and Lajoie2008) or Lucy (Wang & Petrina, Reference Wang and Petrina2013), have centered more on pattern drills in certain situations where language learners have limited opportunities to practice diverse meaningful exchanges in target languages. Next, existing chatbots are still considered insufficient as L2 conversation partners, especially in EFL settings. Gallacher, Thompson and Howarth (Reference Gallacher, Thompson, Howarth, Taalas, Jalkanen, Bradley and Thouësny2018) asserted that current chatbots were not fully able to generate follow-up questions or relevant, natural responses to develop continuous conversations.

The present study aimed to investigate the extent to which this task-based EFL chatbot, Ellie, could be effectively implemented as a conversation partner in EFL speaking classes and, further, to gather empirical data for improving chatbot design and performance. Three research questions were addressed:

• RQ1: How did the task-based chatbot assist L2 learners to develop their conversation?

• RQ2: How successfully were the three tasks completed by the participants?

• RQ3: How was using Ellie in EFL class perceived by the participants?

3.1 Participants

The participant were 314 English learners in South Korea, consisting of 177 students (fifth or sixth graders from 10 to 11 years old) from three elementary schools, and 137 first-year high school students (15 years old). The elementary school students included 77 male and 95 female students (five students did not answer the question), and the high school participants were male. Their English proficiency levels varied across different regions of South Korea, approximately ranging from beginners to intermediate-low levels. Individual students had diverse durations of English learning experiences, but they officially began to learn English from the third grade of elementary school. They were selected for the convenience of sampling because the teachers voluntarily decided to implement Ellie in their classes. Since this study focused on the chatbot, and especially the appropriateness of its design and performance, user factors such as proficiency, age, gender, or language learning background were not strictly identified or filtered out.

3.2 Materials

3.2.1 An AI chatbot for English learning

The L2 learning voice chatbot named “Ellie” is an ongoing project funded by the National Research Foundation of Korea and was under development for three years by the authors. The chatbot was developed using the chatbot builder API platform Dialogflow ^TM by Google, whose fulfillment library is written in JS nodes. This API platform enabled developers to create and keep training chatbot agents by accumulating and analyzing user data through its own machine learning algorithms. The name of the chatbot was selected by researchers seeking English names both easily pronounced by Korean people and accurately recognized by speech-to-text technology of the chatbot. Since its first pilot test in spring 2019, the chatbot has been undergoing continuous advancements with accumulated user data and machine learning training. For more human-like conversation, the chatbot character was predefined as a female native English speaker and EFL teacher living in South Korea. The chatbot supports multiplatforms and web and mobile apps, using both iOS and Android, so that students can use laptops, tablet PCs or mobile phones in any place where a Wi-Fi connection is available. Prior to this study, each task for the chatbot was tested at least 10 times and cross-checked by eight people in the research team in order to ensure its function and capability. In the web platform “Talk with Ellie”, 71 tasks are currently displayed under the name of “Mission Challenge”. As seen in Figure 1, they are sorted into task complexity levels, from the easiest (Level 1) to the most difficult (Level 3), and task types in which users are expected to exchange conversation turns at least three to eight times for task completion depending on task complexity.

Figure 1.

The first three menus of Mission Challenge

The target tasks were identified based on the K–12 English curriculum in South Korea. According to Jeon, Lee and Kim (Reference Jeon, Lee and Kim2018), the Korean elementary-level national curriculum of English was in line with A1 to A2 levels of the Common European Framework of Reference for Languages (CEFR). When comparing the CEFR levels and the elementary-level (Years 3–6) achievement standards embedded in the national curriculum by utilizing a survey (related to communicative functions) and a performance test (related to performance levels), the elementary-level national curriculum showed a high correlation with Levels A1 and A2 of CEFR. Pedagogic tasks were designed considering target users’ proficiency, task types (Willis, Reference Willis1996), and the computer-assisted language learning (CALL) design principle (Chapelle, Reference Chapelle2001).

Based on task-based language teaching (e.g. Skehan, Reference Skehan1998; Willis & Willis, Reference Willis, Willis, Carter and Nunan2001), chatbot tasks suggested goal-oriented and problem-solving tasks, all of which require cognition process and meaning negotiation (Ellis, Reference Ellis2003). Topics and contexts covered real-life situations that young EFL students are likely to encounter in an English speaking context (e.g. a telephone conversation or finding places). No fixed dialogue patterns are given, but rather specific missions that refer to task goals so the users can freely exchange their own meaning with the chatbot.

Three similarly designed tasks of Level 2 were selected from the tasks in the chatbot (see Table 1). The level was defined by complexity of decision trees (DT; Lobo, Reference Lobo2017) and the number of expected conversation turns to complete a task. In developing a chatbot task, the use of DT is effective in establishing task completion scenarios by predicting possible users’ questions during the conversations with the chatbot (Kamphaug, Granmo, Goodwin & Zadorozhny, Reference Kamphaug, Granmo, Goodwin, Zadorozhny, Diplaris, Satsiou, Følstad, Vafopoulos and Vilarinho2018; Shah, Jain, Agrawal, Jain & Shim, Reference Shah, Jain, Agrawal, Jain and Shim2018). The target tasks included six to eight intents, which describe users’ intention for one conversation turn with the chatbot, and a similar conversation flow structure consisting of seven to eight conversation turns at a minimum to fully complete the three specific missions. In the conversation of Task 1 (see Figure 2), for example, students, as a waiter or waitress at a restaurant, were asked to recommend dishes to the chatbot. If a student’s suggestion did not match the chatbot’s preferences, the chatbot would request that the student recommend other dishes.

Table 1.

Three tasks employed in this study

Figure 2.

The screenshot of missions and directions of Task 1

To talk to the chatbot, users choose a menu, click on the “Click and speak to Ellie” button at the bottom of the screen, and speak to the device. The student–chatbot conversations are displayed on the screen. Figure 3 presents examples of user–chatbot conversations for Task 1 and Task 2. It shows how a student negotiated meanings as the student recommended “pepperoni pizza” in response to the chatbot’s talk (“Well, I don’t prefer potato. Anything else?”). To complete all the tasks, the students needed to take multiple turns until they achieved the specific missions, consequently fostering students’ negotiation strategies. The teachers used these tasks in their classes, and user data logs were recorded in all the classes.

Figure 3.

The screenshot of a student conversation with the chatbot for Task 1 and 2

3.3 Data collection procedures

Data were obtained from conversation logs of the student–chatbot conversations and the students’ responses to the questionnaires. To collect the conversation logs, the students had a 10-minute orientation session where the teachers demonstrated how to talk with the chatbot on the first day of their chatbot use. Any questions from the students were resolved to confirm that the students comprehended how to use the chatbot. The frequency and duration of the chatbot use varied across schools. The students (n = 132) from two of the elementary schools used the chatbot only one time in an after-school English class, whereas those of the other elementary school (n = 45) and the high school students had two or three sessions with the chatbot in their regular classes over three weeks. The students then talked to the chatbot in groups of three or four using laptops, tablet PCs, or smartphones. The students took turns to talk with the chatbot because each group had one device. Thus, the conversation was one on one with the chatbot, not among peers. On average, they took approximately 10 to 15 minutes to carry out each task. The conversations were automatically converted to text logs and saved in the database of the chatbot. The text logs were then extracted from the database as Excel files for analysis.

The students’ responses to the questionnaires were collected on the last day of their use of the chatbot. Paper-based questionnaires were distributed to the elementary school students, whereas the high school students took the same questionnaire online. Their responses were prepared as Excel files for subsequent analysis.

3.4 Data analysis

A mixed-methods approach (Creswell & Plano Clark, Reference Creswell and Plano Clark2011) was employed by triangulating the data collected from different sources, including the conversation logs between the students and the chatbot, and the students’ responses to the questionnaires. For the first research question, three elements were examined: conversation session, conversation-turns per session (CPS), and vocabulary level. Conversation session refers to the period from start to end of a user–chatbot conversation. CPS is the average number of dialogue exchanges between the student and the chatbot in a conversation session, and a higher CPS indicates a user’s higher engagement in conversation with a chatbot (Shum et al., Reference Shum, He and Li2018). CPS is often used as a success metric that determines the extent to which a chatbot is successful in maintaining conversations with users.

Students’ vocabulary levels were examined to determine the extent to which the student–chatbot conversations were linguistically appropriate by employing a corpus analysis program: BNC-COCA 25,000 Range Program (Nation, Reference Nation2012). This software can provide lexical profiles by using BNC-COCA 25,000, containing 25,000-word bands extracted from a 600-million-word compiled corpus with the British National Corpus (BNC) and the Corpus of Contemporary American English (COCA). In this study, only the first four graded word lists were applied to the analysis because the first 4,000 words were considered to be adequate for measuring Korean high school students’ competency as non-native English speakers (Joo, Reference Joo2008).

The second research question evaluated the appropriateness of task designs. The three tasks were quantitatively and qualitatively reviewed by measuring task success rates for each task. Task success rate is another metric used to measure chatbots’ performance (Shum et al., Reference Shum, He and Li2018). Adopting this concept, the task success rates of this study were calculated using the following formula:

$$task\;success\;rates = {{{\rm{successful}}\;{\rm{conversation}}\;{\rm{sessions}}} \over {{\rm{total}}\;{\rm{conversation}}\;{\rm{sessions}} - {\rm{invalid}}\;{\rm{conversation}}\;{\rm{sessions}}}}$$

Herein, the term total conversation sessions refers to the sum of all the conversation sessions recorded in user logs for each task. Invalid conversation sessions included the sessions in which the users’ voices were not successfully recognized or in which the users terminated the conversation abruptly for unidentified reasons. The invalid conversation sessions, taking up 45.6% of the total conversation sessions recorded in the user logs, were excluded to calculate the task success rates because such errors were unpredictable and hard to control in classroom settings. Successful conversation sessions indicated the sum of sessions where the users fully completed the three specific missions of each task.

The users’ task success was further examined to determine the reasons for the failure of the users’ successful task completion. Conversation sessions that qualified as unsuccessful task completion were initially extracted from the total conversation sessions. Then, the authors identified a coding scheme that could explain the reasons for unsuccessful task completion. The coded results were cross-checked and confirmed through peer discussion among the two coders.

The third research question about the students’ perceptions of the chatbot was examined through descriptive statistics and content analysis of the questionnaires. Descriptive statistics were computed to analyze the students’ responses to 5-point Likert-scale statements. Content analysis was conducted to analyze students’ written responses to two open-ended questions to obtain more in-depth information about the students’ perception of the chatbot. In the first cycle coding, initial coding was carried out to explore students’ perceptions by assigning categories to each segment (Saldaña, Reference Saldaña2016). The categories emerging from the students’ responses were then classified to create the coding scheme. For the second cycle coding, focused coding was adopted, as the other coder cross-checked the segments in the categories that were defined in the initial coding procedure. Any disagreements between the two coders were resolved through discussion.