1 Introduction
Despite its relevance, the role of annotation is very rarely discussed in Data-Driven Learning (DDL) proposals. Some of them exploit raw, unannotated corpora and use existing applications to manage the language data, either with stand-alone solutions like WordSmith or Monoconc or with online resources such as Spaceless.Footnote 1 Other proposals use annotated corpora for language classroom activities. However, the very notion of annotation has remained bound to the linguistic research paradigm, which has led teachers to identify corpus applications in DDL with the exploitation of grammatical features and text genres. The usefulness of this approach is undisputable, but we were concerned by the fact that the efforts to implement DDL were mediated by a dominating morphosyntactic approach to language data. This becomes even more striking when one reviews mainstream publications in the field of foreign language teaching (FLT) and comes to realize that the morphology or the syntax of a language is driving pedagogy in none of them. So the issue is why Data in DDL should be restricted to morphological tagging. The challenge would be to try and turn Data into a pedagogical construct, just like any other CALL or published material.
In this paper we present the analytical framework that guided our research in providing annotation solutions for the implementation of the SACODEYL system, an effort to deliver the actual voices of European young people online. In particular, it is devoted to the design of the annotation application that has been developed for SACODEYL: SACODEYL Annotator.Footnote 2 The role of pedagogic annotation is central here as the resulting annotation will condition and shape the learning experiences of learners using our system. For reasons of space, areas of the system other than the annotation solution cannot be fully discussed here.
2 Annotation in corpus linguistics
Annotation in the context of linguistics can be seen as both the process and the resulting product of adding information to electronic texts. In the field of corpus linguistics (CL), annotation is primarily conceived of as an add-on, a way to describe the information that matches the needs of language researchers, linguists or corpus users. In this way, Leech (Reference Leech1991) equates annotation with analysis, while McEnery and Wilson (Reference McEnery and Wilson1996) describe corpus annotation in terms of processing. Within this scheme, the whole point of annotating corpora lies in the fact that annotation allows corpus users both refined information retrieval capabilities and subsequent treatment of the data. Once captured, this information may serve a very wide array of purposes, from language description to natural language processing.
There exist different approaches to the conceptualization of the annotation process. Annotation can be (a) automatic, semi-automatic or manual, depending on the degree of human intervention in the process, or (b) it can be done by one single annotator or a group of annotators. In either case, its raison d’être inevitably (c) reflects the different nature of the ultimate aim of the meta-information being added to the corpus. As an illustration, annotators’ needs may range from an interest in non polysemic ambiguity (Poesio & Artstein, Reference Poesio and Artstein2005) to an interest in L2 speakers’ errors (Abe & Tono, Reference Abe and Tono2005). Within the CL research tradition, the efforts of annotation specialists have been geared towards grammatical tagging,Footnote 3 including morphological and syntactical information, (Garside, Leech & McEnery, Reference Garside, Leech and McEnery1997). Biber, Conrad and Reppen (Reference Biber, Conrad and Reppen1998) is, for example, a valuable introduction to corpus annotation issues, whereas Leech (Reference Leech1993) offers a more comprehensive treatment through a description of annotation procedures.
McEnery and Wilson (Reference McEnery and Wilson1996: 57) have pointed out that annotation poses important challenges to researchers, who predictably will need to find the right balance between “consensus-based-theory-neutrality” and problem solving issues in connection with the ultimate purpose of their annotating. Therefore, it follows that annotation in CL is well rooted, although not restricted, to traditional linguistic research paradigms. Teachers and language researchers have promoted the use of L1 representative corpora in the FL classroom. Examples of this interface are studies such as Santos Pereira (Reference Santos Pereira2004: 110), which reports on the use of a Portuguese language lexicon with “grammatical and quantitative information” as a “new approach to understanding […] meaning and semantic disambiguation, real differences between near-synonyms” and other features of language in actual usage. The tide of CL has reached the classroom shores, and now it has become more and more common to find otherwise specialised jargon such as ‘concordance’ and ‘concordancer’ (Flowerdew, Reference Flowerdew1993; Owen, Reference Owen1996; Gavioli & Aston, Reference Gavioli and Aston2001; Weber, Reference Weber2001; Frankenberg-Garcia, Reference Frankenberg-Garcia2005) or DDL (Mishan, Reference Mishan2004) not only in linguistics or CL, but also in FLT journals. Everything has gone the way it was predicted by Biber and Finnegan (Reference Biber and Finnegan1991), who expected that increased computer literacy among linguists and the overwhelming presence of computers in our daily work could contribute to the expansion of the field and its applications.
Although the situation is far from being one where CL has become mainstream in FLT, it is nonetheless true that major efforts have been made to voice the enormous potential of CL in the foreign language classroom (McCarthy & O’Dell (Reference McCarthy and O’Dell2006) or O’Keeffe, McCarthy & Carter (Reference O’Keeffe, McCarthy and Carter2007)). Let us consider some of these attempts.
3 Annotating corpora for the FL classroom
3.1 Corpora in the FL classroom
In recent years various collected volumes have been devoted to the use of corpora in the language classroom (Sinclair, Reference Sinclair2004; Braun, Kohn & Mukherjee, Reference Braun, Kohn and Mukherjee2006; Hidalgo, Quereda & Santana, Reference Hidalgo, Quereda and Santana2007). Although their scope is wide, and very often interdisciplinary, we can find in all of them some common ground as to the need for the FL community to develop specific corpora that suit the challenges of the language classroom. Notwithstanding, this is not a new original claim. Chambers (Reference Chambers2007) takes up Leech and Candlin’s (Reference Leech and Candlin1986) pioneering work on CALL to remind us that corpora and CL methods that are not adapted to the specific classroom environment may be unsuccessful. In the same volume, Leech (Reference Leech1986) gave a longer-term perspective on the integration of corpora in CALL: “The CALL and CBELT programs will be able to make use of corpora of graded and classified English language texts, and language databases in the form of a grammar and a lexicon”. Twenty-one years after this statement, and twenty-three after the Lancaster Symposium ‘Computers in English Language Education and Research’ where these ideas were first advanced, the CL and CALL community are still in the process of normalizing the use of corpora in the language classroom (Bax, Reference Bax2003).
Chambers (Reference Chambers2007: 12) has detected “major obstacles” for the popularisation of corpora, including attitudinal factors as well as gaps in teacher training and familiarization in secondary education with CL methodology. Braun (Reference Braun2007b) has explored corpus integration in the secondary school curriculum and has outlined the methodological challenges she encountered during her research. The author concludes that for corpora to be integrated in mainstream formal education it is necessary to move from DDL to needs-driven corpora.
It appears that, despite the benefits (Gavioli & Aston, Reference Gavioli and Aston2001; Bernardini, Reference Bernardini2004), CL methods have a long way to go before establishing themselves in the mainstream FL classroom. On this topic, Mauranen (Reference Mauranen2004: 99) points out that for a teaching method to become an important innovation, it has to “make its way to the normal classroom where teachers and students can use it as part of their everyday routine, with not too much extra hassle”. Certainly, the reasons that underlie the invisibility of corpora in mainstream formal education lie outside the scope of this research. For the time being, the natural corpus playground continues to be tertiary education. However, some pioneering work is spearheading significant innovation.
3.2 Annotating with learning in view
Within the context of advanced use of the English language, Braun (Reference Braun2007a: 43) has pointed out that a pedagogically motivated corpus may be instrumental in assisting learners in their discovery of “interesting items” in their materials and their further exploration “with corpus-based methods of investigation”. Braun (Reference Braun2006a: 29 ff.) builds on previous DIY corpus literature and suggests that ad hoc spoken corpora in FLT may (a) provide a more systematic range of material than individual texts or scattered collections of activities and, if well-designed, (b) offer a wider range of idiolects than the average material. She states that thematic annotation, including topic keys and section titles, are particularly useful in the implementation of pedagogically motivated corpora. Thus, by adopting a classroom perspective, the morphological or syntactic features of the language are not necessarily driving the annotation of the corpus and, consequently, are not conditioning the type of learning experiences that a learner could engage in when working on these corpora.
As already stated, the annotation in a corpus describes the information that matches the needs of researchers, linguists or corpus users. It goes without saying that annotation driven by language description standards may have a very positive impact on the language classroom. However, it seems ironic to think that this type of language research-oriented annotation, which heavily relies on traditional grammar paradigms, should be instrumental in a FLT communication-oriented methodological context in which the grammar-translation method or the structural method are no longer dominating the arena. Campbell et al. (Reference Campbell, Mcdonnell, Meinardi and Richardson2007) are a good example of how ad hoc, FLT-oriented corpora can be built around interests other than morphosyntactic units.
Braun (Reference Braun2005, Reference Braun2006a) has discussed the creation of one topic-driven corpus: the English Language Interview Corpus as a Second-Language Application (ELISA), “a collection of video-based interviews with native speakers of different varieties of English, for example, US, England, Scotland, Ireland, Australia, and from different walks of life”.Footnote 4 Taking one of these interviewsFootnote 5 at random, we can find that the root element in the markup 〈session file = “horse_caravanning_ie.xml”〉 is a whole text. By enlarging the element, we obtain the annotated information, like title, metadata (creator, video file, language variety, list of speakers, description, speech rate information), topic and speaker name. The following is an excerpt from the code:
Finding XML in the code has ceased to be a surprise in CL or in CALL.Footnote 6 XML has been used successfully in different CALL applications for a variety of purposes and reasons. Cushion (Reference Cushion2004) sees in XML a powerful technology to exchange language data and language structure and thus prolong the extension of data life. Ward (Reference Ward2002) believes that this technology facilitates the re-usability of resources and is a time-saving option for CALL projects where developers have to adhere to highly structured and repetitive templates. Mishan and Strunz (Reference Mishan and Strunz2003) have laid emphasis on the possibilities of XML to customize language learning and, very significantly, to provide developers with the opportunity to design and implement applications that are driven by pedagogy rather than by technology.
The annotation in Figure 1 tells us that the information that is pedagogically relevant to learners is that which concerns the topic of the oral text as well as the speaker(s) taking part in the communication process. Visually put, one may argue that the tags here are telling us that pedagogy is driving the annotation, the markup text (enclosed in angle brackets) adding the information that will actually guide learners in their exploring and engaging in corpus-based activities. The annotators have a pedagogical use of the text in mind when approaching the annotation stage. The tags 〈topic_title〉, 〈topic_key〉 and 〈content_key〉 highlight the relevance of the communicative purpose of texts, that is, the topics and the contents that characterize them. Also, ELISA being a multimedia corpus, the markup also includes timestamping information on the related video. The screenshot in Figure 2 shows the ELISA users’ interface and the integration of the annotation above.
XML code from ELISA
ELISA web interface
Despite the simplicity of the markup, we can envisage this annotation approach as a powerful problem-oriented type of annotation where the XML is performing the role of separating the corpus, that is, the linguistic interface, from the learning, pedagogical interface. Figure 3 illustrates this approach.
Pedagogy as add-on
This is an example of what simple XML can do for CL in the language classroom. The resulting annotated corpus in Figure 3 can be seen as being integrative of language data and annotated pedagogy. The former is the liquid, the latter is the container. The interesting aspect here is that pedagogy can be annotated and, subsequently, accessed by corpus users. In the next section we review existing annotation tools and discuss their relevance to the field of pedagogical annotation.
3.3 Existing annotation applications
There is a wealth of annotation solutions adapted to different contexts and users’ requirements. In some cases, these tools are geared towards morphological and syntactical language research and, in particular, towards tagging. The tagging adscription can be performed in a manual, semi-automatic or automatic fashion. Examples of the latter include CLAWS (Garside, Reference Garside1987), TreeTagger (Schmid, 1994) and FreeLing (Atserias, Casas, Comelles, González & Padró, Reference Atserias, Casas, Comelles, González and Padró2006). In all of these applications, the end user is presented with a corpus/text which is tagged according to a language theory and a tagset. It is important that we understand that this tagset is given to the researcher, which demands a high degree of adaptation of the specifications of his study to the tagset characteristics. The degree of flexibility and adaptation of this tagset to the user-specific needs is limited, which may deter potential language educators from becoming involved in projects which are not primarily devoted to morphological research.
However, an increasing number of new tools based on XML-aware annotation have introduced powerful user features. Among these applications we can find AGTK (Bird, 2002), NITE (Reidsma, 2005), ACE (Maeda, 2006) and Callisto (Bayer, 2006). These tools allow for a more flexible representation of annotation, although the specification of user-driven annotation characteristics remains a complex issue. Thus, Callisto forces users to define a XML schema whereas ACE, NITE or Palinka (Orasan, 2003) offer users customization of their tagsets via configuration of an external file. This fact hinders the utilization of these tools in a FLT or research environment with no ICT expert support. Dexter (Garretson, 2006) is a step forward in getting annotation tools to be used in a more generic and more flexible way in different contexts, including foreign language teachers who want to prepare teaching materials. Dexter codes the annotation of the linguistic corpus using its own XML solution. To start a simple annotation process with this tool, the annotators need to use a converter to process the text they want to annotate into a native XML format. A similar process has to be followed if we use MMAX2,Footnote 7 another stand-off solution suitable for multi-layer annotation and widely used in discourse analysis (Braun, Kohn & Mukherjee, Reference Braun, Kohn and Mukherjee2006b).
In the context of a multinational, multilingual and multi-corpus project, however, the annotation solutions should cater for more complex needs as the language learning experiences of future users of our system beg for a high degree of sophistication.
4 Annotation challenges in DDL
In the previous section, we had borrowed the notion of problem-oriented tagging from McEnery and Wilson (Reference McEnery and Wilson1996: 56–7). This term is a special label for those cases where non-traditional, standard grammatical annotation is required so as to fulfil the aim and ultimate use of a corpus. It should be noted that the authors refer to this type of tagging as not being exhaustive and of not providing broad coverage, as well as lacking consensus-based theory neutrality. McEnery and Wilson’s (Reference McEnery and Wilson1996) remark is better understood in the context of language analysis and, in particular, in the context of linguistic research, which clearly shows that corpus applications in FLT still need to gain a status in their own right.
While it is true that local problems may call for specific individual solutions, it is undeniable that such a generalization may not hold true for an annotation scheme simply because it is not morphological or morphosyntactic. There are, however, some important challenges for those interested in exploring new horizons in terms of enriching language corpora with pedagogy. These challenges can be classified under three different headings and affect the nature of the pedagogical annotation process: the design of the annotation scheme, the epistemology of the annotation, and the technology that will support the process.
The annotation of a pedagogically relevant corpus should comply with standard practices and, accordingly, good pedagogical annotation should arguably meet Leech’s (Reference Leech1993) quality maxims, that is, it should be possible to remove the annotation from the text; if desired, the annotation could be extracted and should be based on guidelines everyone could reach; it should be made clear how and by whom the annotation was carried out and it should be based on widely agreed and theory-neutral principles.
Figure 3 shows conceptually how feasible it should be for anybody with basic computing skills to perform all of these basic operations. Any XML tagging can be easily extracted and removed with any XML editor and, of course, with find and replace text tools. These design features will give our annotation a rigorous and well-planned structure. The issue of an underlying theory-neutral principle would require extensive treatment, however. As with other areas in applied linguistics, FLT lacks a unified theory that can account for the teaching/learning dichotomy in absolute terms. Ellis (Reference Ellis2005) has drawn together the most important contributions from Second Language Acquisition (SLA) research in the area of FLT. Ellis has stressed the need for learner-centred teaching that focuses on form as well as on meaning and which fosters L2 competence based on both formulaic and rule-based declarative knowledge. Ellis also highlights the well-researched need to offer implicit in addition to explicit knowledge of the L2. Given this situation it would be, therefore, difficult for us to decide on an exclusive perspective that adheres to some areas of FLT methodology and neglects others, especially when we aim to offer a generic solution for pedagogy-driven annotation of natural language.
From a more epistemological perspective, Leech (Reference Leech1993) argues that the end user should be made aware that the corpus annotation is not infallible and that no annotation scheme has the a priori right to be considered as a standard. While accepting the implications of such a statement, and indeed because of it, we believe that the annotation of a DDL application should meet sound pedagogical presuppositions and foundations.
Antecedents in the DDL literature such as Aston (1997), Bernardini (Reference Bernardini2000, Reference Bernardini2004), Meunier (Reference Meunier2002) and Mukherjee (Reference Mukherjee2006) call for DDL solutions that are representative of language situations that resemble real-life tasks and facilitate both inductive and deductive discovery learning. This raises the question whether annotation is instrumental here. We believe that for annotators to annotate such a complex interweaving of pedagogy-related facts, we should strive for a highly-structured approach that could be taken to different foreign language scenarios and which, at the same time, offers enough flexibility to meet the challenges of every specific annotation situation. For a multilingual initiative this means that the annotators of the Lithuanian corpus and the Spanish corpus, to cite two of the languages involved in SACODEYL, may decide on a different annotation structure for the sub-taxonomies of the general, main grammar taxonomy. This is the case because we are not looking at the annotation of every single grammatical feature in the corpus, but rather at those units which are suitable for pedagogical delivery and, accordingly, of interest for further DDL exploitation. A case in point is the pedagogical approach in most FLT textbooks, where the different learning units are not expected to cover or fully describe every single grammar or lexical point that appears in a unit. It is understood that the textbook performs a mediation role between (a) the teaching institution or learning/ teaching initiative and (b) the learner. The role of pedagogically-driven search in this context is of the utmost importance and accordingly the annotation of corpora for the language classroom should bear this subsequent use in mind.
As far as technology is concerned, we conclude that we need a user-friendly tool with multilingual support as our SACODEYL multinational teams of end-users are not computational linguists themselves. Furthermore, we wanted to develop a standard-compliant tool that could be instrumental in more generic contexts and thus favour the re-utilisation and dissemination of the application.
5 Developing annotation solutions
In the previous paragraphs we have implicitly discussed the analysis stage of Colpaert’s (Reference Colpaert2004) Analysis-Design-Development-Implementation-Evaluation (ADDIE) Model. As we have shown, the challenges for pedagogical annotation abound. There is one, however, that will determine the design and implementation of the application: the type of use that we want to put our annotation to, that is, the annotated corpora that we will use so as to feed our DDL system. This can be seen in Figure 4.
Deploying online DDL
To do so, we need to embark on a software design, development and implementation engineering process that will allow us to develop an annotation application that is instrumental for our purposes. In SACODEYL, the annotated corpora will be queried by a search tool which will return information that will read the pedagogically-enriched corpora uploaded to our servers.
Most of the CALL literature concerning application design and implementation (Plass, Reference Plass1998; Levy, Reference Levy1997) adopts a well-known input-output engineering approach. This is in line with mainstream software design principles and is instrumental in getting the application development started (Larman, Reference Larman2002 and Reference Larman2003). Although geared towards language-course development, Colpaert’s (Reference Colpaert2004) Analysis-Design-Development-Implementation-Evaluation (ADDIE) model is an important reference for us. Ward (Reference Ward2006) has stressed the fact that Colpaert’s design phase is probably the most crucial of all the stages. This phase is divided into conceptualisation, specification and prototyping. Table 1 shows the main aims of each stage.
Stages in Colpaert’s (2004) design phase
This conceptualization should include all the facts, considerations, principles, assumptions and requirements to be observed before the implementation of the application itself is started. In software engineering literature the specification of requirements is a central, well-documented stage (Larman, Reference Larman2002). In the light of this, it is essential to avoid any trace of ambiguity in the specification of the requirements. Once these are well-established, we can move on to decide on the type of application architecture that best fulfils them. As a result of this design process, we may obtain eventually (a) the application architecture, (b) an application prototype or even (c) the application itself.
5.1 Requirements specification
In 3 above, we have discussed the challenges that we faced when approaching the main features of an annotation solution for online DDL. Those challenges can be regarded now as shaping the design and implementation of our application. Therefore, our annotation tool can be understood as the output of the software engineering process that started with the analysis stage that gave rise to the specification of the requirements of pedagogical annotation.
In order to analyze thoroughly the conceptualization of SACODEYL Annotator, we have opted for a multi-layer framework analysis. To do so, we have used the models by Lee and Xue (Reference Lee and Xue1999) and Jain et al. (Reference Jain, Vitharana and Zahedi2003). These models integrate different areas that must be carefully reflected upon when dealing with the conceptualization and specification of the features of the tool engineering process. Every dimension in Figure 5 is instrumental in identifying the functional specification of an application.
Dimensions in the analysis process
We have already argued the need for a new bottom-up epistemological approach that shifts the focus of annotation from linguistic research to pedagogy. This new focus also entails a new type of user or users in the annotation debate. In the linguistic-research paradigm, linguists or computational linguists are the end users of tools and technologies such as CLAWS (Garside, Reference Garside1987) or TreeTagger (Schmid, Reference Schmid1995). In DDL, however, the actors are not so easily targeted. Teachers, students and CALL and CL professionals play a role in the making of this type of learning. This will have a highly significant technological effect on the way these new applications are to be designed.
Each type of end-user will demand different non-functional specifications. Language engineers or computational linguists want flexible applications which usually offer complex interfaces, subject-specific and extensible. However, a non-expert annotator such as a language teacher, or even a learner, wants intuitive, easy-to-use interfaces that remain stable and whose learning curve is low. While emphasizing the roles and specifications of the input/ output of the annotation application, we have constantly paid special attention to the needs of the end-users of the application.
In our case, pedagogy and DDL are the specific purposes for the annotation of corpora. Accordingly, such a tool should incorporate, among other things, the possibility for users to annotate discourse topics or the adequacy of a particular text or piece of text for further exploitation in a FLT context. As regards the type and quantity of data to be used as input for the tool, the main consideration here is to find the right balance between data and processing so as to achieve real design efficiency (Banker et al., Reference Banker, Davis and Slaughter1998). It is essential that the tool developers fully understand the profile of the potential users, whether these are linguists, language teachers or language learners.
Table 2 shows the specifications that our annotation approach should meet. Based on these specifications, we are in an optimal position to accomplish the design stage of the engineering process of our own annotation tool.
While actors and data are seen as cross-sectional dimensions, the epistemology, content and technology dimensions match the annotation challenges discussed previously.
5.2 Choosing the application life cycle
In software design, there are numerous approaches to building efficient applications. These approaches, and their methodologies, are commonly referred to as software life cycles. Prototyped life cycles are widely used in developing software products that undergo important changes or modifications in their functionalities. This approach is also used when new technologies are introduced and there is a high degree of uncertainty regarding their impact or acceptance. A prototyped life cycle is advisable if the specifications for the application are not well defined or remain open to further modifications. In this sense, a prototype is a partial product, a provisional artefact which is not released for the end user’s testing. This approach is very cost-effective and understandably is widely established in advanced software development. By developing and evaluating the prototype, more thorough specifications of the tool can be drafted. The drawback here is that the definition and tool building are repeated twice: once for the prototype and a second time for the final product.
A spiral life cycle is anchored to a different view of software engineering. Each spiral is a cycle, and in every cycle the developers fix and improve the functionalities of the application, as illustrated in Figure 6.
SACODEYL Annotator spiral life cycle
This approach is of use when the end user and/ or client is not fully aware of the output he expects from the application. This is very often the case in multi-organisation projects where expectations vary and, obviously, developers cannot wait for end users to evaluate the application once the development cycle is over. The illustration in Figure 6 represents this approach: each loop is a development cycle divided into four squares which stand for different stages, namely, planning, design, implementation and evaluation. Each loop increases the maturity of the application.
6 An annotation solution for online DDL: SACODEYL Annotator
6.1 SACODEYL Annotator
SACODEYL Annotator is one of the applications developed for SACODEYL (System Aided Compilation and Open Distribution of European Youth Language), a Minerva project funded by the European Commission. SACODEYL Annotator is the product of the analysis and design stages discussed in 3 and 4 and meets the specifications outlined in Table 2. Its main functionality is to provide a software-aided annotation solution for users who need to add structured information to electronic texts. To do so, we have adopted an open and extensible standard for the definition of XML documents: the Text Encoding Initiative (Burnard, Reference Burnard1995).Footnote 8 This encoding standard provides our tool with extensibility, interoperability and standardization, three characteristics which we consider of the utmost importance for the re-usability of our annotated corpora (Ward, Reference Ward2002; Cushion, Reference Cushion2004).
SACODEYL Annotator specifications
Among other features, the application allows users to define and work with an open taxonomy set of data. This feature makes the application sufficiently generic to claim that it can potentially be used in a wide range of disciplines and fields, from corpus linguistics to forensic analysis of speech. However, the application as-it-is has been implemented to annotate a corpus pedagogically.
When using the application for the first time, users are prompted to create and add a new corpus container (Figure 7). Once this is done, a user will need to create a taxonomy that specifies the set of tags that will be used in the annotation of texts.
Adding a new corpus to SACODEYL Annotator
In the screenshot in Figure 7, the user has to define a group of TEI-header elements that we have considered necessary for the purposes of SACODEYL. Once a corpus file is created, the user can modify the tree structure. Figure 8 shows a part of the taxonomy tree that has been defined for the Spanish SACODEYL corpus.
Annotation taxonomy tree of the SACODEYL Spanish corpus as displayed in SACODEYL Annotator
This interface allows users to nest and organize the tags in a hierarchical way quite simply. Also, the annotation and the hierarchy tree annotated on a corpus can always be modified. Let us try to exemplify this point. A user has already annotated the lexical features of a text or a group of texts. Later on, she needs to add new information to the corpus, for instance she wants to annotate oral discourse phenomena that are not present in the taxonomy. This can be accomplished by creating a new ‘child’ in the corresponding ‘parent’, something which will allow this teacher or linguist to keep the information previously annotated while extending the range of information added to the XML file.
Note that it is the user that decides on the hierarchy of this element, which means that this new feature, we call it category, can be nested where the annotator deems right. In terms of the generic use of the tool this means that the open annotation we advocate is not conditioned by the discipline or field of research, but rather by the final use or output that we want to obtain.
The application allows users to assign keywords to annotated categories. In SACODEYL Annotator, a keyword is a stretch of language (a word, more than one word or a whole paragraph) that the annotator associates to a category. In Figure 9 we can see three annotated sections of the Spanish corpus. The general taxonomy tree is on the right hand side, while on the left of the interface we can find the text that is being annotated. In Sections 1 and 3 of Figure 9, we can see the keywords associated to the categories that have been annotated. Keyword views and assignment are optional, and thus section 2 in Figure 9 is annotated but keyword view is disabled.
SACODEYL Annotator work areas
SACODEYL Annotator has been developed to meet the needs of a very wide range of users, and as a consequence no a priori knowledge of CL is needed in order to start annotation right away. The tool is very easy to use: tag assignment is performed through drag and drop and keyword assignment is performed through select and click basic operations. To facilitate this process, the application filters out the information shown on screen and so users can decide which highlighted keywords they want to see or hide. Secure deleting of the annotation is also provided. The tool is so intuitive that even learners with no CL background might use it to navigate the annotation.
SACODEYL Annotator supports international coding standards such as ANSI, ASCII, ISO-7866 or Unicode (Needleman, Reference Needleman2000), and thus it can be used to annotate texts in any language. The tool has been designed to run on different operating systems and the only requirement is that users have installed Java Runtime Version 6 (JRE 6) v1.6.0 or higher. The tool uses Java Networking Launching Protocol (JNLP) technology and updates automatically if the machine has an Internet connection. JNLP has proven to be an excellent option in the context of SACODEYL as all the members of the consortium have consistently used the latest development of the application.
6.2 Evaluation
So far, the life cycle of our tool has undergone 25 ‘loops’. These changes have affected areas such as end-user interface design issues, interface multilingual support, refinement of logical and physical views or metadata treatment. Table 3 summarises the aforementioned loops.
SACODEYL Annotator loops
Most of the changes to the tool were prompted by the different users of the tool during its life cycle. On the one hand we find transcribers and annotators, on the other the developers. The former were very concerned with the functionalities of the tools (loops 2, 4 and 14 are good examples), while the latter were more focused on bug-fixing and improving the maturity of the tool (for example, loops 13 and 23). Team-work and very fluent communication gave us the chance to discuss the pros and cons of implementing new changes, which has resulted in improved performance and a high degree of satisfaction among the users of the tool in SACODEYL.
7 Conclusion and further work
SACODEYL Annotator is an annotation solution for those interested in the implementation of pedagogically motivated corpora in the language classroom. It is part of a larger SACODEYL system and, accordingly, its usefulness is better appreciated when used together with the SACODEYL search tools. Far from a weakness, this is a common feature of XML annotation tools which are largely used to create robust, highly-structured searchable files.
From a theoretical perspective, SACODEYL Annotator addresses an important issue raised by McEnery and Wilson (Reference McEnery and Wilson1997: 8): “The needs of the researchers primarily involved with the development of corpora have been subtly different from those which would generally be relevant to classroom research”. Not only is classroom research different from linguistic research, but also applied uses of corpora differ widely from those in the scope of CL. While it is a fact that there is a need for specific uses of corpora in the classroom, it is nonetheless true that the efforts to implement pedagogically motivated corpora are in their infancy.
SACODEYL Annotator is a first technological step towards XML TEI-compliant annotation of pedagogy in FLT, and certainly a solution to meet the needs of the type of teachers and students-as-researchers that our project actively seek to promote in the foreign language classroom.
It must be stressed, however, that having served the purpose of our multi-national team SACODEYL Annotator is expected to become a useful tool for the community of FLT language corpus users in particular, and for those interested in creating multi-layered manually-annotated XML TEI-compliant files, in general. SACODEYL Annotator has proved to be a useful tool in the process of building a mediation effort to bring CL methods to secondary school learners. There is room for improvement in the generalization of the tool to other, less specific fields, including the adaptation of the tool to multi-modal annotation, synchronous and asynchronous collaborative team annotation, export features and the integration of publishing capabilities.
