1 Introduction
Nowadays, specialized-knowledge acquisition cannot be conceived without the use of a corpus. Discovering the lexical units of a given domain is undoubtedly a complex task where sheer introspection, or even the simple analysis of concordances, is often not an effective method.Footnote 1 Indeed, standard frequency criteria only serve to extract a general-purpose vocabulary, thus contributing little to the identification of technical words. This article falls within the field of automatic term extraction (ATE), which represents a priority area of interest for most industry-of-language service providers. Indeed, ATE has two major applications, i.e. as a tool for research or professional purposes (e.g. for terminologists, translators, interpreters, etc.) or as a component of a knowledge-based system (e.g. for document classification, information retrieval, text summarization, etc.). The first application is the focus of this investigation, and more particularly in relation to corpus-based terminology and terminography research. Therefore, our interest lies in the development of software for (a) the study of words and phrases that pertain to particular areas of specialized knowledge, and (b) the compilation of domain-specific lexical units to create language resources such as glossaries or term databases. From this approach, an in-depth study of the state of the art in ATE revealed a number of overarching requirements that newly developed term extractors should meet. It should be noted that, although it is not unusual for existing ATE systems to meet some of the following requirements, all of the requirements can rarely be found in the same system:
Requirement 1. The system should make use of an adequate statistical measure. Ideally, ‘statistical adequacy’ would involve that the system could extract all and only the true terms of the domain. Unfortunately, this assumption is so ambitious that it is unlikely to be realistic. On the one hand, although some reference lists are available as gold standards, it is not reliable to evaluate the performance of ATE systems in terms of recall, since ‘such resources are far from exhaustive and not error free’ (Vivaldi and Rodríguez Reference Vivaldi and Rodríguez2007 : 237). Consequently, if recall is to be evaluated, then all term candidates have to be checked manually, which becomes a time-consuming task. On the other hand, the effectiveness of probabilistic measures is closely dependent on the intrinsic characteristics of the corpus. Therefore, the adequacy of the metric should be reasonably understood as the possibility that the system manages the peculiarities of different corpora by outperforming other metrics. In this regard, one of the main purposes of term extractors is to significantly reduce the amount of noise in the ranked list, since ‘the more time users spend in scanning through term candidate lists the less useful the tool is’ (Thurmair 2003: 6). Thus, a typical way to measure statistical adequacy consists in verifying that little noise (i.e. few false candidates) has been generated with the top-ranked terms output by the term extractor.
Requirement 2. The system should extract simple and complex terms. Although most of the works in this area only rely on multi-word terminological units (cf. Deane 2005; Wermter and Hahn 2005; among many others), ‘one cannot deny the fact that simple terms have a role to play’ (Wong, Liu and Bennamoun 2008: 503). In fact, Zhang, Iria, Brewster and Ciravegna (2008) showed that single-word terms can be as important as multi-word units and occupy a fairly large proportion in certain domains, so ‘algorithms that ignore single-word terms may cause problems to tasks built on top of Automatic Term Recognition (ATR)’ (Zhang et al. 2008: 2108).
Requirement 3. The system should extract the nouns, verbs and adjectives that are used to describe a given specialized domain. Most ATE systems have focused exclusively on noun phrases, under the assumption that they make up the bulk of the terminological inventory. As explained by Justeson and Katz (Reference Justeson and Katz1995 : 9), ‘judging from data in dictionaries of technical vocabulary, the majority of technical terms do consist of more than one word; among these, the overwhelming majority are noun phrases, which constitute the vast majority of multi-word terminological units in probably all domains’. Although this assumption still has a deep effect on some current research experiments (cf. Paulo and Mamede 2004; Pazienza, Pennacchiotti and Zanzotto 2005; Ittoo et al. 2010; Merkel, Foo and Ahrenberg 2013; Zadeh and Handschuh 2014a; Meyers et al. 2015; among others), it is also true that ‘verbs and adjectives, though they have received much less attention, can also be domain-specific’ (Ahrenberg 2009).
Requirement 4. The system should be able to be extended to other languages and other scientific/technical domains. It is reasonable to assume that this adaptability can be favoured if statistical knowledge-poor methods of term extraction are employed, which is in line with the natural language processing (NLP) techniques used in information retrieval, where simple methods such as stopwords and stemming usually yield more significant improvements than higher-level processing, e.g. chunking or parsing, which increase processing and even decrease precision (Brants 2004). However, this approach breaks away from that of mainstream term extractors, where a hybrid approach combines statistical methods with linguistic methods, where part-of-speech (POS) tagging plays a crucial role.
Requirement 5. The system should be provided with a user-friendly interface that can fully integrate the functionalities of corpus management, term extraction and term management, which are required for terminology and terminography research. In other words, if you want to go further than a local experiment performed with a toy implementation of the term extractor, you need a number of tools that can support the complete workflow of term processing in a single platform.
In this context, this research led to the design and development of DEXTER (Discovering and EXtracting TERminology), an online workbench for the recognition, validation and management of the simple and complex terms (namely, unigrams, bigrams and trigrams) extracted from non-structured texts in small- and medium-sized specialized corpora in English, French, Italian and Spanish.Footnote 2 DEXTER not only integrates all of the requirements outlined above but also introduces some improvements in the main modules of the ATE process (i.e. preprocessing, extraction, recognition, weighting and validation), since ‘although ATE has been researched for more than 20 years, there is still room for improvement’ (Conrado et al. 2014: 1). The remainder of this article is structured as follows. Section 2 describes the different stages involved in ATE with DEXTER. Section 3 deals with the evaluation of DEXTER in comparison with BioTex, GaleXtract, Termine and TermoStat. Finally, Section 4 concludes with a review of the five requirements from DEXTER’s perspective.
2 Automatic term extraction in DEXTER
2.1 Graphical user interface application
DEXTER has been developed as a Web-based user-interface application, which is not confined to term extraction in the strict sense but provides an environment suitable for linguistic research, with functionalities such as corpus compilation and textual exploration, among many others. Here, the ATE process consists of a pipeline of five stages: (1) corpus registration, (2) corpus development, (3) candidate extraction, including term recognition, (4) term weighting, and (5) term validation and clean-up. To illustrate the explanations in this and other sections, an experiment was conducted with an English corpus of 200 documents (312,710 tokens), where 1,452 unigrams, 2,143 bigrams and 711 trigrams were extracted as term candidates. This collection of documents, which we will call ‘sample corpus’, was downloaded from a website whose aim is to provide students with basic information about electronics.Footnote 3
2.1.1 Corpus registration
The domain-specific corpus is manually tagged with descriptors such as the name (e.g. Electronics corpus), the language (e.g. English), the content description (e.g. Tutorials for engineering students on all aspects of basic electronics), and the true and false domain(s), which are selected from the IATE database (InterActive Terminology for Europe). With respect to the latter, users must select the ‘true domains’, i.e. the most relevant field(s) of specialized knowledge described in the corpus (e.g. Electrical Industry [6621001] and Electronics and Electrical Engineering [6826, 6826001, 6826002]),Footnote 4 and can also select some ‘false domains’, which serve to discard term candidates that, although they are likely to occur in the corpus, pertain to other specialized domains (e.g. Chemistry [6811, 6811001, 6811002, 6811004], Computer Science [3236001, 3236002], Mechanical Engineering [6821, 6821001, 6821002, 6821003] and Science [36]). True and false domains play an important role in both term recognition and term weighting.
2.1.2 Corpus development
This stage consists of two actions. First, the corpus is compiled. DEXTER has been designed for small and medium-sized corpora (i.e. up to one million tokens). As stated by Koester (2010: 68–69), specialized corpora do not need to be ‘as large as more general corpora to yield reliable results’; since specialized corpora are ‘carefully targeted, they are more likely to reliably represent a particular register or genre than general corpora’. Another feature of the corpus is its sample size, which is closely related to the issue of representativeness. For example, ‘a corpus of a million words or so cannot afford to include whole books which might be up to 100,000 words in length’ (Hunston 2008: 165), since such large documents could result in a disproportionate composition of the corpus. Although the suitability of a sample size depends on the specific task that is undertaken, ‘experience with samples of 20,000 words has shown that on the whole these are sufficiently large to yield statistically reliable results on frequency and distribution’ (Haan 1992: 3), which helped us determine the maximum number of tokens in a single document (i.e. 25,000 tokens). Apart from plain-text files, PDF documents and HTML Web pages can also be uploaded, which are automatically converted into TXT files. Each document can also be manually tagged with descriptors such as the title (e.g. Transistor biasing) and the content description (e.g. Transistor base biasing configurations from a single supply available for a common emitter amplifier).
Second, corpus indexation is performed with Lucene.Net (Hatcher, Gospodnetic and McCandless Reference Hatcher, Gospodnetic and McCandless2010)—the C# port of Lucene, one of the most popular open-source library for high-performance information retrieval. Document indexation enables the user to retrieve the context of all the words derived from a given stemmed ngram. A naïve approach of searching a certain word would have been to sequentially scan each text file for the target word, but this rudimentary method could have generated a bottleneck in case of a large collection of documents.
2.1.3 Candidate extraction and term recognition
This stage consists of four actions. First, each document in the corpus is preprocessed. Each document is tokenized by a simple analyzer, which splits tokens at non-letter characters and then lowercases each token, discarding numbers from the token stream. Then, the tokens are processed by the Snowball stemmer, and unigrams, bigrams and trigrams are derived from the stems. The default frequency threshold is set to three. As usual, a threshold is applied not only to reduce noise but also ‘to avoid producing a too long list that might become a hindrance for the experts evaluating the output’ (Marín Reference Marín2015 : 7).
Second, term recognition is performed with IATE, i.e. the identification of known terms by comparing the list of candidates with a term database. Today there are a number of terminology data banks that are both multidisciplinary and multilingual, such as Le grand dictionnaire terminologique,Footnote 5 TERMIUM Footnote 6 or TermSciences.Footnote 7 For a platform such as DEXTER, one of the best options of gold standard is IATE, because it provides a large coverage of words (i.e. 8.5 million terms) from a high number of domains and subdomains (i.e. 21 domains that are hierarchically structured into 649 subdomains) in a high number of languages (i.e. 24 languages). The main goal of this term database, which can be downloaded in the XML-based TermBase eXchange format,Footnote 8 is to ensure the quality of the terminology employed in the documentation of the EU institutions. In this stage, DEXTER can recognize (a) a large number of extracted ngrams that pertain to the true domains of the corpus, which are automatically tagged as positive candidates, and (b) those ngrams that pertain to the false domains, which are eliminated from the term-candidate list. False domains enable DEXTER to deal effectively with multi-domain discourse, which is particularly common in scientific and technical writing. For instance, in the sample corpus, the system recognized many lexical units typically used in other specialized domains, such as cadmium sulphide (chemistry), star network (computer science), root mean square (mathematics) or absolute position encoder (mechanical engineering). As a result, to make the list of ngrams more domain-focused (i.e. electronics), the stems pertaining to these false domains (i.e. chemistry, computer science, mathematics and mechanical engineering) were discarded just before term weighting, providing that the stem did not belong to some true domain.
Third, DEXTER makes use of a predefined list of functional stopwords (e.g. for, in, the, etc.). Moreover, the system dynamically generates a list of common stopwords (e.g. allow, high, know, type, etc.) that is tailor-made for the domain-specific corpus.Footnote 9 Whereas stopword lists are usually obtained from general corpora (cf. Fox Reference Fox1990) or elaborated for a few scientific domains in a given language (cf. Jacquey, Tutin, Kister, Jacques, Hatier and Ollinger 2013), DEXTER is provided with a multilingual and multi-domain method to automatically discover common stopwords in specialized corpora, as explained in Section 2.3.1.
Finally, a set of lexical filters, which are applied to the stemmed ngrams that were not recognized as relevant domain-specific terms in the second step of this stage, serves to discard many candidates for further analysis. For example, the following types of stemmed ngrams are ignored:
-
• Ngrams containing one single character (e.g. D in ‘In the above circuit, node D is chosen as. . .’)
-
• Ngrams containing one or more non-alphabetical characters (e.g. V1 in ‘Consider two AC voltages, V1 having a peak voltage of . . .’)
-
• Unigrams matching a functional or common stopword (e.g. call in ‘These audio signal transformers are called ‘matching transformers’ . . .’).
-
• Unigrams derived from words starting with a capital letter, provided that they do not belong to the general domain (e.g. Schmitt in ‘. . .the higher threshold value of the Schmitt trigger. . .’); this unsophisticated procedure is intended to identify domain-specific named entities, so a more powerful named-entity recognizer will be used in future versions.
-
• Bigrams containing one or two functional stopwords (e.g. the AC in ‘. . .an important parameter of the AC waveform. . .’)
-
• Bigrams or trigrams containing at least one common stopword (e.g. short length in ‘. . .a short length of wire designed to. . .’)
-
• Trigrams containing a functional stopword at the beginning and/or the end of the ngram (e.g. the form of in ‘. . .in the form of infra-red radiation.’)
-
• Trigrams containing a functional stopword in the mid-position, provided that the stopword is not a preposition (e.g. volt or amp in ‘. . .the signal waveform measured in volts or amps.’)
It should be noted that these shallow lexical filters cannot be equated to standard grammatical patterns used for term-candidate detection (cf. Justeson and Katz Reference Justeson and Katz1995), since the former do not actually require that the content of the corpus be annotated by POS taggers or parsers. As a result, the above lexical filters can be extended to other languages more easily than grammatical-pattern matching techniques, which are heavily language-dependent. Therefore, DEXTER does not adopt a traditional hybrid approach to term extraction, since the linguistic properties of words are not taken into account.
2.1.4 Term weighting
DEXTER employs a parameterized composite metric called SRC (Periñán-Pascual Reference Periñán-Pascual2015), which is grounded on the notions of salience (S), relevance (R) and cohesion (C):
\begin{equation}
SRC(g) = S(g) * \alpha + R(g) * \beta + C(g) * \gamma
\end{equation}
2.1.5 Term validation and candidate clean-up
Term weighting outputs an inventory of SRC-ranked candidates from which users are expected to eliminate irrelevant ngrams. As stated by Oakes (Reference Oakes1998), deleting false positives is a much simpler task than creating the whole list of terms manually, and this validation task becomes much easier when some of the terms have already been recognized with the help of a term database (e.g. IATE). For this reason, it can be said that DEXTER adopts a hybrid approach to the evaluation of term candidates. The semi-automatic process of term evaluation is further described in Section 2.3.3.
2.2 Web service
DEXTER has also been developed as a SOAP/WSDL-based Web service through which you can upload a zipped corpus of TXT documents, together with information about the language and the true and false domains of the corpus. In this case, the system provides a limited set of the features available in the graphical user interface (GUI) application, resulting in an ATE method that consists of only two stages: (1) candidate extraction, including term recognition, and (2) term weighting, as described in Figure 1. Both stages were briefly outlined above. Detailed descriptions of the actions [E] and [G] are given in Sections 2.3.1 and 2.3.2, respectively.
DEXTER Web service: workflow of term extraction.
2.3 Some issues on DEXTER
After a brief description of the stages involved in the term-extraction process within DEXTER, this section provides a detailed account of its most innovative aspects.
2.3.1 The removal of stopwords
A stopword list is a valuable resource in text-data analysis in general, and in information retrieval, information extraction, document clustering and document categorization in particular. Stopwords are ‘words having no significant semantic relation to the context in which they exist’ (Khosrow-Pour Reference Khosrow-Pour2009 : 3112), so they should be removed before processing starts in order to achieve greater effectiveness. The selection of this type of words actually becomes a crucial factor in term extraction, since ‘proper choice and construction of a stoplist can affect results in a way that depends on the task at hand’ (Sinka and Corne 2003: 401). However, despite their importance, stopwords are not so carefully selected as would be expected.
In this search for non-significant words, stopword lists usually consist of two categories of lexical items: (a) functional words, and (b) common words. The task to discover functional words is a non-issue, since they can be easily obtained from the grammar of the language. For example, in the case of (a) in DEXTER, 454 stems were detected for English, including not only functional words (e.g. articles, conjunctions, determiners, prepositions and pronouns, among others) but also Arabic and Roman numerals and common abbreviations used in professional and academic documents (e.g. i.e., c.f., etc or et al). In the case of (b), however, there is no generally accepted list of non-informative words for every language. Although there are a number of stopword lists scattered over the Web, they tend to include only functional words. Moreover, stopword lists have been traditionally hand-crafted from the author’s experience. This situation is further compounded by data obsolescence, as illustrated by one of the most popular stopword lists (cf. Fox Reference Fox1990), which was defined from the Brown Corpus of English (Francis and Kučera Reference Francis and Kučera1982) more than twenty-five years ago. In this context, we developed an adaptive method to automatically identify stopwords in domain-specific corpora. This method is described in the remainder of this section.
In the last decade, most of the NLP research that includes some ATE component has employed stopword lists that were constructed from one of the following methods:
-
• a multilingual and multi-domain method, which automatically generates stopword lists for different languages by means of entropy-based metrics applied to specialized corpora (cf. Sinka and Corne 2003; Zou et al. 2006; Alajmi, Saad and Darwish Reference Alajmi, Saad and Darwish2012; Asubiaro Reference Asubiaro2013),
-
• a monolingual and multi-domain method, which provides a stopword list derived from the general corpus of a given language; in this regard, Salton (Reference Salton1971) and Fox (Reference Fox1990) have been the most popular lists for English (cf. Karystianis, Buchan and Nenadic 2014; Zadeh and Handschuh 2014b; Sajjacholapunt and Joy 2015; Conde et al. Reference Conde, Larrañaga, Arruarte, Elorriaga and Roth2016),Footnote 10 or
-
• a monolingual and single-domain method, where the stopword list is intended for a given specialized domain in a given language (cf. Jacquey et al. 2013), thus substantially restricting the scope of applicability.
Our interest lies primarily in the first type of method, since this research has been developed in a multilingual environment that was designed to process document collections from a wide variety of scientific and technical domains. As shown below, however, a more effective approach to stopword detection is required.
Research on the automatic generation of stopword lists usually aims to measure the importance of a term within a document collection by means of entropy. In this regard, the method typically used to calculate the normalized value of entropy is that of Lochbaum and Streeter (Reference Lochbaum and Streeter1989 : 672):
\begin{equation}
entrop{y_k} = 1 - \frac{{nois{e_k}}}{{{{\log }_2}ndocs}}
\end{equation}
\begin{equation}
nois{e_k} = \sum \limits _{i = 1}^{ndocs} {\frac{{Fre{q_{ik}}}}{{Totalfre{q_k}}}} {\log }_2\frac{{Totalfre{q_k}}}{{Fre{q_{ik}}}}
\end{equation}
As can be concluded from the above, entropy is not a reliable metric to discriminate common stopwords in specialized corpora, so we chose to develop a method based on the distribution of word frequencies in a general corpus. In particular, we used Sun, Shaw and Davis’s (1999: 285) transition point as ‘a threshold value establishing a mark between high-frequency and low-frequency words’, which can be predicted with the following equation:
\begin{equation}
n = \sqrt{D}
\end{equation}
\begin{equation}
{\rm {DEXTER}}: = \left\lbrace \begin{array}{l}\left( {{\rm {CORPU}}{{\rm {S}}_{gen}};\left\lbrace {{\rm {STEM}}{\rm {,FREQ}}} \right\rbrace } \right),\\
\left( {{\rm {CORPU}}{{\rm {S}}_{spe}};\left\lbrace {{\rm {STEM}}{\rm {,FREQ}}} \right\rbrace } \right),\\
\left( {{\rm {STO}}{{\rm {P}}_{fun}};\left\lbrace {{\rm {STEM}}} \right\rbrace } \right),\\
\left( {{\rm {IAT}}{{\rm {E}}_{con}};\left\lbrace {{\rm {ID\_CONCEPT}}{\rm {,ID\_DOMAIN}}} \right\rbrace } \right),\\
\left( {{\rm {IAT}}{{\rm {E}}_{ter}};\left\lbrace {{\rm {ID\_CONCEPT}}{\rm {,STEM}}} \right\rbrace } \right) \end{array} \right\rbrace
\end{equation}
A database scheme D := {R 1,. . .,R n} is usually defined as a set of relation schemes, where a relation scheme R;{A 1,. . .,A n} consists of a finite set of attributes. For example, CORPUSgen and CORPUSspe refer to the general corpus (46,128 stems) and the specialized corpus (3,330 stems), respectively, STOPfun is the list of functional stopwords (454 stems), and IATEcon and IATEter hold 1,842,937 concepts and 256,502 English stemmed unigrams categorized by the IATE database. Moreover, given S⊆R, let t↓{S} denote the restriction of a tuple t over R on S. For example, if the relation scheme is CORPUSgen;{STEM, FREQ} and g = (transistor, 1196) is a tuple over CORPUSgen, then g ↓ {STEM} = (transistor). The remaining symbols in (5)–(10) are used in the standard notation for set theory and symbolic logic. The complexity of the actual database design is certainly underspecified in the scheme (5), which includes only those relations that are relevant for the issues of this section.
The first point to make is that DEXTER takes the stem as the minimum unit of processing, so the inventory of word forms in the general corpus was reduced to 46,128 stems.Footnote 12 Thus, CORPUSgen held these stems and their frequencies. It was found that the transition point n was 215 for this distribution of stemmed unigrams, so DEXTER selected all the non-functional stems whose frequency was within the range from n back to rank 1. Formally, this can be expressed as in the function (6), i.e. the value of STEM in every tuple g of CORPUSgen, where FREQ in g is greater or equal to 215 and there exists no tuple f in STOPfun that contains the value of STEM, where COMMONgen returned 4,195 stems.
\begin{equation}
{\rm {COMMO}}{{\rm {N}}_{gen}}: = \left\lbrace \begin{array}{l}g \downarrow \left\lbrace {{\rm {STEM}}} \right\rbrace |g \in {\rm {DEXTER}}\left( {{\rm {CORPU}}{{\rm {S}}_{gen}}} \right) \wedge \\
g\left( {{\rm {FREQ}}} \right) \ge 215 \wedge \\
\lnot \left( {\exists f \in {\rm {DEXTER}}\left( {{\rm {STO}}{{\rm {P}}_{fun}}} \right):f\left( {{\rm {STEM}}} \right) = g\left( {{\rm {STEM}}} \right)} \right) \end{array} \right\rbrace
\end{equation}
On the other hand, the same procedure was applied to the sample corpus (i.e. CORPUSspe), where n was 64. In this case, COMMONspe returned 358 stems, as described in the function (7), i.e. the value of STEM in every tuple s of CORPUSspe, where FREQ in s is greater or equal to 64 and there exists no tuple f in STOPfun that contains the value of STEM.
\begin{equation}
{\rm {COMMO}}{{\rm {N}}_{spe}}: = \left\lbrace \begin{array}{l}s \downarrow \left\lbrace {{\rm {STEM}}} \right\rbrace |s \in {\rm {DEXTER}}\left( {{\rm {CORPU}}{{\rm {S}}_{spe}}} \right) \wedge \\
s\left( {{\rm {FREQ}}} \right) \ge 64 \wedge \\
\lnot \left( {\exists f \in {\rm {DEXTER}}\left( {{\rm {STO}}{{\rm {P}}_{fun}}} \right):f\left( {{\rm {STEM}}} \right) = s\left( {{\rm {STEM}}} \right)} \right) \end{array} \right\rbrace
\end{equation}
It should be noted that this adaptive procedure to discover common stopwords does only take into account stemmed unigrams, because they are the ones used as the building blocks in lexical filters, which in turn allow the system to discard not only unigrams but also bigrams and trigrams before term weighting.
At this point, it is worthwhile to recall that, when the sample corpus was registered, some descriptors were provided (cf. Section 2.1.1). In particular, the most-relevant IATE domains manually chosen for the sample corpus were Electrical Industry [6621001], Electronics and Electrical Engineering [6826], Electrical Engineering [6826001] and Electronics Industry [6826002]. Therefore, to complete the remaining tasks, it was necessary to identify the 6,863 stems pertaining to such ‘true domains’ in IATE, as described in the function (8), i.e. the value of STEM in every tuple t of IATEter where there exists a tuple c in IATEcon whose ID_CONCEPT is the same as ID_CONCEPT in t and ID_DOMAIN in c is 6621001, 6826, 6826001 or 6826002.
\begin{equation}
{\rm {IAT}}{{\rm {E}}_{electro}}: = \left\lbrace \begin{array}{l}t \downarrow \left\lbrace {{\rm {STEM}}} \right\rbrace |t \in {\rm {DEXTER}}\left( {{\rm {IAT}}{{\rm {E}}_{ter}}} \right)\\
\wedge \left( \begin{array}{l}\exists c \in {\rm {DEXTER}}\left( {{\rm {IAT}}{{\rm {E}}_{con}}} \right):\\
c\left( {{\rm {ID\_CONCEPT}}} \right) = t\left( {{\rm {ID\_CONCEPT}}} \right) \wedge \\
c\left( {{\rm {ID\_DOMAIN}}} \right) \in \left\lbrace \begin{array}{l}6621001,6826,\\
6826001,6826002 \end{array} \right\rbrace \end{array} \right) \end{array} \right\rbrace
\end{equation}
At this stage, the system has five sets available–i.e. CORPUSgen, COMMONgen, CORPUSspe, COMMONspe and IATEelectro, where COMMON gen ⊂CORPUS gen and COMMON spe ⊂CORPUS spe . The core issue now is to decide how these sets can be used to recognize most of the common stopwords in the sample corpus (i.e. STOPcom). In this regard, two possible methods can be applied, which return the sets consisting of all the stems that occur in both COMMONgen and the set resulting from the stems of COMMONspe (9a) or CORPUSspe (9b) that do not belong to IATEelectro.
\begin{equation}
STOP{^{\prime }_{com}} = {\rm {COMMO}}{{\rm {N}}_{gen}} \cap \left( {{\rm {COMMO}}{{\rm {N}}_{spe}} - {\rm {IAT}}{{\rm {E}}_{electro}}} \right)
\end{equation}
\begin{equation}
STOP^{\prime }{^{\prime }_{com}} = {\rm {COMMO}}{{\rm {N}}_{gen}} \cap \left( {{\rm {CORPU}}{{\rm {S}}_{spe}} - {\rm {IAT}}{{\rm {E}}_{electro}}} \right)
\end{equation}
More specifically, these two methods correspond to the functions (10a) and (10b) respectively, which return the value of STEM in every tuple r of COMMONspe (10a) or CORPUSspe (10b) providing that there exists a tuple g in COMMONgen that contains the value of STEM but there exists no tuple i in IATEelectro that contains the value of STEM.
\begin{equation}
{\rm {STOP}}{{\rm {^{\prime }}}_{com}}: = \left\lbrace {r \downarrow \left\lbrace {{\rm {STEM}}} \right\rbrace \left| \begin{array}{l}r \in {\rm {DEXTER}}\left( {{\rm {COMMO}}{{\rm {N}}_{spe}}} \right) \wedge \\
\left( \begin{array}{l}\exists g \in {\rm {DEXTER}}\left( {{\rm {COMMO}}{{\rm {N}}_{gen}}} \right):\\
g\left( {{\rm {STEM}}} \right) = r\left( {{\rm {STEM}}} \right) \end{array} \right) \wedge \\
\lnot \left( \begin{array}{l}\exists i \in {\rm {DEXTER}}\left( {{\rm {IAT}}{{\rm {E}}_{electro}}} \right):\\
i\left( {{\rm {STEM}}} \right) = r\left( {{\rm {STEM}}} \right) \end{array} \right) \end{array} \right.} \right\rbrace
\end{equation}
\begin{equation}
{\rm {STOP^{\prime }}}{{\rm {^{\prime }}}_{com}}: = \left\lbrace {r \downarrow \left\lbrace {{\rm {STEM}}} \right\rbrace \left| \begin{array}{l}r \in {\rm {DEXTER}}\left( {{\rm {CORPU}}{{\rm {S}}_{spe}}} \right) \wedge \\
\left( \begin{array}{l}\exists g \in {\rm {DEXTER}}\left( {{\rm {COMMO}}{{\rm {N}}_{gen}}} \right):\\
g\left( {{\rm {STEM}}} \right) = r\left( {{\rm {STEM}}} \right) \end{array} \right) \wedge \\
\lnot \left( \begin{array}{l}\exists i \in {\rm {DEXTER}}\left( {{\rm {IAT}}{{\rm {E}}_{electro}}} \right):\\
i\left( {{\rm {STEM}}} \right) = r\left( {{\rm {STEM}}} \right) \end{array} \right) \end{array} \right.} \right\rbrace
\end{equation}
Both methods are based on the same two premises. On the one hand, it can be assumed that the stopwords in the sample corpus should be derived from the top-ranked words in the general corpus. The next issue to be considered is the search space in the sample corpus, i.e. COMMONspe in the function (10a) or CORPUSspe in the function (10b). Indeed, this is a critical decision, since the effectiveness of term extraction is closely related to the aggressivity of candidate-space reduction by stopwords. For example, the functions (10a) and (10b) resulted in |STOP′com| = 114 and |STOP″com| = 942, respectively. Both sets were manually reviewed for evaluation, whose results are displayed in Table 1.
Evaluation of stopwords for DEXTER
It should be noticed that recall was evaluated by taking into account all stems in both stopword lists that were not related to the given specialized domain; considering that STOP′ com ⊆STOP′′ com , then the latter has no false negatives. Table 1 demonstrates that STOP”com, i.e. the most aggressive stopword-removal method, was actually the most effective one.
On the other hand, it is also reasonable to believe that stopwords can only be found among the stems that do not belong to any of the IATE true domains of the sample corpus. This assumption raises a non-trivial issue, since neonyms are not the only type of term to appear in domain-specific corpora. According to ISO 704 (2009), terms are formed by applying one of the following methods: (a) creating neoterms (i.e. newly coined technical or scientific words and phrases), (b) using forms that exist in the general language (e.g. through processes such as conversion, terminologization, semantic transfer and transdisciplinary borrowing), or (c) translingual borrowing (i.e. existing terms in one language can be introduced into another language). Therefore, it was found that terminologized words such as face, state or turn (Table 2), which were all present in both COMMONgen and CORPUSspe, became a major source of error that was eliminated with the aid of the term database. Indeed, if the IATE-based component of the functions (10a) and (10b) had been ignored, then |STOP′com| = 263 and |STOP″com| = 1438, but the error rates would have been 0·57414 and 0·35396, respectively.
Instances of terminologization
It should be recalled that the goal is not to automatically build a stopword list for a given specialized domain but only to discover the stopwords that are present in a given domain-specific corpus. To this end, the next task was to evaluate this adaptive method against the static method of applying a list of 571 stopwords used in the SMART information retrieval system (Salton Reference Salton1971) and a list of 421 stopwords proposed by Fox (Reference Fox1990). In this case, the number of stems from non-functional words was 942 for DEXTERstop_com (or STOP″com), 163 for SMARTstop_com and 89 for FOXstop_com. Assuming that the system should ideally detect all and only those stems in these three lists that are not relevant to the sample corpus, the evaluation of DEXTER, SMART and FOX provided the results shown in Table 3. This experiment demonstrates that our stopword-removal method is not only the most aggressive but also the most effective, i.e. high precision and recall with respect to other stopword lists.
Evaluation of stopword lists
Finally, we can assess the impact that DEXTERstop_com has on the processing of the sample corpus. In this regard, we highlight the capacity of our method to substantially reduce the number of irrelevant term candidates, contributing ultimately to the increase in precision. It should be recalled that two of the lexical filters used by DEXTER are based on common stopwords; particularly, (a) the unigrams that match a common stopword and (b) the bigrams or trigrams that contain at least one common stopword are eliminated. Suppose that neither of these two lexical filters is applied. In this case, the inventory of candidates would have increased by 39·34 per cent for unigrams (i.e. from 1,452 to 2,394 candidates), 27·03 per cent for bigrams (i.e. from 2,143 to 2,937 candidates) and 50·03 per cent for trigrams (i.e. from 711 to 1,423 candidates). Figure 2 shows the distribution of the common stopwords as unigrams or as part of bigrams and trigrams in the sample corpus. In the range [1–200], for example, 24 unigrams (e.g. calculate, equation or symbol) would have been recognized as common stopwords, and 64 bigrams (e.g. common use, gate symbol or previous tutorial) and 97 trigrams (e.g. amount of current, ideal for use, nearest preferred value or temporary storage device) would have consisted of one or more common stopwords. Considering that none of these unigrams and bigrams and only eight of the trigrams (i.e. current-limiting resistor, free-wheeling diode, high-pass filter, high-wave rectifier, liquid-crystal display, low-pass filter, magneto-motive force and push-pull amplifier) are true terms, these stopwords would have contributed 0·12, 0·32 and 0·44 to the false-discovery rate in the top-ranked 200 unigrams, bigrams and trigrams, respectively, affecting the precision in this range of candidates.
Distribution of common words.
Therefore, it can be concluded that DEXTER is provided with an effective stopword-detection method that can be adapted not to a few large areas of knowledge (e.g. economics, law, science or technology) but to the whole range of highly-specialized fields of expertise (e.g. civil law, accounting system, space science or mechanical engineering) in twenty-four languages that are represented in IATE.
2.3.2 The SRC metric
The research literature has demonstrated that the combination of multiple ATE algorithms tends to outperform most of the methods that consider only one statistical feature (cf. Zhang et al. 2008; Fedorenko, Astrakhantsev and Turdakov 2013). Indeed, the recent trend does not seek to devise new measures for unithood and/or termhood but to combine statistical features effectively, where
‘Unithood’ refers to the degree of strength or stability of syntagmatic combinations or collocations. (. . .) On the other hand, termhood refers to the degree that a linguistic unit is related to (or more straightforwardly, represents) domain-specific concepts. (Kageura and Umino Reference Kageura and Umino1996 : 260-261)
Moreover, a metric with adjustable parameters allows the system to accommodate to the configuration of the document collection. However, unlike most of the research in ATE, where single metrics are usually combined indiscriminately to produce the best results, SRC is grounded on the theoretical principles of salience, relevance and cohesion. Periñán-Pascual (Reference Periñán-Pascual2015) presented a detailed description of these terminological features.
On the one hand, one of the pillars of SRC is the notion of salience, which is based on the termhood measure tf-idf (Salton, Wong and Yang Reference Salton, Wong and Yang1975; Salton, Yang and Yu Reference Salton, Yang and Yu1975; Salton and Buckley Reference Salton and Buckley1988), i.e. the weight of a term is determined by the relative frequency of the term in a certain document (or term frequency, i.e. tf) compared with the inverse proportion of that term in the whole document collection (or inverse document frequency, i.e. idf). This decision is supported by the fact that the task of automatic document indexation has a clear point in common with that of ATE, since the keywords employed to index a given domain-specific document are usually perceived as terminological units (Pazienza et al. 2005). In this regard, the two most popular weighting measures in automatic keyword extraction are tf-idf and Okapi BM25 (Robertson, Walker and Beaulieu 1998). Thus, the salience of the stemmed ngram g in the document d is calculated in DEXTER by applying the following formula:
\begin{equation}
{S_d}\left( g \right) = TF(g) * IDF\left( g \right) * NORM\left( g \right)
\end{equation}
\begin{equation}
TF\left( g \right) = {f_d}\left( g \right)
\end{equation}
\begin{equation}
IDF\left( g \right) = 1 + \log \left( {\frac{{{N_T}}}{{df\left( g \right)}}} \right),{\rm {where\:}}{ df\left( g \right) > 0}
\end{equation}
\begin{equation}
NORM\left( g \right) = \frac{1}{{\sqrt{\sum \nolimits _{g \in d} {{{\left( {TF\left( g \right) \times IDF\left( g \right)} \right)}^2}} } }}
\end{equation}
The rationale for the three components of the equation (11a) is described as follows. First, TF(g) serves to justify the fact that more weight is given to those ngrams that appear many times in a given document. Second, IDF(g) rewards those ngrams that are concentrated just in a few documents of the corpus. Thus, the value of a rare ngram in the corpus is high, whereas the value of a frequent ngram is low; in other words, less weight is given to ngrams that appear in many documents. Third, the document size is a parameter which can dramatically affect the calculation of weights, since (a) long documents usually use the same ngrams repeatedly, and (b) long documents have numerous different ngrams (Singhal, Buckley and Mitra 1996). Therefore, NORM(g), i.e. document length normalization of ngram weights, is used to remove the advantage of long documents: less weight is given to documents that contain many ngrams. In other words, the normalization factor makes all documents be treated equally important regardless of their size.
The salience of ngrams with respect to the whole target corpus, and not just to a single document, can be calculated as follows:
\begin{equation}
S\left( g \right) = \frac{{\sum \nolimits _{d \in C{P_T}} {{S_d}\left( g \right)} }}{{\sqrt{\sum \nolimits _{{g_j} \in C{P_T}} {{{\left( {S\left( {{g_j}} \right)} \right)}^2}} } }}
\end{equation}
Again, the normalization factor of this formula only takes into account ngrams of the same type.
On the other hand, salience is combined with a measure that quantifies the relevance of ngrams through the contrastive analysis between the target corpus and a reference corpus. Salience measures the prevalence of the term in a particular target domain, but it does not serve to reflect the tendency of term usage across different domains, that is, the relevance of the term. In this context, relevance is calculated in DEXTER as follows, which results from an adaptation of Ahmad, Gillam and Tostevin’s weirdness (2000):
\begin{equation}
R{\left( g \right)^{^{\prime \prime }}} = \frac{{{P_T}\left( g \right)}}{{{P_R}\left( g \right)}}
\end{equation}
\begin{equation}
{P_T}\left( g \right) = \frac{{{f_T}\left( g \right)}}{{\left| {C{P_T}} \right|}},{\rm {iff}}\left| g \right| = 1;{\rm {otherwise}},{P_T}\left( g \right) = \frac{{\@root \left| g \right| \of {{\prod \nolimits _{{k_i} \in g} {{f_T}\left( {{k_i}} \right)} }}}}{{\left| {C{P_T}} \right|}}
\end{equation}
\begin{equation}
{P_R}\left( g \right) = \frac{{{f_R}\left( g \right)}}{{\left| {C{P_R}} \right|}},{\rm {iff}}\left| g \right| = 1;{\rm {otherwise}},{P_R}\left( g \right) = \frac{{\@root \left| g \right| \of {{\prod \nolimits _{{k_i} \in g} {{f_R}\left( {{k_i}} \right)} }}}}{{\left| {C{P_R}} \right|}}
\end{equation}
\begin{equation}
R\left( g \right) = 1 - \frac{1}{{{{\log }_2}\left( {2 + R{{\left( g \right)}^{^{\prime \prime }}}} \right)}}
\end{equation}
Finally, the notion of cohesion was introduced to determine the unithood of complex ngrams. Therefore, whereas salience and relevance serve to measure termhood, cohesion is aimed at quantifying the degree of stability of bigrams and trigrams. Although many association measures have been employed for unithood, such as Φ2 (Gale and Church 1991), χ2 (Nagao, Mizutani and Ikeda Reference Nagao, Mizutani and Ikeda1976), cubic mutual information (Vivaldi, Màrquez and Rodríguez 2001), Dice coefficient (Smadja, McKeown and Hatzivassiloglou Reference Smadja, McKeown and Hatzivassiloglou1996), log likelihood (Dunning Reference Dunning1994), log odds ratio (Everitt Reference Everitt1992), mutual expectation (Silva et al. 1999), (pointwise) mutual information (Church and Hanks Reference Church and Hanks1990), symmetric conditional probability (Silva and Lopes 1999) or t-score (Church et al. 1991), Park, Byrd and Boguraev (2002: 5) reminded us that these measures have two major drawbacks:
First, they evaluate the degree of association between two units and need to apply special techniques to calculate the association of terms with more than two words (. . .). Second, these measures tend to give higher values for low frequency terms, especially mutual information.
Moreover, Korkontzelos, Klapaftis and Manandhar (2008: 249) showed that
approaches which take into consideration the nestedness of a candidate term into others (. . .) have in general superior performance over methods which measure the strength of association among the tokens of a multi-word candidate term.
In this regard, one of our first options was to consider C-value (Frantzi and Ananiadou 1996; Frantzi, Ananiadou and Mima Reference Frantzi, Ananiadou and Mima2000), which is calculated as follows:
\begin{equation}
\textrm {C-value}\left( g \right) = {\log _2}\left| g \right|*{f_T}\left( g \right),{\textrm { iff g is not nested}}
\end{equation}
\begin{equation}
\textrm {C-value}\left( g \right) = {\log _2}\left| g \right|*\left( {{f_T}\left( g \right) - NST\left( g \right)} \right),{\textrm { otherwise}}
\end{equation}
\begin{equation}
NST\left(g\right) = \frac{{\mathop \sum \nolimits _{b \in {S_g}} {f_T}\left(b\right)}}{{\left|{{S_g}}\right|}}
\end{equation}
However, C-value presents a problem in the weighting of bigrams. In particular, in case of a bigram not appearing as a nested sequence of longer ngrams, C-value is equal to the frequency of the bigram, since log2(2) = 1 and the NST component is not taken into consideration. In this context, C-value is incapable of adequately measuring the unithood of the bigrams that do not occur as part of longer ngrams in the target corpus, being unable to discriminate complex terms from recurrent co-occurrences of unigrams with their collocates. It should be noted that most of the research with C-value has focused on long-term candidates in the medical domain, where 4-gram and even 5-gram candidates are relatively frequent, e.g. adenoid cystic basal cell carcinoma (cf. Frantzi et al. 2000). However, this degree of complexity is certainly infrequent in some other specialized domains.
In the light of this evidence, cohesion is calculated in DEXTER as follows, which results from an adaptation of Park et al.’s Term Cohesion (2002):
\begin{equation}
C{\left( g \right)^{^{\prime \prime }}} = \frac{{{f_T}\left( g \right)}}{{\@root \left| g \right| \of {{\prod \nolimits _{{k_i} \in g} {{f_T}\left( {{k_i}} \right)} }}}} \times F,{\rm {iff}}\left| g \right| > 1
\end{equation}
\begin{equation}
F = \left\lbrace {\begin{array}{*{20}{c}}1,&{{\rm {iff }}{f_T}\left( g \right) = 1}\\
{{{\log }_2}\left( {{f_T}\left( g \right)} \right),}&{{\rm {iff }}{f_T}\left( g \right) > 1} \end{array}} \right\rbrace
\end{equation}
\begin{equation}
C\left( g \right) = 1 - \frac{1}{{{{\log }_2}\left( {2 + C{{\left( g \right)}^{^{\prime \prime }}}} \right)}},{\rm {iff}}\left| g \right| > 1
\end{equation}
Therefore, DEXTER integrates the above terminological features through the following SRC equation:
\begin{equation}
SRC\left( g \right) = termhood\left( g \right) + unithood\left( g \right)
\end{equation}
\begin{equation}
termhood\left( g \right) = S\left( g \right) * \alpha + R\left( g \right) * \beta
\end{equation}
\begin{equation}
unithood\left( g \right) = \left\lbrace {\begin{array}{*{20}{c}}0,&{{\rm {iff }}\left| g \right| = 1}\\
{C\left( g \right) * \gamma ,}&{{\rm {iff }}\left| g \right| > 1} \end{array}} \right\rbrace
\end{equation}
Many researchers agree that ‘it is reasonable to expect that there will be no ‘best’ ATR method which would outperform others on all data sets and in all circumstances’ (Knoth et al. 2009: 84), so it is also reasonable to expect that there will be no predefined combination of constant values in SRC, which would outperform others on all data sets and in all circumstances. In this regard, DEXTER can automatically discover the most suitable weights for the SRC coefficients after term recognition with IATE. Thus, taking into account all the permutations of these coefficients, the system gets the combination that provides (a) the highest precision with the top-ranked 200 ngrams, and (b) the most gradual distribution of the terms with respect to four cut-off points (i.e. 50, 100, 150 and 200) along the top-ranked 200 ngrams. In particular, DEXTER examines eleven permutations of the coefficients to calculate the best distribution of unigrams; and in the case of bigrams or trigrams, the permutations are sixty-six. This task can be performed only when the ngrams found in the IATE true domains have been automatically tagged as positive candidates. For example, Table 4 shows how terms are distributed among the unigrams of the four ranges in the sample corpus before human validation. In this case, DEXTER easily discovers that the highest precision is obtained with the first combination in Table 4, i.e. [α = 1; β = 0]. However, when there is no single winning combination, the next step consists in determining which distribution shows the most gradual distribution of the terms along the top-ranked 200 ngrams.
SRC coefficients and term distributions (sample corpus)
The assumption of the gradual distribution of terms is based on what Pazienza et al. (2005: 270) described as ‘the power of each measure in discriminating true and false terms’, i.e. true terms should be assigned to the highest positions in the rank, while the remaining false terms concentrate closer to the bottom of the list. Consequently, although the values of the SRC coefficients can be determined in part by the gradual distribution of only the top-ranked 200 candidates, a decreasing precision trend is expected to be shown in the remaining candidates, so that the discriminating power of the metric produces a significant ranking. In DEXTER, this notion of ‘gradual distribution of terms’ plays a key role when the permutations of the values of the SRC coefficients do not result in a winning combination. One of the most logical choices was initially to use a measure based on the ranks of data. For example, non-parametric measures of rank correlation such as Gamma, Kendall and Spearman could have served to determine the strength of the relationship between the rank (i.e. the cut-off point) and the number of terms that were detected in each rank. However, we soon realized that these measures failed to achieve their goal. By way of example, suppose that DEXTER finds ten combinations of the α, β and γ values that yield the same highest precision, e.g. 140 terms have been recognized by IATE among the top 200 SRC-ranked ngrams. In this hypothetical scenario, Table 5 shows the scores derived from the rank correlation measures, where the resulting distributions have been labelled from A to J.
A hypothetical distribution of terms (rank correlation)
As can be noted in C, D, I and J, it is not uncommon to find two or more ranks with the same number of terms. Although extended measures were used to effectively manage tied ranks (e.g. Kendall’s tau-b), none of the coefficients above showed sufficient discriminating power. For example, E, F, G and H have the same score with any of the three coefficients, so DEXTER is still not able to select a winning distribution of terms. The problem is that the calculation is not based on the actual values of the data points but only on the rank order.
As the gradual distribution of terms is based on the assumption that the bulk of terms is likely to be found at the top of the list and the number of specialized words progressively declines to the bottom, the alternative was to take into consideration the cumulative number of terms and precision in each cut-off point (i.e. 1–50, 1–100, 1–150 and 1–200), where a linear pattern of correlation is usually revealed. For example, Figure 3 displays the data points corresponding to the distributions A and B, whose best-fitting lines are represented by the dashed line and the dotted line, respectively.
Two linear distributions of terms.
In this approach, the distribution of terms can be measured by means of Pearson’s product-moment correlation coefficient, since we are concerned with how closely the points fit to the line. Table 6 demonstrates that Pearson’s correlation coefficient (r) can actually get better results with the distributions in Table 5.
A hypothetical distribution of terms (correlation coefficient)
Finally, a measure was devised whereby the adequacy of a given distribution of terms (a) is calculated primarily on precision (p) and to a lesser degree on the correlation coefficient (r), as shown in the equation (19), where e is the exponential constant.
\begin{equation}
a = {e^p} - \frac{1}{2}r
\end{equation}
To conclude, Table 7 shows the scores assigned to the distributions of unigrams in Table 4.
Scores of term distributions (sample corpus)
2.3.3 Computer-aided term validation
DEXTER adopts a hybrid approach to term evaluation, since term recognition based on the IATE database is integrated with human validation based on experts’ judgements. As described by Pazienza et al. (2005: 265), the evaluation of ATE systems has been traditionally performed by means of one of two methods: (a) an a priori reference list of terms for the specific domain is used as a gold standard against which to measure the system performance, or (b) human experts validate the term candidates extracted by the system. It is important to highlight that both methods have drawbacks. With respect to (a), the system can extract terminological expressions, which are not present in the reference list; in this case, although these candidates are true terms, they are treated as false negatives. With respect to (b), manual validation is a time-consuming task, as well as being prone to the expert’s subjectivity and personal interpretation. DEXTER minimizes these problems by adopting both methods. The remainder of this section deals with the computer-assisted mode of data validation and clean-up.
Once the ngrams of a given type are ranked according to their SRC weight, their validation is performed from the most specific (i.e. trigrams) to the most generic (i.e. unigrams) in relation to their cognitive load. The reason is that the complexity of ngrams is proportional to their information content. For example, the information content of the unigram championship is lower than that of the bigram football championship, and in turn this bigram becomes less informative than the trigram European football championship. False candidates (i.e. true negatives and false positives) are manually discarded. A key issue at this stage of the validation is that DEXTER can provide the context of any term candidate, in the form of a maximum of eighty snippets for each candidate with a maximum of 400 characters per fragment. Moreover, the user can browse the whole document from where a given snippet has been extracted. Since DEXTER is based on the indexation and search capabilities of Lucene technology, context retrieval starts with an ngram-based query to the corpus, where every document that matches the query during the search is assigned a score computing how similar the document is to the query. Indeed, Lucene uses a formula that is primarily based on tf-idf, together with other factors such as coordination, field-length normalization, and term/query boosting, to calculate the relevance of matching documents;Footnote 13 then, DEXTER displays snippets of documents reverse-sorted by this score. It is noteworthy that, although most Key-Word-In-Context concordancing programs (cf. Wiechmann and Fuhs Reference Wiechmann and Fuhs2006) allow the user to determine the amount of co-text to the left and right of the keyword, they cannot guarantee the relevance of the data. It is certainly not a matter of size, i.e. the length of the concordance span, but rather of significance, i.e. if meaning arises within the concordance span. In this regard, the statistical significance of Lucene’s similarity score helps to put in the foreground the most relevant contexts of ngrams.
Finally, candidate space can be reduced on the grounds of lack of term nestedness, that is, when one or more terms are embedded in a larger term. For example, in the case of bigrams and trigrams, you have the option of discarding all their nested ngrams automatically. It is important to make it clear that the degree of nestedness is variable, ranging from full to partial nestedness and to no nestedness at all. For example, capacitor electrolyte serves to exemplify full nestedness, since both capacitor and electrolyte are specialized terms. On the other hand, alternating current illustrates partial nestedness, since alternating is not an actual term in the electronics domain. Finally, Thomson effect does not have any nestedness, since neither Thomson nor effect is a domain-specific term. Therefore, as shown in Figure 4, DEXTER allows you to eliminate false candidates in two ways: (a) in case of partial or full nestedness, you only eliminate a given ngram by clicking ‘remove’, but (b) in case of no nestedness, you can eliminate the false candidate together with all their embedded ngrams by clicking ‘nesting’. Indeed, removing all embedded ngrams at one time certainly results in an effective clean-up method during manual validation, because the number of ngrams that can be discarded with just one click amounts to three with a bigram and to six with a trigram.
Validation and clean-up in DEXTER.
It could be thought that a metric such as C-value, which is described as ‘a method to improve the extraction of nested terms’ (Frantzi et al. 2000: 122), could contribute to automate part of the process of data validation. However, as explained in Section 2.3.2, C-value would not have been as efficient as expected with bigrams and trigrams in small- and medium-sized corpora.
3 Evaluation of DEXTER
The evaluation was conducted by comparing the results achieved by DEXTER with those obtained by BioTex (Lossio-Ventura et al. 2014a),Footnote 14 GaleXtract (Barcala et al. 2007),Footnote 15 Termine (Frantzi et al. 2000)Footnote 16 and TermoStat (Drouin Reference Drouin2003),Footnote 17 whose main features are described as follows:
-
• BioTex and TermoStat can discover simple and complex terms, whereas GaleXtract and Termine recognize just complex terms.
-
• Unlike Termine (English), BioTex (English, French, Spanish), GaleXtract (English, French, Galician, Portuguese, Spanish) and TermoStat (English, French, Italian, Portuguese and Spanish) are multilingual.
-
• BioTex, GaleXtract, Termine and TermoStat make use of TreeTagger for the POS-based filtering of term candidates.
-
• In BioTex, a system for biomedical term extraction, the user can change the number of linguistic patterns used to filter term candidates, as well as the function (i.e. average, maximum or sum) in the metrics F-Ocapi and F-TFIDF-C. In this research, BioTex was configured with the default number of linguistic patterns (i.e. 200) and with the maximum function, since Lossio-Ventura et al. (2014b) demonstrated that this function has the best behaviour for the first 300 terms after manual validation.
-
• GaleXtract and TermoStat employ popular association measures: χ2 (Nagao et al. 1976), log likelihood (Dunning Reference Dunning1994), mutual information (Church and Hanks Reference Church and Hanks1990) and symmetric conditional probability (Silva and Lopes 1999) in the former, and χ2, log likelihood and log odds ratio (Everitt Reference Everitt1992) in the latter. Termine makes use of C-value (Frantzi et al. 2000). BioTex employs LIDF-value, F-OCapi and F-TFIDF-C, where the two latter combine C-value with Okapi and TFIDF, respectively, (Lossio-Ventura et al. 2014b, 2014c).
The evaluation focused on open-access systems with a GUI that could be used to support terminology and terminography research. The software that did not contribute with different metrics was ignored; for example, AntConcFootnote 18 calculates keyness values with χ2 or log likelihood, which are already included in TermoStat. Moreover, the software that did not allow a proper comparative evaluation was not taken into consideration; for example, Translated-Labs Term ExtractorFootnote 19 only returns twenty-term candidates. In fact, although there are a number of so-called term-extraction programs on the Web, most of them (e.g. AnchovyFootnote 20 or Okapi’s Rainbow)Footnote 21 should be regarded as no more than tools that return frequency-sorted lists of ngrams.
DEXTER was evaluated by assessing the precision of the top-ranked 200 unigrams, bigrams and trigrams extracted as term candidates from two corpora of different domains and languages. The following experiments were devised to demonstrate that SRC can outperform not only the metrics of other ATE systems but also the results obtained by the single metrics of S (Salience), R (Relevance) and C (Cohesion). On the one hand, term extraction was performed on our sample corpus, that is, the corpus of 200 English-written documents about electronics. Tables 8–13 present the results of precision after manual validation of the candidates extracted from this English corpus.
Precision with unigrams (English corpus)
Precision with unigrams (English corpus) [continued]
Precision with bigrams (English corpus)
Precision with bigrams (English corpus) [continued]
Precision with trigrams (English corpus)
Precision with trigrams (English corpus) [continued]
On the other hand, term extraction was also performed on a corpus of 100 Spanish texts (273,476 tokens) about odontostomatology, whose true domains were Health [2841], Health care profession [2841001], Health policy [2841002], Illness [2841003], Medical science [2841004], Nutrition [2841005], Pharmaceutical industry [2841006] and Life sciences [3606003]; the false domains were Science [36], Natural and applied sciences [3606] and Applied sciences [3606001], since their terms are commonly found in many scientific disciplines. The documents were obtained from the scientific journal Avances en Odontoestomatología.Footnote 22 In this case, additional preprocessing was required during corpus compilation, where the English abstract and the list of bibliographical references were removed in each document. DEXTER extracted 2,642 unigrams, 385 bigrams and 110 trigrams as term candidates. Tables 14–19 present the results of precision after manual validation of the candidates extracted from this Spanish corpus.Footnote 23
Precision with unigrams (Spanish corpus)
Precision with unigrams (Spanish corpus) [continued]
Precision with bigrams (Spanish corpus)
Precision with bigrams (Spanish corpus) [continued]
Precision with trigrams (Spanish corpus)
Precision with trigrams (Spanish corpus) [continued]
Tables 20 and 21 display the values of each SRC coefficient for the unigrams, bigrams and trigrams in the two corpora.
SRC coefficients for the English corpus
SRC coefficients for the Spanish corpus
Both experiments demonstrated that the best precision was obtained with SRC. Indeed, if ranges are inspected, it can be realized that on just one occasion SRC was outperformed in the twenty-two different cut-off points along the top 200 unigrams, bigrams and trigrams. Moreover, like C-value (Termine) and the metrics based on C-value (BioTex), SRC shows a rather gradual distribution of positive candidates in the extracted list, where true terms tend to be attracted to the top of the list.
Since the values of the SRC coefficients can be different for unigrams, bigrams and trigrams (Tables 20 and 21), it makes sense to have separated lists of term candidates to evaluate the behaviour of the various types of ngrams, providing useful evidence to support corpus-based terminology and terminography research. If we finally choose to compile a global list of candidates, then the challenge is to significantly correlate scores that belong to different normal distributions (i.e. each with a different mean and standard deviation). In this context, Z-score standardization helps to determine which unigrams, bigrams and trigrams are at a similar distance from the mean in their respective distributions, so we can position candidates with similar Z-scores close together in the ranking. To illustrate, the global list of ngrams from the English corpus showed the best precision with the top-ranked 200 ngrams (0·88); in this case, precision was also better in each range: 0·94 for 1–50, 0·98 for 51–100, 0·84 for 101–150 and 0·76 for 151–200.
The evaluation of precision required manual validation, since term recognition is not a fully reliable process. IATE is not a full-fledged terminological database, not to mention duplicates, incomplete entries, misspellings or obsolete data that can be found (Zorrilla-Agut 2014). The key issue is that gold standards in terminology can be recognized as exemplars of quality but not of perfection. Indeed, term recognition with IATE raised both false negatives and false positives; for example, manual evaluation revealed 10 unigrams (e.g. amp or watt), 48 bigrams (e.g. Darlington transistor or Schmitt inverter) and 127 trigrams (e.g. Kirchoffs Current Law or Wheatstone Bridge circuit) as false negatives and 37 unigrams (e.g. device, maximum or unit) and 6 bigrams (e.g. input signal) as false positives among the top 200 SRC-ranked ngrams from the English corpus. In both experiments, manual validation was carried out by three terminologists. The problem is that, ‘since the definition of termhood is pretty vague, it is likely that experts produce different validations, based on their own intuition of term’ (Pazienza et al. 2005: 265). Evaluators need to rely on a clear notion of what a term is. A definition of term such as ‘a designation consisting of one or more words representing a general concept in a special language in a specific subject field’ (ISO 704 2009: 34) is very vague for practical purposes. In order to deal effectively with this problem, it should be recalled that terms can be categorized into neonyms, existing forms or translingual loans. For obvious reasons, neonyms and translingual loans don’t raise any problem during term validation. However, existing forms do not help to determine clear criteria to identify domain-specific terms. Paradoxically, most ATE researchers (cf. Park et al. 2002; Zhang et al. 2008; Knoth et al. 2009) are concerned with the number and/or profile of the evaluators involved in the experiments rather than with a clear definition of what a term is. All experts agree that defining such a criterion is not an easy task, so how can evaluators decide that a given candidate is a ‘good glossary item’ or it is ‘characteristic for the domain’? In the evaluation of SRC, we crystallized this decision-making process into a flowchart (Figure 5), where the most critical question was that of the relevance of definitions.
Flowchart for term validation.
Thus, once the definition of the term candidate was obtained from the corpus itself or from other specialized resources (e.g. Google Books and Wikipedia), the evaluator explored the definiens to find at least one lexical item that could be related to any of the true domains of the corpus; if so, the ngram was validated. As scientific and technical definitions are aimed at giving insights into the subject matter to which the defined term pertains, it can be concluded that the definiens provides a means for the discovery of the specialized domain of the definiendum. Thus, the flowchart was used as a guideline to find verifiable evidence for both single- and multi-word terms. This model of evaluation favours final agreement among human judgements, since decisions must be based on textual evidence.
The flowchart proved to be particularly useful to differentiate between complex terminological units and phraseological collocations. For example, should coil of wire and coil rotation be considered terms of electronics? On the one hand, coil of wire was selected as a term because its definition includes three words, i.e. resistor, voltage and inductor, which undoubtedly pertain to the electronics terminology:
A coil of wire is simply a resistor as far as steady voltage is concerned, but for alternating voltages it behaves as an inductor. (Sinclair Reference Sinclair2011 : 36)
On the other hand, the flowchart did not reveal coil rotation as a term but only as a statistically significant co-occurrence (i.e. collocation). To illustrate, we present some valid and non-valid term candidates related to the domain of the English corpus:
-
• Valid term candidates: (a) unigrams: capacitance, farad, galvanometer or voltage; (b) bigrams: Ohm’s law, PN junction, RLC circuit or square wave; and (c) trigrams: magnetic flux linkage, metal film resistor, solid state relay or voltage divider circuit.
-
• Non-valid term candidates: (a) unigrams: centimeter, theorem, tutorial or vector; (b) bigrams: capacitance value, high frequency, input signal or time constant; and (c) trigrams: changes in temperature, Greek symbol phi, speed of rotation or upper threshold level.
The analysis of non-valid term candidates revealed that false positives usually take the form of (a) common words, e.g. maximum or value, and (b) complex ngrams nested in longer multi-word candidates, e.g. width modulation (pulse width modulation) or permanent magnet DC (permanent magnet DC motor).
Moreover, although it is well known that nouns and noun phrases make up the bulk of term candidates, both experiments showed that some adjectives (e.g. astable, capacitive, inductive, sinusoidal or voltaic [electronics]; birradicular, dentinario, estomatológico, gingival, hemostático, malar, mesiodistal or periodontal [odontostomatology]) and verbs (e.g. amplify, rectify or regulate [electronics]; bruñir or suturar [odontostomatology]) can be closely linked to specialized domains.
4 Conclusions
This article analyzed the term-extraction process that takes place in DEXTER, an online workbench that consists of a suite of corpus and terminological tools with such diverse functionalities as corpus compilation and management, document indexation and retrieval, query elaboration, textual exploration, and term weighting and validation. The components of DEXTER that play a critical role in the ATE process, and that have contributed to make some headway in the field of ATE, are (i) the stopword-detection method, (ii) the SRC metric, and (iii) the IATE database. On the one hand, DEXTER is provided with a multilingual and multi-domain method that aims to reduce the inventory of term candidates by filtering simple and complex ngrams on the basis of the common stopwords detected automatically in the given specialized corpus. On the other hand, SRC is a parameterized metric for term ranking that relies on the theoretical principles of (a) salience, which measures the prevalence of terms in the document collection, (b) relevance, which measures the tendency in the usage of terms between a domain-specific corpus and a general-purpose one, and (c) cohesion, which measures the degree of stability of multi-word terms. Finally, the IATE database not only enables DEXTER to recognize domain-specific terms but, even more important, helps to determine the values of the SRC coefficients by placing best term candidates in the foreground.
This research was able to demonstrate that the requirements described in the introduction were successfully implemented in DEXTER:
Requirement 1. SRC outperforms the results of well-known statistical-significance metrics (e.g. χ2, log likelihood, log odds ratio, mutual information and SCP) as well as C-value and other metrics based on the latter (e.g. F-OCapi and F-TFIDF-C). The effectiveness of SRC, whose parameters enable the system to deal with a wide diversity of specialized corpora, was measured with respect to precision.
Requirement 2. DEXTER can extract simple and complex terms (i.e. unigrams, bigrams and trigrams) with the same metric.
Requirement 3. DEXTER can recognize domain-specific terms of different grammatical categories, e.g. nouns, verbs and adjectives, by means of shallow lexical filters rather than elaborate morphosyntactic patterns. It has been proved that the precision of DEXTER is better than that of those systems that adopt a hybrid method for term extraction (e.g. BioTex, GaleXtract, Termine and TermoStat), where the statistical approach is applied to the output of POS taggers.
Requirement 4. DEXTER is capable of processing corpora in several languages (i.e. English, French, Italian and Spanish) by means of knowledge-poor procedures that do not entail sophisticated NLP techniques. Indeed, this term extractor is provided with a few language-dependent resources–i.e. stemmer, lemmatizer and stopword list. For each language, the stopword list contains (a) functional words that were manually obtained from the grammar of the language, and (b) common words that are discovered at runtime using an adaptive method applied to the domain-specific corpus and a general corpus from the Leipzig Corpora Collection. Together with the IATE database, these resources serve to automatically tune the SRC coefficients for the extraction of specialized lexical units.
Requirement 5. DEXTER integrates corpus management and term extraction in a single platform with a user-friendly interface primarily intended for linguistic research.
Future research is aimed at increasing the number of languages (currently four) and the types of ngrams (currently up to trigrams) that DEXTER can process.
