3.1 Shallow linguistic analysis

The incoming textual message undergoes shallow linguistic processing, which is a process consisting of several sub-processes. The result of this shallow analysis on the example sentence is shown in Figure 4.

Fig. 4.

The example sentence Hij is genezen after shallow linguistic analysis.

The first sub-process that we apply is tokenisation, splitting of all the punctuation signs from the words, apart from the hyphen/dash and the apostrophe.

The next sub-process concerns spelling correction. As the users of the WAI-NOT platform have different levels of illiterateness we decided to perform at least some basic automatic spelling correction. An example message from the e-mail corpus reads dag esra ik wes je en ge lukkig niew jaar adem which supposedly should be corrected to (ignoring capitalisation rules) dag esra, ik wens je een gelukkig nieuwjaar, adam ‘hello esra, I wish you a happy new year, adam’. As we did not find any tools available for automatic Dutch spelling correction, we decided to implement our own approach. We used the freely available and open lexicon from www.opentaal.org, which is the Dutch lexicon used in freely available spell checkers such as Hunspell, the open source spell checker for OpenOffice and MacOS and which received the hallmark for spelling of the Nederlandse Taalunie (Dutch Language Union),Footnote ¹² the official international organisation responsible for the Dutch language policy. Besides the words in the lexicon, e-mail messages often contain first names. In order to avoid automatically and wrongly ‘correcting’ these names into proper Dutch words we have added all male and female first names that occurred more than fifty times in Belgium, last updated in 2009. This information was gathered from the Belgian National Institute of Statistics.Footnote ¹³ For every word that is not in this lexicon nor in the list of first names, we check all variants with one deletion, one insertion or one substitution. For all the variants present in the OpenTaal lexicon, the selection of the correct alternative is based upon a large frequency list (Vandeghinste Reference Vandeghinste, Rodríguez and Araujo2002), containing roughly eighty million words of Belgian Dutch newspaper text. This is in fact a unigram model. In future versions we might consider higher order versions, or more complex approaches towards text normalisation, if deemed necessary.

Then, we apply part-of-speech tagging. We use HunPos (Halácsy, Kornai and Oravecz Reference Halácsy, Kornai and Oravecz2007), a trigram-based open source tagger similar to TnT (Brants Reference Brants2000),Footnote ¹⁴ obtainable without requiring explicit permission, using the D-Coi tagset (Van Eynde Reference Van Eynde2005), which has become the de facto standard part-of-speech tagset since the release of the Corpus Gesproken Nederlands (CGN) (Spoken Dutch Corpus) (Oostdijk et al. Reference Oostdijk, Goedertier, Eynde, Boves, Martens, Moortgat, Baayen, Rodríguez and Araujo2002) and the Lassy-small corpus of written Dutch (van Noord et al. Reference van Noord, Bouma, Van Eynde, de Kok, van der Linde, Schuurman, Sang, Vandeghinste, Spyns and Odijk2013). These corpora have nearly identical tagsets (Van Eynde Reference Van Eynde2005; Van den Bosch, Schuurman and Vandeghinste Reference Van den Bosch, Schuurman, Vandeghinste, Calzolari, Choukri, Maegaard, Mariani, Odijk, Piperidis and Tapias2006). The tagger was trained on these manually corrected corpora. Both resources each contain about one million words with manually corrected part-of-speech tags. We chose HunPos over the Frog-tagger (van den Bosch et al. Reference Bosch, Busser, Daelemans, Canisius, Van Eynde, Dirix, Schuurman and Vandeghinste2007) as the latter proved, at the time of choosing a tagger, unstable in server mode, and is not easily trainable.

As the system is intended to translate e-mail messages for mentally challenged people, messages tend to be short, and mostly consist of merely one sentence. Nevertheless, some of the messages contain more than one sentence, so we apply sentence detection, as the translation engine works sentence based. This is done by rules based on punctuation signs.

A more language specific issue is the fact that Dutch contains separable verbs. These are verbs that have a lexical core and a separable particle. In some syntactic situations the core and the particle are written as one word, while in other situations they are written separately. Particles can have different part-of-speech tags, according to the tagset we use (Van Eynde Reference Van Eynde2005). The most frequent part-of-speech tags for particles are the final prepositions VZ(fin),Footnote ¹⁵ such as in verbs like afwerken (ik werk dit af → I/work/this/off ‘I finish this’). Other particles can be singular common nouns in standard case N(soort,ev,stan), such as in verbs like paardrijden (ik rij graag paard → I/ride/gladly/horse ‘I like horse riding’). Yet another set of particles can be tagged as uninflected adjectives ADJ(vrij), such as in vrijspreken (de rechter sprak hem vrij → the/judge/spoke/him/free ‘the judge acquitted him’). A final category of particles are the adverbs BW, such as in bijeenbrengen (hij brengt geld bijeen → he/brings/money/together ‘He collects money’). Each of the words of a sentence that is tagged as a verb, be it in its finite, infinitive or past participle form, is combined with each of the words tagged with one of the potential tags for particles. The most likely combination according to the eighty million word corpus is selected, indicating what the most likely merger of a verb and its particle is. Additionally, we check whether the compound verb is more likely than the parts kept separately. This is done according to the methodology described in Vandeghinste (Reference Vandeghinste, Rodríguez and Araujo2002), which is based on the number of different particles a verb can have, the frequency of the compound and the frequency of the parts separately.

Then, we apply lemmatisation. We first look up whether the token and part-of-speech tag combination occurs in the already mentioned manually corrected corpora CGN and Lassy-small, and if this is not the case, apply lemmatisation rules implemented as regular expression substitutions conditioned on the part-of-speech label.

In the future versions, we might consider dependency parsing, as available for Dutch in the Alpino-parser (van Noord Reference van Noord, Mertens, Fairon, Dister and Watrin2006) or the Frog-parser (van den Bosch et al. Reference Bosch, Busser, Daelemans, Canisius, Van Eynde, Dirix, Schuurman and Vandeghinste2007), provided that these parsers are robust enough to deal with messages that contain many errors and dysfluencies.

3.2 The direct route

Not all words of a message can be analysed with Cornetto, and not all words should necessarily be converted into pictographs.

As personal pronouns are very frequent in the e-mail messages in our e-mail corpus and not included in the Cornetto lexical-semantic database, we have provided an explicit treatment for them, making sure that they are covered: the part-of-speech tag of the personal pronounsFootnote ¹⁶ contains a detailed list of features, of which the important ones in this case are the three last ones, indicating the values for person, number,Footnote ¹⁷ and gender.

Additionally, we have provided a translation mechanism that uses a dictionary. This mechanism allows to bypass the semantic analysis via Cornetto and provides a direct link between token/lemma/tag and the names of the pictographs, allowing for underspecification of any of the three input fields: the person creating the dictionary can choose whether to specify only the lemma, or to make it more specific, by also adding the part-of-speech tag and/or the token to which the word has to comply in order to allow translation into the pictograph. A detailed description of the dictionary is found in Vandeghinste and Schuurman (Reference Vandeghinste, Schuurman, Calzolari, Choukri, Declerck, Loftsson, Maegaard, Mariani, Moreno, Odijk and Piperidis2014), and the effect of the dictionary is evaluated in Section 4.2. The dictionary entries are based on the most frequent words in the e-mail corpus which could not be appropriately converted through the semantic route. It concerns those words that occur at least fifty times in our email corpus (excluding our evaluation set).

3.3 Semantic analysis

As a first step in the semantic analysis of the source message, we detect words indicating a negative polarity, such as niet (not) and geen (no). When such a word is found, we look for its head. In the case of niet, we look for a verb within a window size of three, i.e. we look for a verb in the three preceding and the three following words. When a verb is found, we add the value negative to the verb’s polarity feature.

As a second step in semantic analysis, we look up all the possible Cornetto-synsets connected to the lemma of each word. On its website Cornetto is described as

‘. . .a lexical semantic database for Dutch, covering 92K entries, including the most generic and central part of the language. The database combines the structure and content of WordNet and FrameNet-like data. It contains both vertical and horizontal semantic relations and combinatorial lexical constraints such as multiword expressions, idioms and collocations on the one hand, and lexical functions and frames on the other. The concepts are aligned with the English WordNet so that ontologies and domain labels were imported.’ (http://wordpress.let.vupr.nl/cornetto/)

In our current system, we have limited ourselves to using the WordNet-like features of Cornetto, not (yet) using the FrameNet-like data. When developing and debugging the system, based on the development set, we made some adaptations in Cornetto, as it is rather biased towards Dutch as spoken in the Netherlands, while our system is mainly intended for the Dutch-speaking part of Belgium. Therefore, we added Belgian synonyms to the synsets from which they were lacking. Furthermore, we removed some links between lemmas and synsets because of the fact that Cornetto contains many synonyms for sexual concepts, such as e.g. genitals. The word mossel ‘mussel’, for instance, is disabled as a synonym for the concept of female genitalia. An evaluator judged the appropriateness of each lemma belonging to sexual concepts: is the first, most intuitive sense of that lemma a sexual sense or not? If not, we disabled it. We also disabled meanings that were not prominent in our development set, for example the word suiker ‘sugar’ as referring to suikerziekte ‘diabetes’, while it is much more frequent in the sugar-reading. Doing so, we improve the coverage of the language of the intended, cognitively challenged users. We discovered several such misconversions through extensive testing the system, not only with the messages from the development set, but also by test usersFootnote ¹⁸ that were asked to report errors.

Cornetto is organised as such that each synset is linked to a number of lexical units (lexunits). Such a lexical unit consists of an identifier and a lemma. Linking words with synsets goes via the lexical units. Most synsets have a part-of-speech category encoded with them, which allows distinguishing between different synsets of the same lemma. We filter the synsets, keeping those where the part-of-speech of the synset agrees with the part-of-speech main category of the word, as labelled by the part-of-speech tagger. It should be noticed that the part-of-speech labels from Cornetto do not correspond in a one-to-one fashion with the labels from the part-of-speech tagger. Not only the labels are different, but so is the granularity. We manually encoded which Cornetto-labels are compatible with which part-of-speech labels according to Van Eynde (Reference Van Eynde2005).

The result of the semantic analysis of the example sentence is shown in Figure 5. Note that by performing semantic analysis, we do not (yet) apply proper word-sense-disambiguation, but use the most common sense, based on DutchSemCor (Vossen et al. Reference Vossen, Görög, Izquierdo, Bosch, Calzolari, Choukri, Declerck, Doğan, Maegaard, Mariani, Moreno, Odijk and Piperidis2012).Footnote ¹⁹

Fig. 5.

The example sentence Hij is genezen after semantic analysis.

3.4 Retrieving the pictographs related to the semantic concepts

As stated in Mihalcea and Leong (Reference Mihalcea and Leong2009), the use of pictographs has limitations. Complex concepts or combinations of concepts cannot always be depicted in a straightforward manner. There are certain properties in natural language which give it more power than pictograph languages. There are also many concepts that are difficult to depict, as their level of abstraction is too high. It is clear that pictograph symbols can be used to represent these concepts, but these concepts need to be learned by the users.

Some characteristics of natural language may not be present in certain pictographic languages, such as articles, inflection, or tenses. Only few auxiliaries are used. In general, no distinction is made between singular or plural. In some cases this is due to the fact that the concepts involved are hard to put into pictographs (like determiners, inflection of a verb), or because the pictographs mainly express the more abstract concept, as expressed by the lemma. In some exceptions the pictograph represents a more specific concept, such as a token in its plural form.

The Text2Picto translator works with two different target languages: Sclera, described in Section 3.4.1, and Beta, described in Section 3.4.2. Section 3.4.3 describes how the pictographs have been linked to the synsets and Section 3.4.4 describes how the pictographs are used in the translation engine.

3.4.1 The Sclera pictograph set

ScleraFootnote ²⁰ is a large set of mainly black-and-white pictographs. Originally these were used as directives (feed the dog, brush your teeth), just like the pictographs we are confronted with in everyday life.

There are currently over 13,000 Sclera pictographs and new pictographs are created every month upon user request. These pictographs are freely available as .png files with a filename indicating their meaning in Dutch, English, French, and Spanish. As shown in Figure 6, they can represent simplex concepts corresponding to single Dutch words, but often they represent more complex concepts corresponding, for instance, to a verb and its objects (Figure 7), to two or more nouns or to nouns and prepositional phrases. Most pictographs are for content words and there are hardly any pictographs for prepositions or adverbs.

Fig. 6.

Simplex Sclera pictographs for dream and large.

Fig. 7.

Verb-object (Sclera) pictograph for eat a sandwich and feed the dog.

Although Sclera mainly contains black-and-white pictographs, some of them are green (indicating that something is permitted or approved) while others are red (indicating a ban or disapproval). In some others another colour is used for contrast or to indicate the colour itself. A ban or disapproval may also be expressed by a (red) cross through the pictograph.

As mentioned above, Sclera was originally used as a means to communicate directives to its users (pupils, residents) with as few pictographs as possible. However, the last decade more and more attention is being paid to the communicative needs of people with cognitive disabilities, focusing on social inclusion.

3.4.2 The Beta pictograph set

The Beta pictograph set used in the WAI-NOT environment consists of more than 3,000 colour pictographs. A licence for this pictograph set can be obtained at reasonable prices. Their black-and-white equivalents are available for free.

Figure 8 shows some Beta pictographs. Easy recognition of the pictographs is one of the main objectives. The pictograph set is built based on a set of rules, mainly determined by colour, form and position. A number of specific figures appear throughout the different pictographs, consistently playing the same role. The colouring in the drawings is done logically, such that the intended concept stands out from its context. Similarly, arrows are used to point to the object, whereas the rest of the pictograph serves as context to disambiguate the pictograph. An empty white arrow represents a change in space or time. Small dashes are used (like in comic strips) to indicate verbs. Figures in verb pictographs are depersonalised, i.e. the head is a simple circle, without any facial features in order to allow ignoring natural gender. A more detailed description of the rules to which Beta adheres can be found on http://www.betasymbols.com/en/images.

Fig. 8.

Beta translation of eet een boterham ‘eat a sandwich’.

This approach is not free from criticism,Footnote ²¹ but understanding pictographs should be seen as a learning process, with regular exposure to the pictographs.

3.4.3 Linking the pictographs to synsets

We have linked the pictographs to the synsets in the Cornetto database for several reasons. The first reason is that by doing this we greatly improve the lexical coverage of the system, as not only the literal filename of the pictograph, but all its synonyms that are included in the database are now covered. A second reason also concerns the coverage. If for a certain word the synset is not covered by a pictograph, we can use the links between the synsets to look for an alternative pictograph with a similar meaning.

We have tried to minimise ambiguity in the process of linking the pictographs to the synsets. The links were created in two phases. First, we started from the pictographs and linked them with the synsets they were representing.Footnote ²² Second, we used the most frequent content words of the corpus, (with a threshold frequency of fifty) excluding our evaluation set, and made sure that links between these words and pictographs were established via the intended Cornetto-synset. For many words, the other possible synsets were never the intended sense of the words. One pictograph is linked to one synset, but note that it is theoretically possible to link different pictographs to the same synset, when they depict the same concept.Footnote ²³ In those cases we selected a prototypical pictograph in order to provide consistent translations and not to confuse our users.

We have manually linked a subset of 5,710 Sclera and 2,760 Beta pictographs to Cornetto synsets (Vandeghinste and Schuurman Reference Vandeghinste, Schuurman, Calzolari, Choukri, Declerck, Loftsson, Maegaard, Mariani, Moreno, Odijk and Piperidis2014), and provide a detailed account of how this was done. As these pictographs sometimes depict complex concepts, they can be linked to one or to more synsets indicating that their meaning combines the meanings of the synsets. In these cases one of the synsets was identified as the head synset, indicating that the other linked synsets are in some kind of dependency relation with the head synset. Table 1 presents how many pictographs are linked to how many synsets. In comparison with Sclera, the Beta pictographs are much more word-based. They rarely do not form a translation of a single Dutch word, so they are more easily attributed to a single Cornetto synset.

Table 1.

Distribution of the number of synsets per pictograph

In cases, where the pictograph meaning was not reflected by one or more synsets, they were often linked to the synset of its hyperonym. Note that, these links between Sclera and Beta pictographs and Cornetto-synsets are freely available to anyone upon simple request.

3.4.4 Using the links between synsets and pictographs

For every synset of every word we distinguish the simplex pictographs from the complex pictographs. The simplex pictographs are linked to one single synset. In the complex pictographs, we make a distinction between the head pictographs, which are linked to the head synset of that pictograph and the as_dependent pictographs, which are linked to the synsets of the dependents of the pictographs.

We decided to further extend the coverage of the pictographs in the system, by using the Cornetto relations between synsets. The HAS_HYPERONYM relation indicates the link between a subcategory and a super-category. Figure 9 shows the synset of the concept vrouw (woman) in the centre. As shown, there is a pictograph attached to that synset. When we have to translate the word buurvrouw (female neighbour) to which no pictograph is attached, we will connect the synset to that word through the HAS_HYPERONYM relation with its hyperonym synset vrouw and display that pictograph.

Fig. 9.

The synset of woman and its links to closely related synsets.

Other relations which allows improving the coverage of the system are the XPOS_SYNONYM or XPOS_NEAR_SYNONYM relations, which indicate a link between similar concepts but with a different part-of-speech. In Figure 9, this is shown by the link between the synsets of the words vrouw and vrouwelijk (feminine).

Yet another relation between synsets that we use is the ANTONYM relation, which indicates that synsets are the opposite of each other. In Figure 9, this is demonstrated by the link between the synsets for vrouw and man (man). Of course, when finding the pictograph of the antonym, this is marked by giving the antonym feature of the word the value TRUE, as shown in Figure 10, which will be displayed by the negation pictograph, as shown in Figure 3. The translation of the message Hij is genezen (E: He has recovered) consists of the following sequence of pictographs (verbatim): ‘Hij’ (He) ‘is’ (is) ‘ziek’ (ill) ‘niet’ (not).

Fig. 10.

The example sentence Hij is genezen with linked pictographs.

Using pictographs through synset propagation, as this is called, is controlled by parameters, for which we describe the tuning in Section 4.1. The penalties for using WordNet relations affect how the system selects the pictographs, as described in Section 3.5.

3.5 Finding the optimal path

Up to this point, we have, for every word in the sentence, looked up whether we can find one or more pictographs for it. We have connected these pictographs, together with their penalty, depending on the number and kind of synset relations we had to go through to connect them to the words.

The final step consists of finding the optimal path, and takes as its input the message annotated with the connected pictographs, as shown for the example sentence in Figure 10. We define the weight or cost of a path in Equation (1).

(1) $$\begin{equation} W(P)=\sum _{i=0}^n\sum _{j=0}^q W(s_{j_{w_i}}) \end{equation}$$

where W(P) is the weight of a path P, n is the number of words, w_i is the ith word, q is the number of pictographs associated with that word, and s_jwi is the jth pictograph associated with w_i . If there are no pictographs associated with word w_i , then we set q = 1 and s_1wi = w_i . The weight of a pictograph W(s_jwi ) is determined by the relations linking the synset associated with w_i with the synset of the pictograph s_jwi . In Algorithm 1 (cf. infra), which is an A* search algorithm (Hart, Nilsson and Raphael Reference Hart, Nilsson and Raphael1968) we show how we search for the best path in more detail. We used A* as it is well-known for its performance and accuracy (Dechter and Pearl Reference Dechter and Pearl1985).

We first show the findBestPath subroutine, which takes as input the queue Q containing path P ₀. Let P ₀ be the path with all words left to process. We shift the first, currently best scoring path (P) from the queue and extend it, as explained in the subroutine extend. This subroutine returns a number of paths which are added to the queue Q. We remove the double paths, i.e. paths leading to the same solution, keeping only the variant with the lowest estimated cost.Footnote ²⁴ We sort the queue Q by lowest estimated cost, and repeat until the first queue element Q ₀ no longer has any words left to process.

Algorithm 1

Find the best path

The extend subroutine works as follows. The input path has an array of words to process W_P . We shift the first word w from W_P and check whether it has complex pictographs C_w attached to it. If this is the case, we check for each of these complex pictographs c ∈ C_w whether each of the list S_c of other synsets that are linked to C_w but not to w are linked through the synsets of the remaining words to process W_P . If this is the case, we call the word to which they are linked w_x with x being the index of this word in W_P . Then, we copy the current path P to a new path P′, and add the complex pictographs to the list of already matched pictographs in P′, and we remove (splice) the word w_x at position x that linked to this synset s_c from the list of words to process W_P . We add the new path P′ to the list of new paths. We have implemented a similar, but simpler, treatment for simplex pictographs. If the word w has simplex pictographs C_w , we create a new path P′ for each s of C_w by copying P and adding the pictograph s to the list of already matched pictographs. In the case where no matching pictographs were found and the list of new paths is empty, instead of adding a pictograph to the path, we copy the current path P to a new path P′, as in the two previous cases, but this time we add the word form as it is to P′ instead of a pictograph (as no matching pictographs were found). P′ is added to the (empty) list of new paths. The extend subroutine returns the list of new paths.

When encountering words that have their antonym feature set to TRUE, we insert the negation-pictograph, as shown in Figure 3, to communicate the correct polarity of the message.

4.1 Parameter tuning

We can distinguish between three types of parameters that influence system behaviour. The first set consists of the parameters that tune the behaviour of using Cornetto relations, as described in Section 3.4.4. As using such relations increases the distance between the textual and the pictographic message, the use of these relations is subject to a series of penalty parameters that determine the cost of each of these relations. We keep track of how many of these synset links we apply and count penaltiesFootnote ²⁵ for using these relations, which results in preferring the words closest to the original meaning. The maximum penalty that is allowed for finding pictographs for a synset can be set as the threshold parameter.

The second set of parameters tunes the behaviour of the system with respect to features of the pictographs. Some pictographs clearly depict a concept either in plural or in singular. For a subset of the Sclera pictographs,Footnote ²⁶ we have manually checked whether they depict the concept either in plural or singular. We have introduced a parameter that determines the cost of using the wrong number. In cases where either the pictograph is not linked to a certain number, or the number of the word is underspecified (cannot be determined on the basis of the part-of-speech tag), we introduce the parameter for using no number.

A third set of parameters determines the behaviour concerning what route to take. The out-of-vocabulary penalty determines the cost of leaving a word untranslated, whereas the direct route advantage is a negative penalty (bonus) for using the direct route (Section 3.2) over the semantic route (Section 3.4).

We have tuned these parameters through an automated procedure. We built a local hill climber that varies the parameters between certain boundaries (Min and Max) and with certain granularity (size of parameter steps) when running the Text2Picto script. We ran this on fifty sentences selected from the development set, that were manually translated into Sclera and Beta, maximising the BLEU score (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002) of the Text2Picto script. BLEU is a precision metric often used in machine translation which compares the system output to one or more reference translations, by counting how many n-grams overlap (with n going from one to four), and correcting forbrevity.

Each trial of the tuning procedure consisted of a local hill climb with random initialisation of each of the parameters, until the BLEU score converged onto a fixed score. For each of the target pictograph sets, we ran five trials with different random initialisation and a granularity of oneFootnote ²⁷ in order to cover different areas of the search space. From these five trials we took the best scoring parameter values. These are presented in Table 2.

Table 2.

Results of parameter tuning

4.2 Automated evaluation

The evaluation test set of 50 Dutch messages that have been sent with the WAI-NOT e-mail system consists of 84 sentences (980 words). For each of these sentences, we created one reference translation in Sclera and one in Beta, translating, to the best of our ability the messages into the respective pictograph sets, focusing on the content of the message and how this content can best (most clearly and unambiguously) be expressed in pictographs, not performing a word-by-word translation, and not by post-editing the system output.

We have automatically evaluated different experimental conditions, progressively activating more features of the system. The first condition is the Baseline condition in which we only replaced words by pictographs if the pictographs have the same filename (without the .png extension) as the input words. In the next condition, we applied lemmatisation: the input sentences were tokenised, pos-tagged and lemmatised, and words that have a lemma that has the same filename as the pictograph are translated. Then, we added the direct route, as described in Section 3.2. This includes a specific treatment of the pronouns plus a set of dictionary entries per translation direction. The following condition uses the synonym sets of the Cornetto concepts: the pictographs are linked to synsets, and the input words are also linked to synsets, and whenever they share a synset, the words are translated into pictographs. The last condition also uses the relations between synsets, as described in the previous sections.

Table 3 shows the respective BLEU, NIST (Doddington Reference Doddington2002), Word Error Rate (WER) and Position-independent word Error Rate (PER) scores for the translation of messages into Sclera and into Beta. NIST is similar to BLEU but gives less credit to high-frequency non-informative n-grams. WER is often used in speech recognition and counts the number of words that are incorrect with respect to the reference translation(s). PER is like WER but treats the words as a bag and was included as there is no language model available for Sclera nor for Beta, so the position of the pictographs is not necessarily what we want to evaluate. For each condition we have included three variants. No spelling correction takes the input as it is and excludes the spelling correction process as described in Section 3.1. Automated spelling correction takes the input as it is and applies the spelling correction process as described in Section 3.1. In the Manual spelling correction-variant, we have manually corrected the spelling of the input to the best of our abilities, and sent this input to the Text2Picto engine. This was done to calculate an upper bound in order to estimate how much more room for improvement there is in the spelling correction process.

Table 3.

Automatic evaluation of Text2Picto conversion with one reference translation for the different experimental conditions

We have added significance levels for the BLEU and NIST scores, by comparing each condition with the condition on the previous line. Significance was calculated using bootstrap resampling (Koehn Reference Koehn, Lin and Wu2004). Although not presented in the tables, we have calculated significance levels between each pair of conditions. These are only used in the discussion of the results.

As is clear from Table 3 our tool improves translation quality over the baseline. The effect of the spelling corrector is negative or very small (and insignificant for BLEU) in the initial conditions, but it becomes significant for BLEU as well as NIST in the more advanced conditions that use Cornetto (Synonyms and Relations). There is clearly room for further improvement in the automated spelling correction process, as the scores for the upper bound in the advanced conditions are significantly better than the scores for the automated spelling correction process.

Adding lemmatisation results in a substantial improvement. For Sclera, there is significant improvement in comparing Baseline with Lemmatisation in all three variants, both for BLEU and NIST. In Beta, the rise in BLEU and NIST is smaller (because the Baseline has a better score), but still significant for NIST in all three variants, and significant for BLEU in the case of manual spelling correction. Adding the direct route provides a clear and significant improvement for Sclera and Beta in both BLEU and NIST in all three variants. This can be attributed to the proper treatment of high-frequency phenomena. Adding the synonyms results in another significant improvement for both target languages in BLEU and NIST in all three variants, as this step greatly improves the coverage of the systems, as now all synonyms in a synset can be converted to the pictograph associated with that synset. The effect of adding the relations is less straightforward. It does not seem to progress scores (significantly) in translations into Sclera, in any of the three variants in any of the metrics. For Beta, we see that there is a significant improvement for BLEU and NIST when the relations are added to the system, for all three variants.

Additionally, there is quite a large gap between the results for Sclera and the results for Beta. To find an explanation for this gap, we dug deeper and performed more experiments. The Sclera pictograph set consists of a much larger amount of pictographs than Beta, and it is therefore much more difficult to manually translate text into Sclera messages. Several different paraphrasing translations are possible, resulting in a less accurate measurement of translation quality by BLEU or NIST. In general WER and PER results are consistent with the results for BLEU and NIST.

To test this hypothesis, we have created a second reference translation for Sclera, based upon post-editing the system output. Figure 11 shows the effect this has on the BLEU score, comparing with the evaluation with only one reference, as presented in Table 3. For two references the effect of adding the synsets is significant in all three variants. The effect of adding the relations however remains insignificant for NIST and BLEU for all variants, apart from NIST on the automated spelling correction variant.

Fig. 11.

Bleu scores for Sclera for all experimental conditions in the three variants.

Although we would like to compare our work with the work of others, the only more or less comparable system that we found in literature is the system by Mihalcea and Leong (Reference Mihalcea and Leong2009), but they provide a completely different type of evaluation, performing more psycholinguistic experiments with human subjects estimating the understandability of messages in which some words had been replaced by pictographs, which is not what we intend to measure. As they do not convert adjectives or adverbs at all, and only convert a selection of verbs, their conversion accuracy would result in a rather low recall compared to our system.

4.3 Manual evaluation

We have performed a manual annotation with one judge, who removed untranslated words that were considered not to contribute to the content. This allows calculating the recall. For each of the translated words, she judged whether the pictograph generated was the correct pictograph, in order to calculate precision. Results are presented in Table 4. These results differ from the results presented in Vandeghinste and Schuurman (Reference Vandeghinste, Schuurman, Calzolari, Choukri, Declerck, Loftsson, Maegaard, Mariani, Moreno, Odijk and Piperidis2014) as we redid the manual evaluation, but now on the tuned system (cf. Section 4.1), and using a more systematic and objective approach to manual evaluation.

Table 4.

Manual evaluation of the Text2Picto translation engine

As proper names occur rather frequently in e-mail messages, we have calculated recall and F-score with and without proper names, in the latter case removing all proper names from the output. Precision remains the same in both conditions. In the case where proper names are included, they are not converted into pictographs, affecting recall negatively. In the WAI-NOT environment, proper names occurring in the contact lists of the users are converted into the pictures attached to these profiles, resulting in more personalised messages.

The improvements for Sclera are very large, especially the rise in recall, although we still have a substantial rise in precision as well. The Beta baseline system was already much better than the Sclera baseline system, so the improvements for Beta cannot be of the same magnitude. Nevertheless, there is still a substantial rise in recall, and a small rise in precision.

The F-scores for both target pictograph sets, while very different in the baseline, are now all around eighty-seven per cent. The difference in performance of the baseline systems led to the fact that Sclera was nearly unusable. This is now resolved, with both the translation of text into Beta and Sclera reaching similar levels of accuracy.

4.4 Extrinsic evaluation

To have an idea of the real-life effects of the improvements of the system requires an extrinsic evaluation. This can be a quantitative evaluation, using actual usage numbers, although changes in usage can never be exclusively attributed to the changes in the text-to-pictograph conversion process. Other possible reasons for change are differences in promotional activities, internal competition (other changes in the WAI-NOT environment allowing new possibilities to communicate, such as audio messages, or a WAI-NOT social network), and external competition (other communication environments and possibilities, outside of WAI-NOT).

We have collected the WAI-NOT statistics for over four years of usage, and if we look at usage number for the two years since the Text2Picto system’s first version (as described in Vandeghinste (Reference Vandeghinste2012)) has been implemented and compare them with the two years before that, there is a relative rise of 38.55 per cent in number of e-mail messages that has been sent with the system.

Up till now we have not systematically collected qualitative feedback. Nevertheless we have received feedback that the biggest improvements of the system over the baseline are due to the translation of conjugated verbs instead of just infinitives, and of nouns, not only translating singular nouns, but also plurals and diminutives. The communication between Beta and Sclera users has improved due to the better conversions, and especially the big improvement in recall makes the WAI-NOT communication platform much more usable.

There have also been a number of improvements that are not inherent to the Text-to-Picto translation engine, but that were implemented at the same time, as the result of brainstorming sessions between the authors and the people of WAI-NOT, such as the use of actual photos from the address book in the case of proper nouns, making the messages much more personal.