2.1 Language variability considerations in natural language generation

Power, Scott and Bouayad-Agha (Reference Power, Scott, Bouayad-Agha and Gelbukh2003) describe an algorithm for generating a variety of texts that express the same content by considering a set of linguistic constraints. These constraints are parameters like paragraph or sentence length, frequency of connectives, pronouns or punctuation marks, use of technical terms, etc. This work illustrates how to generate different texts that are equally correct for a specific communicative goal.

Reiter et al. (Reference Reiter, Sripada, Hunter, Yu and Davy2005) present a corpus-based analysis of how humans write weather forecasts. Their results show that the choice of words used to describe particular data depends on, among other things, the preferences of individual authors. They argue that the generation of texts by taking into account personalization would allow readers to get texts that are optimized to their personal style, thus making these texts more understandable for people with limited literacy skills or under some kind of stress (for example, when receiving medical information about operations or medication).

Paiva and Evans (Reference Paiva and Evans2005) present a natural language generation system whose behavior is controlled by correlating internal decisions with the surface stylistic characteristics expected in the resulting outputs. This approach is exemplified by a system that generates short medical information texts with stylistic variation in the sense of Biber (Reference Biber1988). Due to their implementation, the generator decisions and the stylistic characteristics of the texts are quite independent of each other.

Mairesse and Walker (Reference Mairesse and Walker2011) present a parameterizable statistical generator whose parameters are based on psychological findings about the linguistic reflexes of personality (extroversion, emotional stability, agreeableness, conscientiousness, and openness to experience). The system is trained on personality-annotated data to perform the generation decisions required to create a text aimed at a particular personality: given a target personality, the system estimates values for each of its parameters and creates the final text according to them. Although it is a very powerful system, the definition of personalities is based on psychological aspects not related to language.

As diverse as they are, all these works seem to agree about the usefulness of allowing different possibilities during the NLG process in order to produce more natural texts. These studies also build on the idea that different people use language in their own different ways and that this diversity could be useful to develop more human-like NLG systems. In this paper, we try to contribute to these relatively unexplored path with a systematic approach.

2.2 Language variability considerations in referring expression generation

REG is one of the most studied problems in the NLG process. Many different solutions have been proposed for this task, each one taking into account different considerations and approaches. A detailed survey on the topic can be found in Krahmer and van Deemter (Reference Krahmer and van Deemter2012). Although some systems may differ, REG is usually considered to be a two-step procedure. The first step, usually called attribute selection, is to choose the conceptual information to be expressed. This information should be enough to distinguish the entity being referred from any other distracting elements. Then, it is necessary to decide how that information will be expressed in the text. This step requires selecting which words or expressions are more suitable for each part of the expression. This is mostly a task for the lexicalization stage which usually happens later in the generation process, although it is closely related to the generation of references.

Although different REG systems have tried to model the ‘right way’ of performing this task, it is clear that there are many valid approaches (Dale and Viethen Reference Dale and Viethen2010). For example, each person has a different way of speaking or writing, and this could result in several possible referring expressions that are equally good for the same intended referent. This idea is gaining strength in the REG community.

Several works (Dale and Viethen Reference Dale and Viethen2009Reference Dale and Viethen2010; Viethen and Dale Reference Viethen and Dale2010) have focused on the attribute selection part of the process. They study two corpora of human-produced referring expressions in visual scenes and find that individual author choices play a very important role in creating the semantic content of referring expressions in identification tasks.

Bohnet propose different algorithms (Bohnet Reference Bohnet2008Reference Bohnet2009) for attribute selection and lexicalization of references. All of them based on the individual profiles of each of the participants in the TUNA experiment. For the attribute selection, the system tries to create the set of attributes for an intended referring expression depending on the person who created the reference for the same example during the experiment. Taking into account other references made by a certain person, it tries to approximate the attribute selection shown by this person in other expressions. From the point of view of lexicalization, different models of vocabulary and syntactic expressions are created for different people. Information about frequent choices in the use of determiners, favorite words for values and syntactic preferences are stored.

Di Fabbrizio, Stent and Bangalore (Reference Di Fabbrizio, Stent and Bangalore2008) try also to take into account differences between speakers, along with the relation between semantic and word order information. For the attribute selection task, they study the corpus in order to determine what were the most frequent usages of attributes for each of the participants in the creation of the corpus. From the point of view of lexicalization, they consider the different situations in the corpus as templates, but in such a way that if a set of attributes not present in the corpus is being lexicalized, the system fails because of lack of a suitable template.

Hervás et al. (Reference Hervás, Francisco and Gervás2013) carried out an experiment that shows how the performance of a lexicalization algorithm that tries to imitate human-generated referring expressions improves greatly by modeling the particular choices of a specific person. A referring expression lexicalizer was tested following two different approaches: one that considered the lexical choices of the person who had originally generated a referring expression in the corpus, and another that did not. For the approach that does not take into account personal choices, the training set was composed by all the examples in the corpus at the same time. In the second approach, the idea was to reproduce the lexicalization choices of different people so the corpus was divided in several training sets, one for each individual author.

All these approaches acknowledge the importance of considering personal choices in the generation of referring expressions and build on the idea that each person tends to be consistent with his own set of choices when he is expressing through language. However, they miss the broader idea of sets of choices that can be shared by different people, and idea that could improve the applicability of this kind of systems.

3.1 The TUNA corpus

One of the most important projects in the field of REG is the TUNA project (van Deemter, van der Sluis and Gatt Reference van Deemter, van der Sluis and Gatt2006) and its utility for evaluation as shown in Gatt et al. (Reference Gatt, van der Sluis and van Deemter2007). Under this project, a corpus of referring expressions for visual entities in the domains of people and furniture was developed. This corpus was obtained during an experiment in which participants were asked to write textual descriptions for objects (target entities) in visual domains by typing and submitting referring expressions that distinguished them from other objects that were shown simultaneously (distractors). An experimental trial consisted of one or two targets, plus six distractors (other entities from the same domain), without any kind of additional context. Each description was later annotated with semantic information, including information about all the other objects that appeared in the scene. Each referring expression from the corpus is accompanied by the conceptual representation of the situation in which it was generated. An exhaustive description of the corpus can be found in Gatt, van der Sluis and van Deemter (Reference Gatt, van der Sluis and van Deemter2008a).

For this work, we have used the subset of references to singular targets which contains 398 data files (XML documents) for the furniture domain and 340 for the people domain. Each file consists of a single instance of the corpus, that is, a pair consisting of a single situation (the representation of entities and their attributes) and a referring expression that describes an entity in that situation (the target).

Figure 1 presents an example of one of these XML documents. The basic format of the instances of the corpus consists mainly of the following nodes:

• • DOMAIN: Representation of entities in terms of their attributes.

• • STRING-DESCRIPTION: The string describing the target referent in the domain.

• • ATTRIBUTE SET: The set of domain attributes included in the description. Attributes represent the characteristics of an entity by using attribute-value pairs. The possible attributes and values for both domains are shown in Table 1. Empty cells represent attributes that are not used in the domain. x-dimension and y-dimension correspond to the coordinates of the referent in a 5 (column) x 3 (row) matrix in which the objects were presented during the experiment.

• • DESCRIPTION: The string in STRING-DESCRIPTION where the relevant substrings are annotated with attributes from the ATTRIBUTE SET. The substrings corresponding to attributes could be single words (big) or phrases (that is big).

Fig. 1.

Example of referring expression from the TUNA corpus in the furniture domain.

Table 1.

Attributes and their values in the two domains of TUNA

There were 57 authors per domain who were identifiable via a unique identification number (id). Each author created 7 expressions for the furniture domain and 6 for the people domain. Although this corpus is thoroughly annotated with conceptual information about the attributes used in the referring expressions, it lacks information about the lexical and syntactic form these attributes take.

3.2 Heterogeneity of the lexicalizations of the TUNA corpus

Hervás et al. (Reference Hervás, Francisco and Gervás2013) analyzed the TUNA corpus in order to study the degree of heterogeneity of the referring expression lexicalizations that conform it. Lexical variation was analyzed from two different points of view: variation in the different lexicalizations for specific attributes, and variation in the different lexicalizations used by individual authors.

Table 2 shows the lexical variation found for the different attributes in both domains (number of different lexicalizations used per attribute versus total number of mentions to this attribute). We have considered that two lexicalizations are different if they are not exactly the same in a typical string comparison. The number of possible values for each attribute is also included in the table. Note that due to the nature of the corpus both lexical and syntactic structures used for the attributes are considered as lexicalizations. For example, the value ‘big’ was sometimes lexicalized using a single word (e.g., large), and sometimes using a phrase (e.g., that is big). In this case, referring expression lexicalization is considered as both lexical choice and syntactic choice as they are interlaced in the TUNA corpus.

Table 2.

Lexical variation in the TUNA corpus for specific attributes (Hervás et al. Reference Hervás, Francisco and Gervás2013)

For each attribute the distribution of lexicalizations is mostly homogeneous for the different values. It is interesting to note that the attributes orientation, x-dimension, and y-dimension present high variation in both domains. In the people domain attributes hasShirt, hasSuit, and hasTie have a 100 per cent variation, with a different lexicalization for each of their mentions in the corpus (although the number of mentions is very low). All those attributes with high lexical variation correspond to features that do not have an easy and fixed lexicalization.

Table 3 presents variation by author on average (Var. column) for both furniture and people domains. The total number of mentions for each attribute is shown for easier comparison of results. A value of 0 per cent in an attribute means that authors were always consistent with themselves in that attribute, that is, every time an author used the same value for an attribute he used the same lexical expression. This kind of coherence is only perfect in attributes like hasHair or hasTie that were barely used. All authors present a degree of variation lower than 50 per cent (on average) for all the attributes, and most of these values are between 0 per cent and 25 per cent.

Table 3.

Lexical variation in the TUNA corpus considering individual authors (Hervás et al. Reference Hervás, Francisco and Gervás2013)

More details about this analysis can be found in Hervás et al. (Reference Hervás, Francisco and Gervás2013).

4.1 Annotation schema

The annotation schema proposed adds a new layer of information to the TUNA corpus. This layer aims to reflect information about the different lexicalizations of the referring expressions. Therefore, all aspects that were considered useful in order to provide extra information about the way in which references are expressed were initially added to the annotation. This includes syntactic information when it is relevant in the lexicalization of the referring expression. The list that follows contains the identifiers, values, and corresponding descriptions of all the new annotations that have been added. New annotations were organized in four different groups. The first group deals with general grammatical choices that appear in all referring expressions. The other three groups correspond to choices when expressing position (x-dimension and y-dimension), orientation (orientation), and the rest of attributes. Position and orientation have been chosen separately because both present more than a 40 per cent of variation and a significant number of mentions (see Table 2 in Section 1). An exception is orientation in the people domain, but it has been included for completion. To facilitate later data analysis, all the annotations are represented by binary values; that is, yes or no are the only possible values.

4.1.4 Choices for the expression of non-positional attributes

The following six elements represent how the attributes of the referent (other than position and orientation) are expressed. Only those attributes that correspond to the referent are considered.

• • Premodifier. The attributes of the referent are described using a premodifier, without using any kind of connectives or auxiliary phrases (e.g., the bald man).

• • Gerund Phrase. The attributes of the referent are described using a gerund phrase (e.g., the man wearing glasses).

• • Indicative Clause. The attributes of the referent are described using an indicative clause (e.g., it is a fan in the middle and it is small).

• • Relative Clause. The attributes of the referent are described using a relative clause (e.g., the man who has black hair).

• • Using In. The attributes of the referent are connected to the referent by the preposition ‘in’ (e.g., The man in suit).

• • Using With. The attributes of the referent are connected to the referent by the prepositions ‘with’ or ‘without’ (e.g., the man with glasses).

Other possible values like participle phrase or the connector ‘from’ were not considered because they do not appear in the original corpus. If any example of the TUNA corpus presents two or more of the defined situations simultaneously each of the previous elements is annotated accordingly. When more than one form of expression are nested, only the main one is considered (for example ‘. . .who is wearing’ is considered Relative Clause, ignoring the following gerund form). Although the word chosen for the referent may contain information about some of its attributes (e.g., ‘the man’ contains information about the gender), that phenomenon is not considered as an attribute per se.

Table 4 summarizes the new annotation elements and shows their abbreviated names, which also correspond to the XML elements in the corpus. Figure 2 presents an example of a referring expression from the original TUNA corpus with the new annotations.

Table 4.

Annotation elements and their abbreviated names

Fig. 2.

Example of referring expression from the TUNA corpus reannotated with lexical information.

4.2 Annotation process

Two annotators (authors of this paper) annotated the 21 features of the annotation schema for the total of 738 samples in the corpus. As the annotation advanced, the annotation schema evolved and it became necessary to recheck the annotation performed so far, sometimes by adding new elements, or even revising previous choices when a change of interpretation was made. Once the annotation was done, we tested whether different annotators produced consistently similar results. If annotation data is reliable then we can infer that they have internalized a similar understanding of the annotation guidelines, and we can expect them to perform consistently under this understanding. Reliability is thus a prerequisite for demonstrating the validity of the annotation scheme (Artstein and Poesio Reference Artstein and Poesio2008).

The simplest measure of agreement between two annotators is defined as ‘the percentage of judgments on which the two annotators agree when coding the same data independently’ (Scott Reference Scott1955). The literature has reached the conclusion that this measure must be adjusted for chance agreement in order to get figures that are comparable across studies. Simple kappa coefficient is a statistical measure or inter-annotator agreement for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation since it takes into account the agreement occurring by chance. However, Simple kappa (Randolph Reference Randolph2005) is known to be influenced by prevalence and bias, which can lead to the paradox of high agreement but low kappa. It also assumes that raters are restricted in how they can distribute elements across categories, which is not a typical feature of many agreement studies. We used Randolph’s free-marginal multirater kappa (Randolph Reference Randolph2005Reference Randolph2008) to analyze interevaluator agreement for each element annotated in the corpus. We selected this measure because Brennan and Prediger (Reference Brennan and Prediger1981) suggest using free-marginal kappa when annotators are not forced to assign a certain number of elements to each category. Values of kappa can range from − 1.0 to 1.0, with − 1.0 indicating perfect disagreement below chance, 0.0 indicating agreement equal to chance, and 1.0 indicating perfect agreement above chance. A rule of thumb is that a kappa of 0.7 or above indicates adequate inter-rater agreement.

The value of the Kappa coefficient for almost all the elements is 1.0 which indicates there is almost perfect agreement above chance. Elements with lower agreement are: STRUCT-PRED (0.9), STRUCT-NOM (0.8), and POS-WITH-RELATION (0.9), but in all cases the value is greater than 0.7 which is considered an acceptable agreement.

4.3 Adjudication of the annotation

Once the two annotations were ready, an adjudication process for the annotation was needed to obtain a gold standard. In those cases where the Kappa coefficient was 1.0 (perfect agreement above chance) the adjudication was done automatically but for those attributes with a Kappa coefficient less than 1.0, it was necessary to adjudicate the double-annotated expression into a gold standard. Adjudication of differences was conducted by discussion between the two annotators; the first author of this paper moderated these discussions and settled strong disagreements. The result of this adjudication is the whole set of singular referring expressions in the TUNA corpus annotated with information about their lexical characteristics.

4.4 Notes on the annotation process

In addition to the elements presented in this section, the corpus was annotated with many more elements that finally were not taken into account for this work because they were not directly related to lexicalization. For example, the presence of new attributes not considered in the attribute set of the original corpus was also annotated (e.g., the smiling man next to the angry man), along with information about whether the referring expression contained information that refers to the conceptual frame of the TUNA’s experiment itself instead of the object of the referring expression (e.g., the chair at the top right surrounded by a border) or the presence of orthographic errors (e.g., the chair in the midlle).

Several issues appeared during the annotation process concerning strange or ambiguous situations. For example, sometimes the word ‘frontal’ (‘gray frontal table’) can be considered an expression of orientation or an expression of position (although it is more plausibly an expression of orientation). In addition, sometimes the referent does not appear in the referring expression we are annotating. Usually in those cases, the referring expression is considered as not having a valid syntactical structure (e.g., ‘right middle’).

The extended corpus is available in http://nil.fdi.ucm.es/index.php?q=node/567, including all the XML files in the original TUNA corpus reannotated with the new information, along with a document describing the annotation process.

5.1 Cluster analysis

Cluster analysis (Jain, Murty and Flynn Reference Jain, Murty and Flynn1999) is a well-known statistical technique that will serve us for the purpose of finding groupings of similar referring expressions. Let x ₁, x ₂, . . ., x_n be a set of observations (in our case referring expressions) where each observation x_i is a d-dimensional vector and d is the number of variables that characterize the observation. In our case, the referring expressions will be represented by means of d binary variables. Each binary variable will correspond with an annotation element. For example, given four annotation elements (DETERMINER-DEF, DETERMINER-INDEF, STRUCT-PRED, STRUCT-NOM), the expression it is a red chair with the seat facing right will be represented by the following 4-dimensional vector of binary variables (0, 1, 1, 0).

The clustering algorithm used in our work is the k-means algorithm (MacQueen Reference MacQueen, Cam and Neyman1967). The k-means clustering aims to partition the set of observations into k sets or clusters S = {S ₁, . . ., S_k} so as to minimize the within cluster sum of squares

(1) \begin{equation} \arg \min _{\bf S} \sum _{j=1}^k \sum _{x_i\in S_j} \Vert x_i - c_j \Vert ^2 \end{equation}

The distance between two referring expressions a and b described by d binary variables is

(2) \begin{equation} \Vert a - b \Vert = \sqrt{\sum _{i=1}^d (a_i -b_i)} \end{equation}

The result of the clustering process is a partition of the referring expressions into k groups (or clusters). Expressions in the same group are more similar (according to the representation and the distance used) to each other than to those in other groups. The number of clusters k will be chosen after assessing the output obtained with different k values. The assessment will take into account the size and the degree of homogeneity of each cluster, and will use the domain knowledge as a guidance. In other words, a clustering result will be considered suitable if it produces homogeneous and sufficiently large clusters that can be associated to groupings of similar lexical choices that are sound in the context of the corpus.

The clustering method described will be used to carry out two analyses of the variability in the corpus. The first will only consider general grammatical choices that are shared by all referring expressions, while the second one will focus on specific grammatical choices used for referencing non-positional attributes and position.

5.2 Finding groupings according to general grammatical choices

The variables considered in this case are the annotation elements DETERMINER-DEF, DETERMINER-INDEF, STRUCT-PRED, and STRUCT-NOM. These variables characterize general grammatical choices that all referring expressions have: the use of determiners and their grammatical structure.

The cluster analysis tells us which preferences (that is, combination of annotation elements) are more popular in the corpus used. The null hypothesis would be that the group of referring expressions is homogeneous and that in the TUNA corpus all referring expressions were created in the same way. We carried out a cluster analysis in the furniture and people domains independently. In both domains, the number of clusters considered was k = 6, because on one hand, with less clusters we obtained a cluster with most of the references and the rest of them with only a few expressions, and in the other hand, with more clusters we obtained small clusters with few referring expressions that did not represent related or similar choices.

Tables 5 and 7 show the values of the centroids in the six resulting clusters for the furniture and people domains, respectively. The value of the centroid for a variable is the value that appears most often in the expressions that belong to that cluster. In parentheses, the percentage of referring expressions in the cluster for whom the values of the characterizing variables are equal to those of the centroid is shown. These percentages illustrate the homogeneity of each annotation element in each cluster. A value of 100 per cent means that all the referring expressions in the cluster have the same value for the considered value.

Table 5.

Centroids of each cluster of the furniture domain

Tables 6 and 8 show the number of referring expressions in each cluster for the furniture (398 expressions) and people domains (340 expressions). We can see that GC1 and GC2 are the most popular ways of creating referring expressions. The second line of these tables also shows the within-cluster variability for each cluster S_j:

(3) \begin{equation} \sum _{x_i\in S_j} \Vert x_i - c_j \Vert ^2 \end{equation}

Table 6.

Number of observations, within-cluster variability and mean within-cluster variability in each cluster of the furniture domain

Tables 5–8 show that the clusters obtained in both domains are almost equal in terms of centroids, size, and variability, because we can find the same sets of similar choices in terms of the grammatical structure of the referring expression and they are used in the same proportion. The result shows that in this experiment and considering the annotation elements mentioned, the domain was not a key factor as the choices of the participants in the TUNA experiment were consistent in both domains. Clusters GC1 and GC2 represent the most popular ways of creating referring expressions, while clusters GC5 and GC6 represent the least popular ones.

Table 7.

Centroids of each cluster of the people domain

Table 8.

Number of observations, within-cluster variability and mean within-cluster variability in each cluster of the people domain

Another conclusion that can be drawn from the tables is that within-cluster variability is really low. This result is not surprising giving that we have 6 clusters and that there are only 9 possible combinations of the 4 annotation elementsFootnote ¹. Thus, it is easy to understand that some of the clusters present not variability because all its referring expressions have the same annotation values.

Thus, we have found a grouping structure and it is not a naive one. If there were no grouping structure, we would find only one group of homogeneous referring expressions. A simpler grouping structure would be 9 groups of approximately the same size (i.e., referring expressions would be uniformly distributed among the 9 possible combinations of annotation values). However, we have found a more sophisticated clustering structure that is the same in both domains. The clusters found were:

• • Cluster GC1 corresponds to expressions with nominal structure that use neither definite nor indefinite determiners. Some examples are: plain wooden desk and man at the bottom with glasses.

• • Cluster GC2 corresponds to expressions with nominal structure and where the definite determiner is used. This cluster is roughly as big as cluster GC1. Some examples of the referring expressions in the cluster are: the red chair on the right and the picture top right.

• • Cluster GC3 corresponds to expressions with nominal structure where the referent is referred to using an indefinite determiner. This cluster is similar to GC1 and GC2, but much smaller than them. This means that the use of indefinite determiners in noun phrases is much less frequent than the use of definite determiners or the use of no determiners at all. Some expressions found in cluster GC3 are: a red chair which is facing the left-hand side of the screen and a gray desk.

• • Cluster GC4 corresponds to the simplest grammatical structure employed: no structure and no determiners. These expressions are composed by one word or several disjointed words. Some examples are: 2nd chair 3rd row and with glasses.

• • Cluster GC5 corresponds to expressions with predicative structure and definite determiners. A closer look at the expressions in this cluster reveals that all of them use some kind of meta-information, that is, they describe the object using information of the context of the experiment (i.e., the red border surrounding the object). In all of the expressions, the subject is the element being described while the predicate always refers to the red border that surrounds the object. Some examples of the referring expressions in the cluster are: the red chair in the bottom is boxed in red and the guy on the bottom row is in red box.

• • Cluster GC6 corresponds to expressions with predicative structure and indefinite determiner. The referring expressions have the structure ‘it is a. . .’ and they are very unusual in the corpus. Some examples are: it is a red chair with the seat facing right and it is an older man with glasses.

If we assume that each of the above clusters represents a set of similar grammatical choices for referring to an object, it would be interesting to analyze whether each person stayed true to one set or switched between different sets during the whole experiment.

In the furniture domain, 56 authors wrote seven expressions and one person wrote just six expressions. 63 per cent of them (36 individuals) only used one set of choices. However, 25 per cent and 10 per cent of the authors switched between two or three different sets, respectively. Only one individual used four sets. In the people domain, 56 authors wrote six referring expressions and one person just wrote four expressions. 58 per cent of them (33 individuals) only used one set of choices, while 33 per cent and 9 per cent switched between two or three different sets, respectively. No individual used four sets or more. We can again see that the results are quite similar in both domains. Therefore, although most of the grammatical variability in the corpus is produced by authors that have a personal way of expression, we also find that some authors make different choices in different expressions. These results also show that sets of these choices are not unipersonal but shared by different authors.

Tables 9 and 10 show the number of consistent individuals (persons who always create referring expressions using the same lexical choices) per cluster in the furniture and people domains, respectively. The number in parentheses is the relative number of referring expressions written by consistent individuals. It can be seen that the numbers obtained in both domains are quite similar. There is an interesting difference between GC1 and GC2. While referring expressions in cluster GC2 belong mostly to consistent authors, in cluster GC1 the percentage of referring expressions written by consistent individuals is still substantial but significantly smaller. Interestingly, nobody wrote all the referring expressions using the choices represented by cluster GC4 (the non-structured set) nor cluster GC6. Cluster GC5, while not very representative, is mainly shared by two different authors that use the same choices.

Table 9.

Number of consistent individuals in each cluster in the furniture domain

Table 10.

Number of consistent individuals in each cluster in the people domain

A closer look at the tables of the centroids (Tables 5 and 7) and the tables of the numbers of consistent individuals (Tables 9 and 10) reveals that all the clusters characterized by the use of the definite determiner (GC2 and GC5) are the ones with a greater percentage of referring expressions written by consistent individuals. This could be due to the appropriateness of definite determiners to lexicalize all kinds of referring expressions. For example, expressions where comparatives or superlatives must to be lexicalized (e.g., the smaller of the two blue fans). However, the use of indefinite determiners in these and other similar situations do not make it possible to generate correct or natural expressions. Thus, it is more difficult to stay true to choices that include such constraints. A similar explanation can be given to the absence of consistent individuals in cluster GC4. The use of non-structured expressions may be not suitable to lexicalize all the information needed to refer to an object under any circumstances. As a result, nobody stayed true to these choices through all their referring expressions. Cluster GC6 only contained two expressions, not enough for having a consistent author.

The results reveal that variability is present in the lexicalization of referring expressions, even in a very restricted context such as the one considered here and with this few annotation elements. This variability gives rise to interesting manifestations, such as popular and rare preferences, or consistent and non-consistent individuals. In the next subsection, we will go deeper into the description of the variability of the lexicalization in the TUNA corpus focusing on more annotation elements.

5.3 Finding groupings according to specific grammatical choices

The previous analysis showed that it is possible to find very homogeneous groupings of general grammatical choices for lexicalized referring expressions. However, the expressions in each cluster showed a degree of heterogeneity in terms of other grammatical choices that were not taken into account in the analysis carried out in Section 5.2. These grammatical choices are specific to the attributes chosen to refer to the intended referent in each expression. The main attributes we considered are position and orientation and the rest of attributes (henceforth referred to as ‘non-positional’ attributes for the sake of clarity). This division is due to the high variability found in the lexicalization of attributes that express position and orientation (see Table 2 in Section 1) explained in Section 4.1. Obviously, not all the expressions in the corpus use the same attributes to refer to the intended referent and some of them use more than one attribute. Table 11 shows the number of expressions in each domain that use each of these three kinds of attributes. As can be seen, most of the expressions used non-positional attributes. Around 40 per cent of expressions in both domains used the position in the grid of the experiment to refer to the person or piece of furniture. Interestingly, orientation was seldom used in the people domain, while it was more popular in the furniture domain.

Table 11.

Use of position, orientation, and non-positional attributes in the attribute set

In this section, we will aim to find groupings of expressions that share similar grammatical choices to represent one of the attributes mentioned. We will show the results obtained for the cluster analysis of the grammatical choices when using non-positional attributes in the people domain and when using position in the furniture domain. The other possible analyses are position and orientation in the people domain, and orientation and non-positional attributes in the furniture domain. Orientation cannot be analyzed in the people domain because only 9 referring expressions use orientation. Interestingly, the use of non-positional attributes in the furniture corpus shows no variability, because 99 per cent of the expressions represent the attribute using an adjective as premodifier. Due to this lack of variability, it made no sense to carry out further cluster analysis for this case. The conclusion that can be drawn is that the expression of non-positional attributes for ‘things’ does not exhibit the same level of richness as in the case of people. The other two clustering analyses that were carried out were the use of orientation in the furniture domain and the use of position in the people domain. These analyses provide descriptive information about the referring expressions in the TUNA corpus, but do not provide further insight about lexical choices beyond that provided by the two analyses that will be detailed in the next sections.

5.3.1 Non-positional attributes in the people domain

In the people domain, 251 out of 340 referring expressions use attributes different from orientation and position to refer to the required person. The non-positional attributes used are mainly age, hair, hair color, beard, glasses, and clothes. The annotation elements that represent the grammatical choices for expressing a non-positional attribute are the ones enumerated in Section 4.1.4.

Table 12 shows the frequency of use for each of these grammatical choices for expressing a non-positional attribute in this domain. The most popular way of expressing attributes is the use of ‘with’ (79 per cent), while the premodifiers and the use of gerunds are much less frequent (25 per cent and 10 per cent, respectively). The other ways of representing non-positional attributes are barely used.

Table 12.

Frequency of use of the different ways of expressing non-positional attributes in the people domain

Table 13 shows the distribution of the 251 expressions that use non-positional attributes among the six clusters identified in Section 5.2. The percentage that these referring expressions represent in the original cluster is shown in parentheses. The table shows that in all the clusters a fair share of expressions use non-positional attributes to refer to a person. In cluster GC4, which is the cluster with no structured expressions, the percentage is smaller (55 per cent) because this cluster contains many referring expressions such as top left or second top where only the position is used.

Table 13.

Distribution of the expressions with non-positional attributes in the six clusters found in the people domain using the variables of the general grammatical choices

The k-means clustering method was applied to find groupings of expressions with similar grammatical choices. The variables considered were the four annotation elements that represent grammatical choices and the six that represent different ways of expressing the non-positional attributes of an element. After different attempts, the number of clusters considered was k = 7 because it produced meaningful and homogeneous clusters.

Table 14 shows the centroid values for the seven resulting clusters (the percentage of observations of the cluster whose value is equal to that of the centroid is shown in parentheses). The number of referring expressions in each cluster and the variability measures are shown in Table 15.

Table 14.

Centroids of each cluster of the people domain

Table 15.

Number of observations, within-cluster variability and mean within-cluster variability in each cluster of the people domain

The sets of similar choices found in this new clustering for the people domain are:

• • Cluster AC1 mainly has nominal structure, uses no determiners and represents the non-positional attributes as premodifiers, like in balding gray haired gray bearded man or white bearded man. The cluster has 30 expressions. However, the cluster is heterogeneous. 27 per cent of the expressions have no structure at all (e.g., glasses and white beard) and 30 per cent of the expressions use a definite determiner (the white bearded man on the left).

• • Cluster AC2 has only 15 expressions. They have nominal structure and represent the non-positional attribute by using a gerund form. The use of determiners in the cluster is heterogeneous. 13 per cent of the referring expressions have an non-positional attribute represented as a premodifier. Examples of this cluster include: man wearing glasses or elderly gentleman wearing heavy framed spectacles. The gerund is in all the cases ‘wearing’ and always refer to glasses.

• • Cluster AC3 corresponds to expressions with nominal structure, which use the definite determiner and represent non-positional attributes using ‘with’. This is the biggest cluster with 80 expressions. Examples of this cluster are the man with a beard and the man with black hair and glasses. 11 per cent of the referring expressions in this cluster have predicative instead of nominal structure, for example, the guy with glasses in the middle of the top row is in red box.

• • Cluster AC4 has 28 expressions with nominal structure that represent at least two non-positional attributes, one of them using ‘with’, while the other is represented as a premodifier. Some examples of this cluster are dark haired man with gray beard or a young man with glasses. 11 per cent of the expressions also represent a third non-positional attribute by using a gerund form as in elderly gentleman with a white beard wearing spectacles. The use of the determiner is not homogeneous in the cluster.

• • Cluster AC5 has only four referring expressions, all of them written by the same author. The referring expressions mainly have nominal structure, use the indefinite determiner and express one non-positional attribute by using a relative clause and in 75 per cent of the cases a second non-positional attribute as a premodifier. An example is a young man who is wearing glasses and who is looking forward. This cluster represents the set of choices of one person that exhibits a great deal of consistency. However, these choices are shared by no one else in the corpus, which suggests that they are not usual.

• • Cluster AC6 represents the expressions with nominal structure and an indefinite determiner that use ‘with’ for the non-positional attribute. This cluster has 32 expressions such as a man with glasses and black hair or a man with glasses and a white beard.

• • Cluster AC7 is characterized by nominal structure, the use of no determiners and the use of ‘with’ to represent the non-positional attributes. It is the second biggest cluster with 62 referring expressions. Some expressions found in this cluster are: man with gray beard or male subject on bottom with beards. Some of the expressions of the cluster use an additional non-positional attribute using a gerund form (man with beard and wearing glasses) or with ‘in’ (man in tie and glasses with dark jacket).

The consistency of the clusters does not offer as much information as in the case of general grammatical choices. This is due to the fact that the number of referring expressions written by each person using non-positional attributes is not always the same. In addition, although a non-positional attribute can be expressed in different ways, they sometimes depend on the attribute itself; for example, there is no direct way to say ‘with glasses’. This makes more difficult to find consistency. As a result, if we study the data looking for consistent individuals that have written more than one referring expression with non-positional attributes, we find that there are only 10 consistent individuals. Interestingly, eight of those individuals belong to cluster AC3 (for a total of 34 referring expressions out of 80 referring expressions in that cluster) and two individuals belong to cluster AC7 (for a total of 10 referring expressions out of 62). It is not surprising to find consistent individuals in the two biggest clusters. Most of the remaining individuals wrote referring expressions that belong to two clusters. Interestingly, the individual that wrote the four referring expressions that make up cluster AC5 is not a consistent individual, because he wrote two other referring expressions that belong to other clusters.

5.3.2 Position in the furniture domain

In the furniture domain, 150 expressions out of 398 (i.e., close to 40 per cent of the corpus) include the position of the referred piece of furniture in the experiment grid. They reveal that there is a great variety of ways to express the position. The new annotations that represent the grammatical choices for expressing the position are the ones enumerated in Section 4.1.2.

First, we studied the frequency of use for each of the position variables employed. Table 16 shows their absolute and relative frequency. It is interesting to see that close to 90 per cent of the expressions represented the position using top–bottom (left–right) and 33 per cent of the expressions used rows and columns to refer to the piece of furniture indicated. On the contrary, the use of cardinal points and cardinal numbers in the corpus is negligible as they were used only once.

Table 16.

Frequency of use of the different ways of representing position in the furniture domain

Table 17 shows the distribution of the 150 expressions that use position among the six clusters identified in Section 5.2. The percentage that these referring expressions represent in the original clusters is shown in parentheses. In the two biggest clusters, GC1 and GC2, 35 per cent and 44 per cent of the expressions use position. However, only 4 per cent of the expressions of cluster GC3 (that is, 2 out of 53) use position. This may be due to the fact that cluster GC3 is characterized by the use of indefinite determiners and this kind of determiners is not suitable to express position. On the contrary, 93 per cent of the expressions of cluster GC5 use position and cluster GC5 is characterized by the use of the definite determiner. In cluster GC4, the cluster without structure, we find that 55 per cent of expressions use position. Only 1 out of 3 expressions of cluster GC6 used position, but given such a small number, the result is not significant.

Table 17.

Distribution of the expressions using position in the six clusters found in the furniture domain using the variables of general grammatical choices

We applied a k-means clustering method to group referring expressions with similar grammatical and position choices. The variables considered included: the four variables that represent grammatical choices, and the six variables that represent different ways of expressing the position of an element in the grid of the experiment. The clustering was carried out for different values of k, i.e., number of clusters. After a careful analysis, we considered that the most meaningful results were obtained for k = 6. Table 18 shows the centroid values for the six clusters obtained (the percentage of observations of the cluster whose value is equal to that of the centroid is shown in parentheses). The number of referring expressions in each cluster and the variability measures are shown in Table 19.

Table 18.

Centroids of each cluster of the furniture domain

Table 19.

Number of observations, within-cluster variability and mean within-cluster variability in each cluster of the furniture domain

The sets of similar choices found in this new clustering for the furniture domain are:

• • Cluster PC1 is a small cluster that represents the expressions with nominal structure that express the position using a relation (e.g., the red chair above the blue one). Some of these expressions use definite or indefinite determiners and some of them use no determiner at all. More interestingly, 56 per cent of the expressions represent the position using top–bottom (e.g., red sofa bottom right corner of screen next to blue sofa and below green desk).

• • Cluster PC2 represents a quite homogeneous way of referring to objects that consists of nominal structure, no determiners, and position represented with top–bottom. It is the second biggest cluster with 38 referring expressions. Prototypical examples of PC2 are blue chair on top or top blue fan. However, in this cluster 24 per cent of the expressions also use a reference to the row or the column, such as in the case of green desk in middle row.

• • Cluster PC3 is very small (only 4 expressions), but all of them are homogeneous in terms of general grammatical choices (they use predicative structure and the definite determiner) and all of them use row-column to represent the position (e.g., the chair in the top row is in a red box). However, this cluster does not represent shared choices as all its expressions were written by the same individual.

• • Cluster PC4 has as prototype a referring expression with no structure and no determiners and that uses top–bottom to represent the position. For example, expressions such as right bottom or blue chair center. However, expressions with the definite determiner and predicative structure are also an important part of the cluster (e.g., the chair at the top right is surrounded by a border or the red sofa at the top left is surrounded by a border). These two different kinds of expressions represent the bulk of the 22 referring expressions in the cluster.

• • Cluster PC5 mostly represents expressions with nominal structure that do not use determiners and where the position is represented with row-column and ordinal numbers (e.g., first picture on third row). However, 29 per cent of the expressions have no structure at all (e.g gray desk first row). Note that the grammatical difference between the two kinds of expressions is the use of a preposition. 18 per cent of the referring expressions in the cluster use the definite determiner (e.g., the gray chair in the second row). Regarding position, 18 per cent of the expressions in the cluster use top–bottom, in addition to the features mentioned above, to represent it (e.g., the red chair in the middle of the first row or top row first fan only).

• • Cluster PC6 uses definite determiners and nominal structure and expresses the position using top–bottom. It is the biggest cluster obtained, with 60 expressions. An example of referring expression included in this cluster is the one in the middle top. 30 per cent of the expressions in this cluster also use row and column (e.g., the blue chair in the bottom row). This cluster represents almost all expressions in cluster GC2 (60 out of 65) obtained by considering only the grammatical choices (see Table 16). Thus, it can be said that the joint use of definite determiner, nominal structure, and position using top–bottom is very popular in the corpus.

As happened in Section 5.3.1, the study of the consistency of these clusters offers less information than in the general grammatical choices analysis. We found 11 individuals who wrote all their referring expressions with position using the same set of choices. Six of these individuals belong to cluster PC6, which is the biggest cluster. Those individuals wrote 32 of the referring expressions of the cluster, which means more than 50 per cent of the cluster referring expressions. The rest of the consistent individuals belong to middle size clusters: two to cluster PC2 (for a total of 7 referring expressions), one to cluster PC4 (5 referring expressions) and one to cluster PC5 (3 referring expressions). An individual can be consistent with 3 or 5 referring expressions if they are the only ones in which he used the position attribute we are studying. As happened in the study of attributes for the people domain, the individual who wrote all the referring expressions of a small cluster, PC3 in this case, is not a consistent individual, because he wrote other expressions that belong to other clusters. In this case, it is also observed that most individuals write referring expressions switching between two different sets of lexical choices.