3.1 Spatial relations

Table 1 summarizes the definitions we apply for determining a spatial relation between two objects. Our inventory of spatial relations is a refined version of those presented in Elliott and Keller (Reference Elliott and Keller2013) and Ortiz et al. (Reference Ortiz, Wolff and Lapata2015) so as to account for the wide variety of object interactions attested in open domain real-world scenes. Elliott and Keller (Reference Elliott and Keller2013), in conjunction with human annotators, developed the relations with the goal to discriminate between mere object co-occurrence and action-related object interactions. For details, see Elliott (Reference Elliott2015, p. 12ff). We introduce relations 3 and 5, and additionally allow combinations of several relations. Parameters in the definitions of relations were based on those deemed optimal in previous work (Ortiz et al. Reference Ortiz, Wolff and Lapata2015; Elliott and Keller Reference Elliott and Keller2013). Relations 1–6 are mutually exclusive. If one of the conditions for relations 7–8 applies to a pair of objects, we concatenate the relation with the chosen relation from 1–6 to form a finer relation (e.g., on_below). We estimate the relative Euclidean distance between the centroids of objects s and t in relation to the length of the image’s diagonal (relations 3–6 in Table 1).

Table 1.

Definitions of spatial relations between objects s and t based on pairwise relationships between bounding boxes B or their centroids C

To compute the angle between centroids, we follow the definition of the unit circle, i.e., 0° lies to the east of its center and 90° to the north.

We also experimented with learning abstract relations from data through a k-means clustering approach, which used as features three geometric properties, namely pixel overlap of regions, and the angle and the distance between regions (see the definitions in Table 1). Experimental results revealed, however, that tree representations which used these induced relations performed worse (see the caption generation experiments in Section 6).

Finally, for an automatic induction of more elaborate and generic relations (e.g., attached to; moving towards; instrument for), which go beyond purely spatial ones, datasets of, e.g., images and their descriptions, would not suffice. This goal would not only require human annotations, but would further lead to arbitrary data (e.g., Johnson et al. (Reference Johnson, Krishna, Stark, Li, Shamma, Bernstein and Fei-Fei2015)).

3.2 Trees

Previous work (Lan et al. Reference Lan, Yang, Wang and Mori2012; Ortiz et al. Reference Ortiz, Wolff and Lapata2015) represents an image as a set of binary tuples between objects and their spatial relations. Figure 1 (bottom left) gives an example. Representations based on object pairs cannot express ditransitive verbs. Furthermore, tuples with more than two objects are ambiguous with respect to the scope of the spatial relations, unless implicit conventions on the interpretation of their order are applied. For example, in the tuple 〈person surroundscarrot on_beside person closegiraffe〉, it is not clear whether the giraffe is close to the person, the carrot, or both.

Fig. 1.

Top: example of image and its descriptions collected from humans. Bottom: types of representations (tuples, dependency, and constituency trees; dining_table is abbreviated to d_table to save space). Tuples in parentheses denote duplicates—both its objects are part of a previous tuple. Spatial relations were determined from image regions (Section 3.1).

We opt to represent objects and their relations through the use of tree structures (Elliott and Keller Reference Elliott and Keller2013) as they allow us to capture an arbitrary number of scene participants and can express complex relationships involving, e.g., transitive and ditransitive verbs. In addition, they allow for the quantification of multiple object class instances (see Figure 1 for an example). In analogy to syntactic representations of sentences, we formulate two different types of visual trees based on dependency and constituency relations. We first formally introduce visual trees and then explain how these are automatically constructed.

Constituency Trees: Our visual constituency trees are based on a context-free grammar G _c = 〈R, T, N, S〉. S is the start symbol and T are terminal symbols comprising all spatial relation labels (e.g., below, surrounds_above) and object class labels (e.g., umbrella); N denotes non-terminals which in our case are object phrases (NPs; e.g., skateboard, or skateboard on_below person), spatial relation phrases (SRs; e.g., on_below person), and spatial relations (Rs; e.g., on_below). The grammar rules for G _c are defined as follows:

Figure 1 (right) shows an example tree. A spatial relation phrase can be interpreted as a stand-in for a verbal or prepositional phrase in a linguistic constituency tree.

Dependency Trees: A dependency grammar encodes the structure of sentences by pairwise asymmetric relations between terminal symbols known as dependency (or head-dependent) relations. A dependency structure is a directed acyclic graph which contains the terminal symbols as its nodes, and their dependencies as edges. The (finite) verb is the center of the structure to which all other nodes are connected through the dependencies. In a visual dependency tree, nodes are object labels and edges correspond to spatial relations. The structural center of the visual dependency grammar is an individual object assumed to play a central role in the depicted scene. This definition is similar to Elliott and Keller’s (Reference Elliott and Keller2013) VDR; however, our trees are ultimately different due to another construction procedure which we outline below. The dependency grammar G _D in Table 1 defines the set of spatial relations we employ and Figure 1 (bottom left) provides an example of a visual dependency tree, with the center (person) being the dependent of the root label. Dependencies can capture nuanced semantics of seemingly related images. For example, an image showing a man standing on a flying airplane is representated by below_veryclose(airplane, person), whereas the tree for an image depicting people boarding a plane would be surrounds(airplane, above_veryclose(person, person)). Alternatively, if the image showed people standing by an airplane, the appropriate representation would be veryclose(airplane, above_veryclose(person, person)).

Visual dependency trees capture object interactions via pairwise spatial relationships. The latter are part of the grammar which describes the language. Constituency trees explicitly group objects in spatial proximity into phrases. Spatial relations and object labels are both terminal symbols, i.e., words of the language, and differ in terms of their grammatical categories (formally, NP and R). Intuitively, constituency trees may be better suited at modeling phrases (i.e., groups of objects) and their compositionality.

3.3 Tree creation

For each image, we construct a tree automatically in a deterministic manner. Assuming objects have been identified with an object detector, we rank all possible object pairings in ascending order of the Euclidean distance between their centers. Starting with a forest where each node (i.e., object) is a separate tree, we create a minimum spanning tree (MST). An MST is a subset of the edges of a connected, edge-weighted graph which connects all the nodes of the graph together and whose sum of edge weights is as small as possible. We process the list of object pairs top–down, connecting the corresponding nodes if they are not yet in the same tree, until all objects have been linked. We then transfer the MST to a constituency or dependency tree. In our MST construction procedure, we follow Kruskal’s (Reference Kruskal1956) MST algorithm.Footnote ¹ We use this algorithm due to its simplicity and general applicability to a variety of graph structured problems; however, other well-known algorithms (Prim Reference Prim1957; Karger, Klin and Tarjan Reference Karger, Klin and Tarjan1995) could have been employed instead.

Figure 2 illustrates the construction process. The five objects identified in image (a) constitute nodes in the graph (b); graph edges denote spatial proximity; as mentioned earlier edge weights correspond to the Euclidean distance between their objects’ centers. Filled nodes in the graph represent the MST which can be straightforwardly converted to a dependency tree as follows: we first select the root node according to a criterion such as the prediction score of the object detector, or the top-ranked object as determined by a content selection model (see Section 4.2). This node then determines the direction of all edges which we label with their respective visual relations. For example, to construct the tree in Figure 2(d), we first link the object pizza which has been deemed the central object (subfigure (b)) to the root. We next insert the node dining_table as a dependent of pizza, use the definitions in Table 1 to compute their relation and link the nodes with a corresponding edge (on_beside). dining_table is in turn linked to person in the MST, and we insert the node and the corresponding edge applying the same procedure as before.

Fig. 2.

Illustration of tree creation process. Given a graph G with nodes denoting the objects in an image and edge weights corresponding to their Euclidean distance, we compute a minimum spanning tree for a subgraph G′⊆G (filled nodes and plain edges in (b)) and use it to build a constituency (c) and dependency (d) tree, starting from an individual node in the spanning tree (black node in (b)).

We also construct constituency trees based on the MST. We use grammar G _c (defined in Section 3.2) and build a tree with a deterministic bottom–up parse. As the leftmost node of the tree, we select n _o from the edge 〈n _o, n _i〉 with the least weight in the MST (e.g., NP(pizza) in 〈NP(pizza), NP(dining_table)〉 in Figure 2). This node is again selected according to some criterion, as explained in the dependency tree creation. We use the definitions in Table 1 to compute the spatial relation between the nodes n _o and n _i, and call rules 3 and 4 of grammar G _c (in the example in Figure 2, this results in NP(pizza), R(on_beside), NP(dining_table)). Rule 2 of G _c can then be applied, which reduces the corresponding relation and n _i to a spatial relation phrase (e.g., SR(R(on_beside), NP(dining_table))). Next, rule 1 from G _c reduces SR and the NP which dominates node n _o to an NP (e.g., NP(NP(pizza), SR(R(on_beside), NP(dining_table)))). Since no other rules apply, the creation proceeds with the next edge 〈n _j, n _k〉 of the MST which has the least weight and is connected to the already processed nodes (e.g., 〈dining_table, person〉, with n _j = =n _i). It calls rules 3 and 4 (e.g., R(below_close), NP(person)), and rule 2 (e.g., SR(R(below_close), NP(person))). Again, rule 1 is called to reduce the NP, which dominates node n _j, and the SR to an NP (e.g., the tree in Figure 2). Tree creation continues until all edges have been processed.

In sum, we use the MST to guide our tree creation procedures. The MST is derived on the basis of the Euclidean distance between the objects’ centers, and as a result, a tree’s adjacency relations between objects reflect closeness in the visual modality—objects that are in a direct head-dependent relationship in a tree appear relatively close to each other in the image.

4.1 Parallel corpus creation

The training of an SMT model requires a parallel corpus which consists of pairs of descriptions and visual trees, where each object or entity in the tree is mentioned in the description, and each situation or event can be grounded in the image. Such grounding is absent from most existing datasets (but see Krishna et al. 2016; Plummer et al. 2015, for exceptions) and must be somehow inferred. We detail below how we obtain this information on the Microsoft COCO Captions dataset (Chen et al. Reference Chen, Fang, Lin, Vedantam, Gupta, Dollár and Zitnick2015) which we use in our experiments (see Section 6 for details), but our approach is general and can be applied to any corpus of images and descriptions.

We preprocess the descriptions with a semantic role labeler and align the identified semantic arguments with the objects detected in the image. Specifically, we parse the descriptions with PathLSTM, a state-of-the-art SRL system which is based on dependency path embeddings (Roth and Lapata Reference Roth and Lapata2016). We extract all predicates (e.g., eat) along with their arguments (e.g., constituents the girl and pizza bearing labels A0 and A1 in Figure 3, Example (2)) and modifiers.

Fig. 3.

Example of predicate-argument structures grounded in the images by finding alignments (colored) between the detected objects (left) and the arguments (right). (A0 labels denote agents, while A1 labels denote patients or themes (Palmer, Gildea and Kingsbury Reference Palmer, Gildea and Kingsbury2005).)

Let A denote a PAS extracted from the description of image i containing objects $\mathcal {O}(i)$ (e.g., person in Figure 3, Example (2)). To ground A in image i, we find an alignment α which links each argument a ∈ A (with n = |A|) to its referring object(s), $s \in \mathcal {O}(i)$. Function (1) below scores the alignment between object s of class l(s) and argument a:

(1) \begin{equation} \operatorname{score_{al}}(s,a) = \operatorname{e}(l(s)|a) \operatorname{d}(s) \end{equation}

where d(s) is the class detection score of s, and e(l(s)|a) the probability of class l(s) being aligned to argument a. If there is a direct string match between the object class (or its category, aka hypernym) and the argument, we set the probability e to 1. Otherwise, we estimate e from object–noun alignments which we obtain by running giza++ (Och and Ney Reference Och and Ney2003) on a corpus containing pairs of object class labels detected in an image and all nouns found in its corresponding description.

Let σ denote a permutation on a subset of $\mathcal {O}(i)$, and |a| the minimum number of a’s referents (e.g., |a| = 2 for two dogs, men). An alignment α is obtained by the permutation on the object–argument pairings (σ(s ₁), a ₁), . . ., (σ(s _{m + n}), a _n), $m=\sum ^n_{j=1} (|a_j|-1)$, which maximizes the sum of the individual alignment scores:

(2) \begin{equation} \begin{split} \alpha ^*=& \max _{\sigma } \frac{1}{n}\sum _{j=1}^{n}\sum _{k=0}^{|a_j|-1}\frac{1}{|a_j|} \operatorname{score_{al}}(\sigma (s_{j+k}),a_{j}) \end{split} \end{equation}

Overall, we keep those alignments which link each argument of a PAS and have a score greater than a threshold β (tuned experimentally on validation data). We leave the linking of modifiers optional. Figure 3 shows two examples of images, their descriptions annotated with PASs, and the established alignments.

Grounded PASs are paired with simplified image descriptions. We combine the surface strings of the predicate and the aligned arguments and remove any ungrounded components of the arguments, including adjectives or noun adjuncts. For instance, in Figure 3, for Example 2, we create the sentence the girl is enjoying her pizza for the predicate enjoying and its arguments A0 and A1, and in Example (1) we build the descriptions two dogs sit and two dogs perched on a bench. We furthermore create descriptions for the content captured by multiple grounded PASs, such as in Figure 3 (Example 1) for which we obtain the description two dogs sit perched on a bench. Finally, by applying simple rules based on the dependency parses and the PoS-tags of the captions, we create additional descriptions with alternative referring expressions for quantity groups and for noun phrases with coordinating conjunctions. For example, in the description a group of people are gathered around a table, we replace the subject a group of people by its semantic head, i.e., people, which results in the alternative people are gathered around a table. And we split the caption a table that has some food and a drink into two alternatives: a table that has some food and a table that has a drink. As a result, it is possible to obtain multiple descriptions for every original caption of an image. Note that in the parallel corpus each visual tree which corresponds to an individual caption is built such that it only contains the objects mentioned in the caption. In other words, the semantic content of each caption is licensed by the tree and vice versa. And since we construct the corpus on the basis of PASs, this information corresponds to the verbalization of the objects’ interactions or relationships.

4.2 SMT model

Surface realization: SMT formulates the problem of translating from a source to a target language as finding the translation t that maximizes the conditional probability:

(3) \begin{equation} t^{*} = {arg\,max}_t p(t|s) \end{equation}

where p(t|s) is approximated by a conditional log-linear model. Syntax-based SMT (Huang Reference Huang2006) defines p(t|s) in terms of the probabilities of individual derivations d ∈ D(s, t) in a synchronous grammar. The objective function is typically

(4) \begin{equation} d^* = {arg\,max}_d \left(\sum _{k=1}^K \lambda _k h_k(d)\right) \end{equation}

where h _k denote feature functions which define different models, such as a language model, a translation table and a word penalty model. The constants λ_k are tuned during training and used to scale the different models. In our experiments, we used the tree-to-string SMT framework implemented in Moses (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007) to train our models.

Content selection: At test time, we must decide what to talk about, i.e., which objects to focus on. We use logistic regression with l ₂-regularization and one hidden layer to predict whether a detected object s is likely to be relevant for the scene, while taking into account other detected objects. This way we do not impose any restrictions on the number of relevant objects, and allow tree-based representations of arbitrary size. The regression model is trained on positive and negative instances obtained from the parallel corpus described in the previous section.

We create a positive training example for each object which was aligned to an argument in the SRL-parsed image descriptions. As negative examples, we select objects with classes different from those of the aligned objects. We represent objects as vectors using a mixture of unary and binary features. The former include the detection score D _t of any object t, and the relative size S _s of target object s in relation to the average relative object size of its class:

(5) \begin{equation} S_{s_i} = \frac{\text{area}(s_i)}{\text{area}(i)} \left\bad. \left[\frac{1}{|O(l_{s_i})|}\sum _{t_j \in O(l_{s_i})} \frac{\text{area}(t_j)}{\text{area}(j)} \right]\right. \end{equation}

where area(s _i) is the area of object s _i in image i. S _s is higher if object s (rendered by its bounding box) is larger than we would expect for its object class l _s, which we estimate from all instances O(l _s) of class l _s in our training data (see Section 6.1 for details). Intuitively, S _s is a means to express the salience of an object in a specific scene. Binary features include the relative distance D _{s, t} between objects s and t, and their normalized co-occurrence frequency F _{s, t}. We estimate D _{s, t} as the relative Euclidean distance between the centers of s and t (in relation to the length of image i’s diagonal). F _{s, t} is estimated from the object occurrences in the parallel training. We furthermore include the spatial features used for the determination of the spatial relation between two objects, i.e., the relative area of s’s bounding box (unary), the intersection-over-union of s and t, and the normalized angle between s and t.

6.1 Experimental setup

Datasets: For both tasks, we used the publicly available Microsoft COCO Captions datasetFootnote ³ (Chen et al. Reference Chen, Fang, Lin, Vedantam, Gupta, Dollár and Zitnick2015) which contains more than 200,000 images (80 K training, 40 K validation, and 80 K testing images). All training and validation images are annotated with multiple object instances from eighty object classes (e.g., horse, airplane, banana), their categories (e.g., Animal, Vehicle, Food), and five captions, inter alia. The captions were collected via Amazon’s Mechanical Turk by asking annotators to describe the important parts of an image. We used the COCO training set in order to create a parallel corpus as described in Section 4.1 for training the SMT-based description generation models and the content selection model. In total, our training data contains approximately 50 K instances. The mean depth of the trees is 3.5 (std = 0.99; constituency) and 3.3 (std = 0.85; dependency). (For technical reasons, we represent edge labels by separate nodes, i.e., a dependency tree with two objects has depth 3.) On average, the training descriptions verbalize the interactions between two and three objects.

The reference descriptions for the COCO test data are not publicly available, and the performance of a model on the test data can be benchmarked by uploading its generated captions to the COCO test server. Note, however, that only automatic evaluation metrics, such as BLEU (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002), are used, which compare reference and system captions automatically on the basis of, e.g., n-gram agreement. Since we want to evaluate the models against human judgments in addition to reporting results using automatic measures, we created a test set of reference captions for N = 208 images from the COCO 2015 test set. In view of our interest in the verbalization of the relation between objects, images were sampled so that they contained highly frequent (seventy per cent of all test instances), moderately frequent (fifteen per cent), and infrequent object pairs (fifteen per cent; see Figure 5 for examples). Following the Amazon’s Mechanical Turk experimental design described in Chen et al. (Reference Chen, Fang, Lin, Vedantam, Gupta, Dollár and Zitnick2015), we elicited five reference descriptions for every sampled image.

Fig. 5.

Examples of images with frequent (left), moderately frequent (middle), and infrequent (right) object pairs from the COCO 2015 test set.

There exist many datasets created on top of the COCO images for tasks such as visual question answering (Antol et al. Reference Antol, Agrawal, Lu, Mitchell, Batra, Lawrence Zitnick and Parikh2015) or referring expressions interpretation (Yu et al. Reference Yu, Poirson, Yang, Berg and Berg2016). As far as content-based image retrieval is concerned, to the best of our knowledge, there is no corresponding extension of COCO. Elliott et al.’s (Reference Elliott, Lavrenko and Keller2014) work on predicate-centered content-based image retrieval, in turn, was limited to only ten actions. As we explain along with the discussion of our results, we therefore created a dataset from COCO by extracting the predicates from the image descriptions.

Preprocessing: We used Girshick’s (Reference Girshick2015) fast region-based convolutional neural network to detect objects in images. The system takes as input the whole image and region proposals and classifies and refines the proposals. Specifically, for each proposal, it outputs a softmax probability distribution over K + 1 object classes as well as refined bounding box positions for every class. We used the publicly available pretrained ImageNet (Russakovsky et al. Reference Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg and Fei-Fei2015) model CaffeNetFootnote ⁴ and fine-tuned it on the eighty object classes of the COCO object detection challenge using fast region-based convolutional neural network. Region proposals were extracted with selective search (Uijlings et al. Reference Uijlings, van de Sande, Gevers and Smeulders2013). We filtered overlapping boxes with a symmetric intersection-over-union of more than fifty per cent. Finally, we extracted the top fifteen detected locations from the images and filtered overlapping boxes rendering the same object class with a 0.3 intersection-over-union threshold.

6.2 Comparison models

Bag-of-objects: This baseline makes use of detected objects only, and does not exploit any structural information. The visual representation of an image is simply a bag of the top-N predicted objects. To evaluate this model on the image retrieval task, we create a flat tree by linking each object to a root node. For the image description task, we train an SMT model on a parallel corpus of bag-of-objectsFootnote ⁵ and COCO descriptions.

Tuples: Our second comparison model is the approach put forward in Ortiz et al. (Reference Ortiz, Wolff and Lapata2015) who represent visual scenes via tuples (see the example in Figure 1). Content selection in their model is implemented as an integer linear program which selects description-worthy pairs of objects using features such as their relative distance D _{s, t} and their normalized co-occurrence frequency F _{s, t} (see Section 4). To generate descriptions with this model, we adapt the pipeline described in Section 4.2, to the tuple-based framework. We create parallel data of visual tuples and descriptions from our cross-modal corpus of object–argument alignments and use it to train a phrase-based SMT model. We then run the integer linear program on each test image to obtain relevant object pairs whose interactions we wish to describe, convert them to visual tuples, and use the SMT model to transfer them to their verbalizations. For the image retrieval task, we transfer the tuples to trees in which two objects are direct children of their spatial relation. Tree kernel methods are then used to measure image similarity.

Templates: In our description generation experiments, we further compared against the template-based model of Elliot and deVries (Reference Elliott and de Vries2015) which generates a description for an image by filling in the template ‘DT o _s is V DT o _t’. The slot DT takes a determiner (a or an), o _s and o _t are filled with the class labels of the parent and child object nodes in the corresponding visual tree representation (e.g., o_s = dog and o_t = frisbee). Slot V is filled with the verb v that maximizes the conditional probability: v* = arg maxp(v|head, child, sr), where sr is the spatial relation between the objects o _s and o _t (e.g., p(catch|dog, frisbee, very_close), p(eat|person, pizza, surrounds_above)). We used the parallel corpus described in Section 4.1 to derive these probability estimates. In cases where we cannot find a verb to fill slot V, we default to the spatial relation sr (e.g., surrounds or is near).

NIC: Finally, we compare our approach to description generation against the Neural Image Caption (NIC) model of Vinyals et al. (Reference Vinyals, Toshev, Bengio and Erhan2015).Footnote ⁶ We selected NIC as an example of Recurrent Neural Network-based approaches. It was also trained on COCO and performs reasonbaly well even against a nearest-neighbor (NN) baseline (Vinyals et al. Reference Vinyals, Toshev, Bengio and Erhan2015). The latter has proven very competitive on COCO due to the images being similar and as result giving rise to similar linguistic descriptions (Devlin et al. Reference Devlin, Gupta, Girshick, Mitchell and Zitnick2015). In Vinyal et al.’s approach, a convolution neural network (CNN) encodes an image into a compact representation, which is then fed to a long short-term memory decoder to generate a sentence. The long short-term memory is trained to maximize the likelihood of the correct description given the image.

VGG: For our image retrieval experiments, we employ a NN approach (Devlin et al. Reference Devlin, Gupta, Girshick, Mitchell and Zitnick2015) which finds the k NNs using cosine similarity and represents images with a 4,096-dimensional vector. The latter is extracted from the image activations from the penultimate layer of a pretrained CNN model (VGG-16, Simonyan & Zisserman (Reference Simonyan and Zisserman2014).

6.3 Parameter settings

All parameters, including the choice of spatial relations, the features for the content selection models, and the parameters for grounding PASs, were tuned on the COCO validation data. All SMT models were trained on the corresponding parallel corpus obtained from COCO. Six-gram SMT language models were trained on both Flickr30K descriptions (Young et al. Reference Young, Lai, Hodosh and Hockenmaier2014) and COCO descriptions with modified Kneser–Ney smoothing using KenLM (Heafield et al. Reference Heafield, Pouzyrevsky, Clark and Koehn2013).

6.5 Error analysis

Image retrieval: We manually inspected the output of various image retrieval systems on the COCO test data. Figure 8 presents examples of typical errors we found. It shows eight query images (first column) and the one-best image retrieved for each system (columns 2 to 5; for Query 6 we show the two best images). Our analysis reveals that models which use symbolic image representations (i.e., the eighty COCO object class names; all models except for VGG) are susceptible to errors made by the object detectors (e.g., Queries 3 and 6 for Tuples, Query 7). Similarly, objects which are unknown to the detectors result either in detection errors or cannot be captured by the representations at all (e.g., the tree in Query 1). The effect of object detection is most pronounced in the bag-of-objects system which treats all objects equally, lacking any structural information, e.g., based on the spatial relations. For instance, representations for Query 5, which shows two bears walking beside each other, were built on the basis of the objects bear, bear, cell_phone, and bird, causing bag-of-objects to retrieve an image of a bird as its NN. Note that the image retrieved by VGG for Query 5 is visually similar, but does not depict bears either. Recall that VGG represents images by their CNN-based feature vectors, and does not directly rely on discrete object detections. Query 2 illustrates another source of typical errors for the structural approaches: They struggle with images which do not explicitly depict object (inter)actions. VGG is better at retrieving answers which are visually similar to the query, i.e., which depict visually similar objects and background scenes. But it has difficulty finding images which capture the same semantic content as object interactions depicted in the query. For example, Query 3 shows a person and a cake, which all models (except Tuples) get right. In VGG’s answer image, however, the person is not cutting the cake. For Query 5, VGG’s answer shows formally dressed people, but neither of them is holding a glass.

We also observe that for the given dataset (a collection $\mathcal {I}$ of 868 images; see Section 5), our relatively few spatial relations may be too fine-grained. For example, Query 1 was represented as a person being above_veryclose to a frisbee. $\mathcal {I}$, however, only contains an image of a person being above_close to a frisbee.

Description generation: Figures 6 and 7 provide examples of the systems’ output on the description generation task. As observed with the retrieval task, since all systems (except for NIC) use symbolic representations, they are affected by errors of the object detectors and propagated to the generated descriptions. Furthermore, since we focus on objects and their interactions and do not recognize attributes or scenes, descriptions produced by our models often contain less detail compared to human authored ones. For example, they do not mention parts of objects or attributes (see Figure 6 first image). Perhaps, unsurprisingly, SMT-based models tend to generate more fluent and natural descriptions than template (see Figure 6 first row, right-most column; Figure 7 first image). Constituency tends to be better than dependency in handling uncommon object co-occurrences for which it more often produces grammatical descriptions (see first image in Figure 6 left-most column), or verbalizes only parts of the representation.

Fig. 6.

Examples of system output. Numbers denote the relative rank given by AMT participants.

Fig. 7.

Examples of system output. Numbers denote the relative rank given by AMT participants.

As far as NIC is concerned, we observed that thirty-three per cent of the descriptions it generated are duplicates, i.e., they are identical with captions produced for other images. The proportion of duplicates is considerably less for other models, i.e., between nineteen per cent (tuples) and thirteen per cent (templates). An explanation for NIC’s high proportion of duplicates may be that it has a generation mechanism similar to retrieval-based captioning systems—it reproduces the captions from the training images which are visually very similar to the target image. We give an example for Query image 4 (which shows a woman holding a wine glass) in Figure 8. Recall that NIC receives the same input as the VGG retrieval model. For Query 4, NIC generated the description a couple of men standing next to each other, which turns out to be an accurate description of the image which was retrieved by VGG.

Fig. 8.

Query image (first column) and one-best image retrieved from the test data. The second column shows the output of the bag-of-objects baseline, the third column shows the output of the tuples baseline, and so own. For Query 6, we show the two best retrieved images.