1 Introduction
Image caption generation is the task of generating a natural language description of the content of an image (Bernardi et al. Reference Bernardi, Cakici, Elliott, Erdem, Erdem, Ikizler-Cinbis, Keller, Muscat and Plank2016), also known as a caption. One way to do this is to use a neural language model, typically in the form of a recurrent neural network (RNN), which is used to generate text (illustrated in Figure 1). Given a sentence prefix, a neural language model will predict which words are likely to follow. With a small modification, this simple model can be extended into an image caption generator, that is, a language model whose predictions are conditioned on image features. To do this, the neural language model must somehow accept as input not only the sentence prefix, but also the image being captioned. This raises the question: At which stage should image information be introduced into a language model?
How RNN-based neural language models work. Legend: RNN – Recurrent Neural Network; FF – Feed-Forward layer; wordi – the ith generated word in the text; wordstart – the START token which is an artificial word placed at the beginning of every sentence in order to still have a prefix when predicting the first word (likewise there is an END token to predict the end of a sentence). Note that state1 represents the prefix ‘wordstart’, state2 represents the prefix ‘wordstart word1’, etc. After processing a prefix, the RNN passes its final state statet to a feed-forward layer, which then predicts how likely each known word is to be the next word in the prefix.
Recent work on image captioning has answered this question in different ways, suggesting different views of the relationship between image and text in the caption generation task. To our knowledge, however, these different models and architectures have not been systematically compared. Yet, the question of where image information should feature in captioning is at the heart of a broader set of questions concerning how language can be grounded in perceptual information, questions which have been addressed by cognitive scientists (Harnad Reference Harnad1990) and AI practitioners (Roy Reference Roy2005).
As we will show in more detail in Section 2, differences in the way caption generation architectures treat image features can be characterised in terms of three distinct sets of design choices:
Conditioning by injecting versus conditioning by merging: A neural language model can be conditioned by injecting the image (Figure 1 a) or by merging the image (Figure 1 b). In ‘inject’ architectures, the image vector (usually derived from the activation values of a hidden layer in a convolutional neural network) is injected into the RNN, for example, by treating it on a par with a ‘word’ and including it as part of the caption prefix. The RNN is trained to encode the image–language mixture into a single vector in such a way that this vector can be used to predict the next word in the prefix. On the other hand, in the case of ‘merge’ architectures, the image is left out of the RNN subnetwork, such that the RNN handles only the caption prefix, that is, handles only purely linguistic information. After the prefix has been encoded, the image vector is then merged with the prefix vector in a separate ‘multimodal layer’ that comes after the RNN subnetwork. Merging can be done by, for example, concatenating the two vectors together. In this case, the RNN is trained to only encode the prefix and the mixture is handled in a subsequent feed-forward layer.
The inject and merge architectures for caption generation. Legend: RNN – Recurrent Neural Network; FF – Feed-Forward layer.
In the terminology adopted in this paper: if an RNN’s hidden state vector is somehow influenced by both the image and the words then the image is being injected, otherwise it is being merged.
Early versus late inclusion of image features: As the foregoing description suggests, merge architectures tend to incorporate image features somewhat late in the generation process, that is, after processing the whole caption prefix. On the other hand, some inject architectures tend to incorporate image features early in the generation process. Other inject architectures incorporate image features for the whole duration of the generation process. Different architectures can make visual information influence linguistic choices at different stages.
Fixed versus modifiable image features: For each word predicted, some form of visual information must be available to influence the likelihood of each word. Merge architectures typically use the exact same image representation for every word output. On the other hand, injecting the image features into the RNN allows the internal representation of the image inside the hidden state vector to be changed by the RNN’s internal updates after each time step. Different architectures allow for different degrees of modification in the image features for each generated word.
The main contribution of this paper is to present a systematic comparison of the different ways in which the ‘conditioning’ of linguistic choices based on visual information can be carried out, studying their implications for caption generator architectures. Thus, rather than seeking new results that improve on the state of the art, we seek to determine, based on an exhaustive evaluation of inject and merge architectures on a common dataset, where image features are best placed in the caption generation and image retrieval process.Footnote 1
From a scientific perspective, such a comparison would be useful for shedding light on the way language can be grounded in vision. Should images and text be intermixed throughout the process, or should they initially be kept separate before being combined in some multimodal layer? Many papers speak of RNNs as ‘generating’ text. Is this the case or are RNNs better viewed as encoders that vectorise a linguistic prefix so that the next feed-forward layer can predict the next word, conditioned on an image? Answers to these questions would help inform theories of how caption generation can be performed. The architectures we compare provide different answers to these questions. Hence, it is important to acquire some insights into their relative merits.
From an engineering perspective, insights into the relative performance of different models could provide rules of thumb for selecting an architecture for the task of image captioning, possibly for other tasks as well such as machine translation. This would make it easier to develop new architectures and new ways to perform caption generation.
The remainder of this paper is structured as follows. We first give an overview of published caption generators based on neural language models, focusing in particular on the architectures used. Section 3 discusses the architectures we compare, followed by a description of the data and experiments in Section 4. Results are presented and discussed in Section 5. We conclude with some general discussion and directions for future work.
2 Background
In this section, we discuss a number of recent image caption generation models with emphasis on how the image conditions the neural language model, based on the distinction between inject and merge architectures illustrated in Figure 2. Before we discuss these models, we first outline four broad sub-categories of architectures that we have identified in the literature.
2.1 Types of architectures
In Section 1, we made a high-level distinction between architectures that merge linguistic and image features in a multimodal layer, and those that inject image features directly into the caption prefix encoding process. We can in fact distinguish four theoretical possibilities arising from these, as illustrated in Figure 3 and described below.
• Init-inject: The RNN’s initial hidden state vector is set to be the image vector (or a vector derived from the image vector). It requires the image vector to have the same size as the RNN hidden state vector. This is an early binding architecture and allows the image representation to be modified by the RNN.
• Pre-inject: The first input to the RNN is the image vector (or a vector derived from the image vector). The word vectors of the caption prefix come later. The image vector is thus treated as a first word in the prefix. It requires the image vector to have the same size as the word vectors. This too is an early binding architecture and allows the image representation to be modified by the RNN.
• Par-inject: The image vector (or a vector derived from the image vector) serves as input to the RNN in parallel with the word vectors of the caption prefix, such that either (a) the RNN takes two separate inputs; or (b) the word vectors are combined with the image vector into a single input before being passed to the RNN. The image vector does not need to be exactly the same for each word (such as is the case with attention-based neural models); nor does it need to be included with every word. This is a mixed binding architecture and, whilst allowing some modification in the image representation, it will be harder for the RNN to do so if the same image is fed to the RNN at every time step due to its hidden state vector being refreshed with the original image each time.
• Merge: The RNN is not exposed to the image vector (or a vector derived from the image vector) at any point. Instead, the image is introduced into the language model after the prefix has been encoded by the RNN in its entirety. This is a late binding architecture and it does not modify the image representation with every time step.
Different ways of conditioning a neural language model with an image. The feed-forward layer was left out to save space.
With these distinctions in mind, we next discuss a selection of recent contributions, placing them in the context of this classification.
Init-inject architectures: Architectures conforming to the init-inject model treat the image vector as the initial hidden state vector of an RNN (Devlin et al. Reference Devlin, Cheng, Fang, Gupta, Deng, He, Zweig and Mitchell2015; Liu et al. Reference Liu, Zhu, Ye, Guadarrama and Murphy2016). Wang et al. (Reference Wang, Song, Yang and Luo2016) combine two RNNs in parallel, both initialized with the same image.
A similar architecture to init-inject is used in traditional deep learning machine translation systems (Sutskever, Vinyals and Le Reference Sutskever, Vinyals, Le, Ghahramani, Welling, Cortes, Lawrence and Weinberger2014) where a source sentence is encoded into a vector and used to condition a language model to generate a sentence in another language. This is the basis for the system described by Ma and Han (Reference Ma and Han2016), who first extract a sequence of attributes from an image, then translate this sequence into a caption.
It is also used in attention mechanisms in order to provide a vector representing information about the whole image, whilst parts of the image that are attended differently during each time step are provided via par-injection. For example, Xu et al. (Reference Xu, Ba, Kiros, Cho, Courville, Salakhutdinov, Zemel and Bengio2015) initialize the RNN with the centroid of all image parts before attending to some parts as needed.
Pre-inject architectures: Pre-inject models treat the image as though it were the first word in the prefix (Nina and Rodriguez Reference Nina and Rodriguez2015; Vinyals et al. Reference Vinyals, Toshev, Bengio and Erhan2015; Rennie et al. Reference Rennie, Marcheret, Mroueh, Ross and Goel2017). Image attributes are sometimes used instead of image vectors (Wu et al. Reference Wu, Shen, van den Hengel, Liu and Dick2015; Yao et al. Reference Yao, Pan, Li, Qiu and Mei2017). Yao et al. (Reference Yao, Pan, Li, Qiu and Mei2017) also try passing an image as the first two words instead of just one word by using the image vector as the first word and image attributes as a second, or vice versa.
Just like init-inject, pre-inject is also used to provide information about the whole image in attention mechanisms (You et al. Reference You, Jin, Wang, Fang and Luo2016; Zhou et al. Reference Zhou, Xu, Koch and Corso2016).
Par-inject architectures: Par-injection inputs the image features into the RNN jointly with each word in the caption. It is by far the most common architecture used and has the largest variety of implementation. For example, Donahue et al. (Reference Donahue, Hendricks, Guadarrama, Rohrbach, Venugopalan, Saenko and Darrell2015) do this with two RNNs in series and find that it is better to inject the image in the second RNN than the first. Yao et al. (Reference Yao, Pan, Li, Qiu and Mei2017) par-inject the image whilst pre-injecting image attributes (or vice versa); and Liu et al. (Reference Liu, Zhu, Ye, Guadarrama and Murphy2016) par-inject attributes from the image whilst init-injecting the image vector. Other, less common instantiations include par-injecting the image, but only with the first word (this is not pre-inject as the image is not injected on a separate time step) (Karpathy and Fei-Fei Reference Karpathy and Fei-Fei2015; Hessel, Savva and Wilber Reference Hessel, Savva and Wilber2015); and passing the words through a separate RNN, such that the resulting hidden state vectors are what is combined with the image vector (Oruganti et al. Reference Oruganti, Sah, Pillai and Ptucha2016).
Many times this architecture is used in order to pass a different representation of the same image with every word so that visual information changes for different parts of the sentence being generated. For example, Zhou et al. (Reference Zhou, Xu, Koch and Corso2016) perform element-wise multiplication of the image vector with the last generated word’s embedding vector in order to attend to different parts of the image vector. Oruganti et al. (Reference Oruganti, Sah, Pillai and Ptucha2016) pass the image through its own RNN for as many times as there are words in order to use a different image vector for every word. Chen and Zitnick (Reference Chen and Zitnick2014, Reference Chen and Zitnick2015) use a simple RNN to try to predict what the image vector looks like given a prefix. This predicted image is then used as a second image representation, which is par-injected together with the actual image vector.
More commonly, modified image representations come from attention mechanisms (Xu et al. Reference Xu, Ba, Kiros, Cho, Courville, Salakhutdinov, Zemel and Bengio2015; You et al. Reference You, Jin, Wang, Fang and Luo2016; Rennie et al. Reference Rennie, Marcheret, Mroueh, Ross and Goel2017). Rennie et al. (Reference Rennie, Marcheret, Mroueh, Ross and Goel2017) inject the image not as an input to the RNN but use a modified long short-term memory network (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997), or LSTM, which allows them to inject the attended image directly inside the input gated expression (the part of the LSTM which is multiplied by the input gate).
Like init-inject and pre-inject, par-inject is sometimes used to provide information about the whole image in attention mechanisms whilst the attended image regions are merged (Lu et al. Reference Lu, Xiong, Parikh and Socher2017).
Merge architectures: Rather than combining image features together with linguistic features from within the RNN, merge architectures delay their combination until after the caption prefix has been vectorised (Mao et al. Reference Mao, Xu, Yang, Wang and Yuille2014, Reference Mao, Xu, Yang, Wang, Huang and Yuille2015a, Reference Mao, Xu, Yang, Wang, Huang and Yuille2015b). Hendricks et al. (Reference Hendricks, Venugopalan, Rohrbach, Mooney, Saenko and Darrell2016) use a merge architecture in order to keep the image out of the RNN and thus be able to train the part of the neural network that handles images and the part that handles language separately, using images and sentences from separate training sets.
Some work on attention mechanisms also uses merge architectures with attention mechanisms by merging a different image representation at every time step. You et al. (Reference You, Jin, Wang, Fang and Luo2016) and Xu et al. (Reference Xu, Ba, Kiros, Cho, Courville, Salakhutdinov, Zemel and Bengio2015) merge as well as par-inject the attended visual regions, whilst Lu et al. (Reference Lu, Xiong, Parikh and Socher2017) only merge the regions whilst par-injecting a fixed image representation.
Though they do not use an RNN and hence are not focussed on in this review, caption generators that use log-bilinear models (Mnih and Hinton Reference Mnih and Hinton2007) usually merge the image with the prefix representation (Kiros, Salakhutdinov and Zemel Reference Kiros, Salakhutdinov and Zemel2014a, Reference Kiros, Salakhutdinov and Zemel2014b; Song and Yoo Reference Song and Yoo2016).
2.2 Summary and outlook
While the literature on caption generation now provides a rich range of models and comparative evaluations, there is as yet very little explicit systematic comparison between the performances of the architectures surveyed above, each of which represents a different way of conditioning the prediction of language sequences on visual information. Work that has tested both par-inject and pre-inject, such as (Vinyals et al. Reference Vinyals, Toshev, Bengio and Erhan2015), reports that pre-inject works better. The work of (Mao et al. Reference Mao, Xu, Yang, Wang, Huang and Yuille2015a) compares inject and merge architectures and concludes that merge is better than inject. However, Mao et al.’s comparison between architectures is a relatively tangential part of their overall evaluation, and is based only on the BLEU metric (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002).
Answering the question of which architecture is best is difficult because different architectures perform differently on different evaluation measures, as shown for example by Wang et al. (Reference Wang, Song, Yang and Luo2016), who compared architectures with simple RNNs and LSTMs. Although the state of the art systems in caption generation all use inject-type architectures, it is also the case that they are more complex systems than the published merge architectures and so it is not fair to conclude that inject is better than merge based on a survey of the literature alone.
In what follows, we present a systematic comparison between all the different architectures discussed above. We perform these evaluations using a common dataset and a variety of quality metrics, covering (a) the quality of the generated captions; (b) the linguistic diversity of the generated captions; and (c) the networks’ capabilities to determine the most relevant image given a caption.
3 Architectures
In this section, we go over the different architectures that are evaluated in this paper. A diagram illustrating the main architecture schema, which is the basis of every tested architecture, is shown in Figure 4. The schema is based on the architecture described by Vinyals et al. (Reference Vinyals, Toshev, Bengio and Erhan2015), without the ensemble. This architecture was chosen for its simplicity, whilst still being the best performing system in the 2015 MSCOCO image captioning challenge.Footnote 2
An illustration of the main architecture schema that is instantiated in the four different architectures tested in this paper. Legend: ‘FF’ – fully connected layer with bias; ‘FFimg’ – layer projecting the image vector (may or may not have an activation function); ‘FFout’ – layer projecting into the softmax output; ‘l’ – the layer size (which is the same for three different layers); ‘v’ – the vocabulary size (which is different for different datasets). Only one of the dashed arrows is used depending on whether the architecture is one of merge or inject.
Word embeddings: Word embeddings, that is, the vectors that represent known words prior to being fed to the RNN, consist of vectors that have been randomly initialised. No precompiled vector embeddings such as word2vec (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013) were used. Instead, the embeddings are trained as part of the neural network in order to learn the best representations of words for the task.
Recurrent neural network: The purpose of the RNN is to take a prefix of embedded words (with image vector in inject architectures) and produce a single vector that represents the sequence. A gated recurrent unit (Chung et al. Reference Chung, Gülçehre, Cho and Bengio2014), or GRU, was used in our experiments for the simple reason that it is a powerful RNN that only has one hidden state vector. By contrast, an LSTM has two state vectors (hidden and cell states). This would make architecture comparisons more complex, as the presence of two state vectors raise the possibility of multiple versions of the init-inject architecture. By using an RNN with a single hidden state vector there is only one way to implement init-inject.
Image: Prior to training, all images were vectorised using the activation values of the penultimate layer of the VGG OxfordNet nineteen-layer convolutional neural network (Simonyan and Zisserman Reference Simonyan and Zisserman2014), which is trained to perform object recognition and returns a 4096-element vector. The convolutional neural network is not influenced by the caption generation training process. During training, a feed-forward layer of the neural network compresses this vector into a smaller vector.
Output: Once the image and the caption prefix have been vectorised and mixed into a single vector, the next step is to use them to predict the next word in the caption. This is done by passing the mixed vector through a feed-forward layer with a softmax activation function that outputs the probability of each possible next word in the vocabulary. Based on this distribution, the next word that comes after the prefix is selected.
The four architectures discussed in the previous section are evaluated in our experiments as follows:
• init-inject: The image vector is treated as an initial hidden state vector for the RNN. After initialising the RNN, the vectors in the caption prefix are then fed to the RNN as usual.
• pre-inject: The image vector is used as the first ‘word’ in the caption prefix. This makes the image vector the first input that the RNN will see.
• par-inject: The image vector is concatenated to every word vector in the caption prefix in order to make the RNN take a mixed word-image vector. Every word would have the exact same image vector concatenated to it.
• merge: The image vector and caption prefix vector (RNN state vector) are concatenated into a single vector before being fed to the output layer.
4 Experiments
This section describes the experiments conducted in order to compare the performance of the different architectures described in the previous section. TensorflowFootnote 3 v1.2 was used to implement the neural networks.
4.1 Datasets
The datasets used for all experiments were the version of Flickr8K (Hodosh, Young and Hockenmaier Reference Hodosh, Young and Hockenmaier2013), Flickr30K (Young et al. Reference Young, Lai, Hodosh and Hockenmaier2014), and MSCOCO (Lin et al. Reference Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár and Zitnick2014) distributed by Karpathy and Fei-Fei (Reference Karpathy and Fei-Fei2015).Footnote 4 All three datasets consist of images taken from Flickr combined with between five and seven manually written captions per image. The provided datasets are split into a training, validation, and test set using the following number of images respectively: Flickr8K – 6000, 1000, 1000; Flickr30K – 29000, 1014, 1000; MSCOCO – 82783, 5000, 5000. The images are already vectorised into 4096-element vectors via the activations of layer ‘fc7’ (the penultimate layer) of the VGG OxfordNet 19-layer convolutional neural network (Simonyan and Zisserman Reference Simonyan and Zisserman2014), which was trained for object recognition on the ImageNet dataset (Deng et al. Reference Deng, Dong, Socher, Li, Li and Fei-Fei2009).
The known vocabulary consists of all the words in the captions of the training set that occur at least five times. This amounts to 2539 for Flickr8K, 7415 for Flickr30K, and 8792 for MSCOCO. These words are used both as inputs, which are embedded and fed to the RNN, and as outputs, which are assigned probabilities by the softmax function. Any other word which is not part of the vocabulary is replaced with an UNKNOWN token.
4.2 Hyperparameter tuning
For the results to be reliable, it is important to find the best (within practical limits) hyperparameters for each architecture so that we can judge the performance of the architectures when they are optimally tuned, rather than using one-size-fits-all hyperparameter settings that might cause some architectures to under-perform. For this reason, we used a multi-step process of hyperparameter tuning, which is described below. We optimized the hyperparameters in order to maximize caption quality on the Flickr8K validation set, using beam search as a generation method and CIDEr as the objective function. The optimal hyperparameters were then fixed across all datasets. Mao et al. (Reference Mao, Xu, Yang, Wang, Huang and Yuille2015a) also used Flickr8K for hyperparameter tuning and CIDEr was shown by Rennie et al. (Reference Rennie, Marcheret, Mroueh, Ross and Goel2017) to be a useful metric to optimise on, yielding an improvement on other quality metrics when used as the objective function.
The following hyperparameters were fixed across all architectures:
• Parameter optimization is performed using the Adam algorithm (Diederik and Ba Reference Diederik and Ba2014) with its hyperparameters kept as suggested in the original paper: α = 0.001, β1 = 0.9, β2 = 0.999, and ε = 10−8.
• Loss function is the mean of the cross-entropy of each word in each caption in a minibatch.
• An early stopping criterion is used, such that the geometric mean of the language model perplexity on the validation set is measured and as soon as one epoch results in a worse perplexity than the previous epoch, the training stops. A maximum number of epochs are still used to prevent training from going on for too long (more on this later).
• During caption generation, the caption must be between five and fifty words long. Beam search will not end a sentence before there are at least five words in it and will abruptly stop a sentence that is fifty words long.
• All biases are initialized to zeros.
The following are hyperparameters that were tuned (the ranges of values were minimized in order to keep the search space tractable):
• The weights initialization procedure (normal distribution or xavier (Glorot and Bengio Reference Glorot and Bengio2010) with normal distribution).
• The weights initialization range (−0.1 to 0.1 or −0.01 to 0.01).
• The size of the layers for embedding, image projection (FFimg in Figure 4), and RNN hidden state vector (64, 128, 256, or 512), all three of which are constrained to be equal.Footnote 5
• Whether to normalize the image vector before passing it to the neural network.
• Whether to use ReLU after the image projection (FFimg in Figure 4) or to leave it linear.
• Whether to use an all-zeros vector as an initial RNN hidden state vector or to use a learnable vector (not applicable to init-inject since its initial hidden state vector is the image projection).
• Whether to use L2 weights regularization with a weighting constant of 10−8.
• Whether to apply dropout regularisation at different points in the architecture (in Figure 4: after ‘image’, after ‘FFimg’, after ‘embed’, and/or after ‘RNN’). Each application of dropout (if any) has a dropout rate of 0.5.
• The minibatch size (32, 64, or 128).
The following steps were followed in order to tune these hyperparameters, which were evaluated by training a neural network for a maximum of ten epochs, generating captions with a beam width of 2, and evaluating the captions using CIDEr:
(1) Randomly generate 100 unique hyperparameter combinations and record their performance.
(2) Use Baysian optimization via the library GPyOptFootnote 6 for 100 iterations and record each generated candidate combination’s performance. Use the combinations from step 1 to initialize the search.
(3) Use trees of Parzan estimators via the library hyperoptFootnote 7 for 100 iterations and record each generated candidate combination’s performance.
(4) Take the best combination found in all of the previous steps and fine-tune it using greedy hill climbing and record each modified combination. This is to check if changing any one hyperparameter will improve the performance.
The previous steps do not have very reliable CIDEr scores associated with them as their score was produced using just one training and generation run and so might coincidentally be an unusual score (far from the mean score if we trained the same neural network several times). Ideally, we would have tested each hyperparameter combination three times and taken the mean of the resulting CIDEr scores. Ideally, we would have also tried different values for maximum number of epochs and beam width. This, however, would have been extremely time consuming. Thus, we only apply the procedure to a subset of the best performing combinations from the previous steps. We ensure that the subset is diverse by only choosing combinations that are dissimilar from each other, as follows:
(5) Take all duplicate combinations generated in all of the previous steps and replace them with a single combination with their average CIDEr score. Take the top ten scoring combinations.
(6) Out of the selected ten combinations take the three combinations that are most different from each other in terms of Hamming distance. Ensure that one of these three combinations is the best combination found in the previous step.
(7) Take the three combinations selected and try different maximum epochs (ten and 100) and beam widths (1, 2, 3, 4, 5, and 6) on them. Each evaluation is measured using the average CIDEr score of three independent training and generation runs.
(8) Return the best combination found in the previous step.
In Section 5.1, we will discuss the optimal hyperparameters found.
4.3 Evaluation metrics
To evaluate the different architectures, the test set captions (which are shared among all architectures) are used to measure the architectures’ quality using metrics that fall into three classes, described below.
Generation metrics: These metrics quantify the quality of the generated captions by measuring the degree of overlap between generated captions and those in the test set. We use the MSCOCO evaluation codeFootnote 8 that measures the standard evaluation metrics BLEU-(1,2,3,4) (Papineni et al. Reference Papineni, Roukos, Ward and Zhu2002), ROUGE-L (Lin and Och Reference Lin and Och2004), METEOR (Banerjee and Lavie Reference Banerjee and Lavie2005), and CIDEr (Vedantam, Zitnick and Parikh Reference Vedantam, Zitnick and Parikh2015).
Diversity metrics: Apart from measuring the caption similarity to the ground truth we also measure the diversity of the vocabulary used in the generated captions. This is intended to shed light on the extent to which the captions produced by models are ‘stereotyped’, that is, the extent to which a model re-uses (sub-)strings from case to case, irrespective of the input image.
As a limiting case, consider a caption generator that always outputs the same caption. Such a generator would have the lowest possible diversity score. In order to quantify this, we measure the percentage of known vocabulary words used in all generated captions and the entropy of the unigram and bigram frequencies in all the generated captions together, which is calculated as
\begin{align}
\text{entropy}(F) &= -\sum _{i=1}^{|F|} {P_i(F) \log _2{P_i(F)}}
\end{align}
\begin{align}
P_i(F) &= \frac{F_i}{\sum _{j=1}^{|F|} {F_j}}
\end{align}
Entropy gives a measure of how uniform the frequency distributions are (with higher entropy for more uniform distributions). The more uniform, the more likely that each unigram or bigram was used in equal proportion, rather than using the same few words for the majority of the time, hence the greater the variety of words used.
Finally, we also measure the percentage of generated captions that already exist in the training set, as an estimate of the extent to which a model evinces ‘parroting’, or wholesale caption reuse from the training set.
For these diversity metrics, we obtain a ceiling estimate by computing the same measures on the test set captions themselves. We take the first caption out of the group of human-written captions available for each image in the test set and apply these diversity metrics on them.
Retrieval metrics: Retrieval metrics are metrics that quantify how well the architectures perform when retrieving the correct image out of all the test set images in the test set given a corresponding caption. A conditioned language model can be used for retrieval by measuring the degree of relevance each image has to the given caption. Relevance is measured as the probability of the whole caption given the image (by multiplying together each word’s probability). Different images will give different probabilities for the same caption. The more probable the caption is, the more relevant the image.
We use the standard R@n recall measures (Hodosh et al. Reference Hodosh, Young and Hockenmaier2013), and report recall at 1, 5, and 10. Recall at n is the percentage of captions whose correct image is among the top n most relevant images.
Since this process takes time proportional to the number of captions multiplied by the number of images, the pool of possible captions to consider during retrieval excluded all captions except the first out of the group of captions available for each image in order to reduce the evaluation time. For MSCOCO, we only used the first 1,000 test set images out of 5,000 for the same reason, similar to Flickr8K and Flickr30K that only have 1,000 images.
We also included the language model perplexity. In order to aggregate the caption perplexity of the entire test set of captions into a single number, we report the geometric mean of all the caption’s scores.
5 Results and discussion
Three runs of each experiment, on each of the three datasets, were performed. For the various evaluation measures, we report the mean together with the standard deviation (reported in parentheses) over the three runs. For each run, the initial model weights, minibatch selections, and dropout selections are different since these are randomly determined. Everything else is identical across runs.
5.1 Optimal hyperparameters
We start by discussing the optimal hyperparameters found for each architecture which are listed in Table 1.
The optimal hyperparameters found for each architecture, tuned on Flickr8K with CIDEr as an objective function. Each row is explained in Section 4.2
It is interesting to note that, in every architecture’s optimal hyperparameters, the RNN output needs to be regularized with dropout, the image vector should not have a non-linear activation function or be regularized with dropout, and the image input vector must be normalized before being fed to the neural network. Par-inject seems to need the most help in terms of regularization and even in terms of beam width, whilst the small size of merge means that it needs the least amount of regularization.
The most interesting observation is that the merge architecture is much ‘leaner’ overall. In terms of RNN size, it needs half of what par-inject needs, and only a quarter of what init-inject and pre-inject require for optimal performance. This makes sense, since merge only needs the RNN for storing linguistic information, whilst the other architectures need to additionally store visual information from the image. Using a larger RNN with the merge architecture would likely lead to overfitting.
The implication is that init-inject and pre-inject are much more memory-hungry architectures that require large RNN hidden state vectors in order to function well, whilst merge is more efficient. In fact, the number of parameters for merge is between three and four times smaller than the number of parameters for init-inject and pre-inject. Merge is also about two or three times faster to train.
Of the inject architectures, par-inject has the smallest optimal RNN size. This is probably due to the fact that, in this model, the image is present at all-time steps, thereby necessitating less memory to be allocated to ‘remember’ visual information together with linguistic information, compared to early binding architectures. It’s interesting to note that the par-inject RNN size is equal to the size of the concatenated image and RNN hidden state vector in the merge architecture.
5.2 Quality of generated captions
Tables 2 and 3 display the metrics that measure the quality of generated captions, calculated using the MSCOCO evaluation toolkit and averaged over the three experimental runs.
Results of caption quality metrics (CIDEr, METEOR, and ROUGE-L). Best values are in bold.
Results of caption quality metrics (BLEU-1, BLEU-2, BLEU-3, and BLEU-4). Best values are in bold.
Does merge’s small size impact its performance when generated captions are compared to corpora? The metrics reported here show considerable variability in ranking of the various architectures depending on dataset. For example, CIDEr scores place init-inject at the top for both Flickr8K and MSCOCO, but merge outperforms it on this measure on Flickr30K. Comparing ROUGE-L, METEOR, and CIDEr, init-inject seems to be ranked highest over most datasets (the situation is far more variable with the BLEU scores in Table 3, however). However, the differences among architectures are very small. This is especially true for the larger MSCOCO dataset. Thus, though init-inject often comes out on top, the other architectures are not lagging behind by a wide margin.
5.3 Image retrieval
Image retrieval results across the three datasets are shown in Table 4.
Results of the image retrieval metrics. Language model perplexity was also included here. Best values are in bold.
When it comes to retrieving the most relevant image for a caption, we once again see merge ranked first on Flickr30K, while init-inject is at the top on Flickr8K and MSCOCO, on practically all R@n measures, as well as median rank. Interestingly, in the two sets of cases where init-inject outperforms other architectures, merge is a close second, at least for R@1. In terms of perplexity, the general picture is in favour of inject models, with merge evincing marginally greater perplexity on all datasets. Overall, however, the outcomes mirror those of the previous sub-section: differences among architectures do not seem compelling and although the init-inject model outperforms merge in a number of instances, merge is a close second.
5.4 Caption diversity metrics
Next, we turn to the caption diversity metrics, shown in Table 5.
Results of the caption diversity metrics. The metrics were also applied to the first caption of each image in the human written test set captions. Best values are in bold.
These diversity metrics evince the most dramatic performance differences. If we focus on the proportion of generated captions that were found in the training set, on MSCOCO, this figure ranges from just over forty percent for merge to over sixty percent for par-inject. With the exception of Flickr8K, merge has the lowest proportion of caption reuse overall. If these results are compared to those in preceding sub-sections, the fact that those models with the greatest tendency to reuse captions tend to perform well on corpus-based metrics such as CIDEr suggests that the datasets under consideration are highly stereotyped, perhaps with a significant amount of redundancy and lack of variety.
A similar observation has been made by Devlin et al. (Reference Devlin, Cheng, Fang, Gupta, Deng, He, Zweig and Mitchell2015). In a comparison of retrieval-based and neural architectures for image captioning, these authors found that corpus-based metrics (especially BLEU) tend to give higher scores on test instances where the images were very similar to training instances. Neural architectures performed better for more similar images overall.
The results obtained for the human captions (bottom rows of Table 5) suggest that the level of caption reuse by humans is extremely low compared to the models under consideration, though it stands at seven percent on MSCOCO.
Turning to the extent to which architectures use their training vocabulary, the picture that emerges is consistent with the above. While humans used between twenty-nine percent and forty-seven percent of the known vocabulary (taken from the training set) to describe the test set images, none of the evaluated systems used more than fourteen percent. The merge architecture tops the ranks for all datasets by a small margin, although unigram and bigram entropy is highest for pre-inject (Flickr8K and Flickr30K) and init-inject (MSCOCO).
We interpret these results as showing that neural caption generators require seeing a word in the training set very often in order to learn to use it. From a methodological perspective, this further implies that setting an even higher frequency threshold, below which words are mapped to the UNKNOWN token (the current experiments set the threshold at five), would be feasible and would make relatively little difference to the results.
5.5 Visual information retention
As noted in Section 1, one of the differences between the architectures under consideration is whether they incorporate the image features early or late. This raises the possibility of differences in the degree to which visual information is retained by each architecture in the multimodal vector, that is, the input to ‘FFout’ in Figure 4. This is where information about visual and linguistic input is combined and is the information bottleneck that the output depends on. The question we want to answer is: Do (early-binding) inject architectures tend to ‘forget’ about the image as more words are input into the RNN? Given that the RNN’s memory is finite, it should be difficult to retain information about all inputs as the length of the sequence increases, so information about the image might start fading away as the input sequence gets longer. Merge architectures do not have this problem with visual information as it is kept outside of the RNN and so is fully retained in the multimodal vector regardless of the number of time steps.
To measure how much visual information is retained as the number of time steps grows, we do the following:
(1) Take a trained neural network and input an image and a matching caption.
(2) Record the multimodal vector in the neural network at every time-step.
(3) Replace the image from the original neural network in step 1 with a randomly selected image, paired with the original caption, thus introducing an image-caption mismatch.
(4) Record the new, adulterated multimodal vector at every time-step for the new caption-image combination.
(5) Compare the original and adulterated vectors: if these converge as more words are fed to the model, it implies that the multimodal vector is losing image information, as it would be getting influenced less by the image and more by the prefix.
As a measure of distance between original and adulterated vectors, we use the mean absolute difference, that is, we take the absolute difference between each corresponding dimension in the two multimodal vectors and then take the mean of these differences. Mean absolute difference avoids giving a larger distance to larger vectors and is also intuitive as a measure of difference between vectors. It also keeps the distance between time steps exactly equal for merge, which is desirable since merge does not lose visual information across time steps.
For this set of experiments, we used all twenty-word captions in the MSCOCO test set and measured the mean distance over all twenty-one time steps (the twenty words plus the START token). Twenty-word captions are long enough to see a trend without ending up with too few captions (the mean caption length on the MSCOCO test set is about 10.4). To create a more reliable mean, we repeat this procedure 100 times so that the mean is over all images in the test set using 100 random images per instance. The results are shown in Figure 5.
Difference between multimodal vectors over time, when the caption is held constant but the image is changed. For a trained model, markers show mean absolute difference between multimodal vectors when a twenty-word caption is fed to the model with the original, versus a randomly chosen image. The marker at position 0 gives the multimodal vector difference after feeding in the START token whilst the marker at position twenty gives the multimodal vector difference after feeding in the last word in the caption.
None of the inject architectures maintained a consistent distance between the original and adulterated multimodal vectors. Crucially, the merge architecture also has the largest distance among all architectures, demonstrating that, in this architecture, the words in a caption exhibit a greater dependency on the image to which they pertain (hence, adulterating the multimodal vector with an irrelevant image alters the representation considerably). Par-inject comes in second place in terms of multimodal vector distance. This suggests that it retains more visual information than the other inject architectures, though not as much as merge. It seems that the amount of retention across time steps changes somewhat unpredictably, but tends to decrease overall, which means that information gets lost over time (though not to the extent of init-inject and pre-inject). Init-inject comes third in visual information retention followed by pre-inject, both of which decrease over time. It seems that, in a GRU trained for caption generation, the initial hidden state vector exerts more influence on the final hidden state vector than the first input.
These results predict that if the generated captions needed to be very long, late binding architectures will produce better captions as they will retain visual information over longer time steps, maintaining a tighter coupling between visual and linguistic information.
6 Conclusion
This paper presented a systematic evaluation of a number of variations on architectures for image caption generation and retrieval. The primary focus was on the distinction between what we have termed ‘inject’ and ‘merge’ architectures. The former type of model mixes image and language information by training an RNN to encode an image-prefix mixture. By contrast, merge architectures maintain a separation between an RNN subnetwork, which encodes a linguistic string, and the image vector, merging them late in the process, prior to a prediction step. These models are therefore compatible with approaches to image caption generation using a ‘multimodal’ layer (Mao et al. Reference Mao, Xu, Yang, Wang and Yuille2014, Reference Mao, Xu, Yang, Wang, Huang and Yuille2015a, Reference Mao, Xu, Yang, Wang, Huang and Yuille2015b; Hendricks et al. Reference Hendricks, Venugopalan, Rohrbach, Mooney, Saenko and Darrell2016). While both types of architectures have been discussed in the literature, the inject architecture has been more popular.
Yet, there has been little systematic evaluation of its advantages compared to merge. Our experiments show that on standard corpus-based metrics such as CIDEr, the difference in performance between architectures is rather small. Init-inject tends to be better at generation and retrieval measures. Thus, from the perspective of corpus similarity, early binding of image features in models that view such features as ‘modifiable’ (in the sense outlined in the introduction) appear to be better than the alternatives.
Crucially, however, we also show that inject architectures are much more likely to re-generate captions wholesale from the training data and evince less vocabulary variation. Hence, from the perspective of variation, late-binding models that treat image features as fixed (i.e. not mixed with linguistic features) are better. While this is due in part to the nature of the available corpora, the superior performance of merge on this measure does suggest that, by encoding information from the two modalities separately, merge architectures might be producing less generic and stereotyped captions, exploiting their multimodal resources more effectively.
Our experiments on visual information retention show that, over time, inject architectures tend to loosen the coupling between visual and linguistic features, so that the difference between actual and adulterated multimodal vectors gets smaller. This too supports the view that inject models may, especially for longer captions, tend towards more generic and less image-specific captions, a finding that echoes the observations of Devlin et al. (Reference Devlin, Cheng, Fang, Gupta, Deng, He, Zweig and Mitchell2015), to some extent. In any case, late merging is, by definition, not susceptible to this problem.
From an engineering perspective, there is a significant difference between the required sizes of the RNN hidden state vectors. Whilst merge only requires a hidden state vector size sufficient to ‘remember’ caption prefixes, which depends on the length and complexity of the training set captions, inject architectures require additional memory to also store image information. This means that merge architectures make better use of their RNN memory. They also require less regularization whilst maintaining similar performance as other architectures.
The work presented here opens up some avenues for future research. In future work, we hope to investigate whether the results in this paper would remain similar when the experiments are repeated on other applications of conditioned neural language models such as neural machine translation or question answering.
Furthermore, by keeping language and image information separate, merge architectures lend themselves to potentially greater portability and ease of training. For example, it should be possible in principle to take the parameters of the RNN and embedding layers of a general text language model and transfer them to the corresponding layers in a caption generator. This would reduce training time as it would avoid learning the RNN weights and the embedding weights of the caption generator from scratch. As understanding of deep learning architectures evolves in the NLP community, one of our goals should be to maximise the degree of transferability among model components.
