2.1 Image Pre-Processing

We begin by transforming the images into a readable format for the computer. This process involves representing each image as a numerical array, where each entry depicts a specific pixel value. The left-hand side of Figure 1, for example, illustrates how a 13 (height) $\times $ 13 (width) pixels picture showing a handwritten “1” can be transformed into a matrix of $13\times 13=169$ units, each of them specifying the light intensity of a specific pixel.Footnote ² In the case of a color image, the transformation would produce three matrices of the same size, one for each primary color channel (red, green, and blue) in which the values in each cell represent the intensity of the corresponding color.

Optimizing the images for being used in a CNN involves making sure all files share the same shape, size, and contrast range. The first step is to convert all images into squares of the same size. This is an important requirement, because squared images optimize the linear algebra calculations involved in the convolution (Rosebrock Reference Rosebrock2017). The three most common methods for squaring the image include squashing the image’s largest side, adding black bordering to its shortest one, or center-cropping it. The latter option is the most common practice, because it also reduces the number of pixels that the model will process, leading to faster learning.

We also need to scale all images to the same pixel range. This step prevents the model from being biased toward images with large-value pixels. A common scaling approach is normalization, which constrains the pixels to the $[0,1]$ range by dividing the raw values by 255, the largest possible pixel value. An alternative scaling approach is centering, which divides the difference between each pixel and the mean pixel value in the image by the standard deviation of the pixel values in the data. Unlike normalization, centering transforms the pixel values to have a mean of zero and a variance of one.

The resulting input matrix is the core unit of analysis. The goal of the CNN is to extract the most relevant information from this matrix while gradually reducing its dimensionality. Given the way in which convolution works (see Section 2.2), CNNs downplay the features along the edges of the image. To prevent this problem, it is suggested to apply zero padding, or appending a perimeter of zeros to the input matrixFootnote ³ .

Finally, we dispatch all of the images in our database into four subsets: (1) one that will fit the model (training set), (2) another one that evaluates its performance and allows parameter tuning (validation set), (3) a third one to test the performance of the “best” model from the previous step (testing set), and (4) a final one that will be labeled after training and tuning the model (target set). The training data include those examples that the model uses to learn the patterns corresponding to each outcome category. The validation and test datasets include examples that help us to find the parameters that optimize classification and check the generalizability of the model’s predictions. The model is only allowed to observe and predict the labels of the validation and test data, but it is not allowed to learn from them. Finally, the target set includes data from unlabeled examples and is never used during the training stage. The whole point of a supervised learning method is to train and test a model that helps us label the examples in the target set.

2.2 Feature Representation

The second stage of the CNN begins when the input matrix passes throughout the first CONV. Each convolutional layer contains a set of smaller matrices, called filters, each of them representing a particular visual feature. The filters in the first layer depict basic features, such as straight or diagonal lines (see Figure A.2 in the online Appendix for an illustration). Subsequent layers build upon those features and transition to more complex ones, from lines to contours, to shapes, and to objects. The more layers a CNN has, the more complex features of the image it will recognize (Buduma and Locascio Reference Buduma and Locascio2017; Qin et al.  Reference Qin, Yu, Liu and Chen2018).

The goal of each filter is to extract different information from the input matrix. Every time a filter slides across the width and height of the input matrix, it computes the dot product between itself and the corresponding image subregion, a process similar to a cross-correlation.Footnote ⁴ The resulting dot products of a filter form a new matrix called receptive field, which represents how well the filter “matches” a region of the image.

Convolution requires three hyperparameters to be defined by the researcher. We briefly describe each of them and provide general recommendations to choose their values in Section 3.1. First, the filter size is the product of the filter’s width and height. For example, the filter size in the example described in Figure 1 is $3\times 3=9$ . Small filters capture fine-grained details, but they are likely to mix up the relevant information from an image with its noise. On the other hand, large filters look for details of a larger size at the cost of a lower specificity. Second, the filter stride is an integer number defining how many pixels the filter will slide through the image. The smaller the stride, the more information from the image that is preserved during the convolution.Footnote ⁵ Finally, the layer depth defines the number of filters in CONV. Note that all of the filters set by this parameter have the same size.

We illustrate how a convolution works in Figure 2. The convolution process involves computing the dot products of the filter and the values of every equivalent pixel space in the image. In this example, the dot product between the entries of the filter and the input of the highlighted image area is $2.2$ . Using a stride of 3, the filter slides three steps to the right and again computes the dot product of its entries. The result of this operation is called the feature map, and the convolution process will create as many feature maps as filters specified in CONV.Footnote ⁶

Figure 2

Illustration of the convolution stage.

Since the resulting feature map is a linear transformation of the input matrix, adding more CONV layers at this point would be redundant; the result could be obtained with a single linear product. Such feature maps are unlikely to produce smooth gradients, a necessary input for the learning phase described in Section 2.3. To address this problem, it is necessary to include an activation or ACT layer. This layer applies a nonlinear transformation to the feature maps before being sent to the next convolution layer. The nonlinearity property allows the network to stack multiple layers and extract more information from the image.

The specific transformation that the ACT layer performs on the input depends on the activation function of the neuron. We briefly describe three of these functions below (Figure A.3 in the online Appendix illustrates their graphic representations). The first one, Sigmoid, is simply the inverse of the logistic function, which bounds the activation values to the $[0,1]$ range. A second one is Tahn, a linear transformation of Sigmoid that zero-centers the outputs and bounds them to the $[-1,1]$ interval. For illustration purposes, we use Tahn for our example in Figure 1. One limitation of the aforementioned functions is that their output becomes flat near their boundaries, limiting the network to learn from inputs with either very low or very high activation values. Addressing these issues, the rectified linear unit (ReLU) is a nonsaturated function that keeps the original input value when it is positive and transforms all negative values to 0. $ReLU$ usually increases the learning speed of the network and is now the standard activation function in practice (Nair and Hinton Reference Nair, Hinton, Fürnkranz and Joachims2010).

Given the potentially large size of an image, one of the goals of this stage is to reduce the dimensionality of it while keeping its most relevant information. To reduce the dimensions of the image, it is recommended to include a POOL layer after every CONV $\rightarrow $ ACT sequence (Rosebrock Reference Rosebrock2017). The way in which POOL summarizes the information depends on whether it keeps the largest value (max pooling), the smallest value (min pooling), or the average value (mean pooling) of a specific pixel area. In the example of Figure 1, we apply max pooling to keep the largest value from every $2\times 2$ pixel area of the matrix. The resulting matrix generalizes the properties of the image, forcing the neural network to pay more attention to whether a feature fits in the image regardless of the location of such a feature.

The CONV $\rightarrow $ ACT $\rightarrow $ POOL sequence is repeated for each of the filters in the first layer. The resulting feature maps then become the input for the second CONV. This new layer reproduces all of the steps described above, and its filters slide through the feature maps to now look for more complex features, such as a combination of lines or edges. The process is repeated as many times as CONV are in the network.

2.3 Learning

The last stage of the CNN uses the elements extracted during the last CONV $\rightarrow $ ACT $\rightarrow $ POOL sequence to predict the label of the image. This stage involves a set of fully connected (FC) layers followed by a final ACT, also called output layer, that produces the output value for the prediction. Each neuron in an FC layer receives an input, transforms it, and sends a signal to the neurons to either an additional FC or an ACT layer. The strength of the signal depends on the weight between the emitting and receiving neurons. Positive weights amplify the signals and highlight their contribution to the output. Negative weights, by contrast, weaken the signal to which they are attached.

When the information reaches the output layer, its corresponding nodes deliver the probability that the original input belongs to each label through a given activation function. If the classification task only involves two labels, the output layer estimates the probabilities using an inverse logit. Otherwise, the probabilities are estimated through a softmax layer, the equivalent of a multinomial logistic function. In the example described above, since we want to identify the image from Figure A.1(a) as a digit value, the last layer of our network has 10 neurons or outputs, one for every digit from 0 to 9. Each neuron will provide a probability that the image belongs to each digit, and the model will attach the label with the highest estimated probability.

Until this point, the neural network has performed only a forward propagation—i.e., passing the input data sequentially through each of the layers to generate a prediction. However, for the CNN to learn what features of the image are more likely to belong to each label, it requires a process called backpropagation (Rumelhart, Hinton, and Williams Reference Rumelhart, Hinton and Williams1988). For this operation, the model goes back from the output layer to the input layer with the goal of discovering high-level features of the images. With every step away from the output layer, the model calibrates the weights between neurons to gradually minimize the errors in predictions (Schrodt Reference Schrodt2004; Lucas Reference Lucas2018a). Therefore, the learning process consists of reviewing a set of labeled examples multiple times to find the optimal combination of feature maps and weights that minimize the disparities between true and predicted labels.

We use the loss function to quantify such disparities and as our target for optimization. The smaller the loss, the better the predictions that the classifier is reaching. In more familiar terms, the loss function in an ordinary least squares (OLS) regression is the mean square error, i.e., the mean of the differences between the observed values $y_i$ and the predicted values represented by the regression line, $\hat {y_i}$ . In the CNN context (and nearly all deep learning applications), the default loss function for binary classification is the binary cross-entropy loss, and for more than two categories the categorical cross-entropy loss.

Analytically, backpropagation searches for the local points in a high-dimensional space where the error derivative of the total loss function is zero. To find those saddle points, the CNN modifies the weights between layers and checks for its resulting changes on the error. This estimation is called gradient descent and consists of repeatedly calculating the slope of each weight with respect to the loss function and modifying such weight to minimize the slope until finally reaching the bottom of the function.Footnote ⁷

Similar to the previous subsection, we describe the hyperparameters that have to be specified in this stage, and provide recommendations for setting these values in Section 3. First, given the iterative nature of the gradient descent, it is necessary to specify the epochs, or the number of times that all training examples will pass through the network. The model is more likely to fit its weights, as it has more opportunities to review the training examples. It is expected that the prediction errors in the training and validation data will decrease with the number of epochs. However, when the loss in the validity dataset stops decreasing, we should stop the training process (see Section 3.3). In this way, we prevent the model from starting to learn features of the images that are not generalizable outside the training data.

Second, the researcher needs to define the batch size, or the number of training images that need to pass through the network before it updates its weights. If the batch size is equal to the total number of training images, the model will update its weights only after accumulating the prediction errors across all the training images, a task that can demand a lot of computational memory. It also produces a static error surface, where the gradient descent is likely to get stuck in a local minimum. By contrast, the batch size can be equal to one, where the model updates its weights for each training example. Though updating the model with every example decreases the risk of getting stuck on a flat region, it produces a very noisy signal. A middle-way approach splits the training data into mini-batches, allowing the model to update its parameters several times during an epoch.Footnote ⁸

Finally, we can specify the learning rate, or the size and timing of the steps we take to reach the function’s local minimum. Finding the optimal learning rate in advance presents a conundrum. A large learning rate will jump around the function, overshooting its minimum. By contrast, a very small learning rate is more likely to find a local minimum, but it will take a long time to converge. Ideally, we would use a learning rate that helps the model to start exploring the entire hyperparameter space and to approach the global minimum by gradually making smaller jumps (Buduma and Locascio Reference Buduma and Locascio2017). Yet it is hard to know ex ante when and by how much to decrease the learning rate along the iterations. Moreover, having a unique learning rate to estimate all the parameters could hinder the model to explore the entire space in an efficient way. Fortunately, the researcher can choose from different gradient descent optimization algorithms that lead to faster convergence and that do not require to manually tune up the learning rate.Footnote ⁹ For example, the adaptive gradient algorithm (Adagrad) adjusts the learning rate to each parameter, computing smaller updates for frequently occurring features and larger updates for the less frequent ones. The adaptive moment estimation (Adam) follows the same logic as Adagrad, but it solves the radically diminishing learning rate of the latter (Ruder Reference Ruder2017). That advantage makes it the current default choice in machine learning.

3.1 Building a CNN Architecture

The simplest structure in a CNN includes a single CONV with a unique filter followed by an ACT and a POOL layer, and a single FC with a subsequent ACT that makes the prediction. We can represent more complicated architectures with the following pattern:

where M and K are the number of times that a given sequence is repeated. For example, the simplest structure mentioned above will have $M=K=1$ .

We can change the complexity of the structure in two ways. First, we can make a deeper network by increasing either M or K. The former option implies adding more CONV to the network. The deeper a network is, the more complex the features from the image that the convolutional layers will find. We expect a larger M in models using extensive training data and a complex classification goal. Alternatively, we can deepen the network by increasing K, or the number of FC layers toward the end of the network, allowing for a more complex classifier.

Another element that can vary from architecture to architecture is the inclusion of a POOL layer (therefore, the “?” sign next to POOL in the pattern above). While some models rely on convolutional layers with longer strides to reduce the dimensionality of the data, others use POOL layers that simplify the data between each round of convolutions. Including this type of layers allows the network to deal with simpler data, but also forces it to focus on less specific features of the images. At the end, this step “compresses” the amount of information from which the model learns.

There is not a unique recipe for the values of these hyperparameters. As a starting point, basic structures include $ 1 \leq M \leq 3$ , and $K \geq 1$ (Rosebrock Reference Rosebrock2017). But researchers should consider the size of their data, the number of desired output labels, the similarity between output labels, and the intraclass variance (i.e., how dissimilar are the objects belonging to the same label). We also encourage the reader to experiment with different settings to test the model performance using hyperparameter grid search (see Section 5).

The number and size of the filters in each CONV are other relevant hyperparameters to consider. Here, we go back to the importance of fully knowing the data under analysis to inform that decision. A higher number of filters will make the network look for more features of the image sharing a similar level of complexity. A common procedure to set this number involves increasing the number of filters gradually through the network. Some scholars recommend to double the number of filters every time we add an additional CONV (Shang et al.  Reference Shang, Sohn, Almeida and Lee2016). Other architectures present even larger increments.Footnote ¹⁰

With regards to the filter size, if the pixel space is sufficiently large and the model’s goal is to find large-scale features of the image, filters of size $11\times 11$ or $9\times 9$ would be fitting. However, if the features that distinguish the images are more fine-grained, then small filters ( $3\times 3$ or $5\times 5$ ) will be more appropriate. For example, Han et al. (Reference Han2018) find that detecting whether a person is smiling in a picture relies on broader and more evident identification of features like the strong curvature of the lips that are easily captured with larger filters. In contrast, detecting whether a person is frowning relies on finding light wrinkles in the forehead, a feature that is more efficiently identified with a smaller kernel.

The last considerations have to do with the characteristics of POOL. In very complex networks, pooling generally uses $3\times 3$ receptive fields in the early stages of the model to then reduce its size to $2\times 2$ . Simpler networks use $2\times 2$ receptive fields and a stride of 2 all along its sequence. Finally, while max pooling is the most common pooling layer (Ranzato, Boureau, and Cun Reference Ranzato, Boureau and Cun2008; Boureau, Ponce, and LeCun Reference Boureau, Ponce and LeCun2010), recent findings suggest that some hybrid pooling methods, such as the “mixed max-averaging pooling,” can also improve performance for an extra computational cost (Lee, Gallagher, and Tu Reference Lee, Gallagher and Tu2016).

3.2 Software

There are multiple open-source machine learning packages that researchers can use to design and run a CNN. Some of the most popular are TensorFlow (from Google), Caffe (from UC Berkeley), CNTK (from Microsoft), PyTorch (from Facebook), and MXNet (supported by AWS). Keras is a neural networks API written in Python that supports models like CNNs and recurrent networks, and that a very accessible, efficient, and user-friendly interaction with packages like TensorFlow.

The differences between the aforementioned packages are in terms of speed, energy efficiency, and accessibility. As a quick summary, TensorFlow is faster when running large-scale models, while Caffe is faster with small-scale ones (assuming they are both implemented on a CPU-based platform). Furthermore, while PyTorch is more memory efficient than its counterparts, MXNet requires the least amount of computational energy. Thus, scholars should consider the size and complexity of their data and classification objectives, plus the hardware/computational resources they have at hand. Zhang, Wang, and Shi (Reference Zhang, Wang and Shi2018) offer a great compilation of insights regarding the performance of these packages with respect to speed, memory, and energy.

Beyond these packages, there are other tools that facilitate the design of the architecture of a CNN and also allow researchers to take advantage of pretrained models. In particular, these might be interesting for deep learning beginners or scholars with less advanced programming skills. The options include but are not limited to Amazon AWS Machine Learning Training, Google Cloud AutoML, or Google Cloud Machine Learning Engine. For a review of these platforms and their performance, see Webb Williams (Reference Webb Williams2019).

3.3 Finding the Right Amount of Training

As with any human-performed task, prediction accuracy for a CNN model comes only with practice. A model requires to be trained over multiple iterations, so it can learn the relevant patterns in the training data. Not doing so leads to underfitting, which occurs when the model is too simple to account for all the variance across images. An underfitted model makes strong assumptions when classifying the data, leading to a high bias on its predictions. As we increase the number of training iterations, the model can continue learning the general patters of the training examples and adjust its filter weights accordingly.

On the other hand, training the model for too many iterations will eventually lead to overfitting. This works against generalization, a principle in machine learning that refers to the performance of the model with examples outside the training set. When overfitting occurs, the model starts picking up noise or random fluctuations from the training images as valuable concepts for classification. To check for overfitting, we can monitor model performance in the training and validity sets after each epoch and stop when the loss decreases (or the model’s accuracy no longer increases) in the validity data. Preventing overfitting while allowing the model to learn the most relevant features of the images is a practical dilemma, and it depends not only on the number of training iterations but also on the model architecture. We suggest below a few ways to optimize the training.

3.4 Optimizing Your Training Set

A better learning depends not only on the number of times the model can learn from the examples, but also on the information it extracts from them. We suggest four concepts to keep in mind when choosing the amount and quantity of data that the model will learn from.

3.5 Transfer Learning

Training the model from scratch can be computationally expensive and requires a large dataset to avoid overfitting. An alternative approach involves transfer learning, where an already trained model can be directly applied to a new task or used as the starting point for training a new model (Pan and Yang Reference Pan and Yang2010). Transfer learning is an available resource when the researcher is lucky enough to find a model already trained on a dataset with similar content to hers, and that also performs a similar classification task.

To adapt the trained model to the new data, we can deactivate (or “freeze”) its initial layers. This way the model takes the weights previously learned and calibrates their values based on the new target categories. For example, Zhang and Pan (Reference Zhang and Pan2019) used transfer learning to fine-tune VGGNet, a canonical CNN model trained on a set of 1.2 million images, to identify collective action events. Because those categories do not cover human faces or crowds, crucial elements of protests and demonstrations, the authors “froze” the first 12 layers of a VGGNet and retrained the last four layers with a set of images containing their target elements.

In Table A.1 in the online Appendix, we provide a summary of popular architectures that can serve as the basis for transfer learning. We include technical details such as the number of layers and size, performance indicators, and the references to the articles that introduced them. See also Webb Williams et al. (Reference Webb Williams, Casas and Wilkerson2020) for a detailed explanation of this method, as well as some resources available when using this approach.

3.6 Validate and Check the Results

Similar to textual analysis, a key principle of machine-coded visual analysis is “validate, validate, validate” (Grimmer and Stewart Reference Grimmer and Stewart2013, 269). Validating the results will reveal potential sources of errors and provide information about the model fit. An insightful way to improve the model is to review the misclassified images in the validity set. As we will show in the example below, this is a helpful practice to find potential problems in the model. Another way to check the validity of the model’s predictions is to visualize the most relevant features driving the predictions (Zeiler and Fergus Reference Zeiler and Fergus2014; Won et al.  Reference Won, Steinert-Threlkeld and Joo2017). Similar to finding the most important coefficients in a regression, this exercise provides information about the mechanisms behind the predictions. For example, some of these tools provide “maps” that reveal what parts of the image or specific features were most determinant in reaching a prediction. This in turn can provide the researcher with information to help complement or tune the samples included in the training pool.