Data repeats itself, the first time as training, the second as prediction. Datasets bear upon datasets in the recursive and nested logic to machine learning systems, and in the types of vision these systems afford. Keeping with the example of MbM, let us turn to how machine vision works by induction, and the division of labour this design entails.

Let us start with one of the most common tasks in computer vision: object detection. This was the first technique applied to the BBC television dataset in conjunction with automated captioning. We used an implementation of DenseCap (short for ‘dense captioning’), which is a method designed to ‘localize and describe salient regions in images in natural language’ (Johnson et al, 2015, p. 1). It consists of an integrated architecture of artificial neural networks specialised in both object detection and image captioning. The aim of DenseCap is to integrate these tasks into a unified system that could describe the objects of a scene and their relations.

For any given image passed as an input, DenseCap outputs several regions corresponding to detected objects (that can be visualised through the familiar coloured bounding boxes overlays), a confidence score of these detections, and a list of descriptions in natural language. The result is an annotation not just of objects in an image, but of relations between objects, actions, and places, such as ‘a parked motorcycle’, ‘a man on a bicycle’, ‘a window on the building’, and so on. Each clip in the BBC television dataset was annotated using the descriptions generated by DenseCap, at a three second sample rate and only including instances detected above a high threshold of confidence.