The Turn to Visual Evidence

The use of visual evidence to document conflict and human rights abuse is as old as photography itself. Archetypal images like slaves with whipping scars on their backs, Jewish concentration camp prisoners, or Nick Ut's photograph of Phan Thi Kim Phuc running down the street naked having been burnt in a US napalm attack have played a prominent role in shaping the way we think about violations of human rights.

While these images pack a powerful emotional punch, they present only a single moment in time and offer little by way of context. Thus, they must be viewed and used carefully. Law, at least, has never been a naïve consumer of visual evidence (Mnookin Reference Mnookin2000). From the very moment that photographs were first introduced into the courtroom, legal actors had concerns. Judges, juries, and commentators held the potential of photographs to provided privileged access to the world as it actually is in tension with the power of the image to mislead the fact-finder, either intentionally or unintentionally. The fact that most judges initially analogized photography to other forms of illustrative evidence such as maps, models, and diagrams, rather than as something demonstrative and disconnected from human production, suggests that they understood—and maybe even feared—the power of the photographic image (Mnookin Reference Mnookin2000).

Over time, however, courts came to rely on photographic images to do more than merely visually reinforce the testimony of witnesses as to matters of fact. Photographic images were soon seen to have probative value as demonstrative evidence—that is, material evidence that could provide information to legal fact-finders above and beyond that which could be attested to by a witness. Thus, the older models, maps, and diagrams that judges once used as analogies to photography—to admit photographs into court but contain their power—ultimately achieved the status of evidence that has inherent probative value thanks to the normalization of photographic evidence.

Legal scholars have begun to examine how this turn toward visual evidence has changed legal reasoning, which has been text based for much of modern history (Porter Reference Porter2000). Richard K. Sherwin (Reference Sherwin2000) tells participants in legal proceedings and investigations to pay close attention to how images and video actually form the basis for constructing reality, from both a sensory and an emotional perspective. Images and video give us the sense that we are experiencing reality, not a reconstruction, and we can often identify with being there in the moment, even though this is only a sensation produced in our mind. We need to be aware of how our brains and our culture condition the way we make meaning from images. He reminds us that videos “place viewers (like judges and juries) in the seeming position of witnesses, even though their position is limited by the camera's lens and position” and that the narrative framing and contextualization of a visual representation shapes the way it is interpreted (Sherwin Reference Sherwin2000, 37). The playback of a video can also impact its reception—for example, by speeding it up or slowing it down, zooming in or zooming out, to alter the flow of action in a way that benefits a particular interpretation.

Sherwin's analysis raises crucial questions that we must ask as visual evidence becomes an increasingly prevalent aspect of human rights documentation and advocacy. Most importantly, we must always remain skeptical about visual claims (especially those that appear to be the most compelling), and in addition to taking note of their potentially prejudicial effects (Feigenson and Spiesel Reference Feigenson and Spiesel2000), heed Ariella Azouley's reminder to ask why visual representations are produced, who produces them, the circumstances under which they are circulated, and how they are framed and contextualized (Azouley Reference Azouley2000).

We must also keep in mind that even with a large volume of visual evidence, facts can remain elusive. Eyal Weizman describes this problem as the “threshold of detectability” (2017, 20). This threshold occurs when a phenomenon or event in question leaves a trace on some medium, but its existence cannot explicitly be verified. The example he gives is when an object recorded on a photograph (say a person or damage to a building from a projectile strike) is close to the same size as the material element that records it (either a single salt grain in a photographic negative or a single pixel in digital imagery). This condition, he writes, “forces us to remember that the negative [or digital image] is not only an image representing reality, but that it is itself a material thing, simultaneously both representation and presence” (Weizman Reference Weizman2000, 20).

This article examines the possibilities, as well as the limitations, of machine learning and computer vision for human rights advocacy and accountability. In doing so, I further Weizman's call for a “counterforensics,” which takes tools for which the state has a near monopoly and puts them into the hands of human rights researchers who can monitor and, when appropriate, call out the behavior of states and non-state actors alike.

Social and Legal Value

As noted above, machine learning and computer vision have the potential to find revelatory moments in large volumes of video. Perhaps more importantly, their ability to annotate large video collections and synchronize video efficiently can also limit the ability of historical revisionists or partisans to pick and choose decontextualized moments of video that obfuscate or misrepresent the past. An actor wishing to obscure the truth might show a video of protesters throwing Molotov cocktails at police without showing earlier video of police maliciously attacking the very same protesters the day before. The actions depicted in videos must be placed in context to be understood fully, and this can occur only with reasonably complete video archives that can be searched by time and place along with other, more traditional forms of forensic evidence and eyewitness testimony.

In the case of police brutality, for instance, it is important to know about previous interactions between the police unit involved and the person or people in question. Was the interaction being filmed a continuation of a set of events in which the previous events were not captured on film? Had either party issued threats that were not captured on film? Was the police unit in question normally restrained, or was the violent event captured typical of its behavior? At the same time, the ability to analyze large volumes of video does not in any way guarantee that the objective truth will be uncovered—videos, after all, still provide only a perspective on events, not an omniscient view or master narrative. At best, having access to many videos from an event, rather than a few, allows many perspectives to be shown side by side.

At minimum, video evidence also makes it much harder for violators to engage in the tactic that Stanley Cohen (Reference Cohen2000, 7) calls “literal denial” and requires them to provide an alternative explanation for their actions or to claim that their actions were justified. The analysis of large volumes of citizen media can also help to discover and amplify alternative, community-oriented narratives that differ from those provided by large media organizations and governments.

A significant concern associated with the application of technology to human rights work more generally, however, is that the ability to mine large video collections tends to be limited to institutions with large staffs or access to expensive, technologically advanced tools and techniques. Thus, analysis of large collections of digital content quickly becomes implicated in longstanding conversations about who owns information and what can be done with it. Well-resourced human rights organizations, generally centered in North America and Western Europe, now have even greater potential to extract information from disenfranchised groups and smaller, more regional organizations and to use it to pursue their desired ends. Sometimes, these ends are at odds with, or at least not the priorities of, the groups or individuals who document conflict and human rights violations at the local level. Indeed, they often reflect the normative frameworks and imaginaries of international institutions and the individuals who staff them (Baylis Reference Baylis2000).

It is crucial to keep this critique of human rights documentation in mind when thinking about the tools and methods described below. One possible way to mitigate this inequity, and to ensure that human rights investigators do not cause harm by applying these tools, is to demystify them and to ensure that their power and limitations can be clearly understood by non-specialists no matter what their level of technical capacity and training. What follows is an attempt to do just this, minimizing jargon and technical nuance to provide an introduction to machine learning and computer vision that is widely accessible to the human rights community. In doing so, I lay out the basic scientific and technical underpinning of these techniques and show how they are directly applicable to the investigation of conflict and human rights violations.

Machine Learning

Machine learning is the science of creating computing systems that are programmed to arrive at logical conclusions about the world through exposure to, and processing of, data (Geitgey Reference Geitgey2000). Machine learning uses algorithms—a set of processes or rules that are performed in a step-by-step manner—to enable systems to sort dynamically through a corpus of data to discover regularities and relationships. In machine learning, computer scientists do not hard code rules of action that a system uses to do the exact same thing every time; rather, they give the system the capacity to recognize patterns, trends, or categories in “training” data provided by the programmer and then to use those discoveries to analyze another set of data that has not yet been processed.

Machine learning systems sometimes use supervised training, in which the programmer gives the system a labeled data set consisting of the thing of interest, and asks the computer to figure out how to recognize similar things in other data sets that it has not seen before. One example would be giving a machine learning system thousands of sample sentences with nouns, verbs, adverbs, and adjectives labeled, and then asking the algorithm to label these entities in sentences to which it had never previously been exposed. The programmer does not tell the system that a word ending in “ing” is usually a verb or that capitalized words appearing anywhere other than the first position of a sentence are generally nouns; instead, she lets the system figure that out based on the labels in the training data. It is important to note that the effectiveness of a supervised machine learning model is directly related to the quality and representativeness of the training data provided to it. The system will pick up on patterns in the data, and it will therefore retain any biases embedded within those data. For instance, if a machine learning system is never exposed to words ending in “ing” that are nouns or adjectives (e.g., king or interesting), but is exposed to many verbs ending in “ing,” it may come to the very logical (but incorrect) conclusion that words ending in “ing” are verbs even when they do not behave like most other verbs. The system can obviously also be trained to recognize nouns by their position in a sentence, but this must be a conscious choice of the programmer.

Scientists can also use unsupervised training, in which the system is provided with an unlabeled data set and is programmed to identify the strongest categories or structuring principles (often called clusters) within it. An example of unsupervised learning would be giving a machine learning system demographic and life history information about thousands of people who receive PhDs in a particular discipline (or, alternatively, have been convicted of a particular crime) and asking it to determine which factor or factors seem most predictive of this outcome. It is important to note that the characteristics of the data set determine what can be learned even if it has no labels. If the PhD data set is made up primarily of humanities professors rather than computer scientists, or if the criminal data set is made up primarily of individuals convicted of possession or sale of methamphetamine rather than a broader set of drug crimes, the system will come to certain conclusions about that particular population that may not be true of the broader population.

Regardless of whether supervised or unsupervised training is used, the output of the machine learning system's analysis becomes a set of “classifiers” that can then be applied to other data sets that have not been used in the training process. Classifiers are reductive—they take the complexity of the world and convert it into decision processes that can be used to identify similar things in other contexts. They are rarely 100 percent accurate, and they can lead users astray if their limitations are not understood and taken into account. For instance, a language processing classifier trained on newspaper articles and official government documents may not do a great job of analyzing transcripts of conversations in slang or regional dialects. One would require training data with these variations to create classifiers that would work well on them. Ideally, machine learning systems should be trained on diverse material to ensure that they are not overspecialized for a single context.

Computer Vision

Computer vision is the analysis of digital visual images to understand both the objects that are depicted within them and the scenes from which they were constructed. Computer vision can involve detection of specific objects, segmentation (separation) of multiple objects within one scene, tracking these objects over time and space, three-dimensional reconstruction of the objects and/or scenes, and determination of the placement of the camera in the scene over time and/or space. Numerous methods are used to carry out these tasks, including color analysis, shadow and illumination analysis, geometrical analysis of curves and edges, and photogrammetry (mathematical and geometric techniques to make measurements within the image) (Szeliski Reference Szeliski2000).

When combined with machine learning, the principles of computer vision can be used to identify objects in, and reconstruct scenes from, digital images. One simple but illustrative example of computer vision is teaching an algorithm to recognize letters or numbers. We know from experience that both typography and human handwriting vary widely, but we learn to recognize even the most challenging writing based on our general understanding of what letters and numbers look like, as well as our contextual knowledge of language and mathematical representation. Even if one or a few individual characters are indecipherable, we can generally infer what words or ideas the writer is representing.

Machine learning systems can be trained to read a huge range of typography and handwriting with the aid of computer vision. The situation becomes more complicated when the letters or numerals are not located on a solid, white background; they may be integrated within a visual scene such as a photograph or painting, be presented at low resolution, or be distorted (as most of us know from solving the Captcha and ReCaptcha puzzles that confirm our identities as humans). Distortion represents a significant computational challenge, especially when the object of interest is more complex than a letter or numeral—such as evidence of a possible human rights violation—and scientists devote a great deal of effort to developing tools to enhance the resolution and clarity of objects within an image.

Event Detection: E-LAMP

Machine learning systems can learn to detect many of the kinds of objects that are of interest to human rights researchers, including tanks, missiles, helicopters, airplanes, military vehicles, particular styles of building, soldiers in camouflage, large crowds, corpses, and visually distinct geographic locations like bridges over water, mountainous terrain, or a desert (Aronson, Xu, and Hauptmann Reference Aronson, Xu and Hauptmann2000). They can also detect and tag sounds like gunshots, explosions, cries, or screams (Liang et al. Reference Liang, Burger, Hauptmann and Aronson2000). This allows a researcher or organization with limited human resources or time to scan a large video collection rapidly to find material relevant to an investigation. This makes it less likely that the investigator will have to rely on a single video to understand an event and also increases the amount of data that can be used to place the event of interest in context.

Event detection is computationally complex because it involves the detection of numerous semantic concepts (i.e., objects, sounds, scenes, and actions) taking place in a dynamic environment. While this task is challenging enough when carried out with video from a fixed camera in a constrained environment, the kinds of videos relevant to human rights investigations are significantly more complex. Typically, the videographer (often an amateur) is in motion and reacting to the events he or she is capturing on camera. The camera thus is unconstrained in time and space and generally offers something akin to first-person vision (Tong et al. Reference Tong, Yang, Jiang, Yu, Lan, Ma, Sze, Younessian and Hauptmann2000). Material drawn from social media contains an almost infinite number of situations that are not bound by clear-cut rules or spatial environments (Tong et al. Reference Tong, Yang, Jiang, Yu, Lan, Ma, Sze, Younessian and Hauptmann2000). Further, social media videos capture an almost infinite variety of human and nonhuman behavior. To mine this resource effectively, an analyst needs to be able to develop novel classifiers on the fly.

One system, developed by researchers at Carnegie Mellon University, that directly addresses these challenges is called “Event Labeling through Analytic Media Processing” (E-LAMP). It works at the most basic level when an operator provides the system with a set of training videos or video shots that depict a particular activity or thing along with a set of null videos that depict other unrelated activities. E-LAMP analyzes these videos for a variety of the kinds of features described above (selected in various combinations based on the need for accuracy, speed, and processing capacity), which can be combined into a computational machine learning model of the relevant action or event. The system then delves into the larger collection of videos to look for other potential examples of this model. It returns a set of videos to the operator that it thinks match the activity in question. The operator confirms whether the proposed matches are correct or incorrect, and E-LAMP takes this information into account and tries again. Once the system returns mostly correct results (which are rarely 100 percent accurate for a variety of reasons), this set of patterns is labeled as a classifier, or event kit, for the particular action (Tong et al. Reference Tong, Yang, Jiang, Yu, Lan, Ma, Sze, Younessian and Hauptmann2000).

This classifier, which can be visual, aural, semantic, or a combination of the three, can then be used to search for particular instances of it in any other video collection. The classifier may need to be modified to work well in these other contexts. In addition to this kind of search capability, E-LAMP can also be used to detect duplicates or near-duplicates, which is valuable in human rights contexts because it is possible to see how similar footage is used in different videos and how various users edit and recombine footage. It also makes it possible to gain multiple perspectives on a single event.

In addition to visual classifiers, E-LAMP can currently perform speech recognition in English and Arabic (together with preliminary versions of Spanish, German, French, Mandarin Chinese, Cantonese, Turkish, Pashto, and Tagalog), and can detect a variety of unique sounds, such as gunfire, explosions, airplanes, and helicopters. Text found in videos, whether as subtitles, titles, or signs, is processed through optical character recognition (OCR) and rendered searchable, although researchers are still working to improve the accuracy of OCR in the kinds of low-resolution contexts found in video.

E-LAMP functions by processing the video once in its entirety, segmenting it by scene, and then taking a visual snapshot (a “keyframe”) at the midpoint of each segment. The feature detection described above is then performed on these frames so that they can be compared with classifiers for the targeted event. The results are returned with probability calculations—the higher the score, the more confident the system is in the call. In most cases analyzed, the higher probability results tend to be accurate, while no systematic claim can be made about the lower probability results. In some cases, they are still accurate, and in other cases they are not (see below). Currently, eight-hundred hours of video would cost approximately $1,872 for computing resources plus 180–200 GB of storage space.Footnote ³

Building Classifiers

To provide a first test of the capacity of the system to aid in conflict monitoring and human rights investigations, Carnegie Mellon researchers sought to identify certain categories of weapons depicted in a set of approximately five-hundred videos that focused on events taking place in Aleppo, Syria, in late 2013. Given the nature of the conflict in Syria, rebel groups routinely filmed their military exploits and regularly reported about their caches of weapons. It is, of course, important to recognize that all such videos are public relations ploys, and that groups avoid distributing footage that highlights their weaknesses or shows them being defeated. One cannot make statistical calculations of any sort based on available social media reports because they are only a convenience sample of data (Price, Gohdes, and Ball Reference Price, Gohdes and Ball2000). That said, one can still gather quite a bit of general information about access and availability of weapons systems from these videos—particularly the presence of weapons systems in a particular region.

At the moment, there are over twenty-five hundred preexisting semantic concepts available in E-LAMP for use in searching a video collection. Because the system was not trained for conflict and human rights analysis, though, classifiers for much of the material of interest to researchers in this domain will have to be built from scratch. Thus, when researchers first uploaded the Aleppo video collection, there was no semantic concept for “mortar launcher,” or for most other weapons systems. There were generic classifiers already built for “weapon” and “machine gun” (both of which were likely developed from video collections that included only recreational shooting, hunting, or gun collecting), but this would be useful for only a fraction of the weapons used in the Syrian conflict.

To try to get around this “cold start” problem, the operator can begin with an analysis of available metadata (which only works with video with at least some metadata preserved), speech recognition, or OCR words (in this case Arabic), or try her luck with the available classifiers. Thus, if the Arabic words for “mortar launcher” are written or spoken in any of the videos in our collection or in their metadata, they will at least theoretically be detected in response to the user's query even if the system has never been trained to recognize an actual mortar launcher. The researcher can also manually select a few videos of mortal launchers that she knows are in the larger video collection.

Assuming that the investigator has a few videos containing mortar launchers, she then can either play the videos in their entirety and select particular “shots” of interest (e.g., moments when the mortar launcher is visible or heard being launched), or use keyframes to select a few positive videos shots containing mortar launchers (Figure 1). These shots and/or keyframes are then fed into the machine learning model. The model will utilize audio, image, and motion signatures extracted from these video shots to create a classifier that will become the basis for a new semantic concept of “mortar launcher” that can be refined through a variety of mechanisms until satisfactory results are achieved in a test data set. In the case of the mortar launcher classifier, for example, the model initially misidentified things like power lines, truck mounted anti-aircraft guns, and other linear objects with similar backgrounds as positive examples.

Figure 1.

Initial Mortar Launcher Keyframes from the First Video We Found that Contained this Weapon System

Notes: Eighteen total keyframes were taken by E-LAMP and we selected a few of these to build the model for our classifier. We also selected keyframes from other videos to broaden the variety of mortar launchers detected and ensure that the angle, positioning, background, and type did not overly limit the machine's final model. [Color figure can be viewed at wileyonlinelibrary.com]

The operator has to determine the ideal sensitivity of the classifier for her purposes. A very sensitive classifier will have a high degree of accuracy in the high confidence range, but will miss many positive but lower confidence cases. A less sensitive classifier will capture a greater percentage of positive video cases (both high and low confidence), but will also capture many incorrect ones as well. This means more time needs to be spent by the operator separating the signal from the noise. Once built, the model can be applied to the entire video collection or any other appropriately processed set of videos, although accuracy may decrease when applied to a new collection.

Figure 2.

Top 100 Mortar Launcher Videos Out of 476 total. [Color figure can be viewed at wileyonlinelibrary.com]

E-LAMP has also been tested on a larger collection of 13,570 publicly available YouTube videos with the goal of identifying more complicated visual phenomena, including helicopters (which were routinely used in the Syrian conflict by the Assad regime to drop barrel bombs on neighborhoods held by antigovernment groups) and corpses, using the process described above. Both were successful in identifying the object in question with a high degree of accuracy in the top results. The most common misidentification for the helicopter classifier was an airplane, although there also appear to be a few incidental images from the scraping process (including a pirate flag with skull and bones that appear to mimic the rotors of a helicopter and a gecko mascot that appears to be falling from the sky with its four limbs spread out in a US insurance company advertisement) that show up toward the bottom of the top 100 results (Figure 2). The “corpse” (defined by Carnegie Mellon researchers as bloodied bodies with visible faces in a horizontal pose with no movement) detector was similarly successful. On the first iteration, ninety-five of the top one-hundred hits were correct. The computational model mistook what appears to be an open artichoke flower for a corpse because of the similarity of the shape and contrast to the face of a corpse, as well as a couple of images of what appear to be pink blossoms against dark green leaves (search results available upon request). It is important to note that this classifier likely missed many cases of corpses with unexposed faces, or those lacking blood, so a separate one would have to be built for those cases.

Carnegie Mellon researchers are currently working with a variety of human rights partners to determine how E-LAMP can be integrated into their organizational workflows. So far, E-LAMP seems to be most useful in acting as a filter to remove irrelevant videos from the analysts’ work queue and pinpointing where particular entities of interest are within a collection of videos. The technology has not advanced to the point where it can be relied on to tag a large video collection and populate a database with the results for investigative and analytical purposes automatically without significant human cross-checking.Footnote ⁴

Face Detection and Recognition

Face detection and recognition is a particular application of machine learning and computer vision. Over the past decade, computer scientists have developed tools that make face detection relatively routine. This should not be particularly surprising: although all human faces are unique, barring unusual circumstances, they share certain common anatomical landmarks that are related to one another in a predictable fashion—two ears at the side of the face and in the center two eyes that sit above the nose, which sits above the lips, which sit above the jaw. Detecting partially occluded faces or faces turned to the side is more challenging, but computer scientists have developed sound methods for solving this problem.

A much more complicated problem is recognition: determining whether faces from different images belong to the same person. If the two images were taken straight on in the same lighting, from the same distance, with the same expression, and in high resolution, then the problem can be solved by measuring enough facial features and knowing how many one needs to combine in order to achieve the desired level of certainty that the match is not a false positive. These conditions might hold for mug shots or driver's license/passport photos, which are taken under standardized conditions, but in real-life situations, they usually do not. As a result, computer scientists have had to develop methods to compare faces in images that differ in terms of lighting, angle, pose, facial expression, composition, resolution, distance, and other characteristics. Most importantly, before any pair-wise comparisons can take place, the faces need to be computationally “transformed” so that the eyes, nose, and mouth are centered and are presented in the same scale, with the same expression, all without introducing any significant distortion or error. It goes without saying that this is a difficult task, and it can lead to problems in lower-resolution images (Geitgey Reference Geitgey2000).

Because geometric feature-based methods can be cumbersome (especially after the kind of transformation just described), scientists have developed photometric techniques that rely on differences at the digital level rather than analysis of physical features. This makes it easier for algorithms to match faces because they can be treated as data rather than particular physical objects. The first successful method compares the intensity of light of each pixel to its neighboring pixel and records the direction from the lighter pixel to the darker pixel. Once these relationships are summarized using various computational techniques, faces can be matched by comparing gradients and their interrelationships. More recent methods use convolutional neural networks to do similar kinds of computations, but more rapidly and accurately (Geitgey Reference Geitgey2000). In addition to photometric methods, computer vision experts are also developing methods to reconstruct faces in three dimensions by combining two-dimensional data extracted from multiple frames showing the face from a variety of perspectives. These methods are not yet well understood or ready for widespread use, but the intelligence community and private companies are actively developing them.Footnote ⁵

In theory, facial recognition has a plethora of potential uses in the human rights domain. Putting aside massive ethical concerns for a moment, and not focusing on the negative applications of the very same technology by states or powerful non-state actors, one could imagine human rights investigators using it to identify perpetrators and victims of a filmed violation, or even bystanders who could then be contacted to provide important information or testimony in an investigation (Georgetown Law Center on Privacy and Technology 2000).

In practice, however, there are technical limitations that make its use in human rights contexts less likely in the near future. Facial recognition is becoming increasingly more accurate from high-resolution photographs and high-resolution surveillance video (which do occasionally become relevant in human rights investigations), but the low resolution and highly processed nature of most videos recovered from social media do not generate enough fine-grained data for facial recognition systems to measure enough characteristics of the face to generate meaningful matches.Footnote ⁶ Such tools may be used to reduce the number of faces that need to be manually checked for a match, but they cannot provide accurate matches. Although methods of image enhancement are in development, they are not yet ready for use.

Along these lines, many available facial recognition systems do a poor job of identifying people of African ancestry compared to Asian or European descendants. Why? Most likely because developers of these systems are of Asian or European descent and, intentionally or unintentionally, tend to use training data sets that do not include people of African descent (Phillips et al. Reference Phillips, Jiang, Narvekar, Ayyad and O'Toole2000; Garvie and Frankle Reference Garvie and Frankle2000; Orcutt Reference Orcutt2000). Over the next few years, facial recognition systems will undoubtedly become better at recognizing faces from around the world, spurred by bad press surrounding their limitations and the likelihood that corporate, intelligence, and military actors will pay for such capabilities.

Second, in many videos related to human rights situations, faces are obscured with adornments like thick beards and head coverings, leaving only a small portion of the face available for scrutiny. There are research groups building algorithms to recognize a face based only on a small visible portion, but such systems are not currently reliable enough for widespread use (Juefei-Xu, Lu, and Savvides Reference Juefei-Xu, Luu and Savvides2000). At the same time, especially when dealing with casualties of war or human rights violations, faces of victims (whether alive or deceased) are often different from the way they look in their last available photograph. Damage from blunt force trauma, drowning, burns, starvation, desiccation, and other factors significantly alter the characteristics of the face to the point that is difficult for even a human investigator to recognize a match. Even visual recognition by family members becomes difficult when bodies are not stored under controlled conditions in hot, moist climates (Morgan et al. Reference Morgan, Sribanditmongkol, Perera, Sulasmi, vanAlphen and Sondorp2000). The tools available to model what the person's face might have looked like pre-trauma are not reliable enough to use in most investigations except to provide general leads.

Event Reconstruction Though Multi-Perspectival Video

Computer scientists are increasingly interested in using machine learning and computer vision to reconstruct events in time and space. This provides a visual narrative that can help fact-finders to understand better what happened during an event, as well as the context in which the event took place (Forensic Architecture and SITU Research 2000, n.d.; Weizman Reference Weizman2000). Early work in this field, most notably Project Rashamon at UC-Berkeley, focused on developing tools for simultaneously displaying several synchronized videos of an event in a way that made it possible to see the event from multiple perspectives at the same time (Lafrance Reference Lafrance2000). Although this tool was an important first step, it was constrained by the need to have timestamps for synchronization, which are generally stripped during upload to social media, and the fact that the system presents several video feeds to the viewer, which can become overwhelming quite quickly.

Recent work in event reconstruction has focused on bringing together multiple streams of data into a single, coherent account that flows through time and space. Forensic Architecture and SITU Research pioneered this work in the human rights context through their reports on human rights violations in Israel/Palestine in collaboration with various local and international human rights practitioners. In describing its approach to event reconstruction in a report on a series of attacks on the city of Rafah in response to the kidnapping of an Israeli soldier by Palestinian forces in August 2014, Forensic Architecture notes that its partner organization, Amnesty International, did not have direct access to Gaza to conduct a fact-finding mission. It was, however, able to collect eyewitness testimony and photographs and videos shot by citizen witnesses and professional journalists remotely though social media and local partners. Amnesty also obtained high-definition satellite imagery from the Pléiades satellite. Given this evidence base, Forensic Architecture developed methods of superimposing visual evidence onto a 3D model of Rafah constructed by parametric modeling of landmarks in photography-based visual surveys of the area and information from the satellite imagery (Amnesty International and Forensic Architecture n.d.)^.

Once visual evidence could be placed geographically, Forensic Architecture synchronized all available visual evidence to a universal clock. This was done by cross-referencing eyewitness testimony, available metadata, and visual information, such as smoke, contained within the images and videos. It also undertook shadow analysis to corroborate time and place calculations. By mapping evidence over time and space, analysts and fact-finders can “explore the spatial and temporal connections between the various sources and reconstruct the unfolding of events” (Forensic Architecture n.d.). This analysis can be conducted at the urban scale (e.g., by tracking movements of groups of people engaged in protest or conflict over geographic space and time) or at the personal scale (e.g., capturing multiple views of a person being hit by a projectile, reconstructing its path from the weapon to impact, and comparing visual evidence to autopsy reports or eyewitness testimony) (Forensic Architecture and SITU Research 2000 n.d.).

Semiautonomous Synchronization and Geolocation

To put together a video archive that is searchable over time and space and that can be mapped onto the location where it occurred with relative accuracy, one must synchronize and geolocate relevant videos. The methods used by Forensic Architecture in its Rafah report are completely manual. This is fine if one has limited video, numerous skilled human analysts, and/or no time deadline. In today's world of widely distributed mobile phones, limited resources, and pressing deadlines, these conditions are rarely met.

In one recent example, Ukrainian human rights lawyers representing families of people killed or injured by national police during the Euromaidan protests in Kiev in 2013 and 2014 were overwhelmed by the dozens of hours of videos that were relevant to their cases. They needed to find evidence to support their claim that security force personnel shot these individuals despite the fact that they were not immediate threats to police or other bystanders. Because most of this material was gathered from social media, it lacked technical metadata. Some video, which was received from news outlets or other official sources, did have reliable metadata and could serve as a basis upon which to synchronize the video lacking metadata.

The legal team initially asked an analyst who was familiar with the events of Euromaidan and the geography of Kiev for help in synchronizing their large video collection. Over the course of eight months, this individual managed to stitch together a small percentage of the total video using visual and audio cues in the recordings and the few videos with reliable time stamps. Because videographers tended to be relatively far away from one another and focused on different points within a larger scene, and because the scenes depicted in the videos tended to be smoky and monochromatic, this task was incredibly challenging and tedious. Often, there are no visual cues that clearly linked two synchronous videos. The analyst put this work together in a nine-channel video grid that stretches 4h37m19s, with large gaps throughout. This was an impressive feat given the circumstances, but it was clear both to the lawyers and the analyst that much of the video could never be synched using manual methods.

The legal team eventually requested the assistance of the computer scientists who created the E-LAMP system described above to develop semiautonomous methods for synchronizing video over time and geolocating the camera. Because visual cues were sparse in these videos and there was no metadata to rely on, the researchers decided to use the kinds of audio signal processing described above. They developed an algorithm that recognized a standardized vocabulary of “features” (e.g., wind, screaming, gunshots, airplane noise, amplified speeches, music, and explosions) and then compared the sequence of these features in each clip to all others in order to look for reasonable matches (see Figure 3).Footnote ⁷ It was necessary to include a certain amount of tolerance for imprecision because each recording device captures sound in a slightly different way due to the specificities of the microphone, the distance from the sound source, and the other noises present in the immediate vicinity of the videographer.

Figure 3.

Audio-Based Synchronization [Correction added on 27 March 2018 after first online publication: the figure caption has been corrected from “Audio-Based Synchronization Pipeline” to “Audio-Based Synchronization”.]. [Color figure can be viewed at wileyonlinelibrary.com]

It is once again important to note that machine learning algorithms are only probabilistic—to ensure accuracy, all synchronizations need to be checked manually by a human analyst. This analyst also needs to adjust the offset that results from videos being shot at different distances from the source of the sound used for synchronization (sound travels more slowly than light). All told, the Carnegie Mellon team was able to synchronize 4h16m13s, solely through their algorithms, some of which overlapped with the existing 4h37m19s, synchronization. The combined total of synchronized video was increased approximately 50 percent, to 6h52m10s, through these efforts. The scientists also determined that 10h40m02s, of the 52h24m00s, of the video analyzed were duplicates of other video in the collection. Once the system was developed, the synchronization took a few days in total rather than eight months.

To aid in the geolocation process, the research team developed an algorithm that used computer vision principles to compare the background scene in a video to a library of accurately geotagged images that were mined from sources such as Google Street View, Flickr, or images taken specifically for this task. Because the results of this system are probabilistic, they also created a tool to allow a human analyst quickly to confirm or reject a match between the scene in a video and a geotagged image. If confirmed, the system quickly moves on to the next video. If rejected, the next most likely location image shows up and the human analyst can repeat the process.

Unfortunately, the utility of this system is dependent on the quality of the videos under review and the availability of accurately geotagged images for comparison. In this first test, the algorithms were only able to geolocate approximately 12 percent of the videos in question. They also determined that 21 percent contained no information that could be used for geolocation (either because they were shot indoors or because no physical landmarks were visible).Footnote ⁸ These results can be improved with refinement of the algorithms developed, but they will likely never replace human knowledge and judgment.