2.1 Internet of things and sensor networks

The IoT paradigm has emerged as means for levering the growing ubiquity of sensor devices, primarily by using the Internet to enable data communication in such a way that these devices are seamlessly integrated into the information network. Although the wide spread of IoT technologies and sensor networks in general is still in its infancy and significant research efforts are still required in this area, there are clear overlaps and synergies with the domain of surveillance systems. Several of such instances are described hereafter.

VigilNet, proposed by Vicaire et al. (Reference Vicaire, He, Cao, Yan, Zhou, Gu, Luo, Stoleru, Stankovic and Abdelzaher2009), is a large scale, outdoor surveillance system composed of tiny power-constrained sensors performing intruder detection tasks. The goal of the research of Vicaire et al. (Reference Vicaire, He, Cao, Yan, Zhou, Gu, Luo, Stoleru, Stankovic and Abdelzaher2009) is to enable the sensor devices to collaboratively increase the system lifetime by employing different energy-efficient strategies. A so-called tripwire service is implemented, consisting of a sensor partitioning phase and a duty cycle scheduling. Results from an actual deployment conclude that the power management techniques proposed are able to increase the lifetime of the network from 4 to 200 days. However, the trade-off between network longevity and performance is an aspect that is currently prescribed during network configuration and needs to be carefully adapted depending on the conditions of the environment.

Li et al. (Reference Li, Lu, Liang, Shen, Chen and Lin2011) describe an IoT architecture, where a multi-hop network of smart homes are organized into a so-called smart community. Distributed community environment surveillance is realized through their neighborhood watch application, based on a classic wireless ad hoc network. Given the static nature of this domain, the global network topology is supposed to be known a priori and loaded to each of the homes’ gateways. Due to the fact that the interconnected homes use wireless communication standards, which are generally highly susceptible to security or reliability faults, the authors suggest some ways to alleviate these shortcomings. For example, detecting unreliable nodes, such as home gateways, could be achieved by having neighboring nodes monitor each other’s retransmissions. Also, filtering for false network traffic may be essential in avoiding false data to be propagated in the community or inefficient bandwidth usage.

2.2 Multi-agent systems (MASs)

The allure of agent technologies for the design and control of surveillance systems can be attributed to several key MAS characteristics, which appear most adequate for this application domain. MAS allows to address challenges of autonomous and decentralized decision-making in a flexible manner, by decomposing complex tasks and assigning subproblems to loosely coupled components that interact and coordinate autonomously to solve system-level design goals (Woolridge, Reference Woolridge2009). A significant part of the work surveyed in this paper applies some form of MAS implementation and below we present some aspects which are more particular to these systems.

Pavon et al. (Reference Pavon, Gomez-Sanz, Fernndez-Caballero and Valencia-Jimenez2007) demonstrate the use of INGENIAS, an agent-based model-driven methodology to develop a multi-sensor surveillance system. Essentially, INGENIAS provides a modeling language to describe the system, based on five viewpoints from which a MAS can be regarded: organization, agent, goals-tasks, interactions and environment. The organization level deals with the definition of roles as functional responsibilities of agents. The agent behavior is described using so-called mental states for managing goals, which need to be mapped to system tasks. Workflows are used to define collaborative plans and specify the interactions between agents. Lastly, the environment description contains details about the data collected by sensors and the possible actions to be performed by actuators. A security guard usecase, where different kinds of alarms can be specified, is depicted using this framework.

A multi-agent architecture for visual sensor networks is presented by Castanedo et al. (Reference Castanedo, Garcia, Patricio and Molina2008). The authors adopt the well-known belief–intentions–desires (BDI) model (Rao & Georgeff, Reference Rao and Georgeff1995) to represent the internal states of the agents. The solution is demonstrated in an indoor tracking application and is implemented using the JADEX toolkit (Braubach et al., Reference Braubach, Lamersdorf and Pokahr2003). In this context, the agent’s beliefs correspond to the information the agent acquires from the environment and the other agents. Desires express the motivation of the agent, in this case to perform tracking, while intentions are the basic steps chosen by the agent to achieve its desire. Additionally, a so-called fusion agent gathers track data from all the agents and determines a performance score for each agent based on the coherence with the rest of the data. This information is feedbacked to the agent, allowing it to use it as external information in order to reason about the confidence of its findings.

Similarly, Bustamante et al. (Reference Bustamante, Molina and Patricio2014) used the BDI paradigm to model an active camera, agent-based surveillance system. The system proposes a three-layer architecture: a sensor management tier, which provides distributed active camera control and management, a data fusion tier for data processing and a user interface tier, where the operator can monitor the system. The focus lies here in using data fusion as an integral part of the architectural design. Regarding the BDI agent model, beliefs are used to represent information about the environment, desires express the goal of the system (e.g. tracking targets), while intentions are used to implement these goals (e.g. controlling the camera). For implementation purposes, the authors define a scenario-specific agent ontology. Implementation is carried out using the JADEX framework. This prototype is evaluated in an indoor tracking application with a limited number of visual sensors. However, the architecture cannot offer support for integrating heterogeneous sensors.

2.3 Service-oriented and specialized middleware

Middleware systems and service-oriented architectures (SOA) are gaining traction as various kinds of sensor networks and their applications increase in complexity. Middleware is designed to abstract hardware-specific issues and facilitate interoperability of various components, such as ad hoc communication, discovery of new devices or establishing new communication links. Essentially, this simplifies development at the application level and exempts developers from handling implementation at the lower layers. SOA have emerged as a popular approach in terms of providing a middleware with standard protocols, common interfaces and well-defined components.

Memory Jog, introduced by Soldatos et al. (Reference Soldatos, Pandis, Stamatis, Polymenakos and Crowley2007), is a non-intrusive digital assistant for meetings, prototyped using an agent-based SOA middleware implemented in JADE (Bellifemine et al., Reference Bellifemine, Caire, Poggi and Rimassa2008). The goal of the service is to provide pertinent information facilitating humans to accomplish tasks during different types of events (i.e. meetings), that are occurring in a room setting equipped with a rich sensing infrastructure. Prototyping the service is done using a middleware designed to boost rapid application development. Whenever a new device is installed, the system instantiates a proxy agent for controlling this device. The room agent is the gateway for accessing all services available in that room. The procedure for a client to request a particular service is straightforward: the client sends a request to the gateway, which searches for a proxy for that service, which in turn finds an available service provider. Moreover, situation models can be defined, specifying the entities, properties and relations to be observed for triggering services. Situation models are expressed in XML format and parsed by a designated watching agent. Defining situation models can nevertheless be a tedious and error-prone task in case of more complex scenarios.

Aguilar-Ponce et al. (Reference Aguilar-Ponce, Kumar, Tecpanecatl-Xihuitl and Bayoumi2007) present an architecture for automated scene surveillance using wireless visual sensor networks. Again, the multi-agent framework is used to model the system components, which can be classified into object processing units (OPUs), scene processing units (SPUs), region agents (RAs) and object agents (OAs). Several OPUs are grouped to form a cluster, where each one contains a cluster head, responsible for collecting the information coming from the other members of the cluster and sending it to the SPU for further analysis. RAs are associated to cameras, while OAs are instantiated once an object is detected. OAs and RAs reside at the OPU level. Communication between OPU and SPU occurs by passing through the network only the object parameters. Finally, once this information reaches the SPU, a decision is taken regarding whether the scene contains a threat or not.

Piette et al. (Reference Piette, Dinont, Seghrouchni and Taillibert2015) propose a method for automatically deploying and configuring applications onto a heterogeneous hardware infrastructure, by taking into account the hardware properties and characteristics. In particular, the authors are focused on modeling ambient systems and ubiquitous applications. Both the hardware infrastructure and the application functionality are represented using graphs. A graph-matching algorithms determines the correspondence between the available hardware entities and the application requirements.

A middleware focused on dynamic reconfiguration in distributed camera networks is introduced by Jovanovic and Rinner (Reference Jovanovic and Rinner2007). Here, dynamic reconfiguration is interpreted in the sense of modifying the system functionality at run-time via exchanging software tasks between the system’s components or altering the Quality of Service (QoS)-levels of tasks. Reconfiguration is achieved using policies, which represent in effect a set of rules, specified based on the Houdini language (Hull et al., Reference Hull, Kumar, Lieuwen, Patel-Schneider, Sahuguet, Varadarajan and Vyas2005) and triggered when changes in the internal or external state of the system occur. The authors argue that a multi-agent implementation can benefit from flexibility and scalability, but fail to flesh-out the details of this implementation, nor do they provide rigorous experimental results to back their claims.

The concept of a task-oriented architecture for collaborative video surveillance is presented by Monari et al. (Reference Monari, Voth and Kroschel2008). Although object tracking is the problem under investigation, the authors claim that the architecture is not application specific, but could also accommodate different tasks. On a similar note with previous works, a cluster-based structure is proposed. However, clusters are dynamically instantiated for each tracking task and not predetermined at design time. This approach has the advantage of overcoming the issues of object handover between sensors. Without going into extensive detail, the authors mention that data obtained from the cluster’s sensors is aggregated using Bayesian fusion and that a cluster manager is used to receive tracking jobs from a human user. The solution was experimentally tested at the Fraunhofer institute using IP-cameras, yet an interesting challenge remains in adapting the architecture to make use of heterogeneous sensors.

2.4 Data fusion in distributed systems

Especially in the case of large-scale systems consisting of heterogeneous nodes, with limited or complementary sensing capabilities, collaboration among the sensor nodes becomes essential in order to improve the reliability of the analysis and to estimate correctly the state of the environment. In contrast, relying on a single node to receive and process all data is becoming increasingly impractical.

Snidaro et al. (Reference Snidaro, García and Llinas2015) provide a comprehensive survey on the importance of context in information fusion processes. The authors distinguish different levels of context representation that can serve as input to the data fusion process. Namely, contextual information can take the form of static data sets (e.g. geographic information system databases), contextual variables from direct observations (e.g. weather conditions) or inferred relationships between entities and the environment. Additionally, context can act as a constraint during the data fusion process or it can bring new problem dimensions, that can influence the possible interpretations of a situation. Overall, this work emphasizes the lack of reference architectures for the next generation of data fusion systems, which can integrate context in a meaningful way.

Collaborative information processing is achieved by Biswas et al. (Reference Biswas, Qi and Xu2008) using a mobile agent-based framework. The idea proposed is that of employing mobile agents that migrate from node to node following a certain itinerary, which could be predetermined or determined on-the-fly, to fuse data locally at each node. Mobile agents are used in an attempt to reduce communication cost and minimize energy consumption across the sensor network. Communication is carried out in a purely peer-to-peer fashion. An agent transfer from one node to another is initiated by the so-called source node. The agent code alongside its state is sent to the new location, where the agent is restored and resumes its activity. Notably, experiments are conducted evaluating the performance of this approach against the traditional client/server architecture. The comparison is focusing on energy consumption and execution time as the main metrics. The authors advocate their solution in the context of large-scale deployments of sensor networks, where the communication overhead increases significantly due to the amount of sensor nodes. Results show that under large-scale network conditions (i.e. large values of the following parameters: size of the network, raw data size and file access overhead) the mobile agent-based approach outperforms the client/server implementation. Namely, given the simulation setup in the paper, MAS performed better both in terms of execution time and energy consumption when the number of nodes in the network exceeded 16. Similarly, the same occurs when the ratio between message size and the mobile agent size exceeds 20. However, to be noted that mobile agent solutions may introduce additional issues such as replication faults, security problems or planning the agents’ itinerary.

Cho et al. (Reference Cho, Lim and Yang2010) introduces a homogeneous architecture with equally powered visual sensor nodes. The goal of the surveillance system is to reason about the occupancy of certain areas. Additionally, a locality structure is used to decrease the computational complexity using clusters. Clustering is performed taking into account the topology of the camera network and more specifically, according to overlapping fields of view (FOVs). Data processing within the cluster entails that all data are collected to a sink node where computation occurs. This brings about the challenge of determining an efficient way to pass data such that the path from leafs to the root is minimized. The solution proposed by the authors to tackle this problem resembles a simplification of the distributed constraint optimization approach of Meisels (Reference Meisels2007), whereby the topology graph is reduced to a spanning tree using a breadth first search and values are propagated from leafs to a root node. However, this only captures part of the problem because attaching many child nodes to one node can reduce the path, but at the same time may create bottlenecks in the case of resource limited nodes. The trade-off between path length and enabling parallelized processing needs to be further explored.

SPIRIT, introduced by Town (Reference Town2007), is a surveillance system based on an ultrasonic location infrastructure to track people and devices in an office building, also relaying on information coming from cameras. A number of vision techniques used in this project include background and object appearance modeling, face detection, segmentation, and tracking. The system has been deployed at Cambridge University. Data fusion is realized via a so-called shared world model, which incorporates static information, such as positioning of sensors, as well as dynamic information, like the location of people or devices. Sensor readings are updated to the model such that it remains consistent and accurate. In order to do so, Bayesian networks are used to perform inference and newly derived hypotheses compete for observations with existing hypotheses. The disadvantage of this approach lies in the fact that people are supposed to carry on them a bat device which can be triggered by radio to emit an ultrasonic pulse, based on which the location of a person can be determined.

2.5 Image processing for computer vision

Computer vision is one of the most active research topics with direct application to surveillance systems. Processing video scenes involves a number of steps of increasing complexity, initiated by modeling the environment, detecting motion, followed by object segmentation, classification and tracking, while more advanced solutions have the capability to recognize activities and understand certain behaviors (Ko, Reference Ko2008). As already mentioned, an in-depth analysis of computer vision techniques is outside the scope of this paper, thus we only describe briefly several notable elements and contributions of this area.

Although motion detection can be achieved in a number of ways, statistical methods, such as Gaussian mixture model (GMM) (Power & Schoonees, Reference Power and Schoonees2002) have the benefit of robustness to noise, shadows or changes in lighting conditions. GMM describes each pixel as a mixture of Gaussians distributions and classifies it as background or foreground. The limitation of this approach is its computational complexity, which can be alleviated by using specialized hardware (i.e. graphics processing unit, field programmable gate array).

Object classification can essentially be carried out using either a shape-based or a motion-based approach. The former uses some form of shape information (i.e. blobs, silhouettes) to distinguish objects, while the later identifies periodic properties of moving entities to categorize them. Tracking methods can also be divided into several categories: region-based tracking, active contour-based tracking, feature-based tracking, and model-based tracking. In practice, exploiting contextual information has proved a helpful approach in improving the accuracy, by constraining the object recognition and tracking problems in the context of a particular application (i.e. vehicles are constrained to roads as opposed to the sky). For an extensive survey on tracking methods we refer the reader to Yilmaz et al. (Reference Yilmaz, Javed and Shah2006).

Finally, behavioral analysis is concerned with the generation of high-level descriptions of actions and interactions for the identified objects and represents an open research challenge. Current techniques range from template-based approaches, such as dynamic time warping (Veeraraghavan et al., Reference Veeraraghavan, Chellappa and Roy-Chowdhury2006), that try to match time-varying data to predefined reference patterns, to state-based methods. Hidden Markov models (Zhang et al., Reference Zhang, Gatica-Perez, Bengio and McCowan2006) is a state-based techniques which has successfully been used to recognize particular classes of activities and which generated a number of variations to the initial model (i.e. PHMM (Vogler & Metaxas, Reference Vogler and Metaxas1999), CHMM (Oliver et al., Reference Oliver, Rosario and Pentland2000), LHMM (Oliver et al., Reference Oliver, Horvitz and Garg2002)). Also various supervised machine learning techniques for classification have been applied for this task, however, in order to obtain a high statistical confidence, large amounts of training data are required. For instance, time-delay neural networks, which essentially append delay units to a general static network, have been applied for hand gesture recognition and lip reading. Other examples include the use of Bayesian networks and context-free grammars (Ryoo & Aggarwal, Reference Ryoo and Aggarwal2008) for describing human actions by recognize composite activities.

2.6 Semantic technologies

Ontology-based approaches are used to provide a shared conceptualization of a given domain of interest. In this manner, meaning can be encoded separately and decoupled from the implementation code or the data being streamed through the system. This allows a machine (but also human) interpretable model, encapsulating the relationships between the terms of the ontology. Because of this principle, ontologies can be further augmented and modified at execution time without the need of re-engineering the application code. Among the most prevalent technologies, we note the RDF (Lassila & Swick, Reference Lassila and Swick1999) and WSDL (Christensen et al., Reference Christensen, Curbera, Meredith and Weerawarana2001) formats, the SPARQL (Prud’hommeaux & Seaborne, Reference Prud’hommeaux and Seaborne2006) query language and the OWL (W3C OWL Working Group, 2009) computational logic-based language, which enables inference capabilities.

In the surveillance domain, there are a number of works explicitly concerned with developing ontologies built on a specific use case. A forensic domain ontology is proposed by Sobhani et al. (Reference Sobhani, Kahar and Zhang2015) to support an automated visual surveillance system. The authors use a top-down approach to construct the ontology by describing activities, entities, relationships, resources and consequences. Notably, the ontology is built to capture outdoor riot scenarios, subject to the data provided by the London Metropolitan Police during the 2011 riots in United Kingdom. Another interesting example is the BOnSAI ontology introduced by Stavropoulos et al. (Reference Stavropoulos, Vrakas, Vlachava and Bassiliades2012), which is specialized for incorporating ambient intelligence concepts for smart building scenarios. It provides the capability to model explicitly functionality, QoS, hardware, context, as well as the user. Hachem et al. (Reference Hachem, Teixeira and Issarny2011) propose an ontology designed to capture the heterogeneity in terms of hardware sensors and devices in the IoT. In addition, they include a representation of real-world objects by modeling them in the ontology as physical concepts. Similarly, the Semantic Sensor Network ontology by Compton et al. (Reference Compton, Barnaghi, Bermudez, Garca-Castro, Corcho, Cox, Graybeal, Hauswirth, Henson, Herzog, Huang, Janowicz, Kelsey, Phuoc, Lefort, Leggieri, Neuhaus, Nikolov, Page, Passant, Sheth and Taylor2012) can describe sensors in terms of capabilities, measurement processes, observations and deployments. These ontologies could prove instrumental in terms of managing complex surveillance infrastructures with heterogeneous sensing equipment.

Mihailescu et al. (Reference Mihailescu, Persson, Davidsson and Eklund2016) propose an IoT system design, based on an internal information representation for sensor-enabled environments. New sensing tasks can be injected into the system, while a semantics-empowered search heuristic is responsible for dynamically assigning data sources for each task. The solution consists of instantiating virtual sensors, which can exploit the existing sensing infrastructure in order to support new value-added services. Solution updates are generated on-the-fly in response to changes in the contextual data. Moreover, the system is demonstrated in a smart building application concerning activity monitoring. For the same task of activity monitoring in office spaces, Nguyen et al. (Reference Nguyen, Venkatesh, West and Bui2014) propose a solution which uses ontological modeling, representation and reasoning. The ontology models activities, spacial contexts, artifactual contexts and the relations between them. Artifacts represents smart things in the environment. The system monitors the state of all artifacts in the environment and reasons with the ontology to recognize which activity is taking place. A time synchronization mechanism is applied in order to generate coordinated sensor readings. The system can handle multiple users, as well as areas.

2.7 Self-organizing and self-adaptive mechanisms

The goal of applying self-organizing processes to surveillance systems is that configuration aspects can be handled autonomously at run-time, such that the system can adapt as the situation demands. Much of the work on surveillance system falls under the assumption of having the network configuration already in place and thus do not deal with this problem. Hoffmann et al. (Reference Hoffmann, Wittke, Hahner and Muller-Schloer2008) are concerned with such an aspect, namely, spatially partitioning an area of interest given a set of cameras. They elaborate a self-organizing mechanism, where cameras communicate with their neighbors and independently calculate their FOV. Maintaining a low-communication traffic and configuring networks containing a larger number of cameras are also the objectives of this work. Essentially, cameras locally adjust their FOV and exchange state information with neighboring cameras in order to minimize FOV overlap and thereby maximizing surveillance coverage. To reduce communication overhead, only improvements over a certain threshold are considered. Yet, the authors do not account for the fact that small local changes could lead to major global improvements. This means that the solution could in fact be arbitrary far from optimal.

Self-adaptation is yet another important feature that can offer surveillance systems the flexibility to provide services in accordance to the situational characteristics of the monitored environment. Such an example is given by Byun and Park (Reference Byun and Park2011), where a surveillance system is used for providing control and energy saving services in buildings. From an architecture standpoint, the system consists of self-adapting gateways and sensors. The gateway provides location-based services and can control a set of devices, while also having information about users, their associated devices and their profile. A user is recognized based on his MAC address and a list of manageable devices becomes available on the user’s smartphone once he enters the room. Hence, specific services are provided to users under given situations, such as building energy monitoring and control services. A test bed has been developed and experiments were conducted showing a moderate reduction in energy consumption.

Also, nowadays, especially in the case of smart buildings, surveillance systems extend beyond a network that is owned and operated by a single stakeholder and instead consist of nodes which are property of multiple owners. In order to accommodate interoperability and heterogeneity between services of increasing diversity, service discovery is essential. Moreover, in a decentralized setting where there is no central repository responsible for the management of services, self-organizing mechanisms can play a central role. Zhu et al. (Reference Zhu, Mutka and Ni2005) survey a number of key service discovery protocols and point out the fact that service discovery design needs to address the challenge of operating in unfamiliar, dynamic contexts without trusting the environment. Thus, self-organization can be instrumental in establishing the structural relations between the system’s components and adapting to changes in the environment in order to improve the system performance.

3.1 System architecture

We identify in the literature several prominent architectural designs that are typically employed and which we propose to characterize over four main dimensions:

• ∙ Decision-making: Refers to the organization and interaction of the system’s components required in order to carry out the system objectives.

  1. o In a centralized setting the information collected throughout the system is concentrated at a single processing point that carries out the analysis. The Server/Client design pattern belongs to the class of centralized architectures and is usually encountered in traditional surveillance systems, where resources such as computing power, memory and storage are concentrated at the server-side.

  2. o In order to minimize communication and computation overhead the architecture can aim to distribute the processing load between the nodes. Distributed architectures are typically organized in a hierarchical manner, having low-level functions carried out as close as possible to the sensing nodes, which are often, in practice, structured into clusters, while only the results of these computations are passed to the upper layers.

  3. o In a fully decentralized architecture, each node retains full controllability of its processes and a system-wide negotiation is conducted to realize the desired goals. Interaction between the systems’ components takes place in a peer-to-peer fashion, where nodes pool their resources together in order to implement some functionality at the system level.

  4. o Various combinations of the previous types of design can pose interesting alternatives leading to hybrid architectures.

• ∙ Model: Refers to the framework that underlies the system’s design.

  1. o SOA represent a good approach to achieve interoperability between heterogeneous devices, making use of services to abstract device-specific functionalities and communication capabilities. A midddleware is then required to enable basic service management such as service discovery and service composition.

  2. o Surveillance systems can be regarded as an area with high potential for the application of multi-agent-based technologies, which provide a natural way to model a complex, distributed system by means of breaking it down into loosely coupled, autonomous components that coordinate to fulfill high-level tasks.

  3. o Several works propose events as the building blocks for describing surveillance systems. For event-driven architectures, detection and classification of events are the key features, while more high-level interpretations (i.e. scene understanding) can be further derived.

  4. o In practice, often times ad hoc solutions are devised to solve particular tasks that do not provide a generic framework, extendable to other functionalities.

• ∙ Interoperability: Concerns the capability of the system’s components to exchange data without or with very little knowledge about the unique characteristics of the other components.

  1. o Closed: This is a limitation present in the majority of existing surveillance systems, which makes appending and upgrading the infrastructure or functionality very difficult, often times requiring to re-engineering the system at large.

  2. o Open: Engineering open system essentially means enabling interaction with other components whose presence could not be determined at design time and regardless of the programming languages used in their implementation, however, abiding to certain requirements or protocols.

• ∙ Extensibility: Refers to the level of effort required to implement the extension.

  1. o Flexible: Allows the addition of new functionality, while minimizing the impact to existing system functions.

  2. o Limited: Hinders systematic reuse in implementing extensions.

3.2 Sensing nodes

Sensing nodes represent the building blocks of the sensing infrastructure and can be primarily distinguished based on their capabilities, as well as the type of data being manipulated in the system. We propose to characterize them over the following dimensions:

• ∙ Processing: Sensors can be classified depending on their on-chip capabilities in terms of the processing power of the device. We can differentiate between devices equipped for running more computationally expensive algorithms, others, where only basic signal processing is performed or even those that provide no processing capability. The former case represents an approach where algorithms, such as those applied in computer vision techniques, are aimed at extracting features or metadata from the incoming sensor stream. Just this information will then be passed on and used in more high-level tasks (i.e. object recognition). This type of systems are also intended to perform data processing in real-time, making it critical that the device is apt to deliver results in a time-frame that remains meaningful for the objective at hand. Sometimes though, it is sufficient that only a simple preprocessing phase is carried out at the sensor level, which typically involves a (temporal) filtering step for smoothing the data or eliminating noisy readings.

• ∙ Storage: Regarding storage, data can either be locally stored, be it in the form of raw readings or some kind of metadata, or directly streamed to a cloud infrastructure. There is also the case when algorithms are directly applied to the stream of data, such that models are learned in real-time without having to actually store this data.

• ∙ Acquisition: The way that data are acquired can be thought of as active feed, when sensors adapt their observation strategies to capture certain aspects in the environment. For instance, sensors can adapt their sleep-cycles to conserve battery-life during periods that show reduced data variability. This can either be an individual, or a more complex collective process, where sensors coordinate their data-gathering actions. Otherwise, it is considered a passive feed process.

• ∙ Privacy: Surveillance systems typically manipulate sensitive information, therefore privacy-preservations needs to be a highly regarded feature. It is often the case that surveillance systems output person-specific information (i.e. MAC address, biometric data), which depending on the circumstances, may or may not need to be anonymized.

• ∙ Quality: Surveillance systems can also be differentiated based on the accuracy of the data provided. In order to bring down the cost of deployment, some systems consist of unreliable, low-cost sensors and thus, noisy data has to be tolerable to the system.

3.3 Sensor network

The sensor network exhibits differences along several lines:

• ∙ Composition: A surveillance system can be regarded as homogeneous if it consists of sensors of the same type, with similar capabilities (i.e. video-based surveillance), or heterogeneous if it represents a mix of sensors with different properties. A list of frequently used sensors includes various types of cameras, static, Pan Tilt Zoom (PTZ), omnidirectional, smart cameras and acoustic sensors, RFID tags, or smart cards.

• ∙ Size: Generally, encompassing more than thousands of nodes, classifies systems as large-scale, otherwise they are considered small-scale.

• ∙ Ownership: If all the sensor nodes in the network are owned by the same entity, the system is considered to have a single owner, and multiple owners otherwise.

• ∙ Communication protocols: Sensor networks are also distinguishable based on the type of communication employed. For instance, Wi-Fi data transmission, as well as ZigBee represent some of the most popular protocols seen in practice. Overall, communication in the network can be classified as restricted, in terms of distance, bandwidth, reliability and security, or non-restricted.

• ∙ Dynamics: We make the distinction between sensor networks, which only consist of sensors that remain static throughout the entire surveillance process and sensor networks that can be considered dynamic. The latter category may incorporate sensor which are mobile or sensors that dynamically appear or disappear in the network.

3.4 Intelligence

Intelligence, both at the system and component level, may become one of the most promising lines of research in surveillance systems and can be understood as the ability of the system to perceive the environment and exhibit not merely a reactive, but a goal-driven behavior that will affect what it can perceive in the future. We outline below several features, which in our acceptation are key toward increasing the level of intelligence of the system and where there also appears to be a lack of contributions.

• ∙ Autonomous behavior: Studying surveillance systems by behavioral means, we distinguish the following classes, that denote different degrees of complexity:

  1. o Reactive: The system responds (i.e. via alerts) to changes that occur in the environment in order to satisfy its design objective. This type of systems display the lowest degree of complexity and do not involve reasoning about the environment, but only map sensory input to an action to be performed (i.e. rule-based).

  2. o Task-oriented: A set of design tasks are concomitantly pursued, while potential conflict or resource constraints are mitigated based on an a priori-defined prioritization scheme. The steps required for delivering a certain task are also made explicit at design time.

  3. o Goal-driven: The system exhibits a proactive behavior in order to satisfy its objective. This means that it is capable to elaborate at run-time a plan, which represents a sequence of available actions that can be executed in order to bring about the desired goal.

• ∙ Data-fusion: Considering a multi-sensory system, data fusion can be carried out at two levels: feature level fusion or early fusion—features extracted from the input data are first combined and then sent as input to a single analysis unit; decision level fusion or late fusion—decisions are obtained from interpreting features using local processing units and then these decisions are further combined to obtain a final decision.

• ∙ Self-configurable: Represents the capability of the system to invoke particular settings or properties over the sensing infrastructure. This can mean anything from altering communication patterns between nodes, to augmenting processing capabilities, to modifying the resolution of incoming data.

• ∙ Context-aware: More sophisticated surveillance systems can distinguish between different modes of operation according to the state of the environment. Especially in situations that require some kind of actuation (e.g. smart buildings), context conditions can trigger the action plan that is most appropriate, acting as precondition. Also, a context can be understood as a higher-level semantic interpretation of the environment, which can be derived from observations aggregated over a spatio-temporal horizon.

• ∙ Sensor characterization: Information representation has an important impact in the way data are processed throughout the system. Solutions that enable a higher level of flexibility in terms of resource management and data processing use ontologies in order to semantically characterize their constituent sensors and data flows running within the system. Otherwise, ad hoc solutions are devised for keeping track of this information.

• ∙ Actuator control: The system is able to perform some level of actuation based on reasoning over the input data, such as optimizing the operation of an elevator in a smart building.

3.5 Environment

Surveillance systems are also distinguishable according to the environment they are designed to operate in.

• ∙ Observability: Assesses the level of perceptibility of the environment.

  1. o Full: Entails that sensor observations can fully capture the environment state without uncertainty.

  2. o Partial: It represents situations in which there is uncertainty about the environment state given the sensors’ observations. In practice most environments are regarded as partially observable.

• ∙ Dynamicity: Assesses the level of dynamism in the environment.

  1. o Static: This type of environment can be assumed to remain unchanged, except for the movement of objects or persons that are the subject of surveillance. Operating under this assumption can significantly simplify the logic of the system. This means that a static reference model can be used to describe the area under surveillance and thus, the task of object detection can be reduced to background subtraction. As an example, deploying surveillance in indoor environments can benefit from a better controllability of the monitored area, as well as a lesser influence from external factors (i.e. weather).

  2. o Dynamic: Such environments are characterized by high variability in scene conditions, requiring more elaborate methods to perform even basic tasks such as object recognition. Given that the background cannot be assumed to remain static, the system is required to continuously update the background model, to account for changes in the environment. Another viable solution is to perform temporal difference by subtracting consecutive frames. Outdoor settings are a typical case of dynamic environments, where algorithms need to handle more complex requirements. For instance, a common challenge in outdoor visual surveillance is to cope with changes in lightning conditions. Also, the topology of the sensing infrastructure and even the architecture is generally influenced by this property of the system.

• ∙ Predictability: Assesses the level of uncertainty in the environment.

  1. o Predictable: An environment is predictable or deterministic when any action has a single guaranteed effect and there is no uncertainty about the state that will result from performing an action.

  2. o Stochastic: Otherwise, the environment is considered stochastic or non-deterministic.

4.1 Tracking via task allocation

Tracking is an interesting application because it requires a combination of real-time tasks including motion detection, object recognition, as well as some coordination between sensor nodes. A tracking scenario with fixed and overlapping FOVs is considered by Nguyen et al. (Reference Nilsson2003). Given that each object is assumed to be visible by multiple cameras, the question becomes one of choosing the most appropriate camera for each object. The paper describes a centralized assignment algorithm to track people, where each camera is connected to a camera processing module running on a computer, while a central module maintains a database with all camera-object pairs. The assignment is based on a simple heuristic using the object-to-camera distance. Apart from being a centralized coordination solution and thus suffering from inherent computational and communication bottlenecks at the central server when the number of cameras increases, the approach also fails to base the assignment on other QoS parameters, or account for the case of having multiple objects tracked by the same camera.

Quaritsch et al. (Reference Quaritsch, Kreuzthaler, Rinner, Bischof and Strobl2007) report the design of a multi-camera tracking system. They propose a decentralized MAS-based approach, where for each object being monitored, a single tracking instance is initiated in the multi-camera system, represented by a mobile agent migrating to the camera which observes the object. Migration is performed using a master/slave handover, having the master create a slave on the neighboring camera where the object is expected to appear. Once the object enters the field of view of the slave, the roles are interchanged, while the old master terminates thereafter. The implementation is done in Java, which introduces severe limitations in terms of the time required to create and launch the slave agent, thus making it impractical for actual usage. Replacing the code migration approach may alleviate this issue and is a prerequisite for real-world deployment.

In the work of Krahnstoever et al. (Reference Krahnstoever, Yu, nam Lim, Patwardhan and Tu2008), tracking is performed by combining in a master/slave configuration fixed cameras with the versatility of PTZ cameras. The novelty consists in specifying a set of application specific quality measures, which are used to define the objective function of the system used to balance the number of captures per subject and quality of captures. The problem is cast to a combinatorial search solved using a heuristic approach.

4.2 Smart buildings

An important application domain for surveillance systems is provided by smart buildings. The growing demand for upgrading building management and automation has led to the deployment of various surveillance systems to this end. In the CMIPS project, Yong et al. (Reference Yong, Qiao, Wilson, Wu, Clements-Croome, Liu, Egan and Guy2007) focused on providing a real-time sensor-based building assessment tool. The infrastructure consists of a wireless sensor network logically divided into zones at design time. A gateway acts a service aggregator by enabling clients to access services provided by these embedded sensors, while masking the complexity of generating tasks for nodes. The core functionality is in assessing the energy consumption of a building. Additionally, a multi-agent model is proposed, where personal agents running on mobile devices are used to capture feedback from occupants in the building on their satisfaction level.

In the ISES project of Davidsson and Boman (Reference Davidsson and Boman2005, Reference Davidsson and Boman2000), the objectives are both energy saving and increasing customer satisfaction through value added services. A MAS is proposed to oversee the control of the system, where room agents devise plans to optimize environmental conditions, based on the occupants’ preferences and energy saving considerations. Thermodynamical characteristics of the rooms are captured in the model and several strategies based on the behavior of occupants are employed, such as pre-heating the rooms according to electronic diaries. Notably, the solution has been deployed in a real-world setting and the authors report up to 40% energy savings.

Patti et al. (Reference Patti, Acquaviva, Jahn, Pramudianto, Tomasi, Rabourdin, Virgone and Macii2014) and Khajenasiri et al. (Reference Khajenasiri, Patti, Jahn, Acquaviva, Verhelst, Macii and Gielen2014) have the goal of converting existing buildings, to the extent possible, into smart buildings by exploiting occupancy information coming from surveillance systems. Similarly to the previous work, they are concerned with energy efficiency. The paper proposes a three-layered SOA, with the main management functions executed via a rule engine. For instance, the logic of the HVAC control is prescribed by a rule, which is reevaluated once modifications in the sensor data occur.

From an organizational standpoint, Silva et al. (Reference Silva, Arakaki, Junqueira, Filho and Miyagi2012) adopt the concept of holons to represent the structure of control in their surveillance system applied to smart buildings. Holons introduce the notion of a hierarchical, nested structure, where groups of individual holons are composed into superior holons. The procedure is designed for smart buildings and a holon here can be either a physical device or a logical component. The paper describes a methodology for mapping the operation of a building onto a holocracy. Special attention is devoted to fault-tolerance requirements, by using Petri nets as a modeling support for performing formal tests on the behavior of the system.

4.3 Traffic

Transportation, whether it relates to vehicles or the road infrastructure itself, is becoming more infused with sensing devices in an attempt to increase efficiency and safety. Haesevoets et al. (Reference Haesevoets, Van Eylen, Weyns, Helleboogh, Holvoet and Joosen2007) introduce a surveillance system design for monitoring traffic conditions on a highway. In terms of the architecture, a MAS overlays the camera network. Agents can communicate with neighboring agents and are differentiated into several roles. A data observer is a software agent that resides at each camera and monitors information about traffic density, intensity and average speed. Data pushers are agents responsible for forwarding the observed data to the data aggregator role. The system is capable to identify areas with traffic jams in a fully distributed way using simple message passing between agents.

Again, another interesting agent-based approach is proposed in the traffic domain, geared toward adapting existing surveillance systems along highways and streets toward semi-automatic surveillance systems, in the work of Rothkrantz (Reference Rothkrantz2013). The paper introduces a similar architecture implemented in JADE, consisting of three types of agents. Identification agents are used for handling request about objects detected by observer agents. This information is then transmitted to a reasoning agent, which tries to infer whether the object has shown a suspicious behavior based on a set of predefined suspicious events. If that happens to be the case, a human operator is alerted.

4.4 Movement assessment in sports

Evaluating the performance of athletes is yet another domain taking advantage from monitorization using video cameras or other sensing equipment such as wearable inertial sensors (i.e. gyroscopes, accelerometers). The goal of these systems can vary from monitoring the potential for injury, to studying team behavior and providing relevant analytics, to detecting fine-grained body movements. An exemplification of the latter is given by Ahmadi et al. (Reference Ahmadi, Mitchell, Richter, Destelle, Gowing, O’Connor and Moran2015), where the authors propose a framework for classifying training activities in a real training environment, designed to feedback data to athletes that can lead to performance enhancement. The study is based on detecting an athlete’s joint angular motion and impact accelerations using off-the-shelf classification techniques with up to 98% accuracy. The discrete wavelet transform is applied to extract discriminative features, while normative data are used to asses the athlete’s deviations from correct movements. Additionally, the focus of the approach is on minimizing processing time and user involvement.

4.5 Domotics

Home automation, also known as domotics, represents the residential extension of smart building and has received considerable interest in recent years. Surveillance systems can harness the built-in infrastructure of smart homes to deliver increased safety and security. An exemplification in this context is presented by Carchiolo et al. (Reference Carchiolo, Longheu, Malgeri, Mangioni and Torrisi2010) and Longheu et al. (Reference Longheu, Carchiolo, Malgeri, Mangioni, Longheu, Carchiolo and Malgeri2012), advancing a solution based on the ZigBee protocol (Yi et al., Reference Yi, Iwayemi and Zhou2011) for detecting and classifying intrusions through video monitoring. The system consists of a supervisor, that can be implemented on a low-cost microcontroller and which manages devices (i.e. IP-cameras) through a coordinator module, that acts as an interface between the supervisor and the ZigBee network. A Gaussian background model is applied for detecting intruders, generating an alarm notification to the operator’s smartphone. Real-time video streaming is also available remotely, allowing the operator to determine whether the alarm has been triggered correctly. Also, the alarm is customizable, enabling the operator to configure the parameters of the algorithm, this task requiring, however, specialized knowledge. With this consideration in mind, we hypothesize that an enhanced version of the system should incorporate the user feedback on false positive or negative triggers and automatically adjust parameters accordingly.

Other smart home approaches, such as the work of Chen et al. (Reference Chen, Chang and Chen2015), are favoring Wi-fi wireless technology for data transmission in comparison to the ZigBee standard due to its reliability and its popularity. This paper introduces a multi-sensor embedded home monitoring framework, including humidity, gas, smoke, and temperature sensors. A Dynamic Host Configuration Protocol sever is adapted to dynamically allocated a virtual IP, thus ensuring that all modules have a wireless link to the Internet. Although promoted as an intelligent system, its functionality appears to be confined to basic threshold detection, followed by SMS alarm pushing to an administrator. Some actuation is also available, such as automatically activating fans once the temperature falls into certain intervals.

4.6 Military applications and crisis management

Autonomous surveillance systems can further be designed to prevent, detect, and aid crisis management. Unmanned aerial vehicles (UAVs) are a good candidate to provide enhanced situational awareness in the event that emergency response is needed (i.e. earthquakes, floods, deforestation). The SIERRA project of Brown et al. (Reference Brown, Wei, Ozburn, Kumar and Cohen2015) is focused on forest fire applications and aims to allow real-time monitoring capabilities for fire-fighters. The project is currently in the prototyping phase. The desired functionality encompasses the capability to approximately encloses the fire area on a map, as well as to incorporate additional data about the environment (i.e. vegetation, moisture) in order to predict fire propagation under given conditions. Preliminary results describe several tests conducted in cooperation with emergency response teams and point out the advantages of UAV assistance.

4.7 Crowd scene surveillance

Crowd-level analysis is an active area of research in video surveillance. A large number of methods have been suggested especially for the problem of crowd counting, that is, estimating the number of people in a scene. This information can be relevant in a wide range of scenarios, for instance, to adapt public transport capacity or to determine crowd density in retail stores. A recent case study is described by Yan et al. (Reference Yan, Chao, Ke Wei and Tao2015) and stands out by deploying a low-cost, non-intrusive counting solution. Infrared distance sensors make this approach adequate for circumstances with privacy concerns. The architecture consists of wireless motes that collect data from the infrared sensors, filter it, determine the direction of the passing targets and eventually forward the result to a base station. However, the accuracy can be negatively impacted by situations when people are passing through the sensors side by side or when someone is standing right in front of the sensors.

4.8 Ambient assisted living

Demographic aging of the population is putting pressure on delivering monitorization solutions that can replace humans in providing one-to-one assistance for the elderly or disabled. Continuously monitoring the state of a person can become particularly important when the person under supervision is unable to actively solicit emergency intervention, such as pressing an alarm button. Unfortunately, these solutions are still prohibitive in terms of pricing. With the objective of increasing efficiency and reducing cost, Meinel et al. (Reference Meinel, Findeisen, Hes, Apitzsch and Hirtz2014) propose a solution for elderly care, that uses a single omnidirectional camera, which is able to cover a 360° view of a scene using a fisheye lens. The idea is to extract virtual perspective cameras from the complete omnidirectional image material for each person entering the scene and use them for tracking during their presence in the observed area. The algorithm detailed performs a combination of blob extraction for detecting people and Kalman filter for tracking, showing that processing can be attained in reasonable time for real-time scenarios.

In a similar manner, Findeisen et al. (Reference Findeisen, Meinel, Richter and Hirtz2015) deploy an omnidirectional stereo sensor for human behavior analysis in complex health-care environments. The geometrical configuration and positioning of their cameras is designed such that in can infer depth data. Acquiring this type information from the environment proves to be instrumental in improving the performance of human behavior analysis algorithms. Several related approaches have also used other 3D sensing technologies for indoor monitoring, such as KinectFootnote ¹ , although this comes with a lower resolution and a narrower field of view.

6.1 Situational and context awareness

Automatically acquiring context models from data is an important open research issue in the realm of surveillance systems. This capability can enable systems to operate in a more flexible and adaptable manner and switch between different modes in response to environment dynamics or user requests.

An effort in this direction is proposed by Fernández-De-Alba et al. (Reference Fernández-De-Alba, Fuentes-Fernández and Pavón2015), through the FAERIE architecture, which provides a distributed approach based on a federated blackboard model. The premise of this work is to enable flexible access to information at any level of abstraction. The underpinning of their solution is the introduction of three types of components: context elements, context containers, and context observers. Context containers implement a distributed virtual blackboard model for data sharing (context elements) among the context observer components, which are responsible for manipulating the data. However, this approach does not currently support real-time processing capabilities.

Bicocchi et al. (Reference Bicocchi, Fontana and Zambonelli2014) propose a starting point for what is termed an awareness framework for knowledge collection. They suggest to use general-purpose algorithms to classify data streams. The architecture consists of three layers, a sensor layer for collecting data, a classifier layer consuming data from the previous layer and an awareness layer based on labels produced during classification. The latter is considered to be central for designing a self-awareness module. Labels are then used to generate a state-based automata which encapsulates a particular context. The solution is demonstrated by means of a life logging application and in the current form, contexts are limited to situations that can involve only one individual.

6.2 Privacy preservation and security

Addressing privacy-related issues is paramount for a mass adoption of surveillance technologies. In spite of this, the trade-offs between privacy preservation and security are consistently overlooked. The literature tackling these problems is scattered and contributions in this area are particularly timely.

Yu et al. (Reference Yu, Sekar, Seshan, Agarwal and Xu2015) argue that traditional host-centric security solutions are fundamentally infeasible for today’s heterogeneous, large-scale networks of sensing devices. The authors propose to rethink network security in order to incorporate context awareness, learning capabilities that can distinguish between normal states of the network and attacks, as well as introducing more dynamic security policies. In addition, a roadmap for tackling these challenges is put forward.

The work of Winkler and Rinner (Reference Winkler and Rinner2014) presents a survey of privacy and security threats and challenges associated with visual sensor networks. The authors classify security aspects into data-centric, node-centric, network-centric and user-centric, and discuss the individual security requirements for each class. Moreover, they identify trends in privacy protection techniques for image data such as blanking, obfuscating, scrambling, abstraction and encryption. However, the conclusion is that a direct use of security techniques developed for related domains is not a straightforward task and a lot of research issues are still open.

6.3 Scalability

A key challenge for current surveillance systems is the capability of deployment in large-scale settings. We describe hereafter several notable efforts in this regard.

Velipasalar et al. (Reference Velipasalar, Schlessman, Chen, Wolf and Singh2006) are concerned with the communication aspects in distributed peer-to-peer camera systems, with special emphasis on reducing the number of messages, as well as the message size. The protocol proposed is designed for object tracking and its applicability in another context remains unclear. One of the desired properties of the system is fault tolerance. Thus, according to this approach, each camera keeps its own track of the objects detected, ensuring redundancy and also a reduced requirement for communication. Collaboration between cameras is constrained to maintaining consistent object labeling. Interestingly, the solution captures the situation when objects merge or split, such is the case of a person entering a car. Messages exchanged between camera are classified as either requests or replies and are processed in a non-blocking fashion avoiding deadlocks or process starvation. Synchronization can, however, become a problem, handled by introducing synchronization points, based on a certain synchronization rate.

A lightweight architecture for online video processing is introduced by Camplani and Salgado (Reference Camplani and Salgado2011), fully developed in C++ based on the OpenCV libraries. Scalability is addressed in the sense of having one processing unit controlling a group of cameras, while a central unit oversees the entire process. Moreover, at the level of each processing unit a parallelization scheme ensures that the operations on the images are pipelined. The pipeline is built as a cascade of stages, which are application specific and thus need to be represented through a task processing design flow. A case study for object detection is presented and although the architecture is suggested for mass deployment, the testbed consists of only three cameras. Results show some improvement in terms of processing speed.

To tackle the challenges of large-scale deployments, Park et al. (Reference Park, Min and Lee2017) propose an automated surveillance system using edge computing. The architecture introduces an inbetween layer, termed cloud of things (CoT), which has the role of interfacing between the data sources and the server. The CoT, which is the core contribution of this work, specifies three types of modules. The manager module is responsible for keeping track and monitoring the alive status of sensors and actuators. The analysis module performs basic processing and filtering of raw data, as well as invoking actuators in real time, thus improving the reaction time between edge and sensors. Finally, the software manager module receives analysis routines from the server, which need to be performed locally.

6.4 Integration

In spite of its prevalence, most of the existing literature on surveillance systems appears to be confined to application-specific settings, often addressing a very limiting aspect posed by a particular scenario (i.e. using ultrasonic infrastructure to track people), but failing to present them in the context of a broader system architecture. In order to be able to move away from experimental, small-scale test-bed cases and toward large-scale end-to-end deployments, it is critical that systems integration aspects are carefully considered.

An example of a more integrative approach developed at Carnegie Mellon University is given by Collins et al. (Reference Collins, Lipton, Fujiyoshi and Kanade2001). The surveillance system is capable to perform a wide range of functions ranging from movement detection, object tracking to semantic classification and human motion analysis. Additionally, different techniques can be seamlessly and alternatively used to perform these functions. The authors correctly point out that in order to coherently interpret sensor data, all observations must be brought into a common frame of reference, which here is represented by the geodesic coordinates. Also, camera calibration is performed using this common coordinate system. Control is centralized in the sense that sensor processing units send a continual stream of time-stamped symbolic object observations to an operator control unit. Thus, a new observation can be inspected whether it corresponds to an object already known. Moreover, since sensors may perceive objects from different viewpoints (i.e. cameras), the observation features should mostly be insensitive to viewpoints (i.e. geolocation). The system operator is capable to task the system in a very specific way by creating triggers (i.e. ‘alert when vehicle enters the east parking lot’). This means that high-level user requirements need to be transposed into a sequence of sensor commands. To achieve this, the authors propose a matrix of cost values, representing the costs of sensors performing tasks. Task allocation is carried out using a greedy optimization approach. Prioritizing tasks is used for conflict resolution. Finally, a simple Web-based reporting system is logging all data that are generated.

The so-called INT3-SDP process introduced by Gascuena et al. (Reference Gascuena, Castillo, Navarro and Fernandez-Caballero2014) deals with multi-sensory monitoring and interpretation of behaviors and situations for intervention in complex and dynamic environments. The process provides, on the one hand, a set of guidelines to support system analysts to capture the customer’s requirements in term of multi-sensory monitoring. On the other hand, a components repository is proposed to aid the implementation phase. These components can be reused and easily adapted to different configurations. Moreover, a proof-of-concept use case depicts the application of INT3-SDP for the task of fall detection in a smart home environment.