RICERCA ITALIANA     

FREE SURF: FREE SUrveillance in a pRivacy-respectFul way

Università degli Studi di Modena e Reggio Emilia

Abstract

Free surf is meant to be a paradigm for the new generation of video surveillance systems, free from controls by human operators, and completely respectful of the privacy. The technological support will be given by emerging solutions of computer engineering both in system architectures for real-time video processing and in innovative techniques of Computer Vision and Pattern Recognition. The majority of the commercial systems only focus on video acquisition and on their visualization in control rooms. The most innovative systems already exploit simple computer vision techniques for motion detection, but with many structural and technological constraints, such as the installation of fixed cameras only, with manual calibration, very simple target models, and, in particular, the lack of inferential capabilities and scene understanding that makes necessary the constant presence of human operators. The “free surveillance” systems aim at overcome these technological constraints, by creating new automatic systems also socially acceptable, since they will be perfectly coherent with the current laws on privacy.

This project aims at developing innovative solutions for detecting people in an automatic way by processing videos in real-time. Original and robust techniques will be applied to installations free from structural constraints, and, in particular, to multicamera distributed systems (with fixed, PTZ and mobile cameras) coordinated with sensors networks. The visual data about people will be used to recognize situations modelled with statistical and machine learning techniques in order to generate alarms in the case of dangerous situations. Information retrieval techniques based on visual content will be employed to detect occurrences of the same people or similar situations in videos acquired in different time or different cameras. This will provide effective tools for the support to investigations and posterity analysis. Face detection techniques will be used to extract and remove biometric data from the videos in real time. By doing this, these systems could be also used to transmit on the web information on the environment without transmitting data protected by privacy laws. They could be installed in public places in a socially-acceptable way, without the “Big Brother” syndrome.

These extermely-innovative proposals will not lead only on theoretical research activity, but also to the concrete development of working prototypes in the two years of the project. This is guaranteed by the composition of the group: the three RUs are involved in computer vision and pattern recognition researches since many years, and they are leading groups in the national and international context, and have active collaborations with the most important international research centres. The project has a strong national relevance because it will allow to develop also in Italy, in a coordinated and synergic way, research on computer vision, research that must be done with a sufficient number of people involved to maintain international excellence. Additionally, the created consortium also includes extra-university groups that work on this field, such as the ITC-iRST of Trento, that, even if not funded, are interested in collaborating to the project and, even more important, the project has the support and the interest of many companies and public entities that want to exploit directly the results of such activity. The project has strong implications of innovation and social impact, and can provide an immediate technological transfer to the companies that already collaborate with the RUs of the project or that have awaked their interest for new solutions to automatic video surveillance. <<<

Principal Investigator

Rita Cucchiara Università degli Studi di MODENA e REGGIO EMILIA

Research Objectives

The FREE SURF project aims at proposing new technologies for the next generations of video surveillance systems oriented to the automatic real-time control of the presence and actions undertaken by people in the environment, without the direct control of a human operator.
The FREE SURF project is born with a twofold aim: first, innovative scientific research in the field of Computer Vision and Pattern Recognition, second, innovative applied research for the development of new generations of video surveillance systems, both effective and socially acceptable with respect to privacy concerns.

The first objective is to conduct a thoughtful research activity in the field of Computer Engineering for video surveillance of people in “structural constraint FREE” systems, that is in systems free from structural and environmental constraints.
The automatic visual control of human presence and actions in a given environment is, indeed, one of the most studied problems in the last decade. Nowadays, a very large literature exists, which presents algorithms and robust implementations for the recognition of single persons, in structured environments: closed environments with controlled illumination, open environments with large field of view (in order to consider people as small rigid moving objects), with few people, with only partially occluded fields of view, controlled by fixed cameras (to segment objects as different from the background), and installed with a precise manual calibration (for an exact 3D reconstruction).

The final objective of the project is to study innovative methodologies and techniques for going further on: the final targets are environments free from structural constraints, in scenes with more people that live together and interact each other, as in parks or tourist areas. The foreseen activities are devoted to the study of new ways to extract visual data, from distributed camera systems, from hybrid systems with active cameras, capable to automatically move toward a target, from moving cameras, and coordinated with networks of sensors. New algorithms will be studied and working prototypes developed for people segmentation and tracking in videos acquired by multiple auto-calibrated cameras, by exploiting geometrical information and appearance (color and texture). Approaches for active camera control and mosaicing of the scene from moving cameras will be studied. Moreover, mobile agents systems will be studied to coordinate cameras and sensor networks in large scenes like archaeological sites. These techniques will all implemented in separated modules by each RU, but they will be coordinated in a single architecture to provide a common interface for the reasoning modules (Figure 1).


Figure 1.

All the previous modules have the common objective to extract visual data on the people in the scene. In particular, trajectory computation with invariants independent of the point of view, people posture analysis and soft biometries are the main data that will be extracted. Differently from projects dealing with biometric analysis, the FREE SURF project is oriented to the automatic visual analysis of the presence and behavior of people independently of their identities, which are not easy to assess in noisy, low-resolution videos with large filed of view, like those typical of distributed video surveillance systems. As a further support, hybrid system with PTZ and mobile cameras can provide, if needed, information with more details, which can be used in “posterity logging” by the experts.
The visual data are provided to modules for dual activities: to monitor dangerous situations in real time, and to annotate interesting situations for future off-line queries. The first is a strategic tool to help the human operator in the prevention and fast responsiveness to facts regarding security, the second provides a valid support to investigations and a-posteriori analysis. These solutions may enable the many existing surveillance systems to provide effective support in a predictive way.

The second objective of the project is to define new solutions to answer to the social requirements of safety and security both for places and people with solutions made “in a privacy-respectful way”. In Italy, as well as in the whole world, cameras are spreading everywhere. This is not a social problem by itself if the cameras were only intelligent sensors, such as fire sensors, capable to simply process videos and generate alarms. The social awkwardness comes from the fact that behind a camera there is a human operator that controls and watches, prejudicing the individual’s privacy. While it is acceptable to use cameras by the public officers for security reasons in public places, the diffusion of private closed-circuit TV systems in department stores, shopping centers or even the diffusion of web-accessible webcams in public tourist places are not. The current laws on privacy are (rightly) very restrictive, but currently there is no technological mean to guarantee that such laws are respected: many installed systems, without video processing, are either not compliant with these laws or simply unusable and used only as a deterrent for criminality since they declare to have a such low resolution or a such deep field of view to prevent people identification.
The aim of this project is, instead, to provide automatic tools for extracting visual information on people that can be, on the one side, used by authorized systems to automatically monitor the scene, and, on the other side, used to guarantee the removal of biometric data and, thus, the compatibility with privacy laws. In particular, robust algorithms for people tracking and face and head detection will be useful to obscure biometric data in real time.

The reported objectives of the project are for sure ambitious, but they are also feasible in the timeframe of the project. Besides the limited funding requests to MIUR, the RUs have made available many man-months of people with already a great experience on the field and can benefit of the resources, both hardware and software, in their laboratories. The consortium is composed of three very active RUs at national level, in particular in the GIRPR (www.girpr.it): all the three RUs have organized both the International Conf. of GIRPR (ICIAP), and the school for PhD students (VISMAC) that has been/will be held in Firenze, Palermo and Modena, in year 1999, 2001 and 2007 (ICIAP) and 1992, 2006 and 2000 (VISMAC). They are, moreover, very active at international level, with frequent PhD students and researchers with research centers in USA and other countries (such as the University of Technology, Sydney, Australia), and they have many active projects in the field.

A very important aspect of the project is that it already contributes to the creation of a network of expressions of interest and collaborations. In particular, ITC-iRST of Trento, one of the most important non-university research centers in video surveillance, will participate to the project; big national companies such as SIRTI and ALCATEL have indicated their interest (see annexes 1 and 2). Public entities such as Regione Emilia-Romagna (annex 3) and Parco Valle dei Templi di Agrigento (annex 4) have declared their interest in the project, as final users.


All. 1


All. 2


All. 3


All. 4 <<<

Timescale

24 months

National and international background

Real time video processing and automatic people detection, localization and tracking are intrinsically complex problems, with a lot of international challenges. Videosurveillance of people and human action control is one of the more discussed arguments in Computer Vision, Pattern Recognition and Multimedia. The first important special issue on this topic was published on IEEE Transaction on PAMI in 2000, in which the first video surveillance systems were presented; for example, it collected proposal like the background suppression algorithm of Stauffer and Grimson[39], that is a reference work in this field, the W4 system from Maryland [42], the PFinder system of Pentland [43].
The current interest on this field is proved by the several conferences presenting papers on this topic such as (CVPR, ICCV, ECCV, ICPR) or devoted to video surveillance like IEEE Int. Conf. on Advanced Video Surv. Systems, IEEE Workshop. on Video Surveillance, and ACM Workshop on Video Surveillance and Sensor Networks.
Intrinsically difficulties of people video surveillance are due to several factors such as shape changing of the human body, its non rigid motion, variable posture and gait, presence of occlusions and interaction between people simultaneously present in the scene, presence of infrastructures in the indoor and outdoor environment (e.g., doors and furniture in indoor scenes, vehicles, trees and urban furniture in outdoor scenes), illumination changes and shadows [51]. In such a situation, well-established and already commercialized research solutions, such as the ones for vehicle surveillance, are not always successfully applicable. From the research point of view, some surveillance systems have been proposed, but only with a lot of constraints such as single and fixed cameras [4], one person at a time, and so on. Only in the two last years more sophisticated and unconstrained multicamera system have been proposed.
The FREE SURF project aims at effectively study innovative solutions on systems free from structural constraints, with distributed moving cameras, coordinated with sensor networks. Visual data extracted from these multimodal systems are then exploited in three different king of applications: to blind biometrical data in real time, to monitor interesting events concerning the safety of the monitored environment, and to recover information later.
A scientific starting point about these topics (corresponding to the project modules depicted on Fig. 1) is now described.

1) Extraction of People surveillance information from fixed cameras, moving cameras, mixed systems of moving and fixed cameras, systems coordinated with sensor networks.

Visual sensors like cameras enable extraction of information more than any other kind of sensor, but automatic and robust video processing techniques working in any condition and in unstructured environments are not yes available. In particular, problems are related to the visual data extraction from the people in the scene.
Among the other problems, this project will focus on three main arguments: assign the same identity to different views (belonging to different cameras) of the same object,. or, in other words, detect that the same person is moving from a camera field of view to another one; segmentation and tracking of people from a single but moving camera, both partially constrained (PTZ, i.e. , Pan-Tilt-Zoom) or completely free; coordination of mixed systems composed by fixed and moving cameras.
A term used to describe the first problem is “consistent labeling”. The approaches proposed to this aim can be classified into three classes: geometrical based methods, both without calibration like in the work of Lanz [1], that is not so robust in presence of occlusions, and with an automatic weak calibration by means of a learning phase as in the works of Khan and Shah [2] and Calderara et al. [3]; appearance based approaches, based for example on the object colour coded with histograms (Krumm et al. [5]), or principal spectral components (Piccardi and Cheng [6]); mixed approaches that combine visual and geometrical information (Kang et al. [7], Dockstader and Tekalp [8]). In this project mixed solutions based on works like [1] and [3] for partially overlapped or disjoint cameras will be proposed.
Differently from the previous case where the complexity of the problem was due to the tracking phase more than the segmentation one, when the video is acquired from a moving camera the object segmentation becomes an extremely critical problem. Previous works on this topic can be divided into two main classes: methods based on an initial egomotion estimation followed by an affine motion compensation (Kang et al [7]) or a mosaic background suppression for PTZ cameras (Prati et al [9]) and methods based on motion vectors clustering (e.g., Lee et al [10]).
There are several facets tackled by current research activities in the case of hybrid systems; the most important ones are: active people following, calibration of PTZ cameras, path optimization for PTZ cameras. Control the motors of the camera keeping an object or a person in the centre of the scene is the first aspect (Barreto et al. [11], Reid and Murray [12], Tordoff and Murray [13]).
Typically, this problem is solved making use of a controller driven by visual information (e.g. optical flow estimation) and based on optical and mechanical knowledge about the camera.
In the case of geometrical calibration of PTZ cameras, focal length and optical centre changes are the main problem since intrinsic and extrinsic parameters are no more fixed. In the past several methods have been proposed to extract the intrinsic parameter (de Agapito et al. [14], Sinha and Pollefeys [15], Wren et al. [16]). Eventually, in a multicamera system, we are interested on determining which camera can faster intercept a particular object or which is the best path to follow when more than one object is moving in the scene. This problem was initially studied from Costello et al. [17] and then improved by Del Bimbo and Pernici [18].
The video surveillance of wide outdoor environments such as archaeological sites [45] or stations and parks [46] using camera only is too expensive. Coordinated solutions with wireless sensor networks (WSN) [44], instead, could be a cheaper and more interesting solution. A sensor network can be considered like a distributed database in which sensors collect, share, and present data to the final user. The most relevant difference between WSN and traditional networks is in term of energy requirements. Since this kind of sensors have to work in hostile conditions without any human action after the deployment, the maximization of the life-time of the network (i.e., the minimization of energy loss) keeping high the reliability and efficiency of the network is a typical project goal [47].
This project proposes two different platforms of sensor network; the first, oriented to low cost applications and with constraints about the resources used, is made of motes; the second, composed by micro-server nodes, useful to reach better performance o to coordinate motes.
Micro-servers are generally more efficient if used to process data, but they are more expensive when processors and radios are in standby. In particular, they can work strictly joint with cameras, actions of which can be sensor guided as described later. As regards data forwarding through nodes, the work proposed by Ortolani et al. [19] describes a new strategy to collect data over sensor networks under energy saving requirements and with a compromise between reliability and efficiency.

2) Automatic selection and blinding of biometrical data in respect to privacy issues
In Italy there are no strong collaborations between technological departments and governmental institutions about privacy issues like in UK. Even if the law is well defined (e.g., “Decalogo della video sorveglianza - DL. 29/04/2004, stadium regulation- S.M. 6 /07/2005)), there are no technical documentation to have an idea when a particular visual information can damage the privacy of a citizen. An example of this king of documentation was provided by the UK police [20]. In this project we’ll give standard tips about the relation between image resolution and identification, and we’ll define privacy compliant surveillance systems exploiting already established collaborations with public safety institutions.
Results obtained in several sociological studies about the surveillance technology impact (as reported by Langheinrich in [21] and Patton in [22]) highlight that the capability to be compliant of privacy issues should be considered as a part of the efficacy of a system. Blind the people faces by means of automatic computer vision processes is a natural way to reach this goal (as proposed by Senior et al. in [23] or by Cucchiara et al. [24]). Several algorithms have been proposed to detect and track faces (Viola and Jones [25], Yang et al. [26], Hjelm and Low [27]). Some of these can be applied in real time directly tracking the faces detected, like in DeCarlo and Metaxas [28], in Cucchiara et al. [29], and in Cucchiara [49], even in presence of illumination changes [50].

3) Real time detection and monitoring of interesting events

Extracted visual information allow an higher level analysis to detect interesting events and situations. Different approaches can be found in the literature about people behaviour classification. In Wu et al. [30] abnormal events are detected by means of a Support Vector Machine (SVM). Cucchiara et al. [31], instead, proposed a method based on Hidden Markov Model to analyse the people posture and to detect falls in monitored environments. Other examples of behaviour modelling for single camera applications are presented by Cupillard et al. in [32], Park and Aggarwal in [33], Bertozzi et al. [34]. The analysis and detection of interesting events in multicamera systems is still a rarely explored research field (Cupillard et al. [35]). Based on machine learning techniques such as kernel sequence-matching PHMM connected with Support Vector Machines (SVM), the work proposed in [36] is an innovative approach.

4) posterity logging and information retrieval

Finally, the last module will be devoted to study techniques to automatically annotate events on stored information, in order to allow query over temporal wide databases (Zhang [37]). In the past this kind of techniques have been used in fields like textual information retrieval and their adaptation to visual data is currently studied. Nowadays it can be considered stable and reliable. The book on Multimedia written by Del Bimbo [38] is one of the most important works on this topic. Nevertheless, only few works address retrieval applications related to video surveillance, in particular on multi camera systems where there is a large amount of data and it’s difficult to choose which feature adopts as search key. A proposed solution is based on the analysis of the people trajectories by means of unsupervised learning techniques (Stauffer and Grimson [39], Buzan et al. [40]). Unfortunately, in complex situations such as metropolitan stations these approaches cannot be used; thus, techniques based on dynamic texture analysis (Doretto et al. [41]) have been explored. In this project we’ll make use of these algorithms integrating them with soft biometry information [48] and with trajectory invariants [52]. <<<