Development of Surveillance Robot Based on Face Recognition Using High Order Statistical Features and Evidence Theory

1. Introduction

In recent years, surveillance has evolved into a critical necessity for safeguarding both lives and assets across various sectors. The growing complexity of security threats, from theft and vandalism to terrorism and cybercrime, has driven the demand for more advanced surveillance systems [1,2]. These systems are increasingly deployed in public spaces, private properties, industrial zones, and government facilities to ensure safety and security.

The integration mobile robots into surveillance systems [3,4], offers a highly flexible and cost-effective solution to enhance safety across various sectors, including industries, the military and home automation. These robots can effectively replace humans in hazardous environments or risky manufacturing processes, minimizing the need for human intervention in dangerous conditions. Their ability to navigate autonomously and perform tasks in real time makes them a valuable asset in ensuring security and safety while reducing operational costs.

Intelligent security robots [5,6] are equipped with cameras and a variety of sensors that allow them to continuously monitor an area with minimal human oversight. They can detect intrusions or issues and alert surveillance personnel in real time with high reliability. Subsequently, surveillance mobile robots have become a significant focus of research, given their potential to enhance security and operational efficiency.

Surveillance technologies have significantly advanced with the integration of modern innovations such as artificial intelligence (AI), computer vision (CV), and the Internet of Things (IoT). These technologies enable real-time monitoring, anomaly detection, and automated responses, making surveillance systems more efficient and reliable. For instance, face recognition, behavior analysis, and motion detection are now commonplace in modern surveillance, allowing for early detection of potential threats.

Recently, the need for secure identification methods has become increasingly crucial. Traditional methods such as passwords and PIN codes are no longer enough to protect sensitive information and secure access to restricted areas [7,8,9]. This has led to the rise of biometric technology, which uses unique physical or behavioral characteristics to verify an individual’s identity. In recent years, many security structures have started implementing biometric systems based on iris, fingerprint [8], voice, and face recognition [10].

Due to the diverse structures and features present in the human face, it has emerged in recent years as one of the most widely employed biometric authentication systems in various applications and domains [7,8,9]. However, a robust face recognition system [10] is developed using three fundamental steps: face detection, feature extraction, and face matching/classification. The process of face detection begins by locating and identifying the human face. The feature extraction step is a critical aspect of the recognition process, responsible for extracting feature vectors for any human face identified in the initial stage. Moreover, successful feature extraction plays a pivotal role in determining the success of subsequent steps. Finally, the face recognition step conducts matching or classification of image features according to predefined criteria and is then compared against all template face databases to establish the identity of a human face. Figure 1 depicts the main components of a typical face recognition system.

Face recognition is a technique used to authenticate or identify individuals by automatically recognizing them based on their facial characteristics. It involves a computer program that can automatically detect or confirm one or more individuals from a digital image or a video frame extracted from a video source. In verification situations, the similarity between two facial images is evaluated, and a decision is made regarding whether they match or not. However, the process of identifying the identity of an unknown facial image from a database of recognized faces is referred to as face recognition. In this scenario, the system computes the similarity between a particular facial image and all facial images stored in a large database, returning the best match as the presumed identity of the person.

In this context, many authors have examined the issue of face recognition by employing diverse techniques [5,6], and many authors and researchers have extensively discussed the challenges and advantages of these hybrid approaches in the context of face recognition [11]. They have explored how combining different techniques can enhance performance under various conditions such as varying lighting conditions, pose variations, occlusions, and facial expressions [7,8,9].

The Internet of Things (IoT) is a recent advancement in surveillance technology designed to bolster security and provide personalized protection. This innovative approach involves integrating a wide range of physical devices—such as cameras, sensors, and alarms into a network connected to the internet. These interconnected devices can communicate with one another and with centralized systems to collect, analyze, and share data in real-time. The advantages of IoT in surveillance extend beyond basic functionalities, offering a range of benefits such as Fast Operation, Automation and Control, Easy Access to Information and Saving Money that enhance overall security and operational efficiency. Overall, the integration of IoT into surveillance systems provides a more dynamic, efficient, and cost-effective approach to security, enhancing both the functionality and economic viability of these systems. However, the integration of face recognition technology with IoT is essential to significantly enhance the effectiveness of surveillance robots and develop more efficient rapid response systems.

In this paper, a prototype of a cost-effective mobile surveillance robot is proposed. The robot’s intelligent security system is controlled by the Raspberry Pi 4 Model B, known for its advanced computing capabilities. Compared to earlier Raspberry Pi models or other microcontrollers such as the Arduino Uno/Mega, the Raspberry Pi 4 Model B delivers significantly improved processing power. This high-performance processor enhances the robot’s capabilities and overall effectiveness in surveillance tasks.

The proposed surveillance robot is outfitted with a PIR (Passive Infrared) sensor and a USB camera to detect the presence of intruders within a designated area. By integrating IoT and face recognition technology, the robot can not only monitor the environment but also identify and track individuals, enhancing its ability to provide accurate and timely security responses. This combination of sensors and intelligent technology allows for efficient surveillance, making it a powerful tool in detecting unauthorized access.

When the PIR (Passive Infrared) sensor detects the presence of a person, the system automatically activates the camera to capture live streaming video and photos. These recordings are then transmitted to the control room through IoT technology, allowing real-time monitoring and analysis of the situation. This automated process ensures that any potential security breach is promptly captured and relayed, enabling quick responses from security personnel.

Face recognition technology based on high-order statistical features and evidence theory is employed to differentiate between known individuals and potential intruders. The proposed method comprises two distinct steps aimed at enhancing recognition accuracy and robustness. Firstly, high-order statistical features are extracted from facial images to capture complex patterns and variations inherent in facial data. Subsequently, evidence theory is employed to integrate these features, providing a comprehensive and reliable basis for face recognition. Hence, this work may be seen as a straightforward additional improvement of the issues proposed by Ben Chaabane et al. [12]. By analyzing and comparing facial features with a pre-existing database, the system can quickly identify authorized personnel while detecting unfamiliar or unauthorized persons. This capability enhances security by allowing more precise monitoring and reducing false alarms, ensuring that only genuine threats trigger a response.

In addition, when an unknown individual is detected, the system immediately sends an email containing the captured image along with an alert notification to the control room via IoT. Furthermore, a web interface has been developed to receive this data and allows for remote control of the robot’s movements over a Wi-Fi connection. This interface provides operators with the ability to monitor the situation in real-time and manually adjust the robot’s position if necessary, enhancing both security and operational flexibility.

The rest of this paper is structured as follows: Section (2) details the proposed robot surveillance system. Section (3) discusses the experimental results, and Section (4) provides the conclusion.

2. The Proposed Robot Surveillance System

The proposed surveillance system is designed to carry out high-risk tasks in environments where it would be unsafe or impractical for humans to operate. By leveraging advanced technology, such as automated drones, AI-powered cameras, or remote-controlled robots, the system can monitor and respond to potential threats or hazardous situations in real-time, making it ideal for use in environments like disaster zones, conflict areas, or high-security facilities. This reduces the need for human involvement in potentially life-threatening conditions, while ensuring continuous and efficient surveillance.

In this context, we propose a prototype of a low-cost mobile surveillance robot based on the Raspberry PI 4, which can be integrated into any industrial environment. This intelligent robot detects the presence of intruders using IoT and face recognition technology, providing a reliable and automated security solution. The flowchart of the surveillance robot is shown in Figure 2.

The system is equipped with a passive infrared (PIR) sensor and a camera, used to capture real-time streaming video and images. These recordings are instantly transmitted to the control room through an IoT network for continuous monitoring.

When the PIR sensor detects the presence of a person, regardless of whether the individual is recognized or registered in the system’s database, the robot automatically activates live-streaming mode and begins capturing photos. This allows for immediate visual data collection, enabling swift response and detailed analysis of potential security breaches or intrusions.

Using the proposed face recognition algorithm, the system can accurately analyze and identify faces by comparing the captured images with the facial data stored in its database.

If the facial features match a stored profile, the system recognizes the person as authorized or familiar. If no match is found, the system classifies the individual as unknown or potentially unauthorized. This real-time comparison enables the system to quickly distinguish between familiar faces and possible intruders, improving the overall security and response time of the surveillance system. Additionally, the system can log interactions with both known and unknown individuals, providing a historical record for future reference or investigation.

If a person is detected as an intruder, the system instantly sends a notification to the control room, alerting security personnel of the potential breach. Additionally, the system can send an email with the intruder’s photo attached, providing visual evidence for further action. In cases where the detected individual is a known person, the system will include their name in the notification, ensuring clarity and a quick response. This dual alert mechanism—both real-time notification and email—helps streamline the process of identifying and addressing unauthorized access.

Additionally, a web application has been developed to allow remote control of the robot, enabling it to approach individuals for better visibility and capture high-quality pictures and videos. Through this application, operators can maneuver the robot to obtain a closer view or focus on specific areas of interest. It’s also worth noting that the robot has the flexibility to function as a stationary surveillance system if needed, offering both mobile and fixed monitoring capabilities. This versatility enhances the robot’s effectiveness in various surveillance scenarios, ensuring comprehensive coverage of the area.

2.1. Face Detection by Dlib

There are two widely used methods for face detection. The first utilizes the haar-like feature combined with an AdaBoost classifier, proposed by Viola and Jones [13]. The second method is based on the HOG (Histogram of Oriented Gradients) and SVM (Support Vector Machine) classifier, introduced by Dalal and Triggs [14].

In the literature [15], a comparison of face detection results is presented between the Viola-Jones algorithm implemented using OpenCV [16] and the HOG algorithm implemented with Dlib [17].

The results demonstrate that for frontal face detection, the performance of the Dlib algorithm significantly outperforms the OpenCV implementation of the Viola-Jones algorithm. Dlib’s use of HOG features and an SVM classifier provides greater accuracy and reliability in detecting frontal faces compared to OpenCV’s traditional haar-like features and AdaBoost classifier. This makes Dlib more effective for precise frontal face recognition tasks. For side face detection, while both Dlib and OpenCV have limited capabilities and exhibit poor overall performance, the Dlib algorithm still produces superior detection results compared to OpenCV. Despite the inherent challenges of accurately detecting side faces, Dlib’s HOG and SVM-based approach manages to outperform OpenCV’s Viola-Jones algorithm, showing better detection accuracy in these situations. Based on the comparison outlined above, the Dlib algorithm is chosen for implementation. Its superior performance, particularly in detecting frontal faces, and its ability to outperform the OpenCV Viola-Jones algorithm in both frontal and side face detection, make it a more effective solution for this application. Therefore, Dlib’s face detection algorithm is adopted for enhanced accuracy and reliability. The images in the ORL dataset [18] are tested, and an illustrative example of face detection and Isolation is presented in Figure 3.

Among the total 400 images in the dataset, two facial images were unable to be detected by the face detection algorithms, resulting in a successful detection face of 398 images. The subsequent experimental results and analyses are based exclusively on these 398 successfully detected images. This indicates a high level of accuracy and reliability in the face detection process, allowing for a robust evaluation of the performance of the employed algorithm. After face detection, face recognition algorithm is implemented in order to distinguish between company personnel and intruders.

2.2. Face Recognition with High-Order Statistical Features and Evidence Theory

Face recognition involves identifying individuals based on their facial features. It encompasses two distinct approaches: biometric identification, which utilizes physical traits for recognition, and behavioral analysis, which assesses actions to verify identity claims. Although both methods have historical use, contemporary practices predominantly favor biometric identification for facial recognition purposes.

Facial recognition is a significant challenge due to the vast volumes of available datasets and various complicating factors. These include facial expressions, lighting conditions, and the presence of accessories like glasses or scarves, which can lead to partial occlusion, further complicating the process.

The objective of a face recognition algorithm is to determine whether two faces are of the same individual or not [19]. Typically, such algorithms consist of three key stages: pre-processing, feature representation, and classifier training. During pre-processing, input images are standardized, noise is eliminated, and essential operations are carried out. The feature representation stage involves extracting relevant features from the images, while the classifier training stage entails training a model based on these extracted features.

The face recognition method proposed in this paper is conceptually different and explores new strategies. The proposed method does not rely on established methods but rather investigates the benefits of integrating various approaches.

The proposed method for face recognition is based on high-order statistical features and evidence theory [20]. In the first phase, the statistical features selection approach is applied to each image in order to select the features based on relevance and redundancy characteristics and to construct the attribute images, whereas the evidence theory is used to merge the different information sources. By integrating multiple information sources, each with its reliability, the proposed method can make more informed decisions in face identification tasks.

Significantly, the proposed face recognition method is structured into two phases. Initially, statistical features are derived from the original images to generate a new image termed as the attribute image [21]. Subsequently, during the second phase, the estimation of the mass functions algorithm [22] and the application of evidence combination and decision rules [23,24,25] are employed to obtain face recognition results. The main components of the proposed face recognition system are depicted in the diagram illustrated in Figure 4.

2.2.1. Feature Extraction

The process of feature extraction from face images is performed using the statistical method. These features capture complex patterns and relationships within the image data that may not be apparent with traditional methods. The idea is to replace the image by the feature extracted from the cooccurrence matrix. The co-occurrence matrix is a common tool in image processing and natural language processing to capture the spatial or relational information between elements in an image or text. From this matrix, various statistical features can be extracted to characterize the relationships between elements. Some common statistical features include:

Mean: the average value of all the elements in the matrix. It provides a measure of the central tendency of co-occurrence relationships.

$$Mean={\displaystyle \frac{1}{N\times N}}{\sum }_{i=1}^N{\sum }_{j=1}^{N}Cooc(i,j)$$

(1)

Variance: Measures the dispersion of values around the mean. Higher variance indicates greater variability in the relationships between elements.

$$Var={\displaystyle \frac{1}{N\times N}}{\sum }_{i=1}^N{\sum }_{j=1}^N{(Cooc\left(i,j\right)-\mathrm{M}\mathrm{e}\mathrm{a}\mathrm{n})}^2$$

(2)

Where $(\mathit{N}\times \mathit{N})$ and $\mathit{C}\mathit{o}\mathit{o}\mathit{c}(\mathit{i},\mathit{j})$ are respectively the size of co-occurrence matrix and the value at the (i,)th position in the co-occurrence matrix.

Standard Deviation: The square root of the variance. It quantifies the amount of variation or dispersion present in the co-occurrence relationships.

Entropy: Represents the amount of information or uncertainty in the co-occurrence matrix. Higher entropy indicates greater randomness or disorder in the relationships between elements.

$$Entrop={\displaystyle \frac{1}{N\times N}}{\sum }_{i=1}^N{\sum }_{j=1}^{N}Cooc\left(i,j\right).\mathrm{l}\mathrm{o}\mathrm{g}(Cooc\left(i,j\right))$$

(3)

Log denotes the natural logarithm.

Energy: Also known as uniformity or angular second moment, it measures the sum of squared elements in the co-occurrence matrix. High energy indicates a high degree of homogeneity in the relationships.

$$Energ={\displaystyle \frac{1}{N\times N}}{\sum }_{i=1}^N{\sum }_{j=1}^N{\left(Cooc\left(i,j\right)\right)}^2$$

(4)

Contrast: Measures the intensity contrast between neighboring elements in the co-occurrence matrix. High contrast values indicate large intensity differences, while low contrast values indicate similar intensities.

Correlation: Measures the linear dependency between pairs of elements in the co-occurrence matrix. It indicates how much one element is related to another. Positive values denote positively correlated elements, negative values denote negatively correlated elements, and zero indicates no correlation.

Homogeneity: Reflects the closeness of the distribution of elements in the co-occurrence matrix to the diagonal. Higher homogeneity values indicate that the elements are concentrated along the diagonal, implying a high degree of uniformity in the relationships.

These are just a few examples of statistical features that can be derived from co-occurrence matrices. The choice of features depends on the specific application and the desired characterization of the relationships between elements. In this work, the features selection is based on their relevance and redundancy characteristics. Features that are relevant for distinguishing between different faces are identified and retained, while redundant or less informative features are discarded. This helps in reducing the dimensionality of the feature space and focusing on the most discriminative features. These statistical features serve as the basis for further analysis and processing in the face recognition task.

2.2.2. Use the Evidence Theory for Classification

The purpose of classification is to classify the images into different classes. The idea of using DS evidence theory for image classification is to integrate the information from multiple sources while considering the uncertainty associated with each piece of evidence. Initially, features are extracted from the input images using statistical methods. Each feature highlights specific characteristics of the original image. The DS evidence theory is then used to fuse the features extracted from the cooccurrence matrix. In DS theory, each piece of evidence (feature) is represented by a belief function, which assigns degrees of belief to different hypotheses (classes, i.e., faces). Finally, based on the combined belief function, a classification decision is made. This decision could involve assigning the image to one or more predefined classes, depending on the evidence provided by the features extracted from the cooccurrence matrix.

Dempster-Shafer Theory (DS) [26,27], also known as the theory of belief functions, provides a framework for reasoning with uncertain or incomplete information. Unlike traditional probability theory, DS theory allows for the representation of uncertainty more flexibly by assigning degrees of belief to sets rather than just single events.

In the present study, the frame of discernment $\Omega$ composed of $n$ single mutually exclusive subsets $H_n$ contains the the clusters or faces ($C_i$), which are symbolized by:

$$\mathsf{\Omega }=\left\{\left.H_1,H_2,\dots ,H_n\right\}=\right.\left\{\left.C_i\right\}\right.$$

(5)

where: $1\le i\le n$.

In order to express a degree of confidence for each proposition A of $2^{\mathsf{\Omega }}$, it is possible to associate an elementary mass function $m\left(A\right)$ which indicates the degree of confidence that one can give to this proposition. Formally, this description of $m$ can be represented with the following three equations:

$$\left\{\begin{array}{c}m:2^{\mathsf{\Omega }}\to \left[\mathrm{0,1}\right]\\ m\left(\mathrm{\varnothing }\right)=0\\ {\sum }_{A\subseteq \Omega }m\left(A\right)=1\end{array}\right.$$

(6)

can be represented with the following three equations:

The purpose of segmentation is to divide the image into uniform regions. The idea of using the evidence theory for image segmentation is to fuse one by one the features coming from the cooccurrence matrix of each image. The Gaussian distribution model [28] is applied to the statistical features to be combined. Then, the mass functions are combined using the Dempster-Shafer (DS) combination theory to obtain the final segmentation results.

Masses of simple hypotheses $C_i$ are derived based on the assumption that the attributes $x$ for the class $i$ follow a Gaussian distribution. For each hypothesis $C_i$, the probability density function (PDF) of the Gaussian distribution is used to describe the probability that a data point belongs to the class $i$. The formula for the mass function based on the Gaussian distribution for the class $i$ is as follows:

$$m\left(C_i\right)={\displaystyle \frac{1}{\sqrt{2\pi {\sigma }_i^2}}}exp(-{\displaystyle \frac{{(x-{\mu }_i)}^2}{2{\sigma }_i^2}})$$

(7)

Where:

$m\left(C_i\right)$ is the mass of hypothesis $C_i$, representing the likelihood that the attribute belongs to class i.

$x$ is the attribute value being evaluated.

${\mu }_i$ is the mean of the Gaussian distribution for class $i$.

${\sigma }_i^2$ is the variance of the Gaussian distribution for class $i$.

The mass function assigned to double hypotheses (i.e., when there is uncertainty between two possible clusters) depends on the mass functions of each individual hypothesis. In the context of evidence theory, such as Dempster-Shafer theory, the mass function for a double hypothesis is derived by combining the masses of the individual hypotheses while accounting for the uncertainty between them.

Let’s denote two simple hypotheses as $C_i$ and $C_j$. The mass function for the double hypothesis $C_i\cup C_j$, which represents the uncertainty that the attribute could belong to either class $i$ or class $j$, can be expressed as:

$$m\left(C_i\cup C_j\right)={\displaystyle \frac{1}{\sqrt{2\pi {\sigma }_{ij}^2}}}exp(-{\displaystyle \frac{{(x-{\mu }_{ij})}^2}{2{\sigma }_{ij}^2}})$$

(8)

Where:

$$\left\{\begin{array}{c}{\mu }_{ij}=({\mu }_i+{\mu }_j)∕2\\ {\sigma }_{ij}=max({\sigma }_i+{\sigma }_j)\end{array}\right.$$

(9)

Once the mass functions of the four attributes are estimated, their combination can be performed using Dempster’s rule of combination (also known as the orthogonal sum) from Dempster-Shafer theory. This rule is used to combine evidence from multiple sources to estimate the final belief or mass function. The orthogonal sum for combining two mass functions $m_1$ and $m_2$ is represented as follows:

$${\mathrm{m}}_{12}\left({\mathrm{C}}_{\mathrm{i}}\right)={\displaystyle \frac{1}{1-\mathrm{K}}}{\sum }_{{\mathrm{A}}_1\cap {\mathrm{A}}_2={\mathrm{C}}_{\mathrm{i}}}{\mathrm{m}}_1\left({\mathrm{A}}_1\right).{\mathrm{m}}_2\left({\mathrm{A}}_2\right)$$

(10)

Where:

${\mathrm{m}}_{12}\left({\mathrm{C}}_{\mathrm{i}}\right)$ is the combined mass function for hypothesis ${\mathrm{C}}_{\mathrm{i}}$.

${\mathrm{A}}_1$ and ${\mathrm{A}}_2$ are subsets of the frame of discernment (the possible hypotheses) such that their intersection is ${\mathrm{C}}_{\mathrm{i}}$.

${\mathrm{m}}_1\left({\mathrm{A}}_1\right)$is the mass function for hypothesis ${\mathrm{A}}_1$ from the first attribute, and ${\mathrm{m}}_2\left({\mathrm{A}}_2\right)$ is the mass function for hypothesis ${\mathrm{A}}_2$ from the second attribute.

$\mathrm{K}$ is the conflict term, which quantifies the conflict between the two mass functions and is given by:

$$K={\sum }_{A_1\cap A_2=\mathrm{\varnothing }}{m}_1\left(A_1\right).m_2\left(A_2\right)$$

(11)

$K$ represents the basic probability mass associated with conflict. This is determined by summing the products of mass functions of all sets where the intersection is an empty set.

In the DS combination theory [29], the mass functions of the four attributes are fused together to generate a single value. After calculating the orthogonal sum of the mass functions for the four attributes, the decisional procedure for classification purpose consists in choosing one of the most likely hypotheses $C_i$. This is typically done by evaluating the final mass functions and identifying the hypothesis with the highest mass (i.e., the most supported hypothesis based on the combined evidence).