Multiple Camera Patient Tracking Method Based on Motion-Group Parameters Reconstruction

1.1. Motivation

Epilepsy is a debilitating neurological condition which is often accompanied by involuntary rhythmic movements of body parts called convulsive epileptic seizures. In certain cases, these seizures can be life-threatening and lead to a fatal outcome, also known as Sudden Unexplained Death in Epilepsy or SUDEP [1]. Special care is required for individuals with untreatable epileptic conditions in order to prevent health complications. This includes prolonged observation in dedicated medical facilities.

Various methods that detect epileptic seizures have been developed and can be found in the literature [2,3,4]. Our work is focused on automated detecting epileptic seizures remotely by using the data from video cameras present in specialized clinics or the patients’ homes. This type of remote detection is fast, non-obstructive, privacy-respecting, easy to set up, accurate and reliable [5,6]. In previous work, we demonstrated a method for remote optical detection of motor seizures using a simple USB camera and an ordinary personal computer [7]. Subsequently, the detection technique is upgraded by adding patient-tracking capabilities to the system [8]. This is done by introducing a dynamic region of interest that tracks the specific patient and helps to reduce background interferences. The tracking method is based on optic flow [9,10,11,12,13,14,15] analysis from the same video feed and uses global motion group parameters for movement estimation. Tracking is accomplished by drawing an initial rectangular region of interest (RoI) around the patient to monitor and update the position and size of the RoI in subsequent frames using the global motion group parameters. Here, we further expand and improve the tracking algorithm by including additional camera sensors. This is done as patients under monitoring move from one room to the other, and coverage must be provided for all living spaces. The new method is based on the same global motion group parameters reconstruction used in the single-camera technique [16], and in the present implementation, it offers some substantial improvements. For one, it allows for a larger detection coverage as using multiple cameras means more overall area under constant monitoring. In areas without camera field of view overlap, our method can transfer the tracking task between cameras in a simple way based on tree traversal. When and where the cameras’ fields of view overlap and capture the same location from two or more different positions, the accuracy and robustness of the tracking algorithm can be increased. We have developed a novel way to fuse the regions of interest of all the cameras monitoring the same area, producing a more accurate RoI.

1.2. Related Work

Multi-camera tracking is an ever-growing topic in the field of computer vision. As computing power and camera quality increase, more and more tracking algorithms for various applications are developed. Multi-camera tracking systems (or networks) are often characterized by increased accuracy, robustness and coverage when compared with single-camera trackers. Developers have established various applications ranging from traffic and pedestrian monitoring, airport security, indoor people counting and tracking, sports analytics and display, augmented reality, and others. Numerous methods and approaches dealing with the multi-camera task exist in the literature. We give a brief overview of original solutions to problems and tasks similar to the one present in this work – mostly indoor people trackers that utilize multiple cameras. In [17], the authors present an indoor person detector and re-identification framework and test it on a pre-labelled dataset consisting of thirteen videos captured by different cameras. In [18] the authors demonstrate a real-time system for tracking people in an airport that uses three cameras placed on top of security checkpoints. They use mixtures of Gaussians to determine foreground pixels, detect pedestrians using a boosted decision tree classifier, and use a KLT optic flow algorithm for tracking. In [19], a study deploys multiple cameras without overlapping the field of view. The aim is to capture people’s indoor movement. Appearance features are extracted from personal images and used by a deep convolutional neural network for similarity comparisons and tracking. Gilbert et al. [20] use four non-calibrated cameras for indoor tracking of people. They rely on the three cues of colour, relative size and movement between cameras to provide a description of objects and their behaviour in the different fields of view. The system can run continuously, updating and building the inter-camera relationships and improving detection and tracking accuracy. In [21] objects are tracked using spatial-temporal and appearance cues. The optimal alignment function between appearance histograms of the tracked objects is estimated using inter-camera colour calibration. Stochastic transition matrices have also been used to describe the motion of moving people [22]. The reader can reference some of the larger survey papers for a more detailed overview of other multi-camera tracking methods and techniques [23,24,25].

An important part of the present work is determining the optimal RoI for each field of view when cameras record the same person (or event) from different positions (overlap). Such a task is also known as sensor fusion [26,27]. This is the process of combining data from multiple sensors. The sensors need not always be of the same type – self-driving cars, for example, often fuse data from regular cameras and LIDARs [28,29,30]. The purpose of integrating different types of data is to increase accuracy and performance on complex systems. As the sensors often provide redundant data, that information can be used to improve reliability and reduce the impact of incorrect measurements of faulty sensors. Occlusions and ambiguities resulting from single-sensor measurements can also be mitigated and potentially resolved by using multiple sensors. Fusion can take place at different levels:

• Raw data can be combined directly, most often in real-time systems.
• Features and properties of events and objects can be fused after raw data collection and processing.
• High-level decision-making for complex systems after substantial calculation and data manipulation.

In the current paper, fusion will be performed at a medium-high level: the video feeds from two or more cameras will be used to determine RoIs of the tracked object. After that, based on a linear model for inter-camera motion, a new region of interest is going to be determined.

1.3. Motivation

The rest of the paper is organized as follows. The next section introduces the novel multi-camera tracking method. Next, we present results and comparisons to the single-camera tracking technique to evaluate the new method’s effectiveness. We also introduce a new way to define the region of interest, enabling us to track objects moving faster. In the Discussions section, we present additional details, plans for future work, limitations, and competitive features of our technique. Finally, in the Conclusions, a summary of the results and their interpretation is given.

2. Materials and Methods

2.1. Multiple Camera Tracking with no Overlap

The method presented in this work extends the one described in [8]. The rectangular region of interest that surrounds the patient is defined by the coordinates of its center point $\left(X_c,Y_c\right)$ and the lengths of the sides $\left(L_x,L_y\right)$. It is presented on Figure 1:

We introduce the following three-element vector $RO{I}_i$:

$$RO{I}_i(k)=\left[\begin{array}{c}{X}_c^i\left(k\right)\\ Y_c^i\left(k\right)\\ \sqrt{L_x^i{\left(k\right)}^2+L_y^i{\left(k\right)}^2}\end{array}\right],$$

(1)

where $k$ is the current frame and $i$ is an index that shows which camera is being used. This vector completely describes the region of interest. Each camera has its own RoI. In the current work, we use one camera per room to provide the required coverage needed to track a person going around an indoor location. This is a strict requirement related to the topology, how rooms are laid out and how they connect, and subsequent camera positioning. Our method will function properly if there are no closed loops in the configuration, implying, for example, that there can be only one connecting path between any two rooms. Such a topology can be described as a n non-oriented tree graph and is illustrated in its simplest two-level setting in Figure 2:

The technique tracks a patient (updates the RoI location and size) around a room until the person exits the area. When that happens (usually at the door frame), the tracking method no longer registers any translational or dilatational movements, and the RoI stops moving at the place of exit. The RoI stays in the same location until it registers new movements due to the patient’s reentry. In this manner there is always at least one RoI which is moving and tracks the patient. An initialisation step consists of manually placing the RoIs around the patient and at the places of camera transitions (most often doors or hallways). When evaluating the method’s performance, the focus is placed on which region of interest tracks the person. A RoI transition occurs after the person leaves one room and enters another. One region of interest remains motionless, and another begins tracking and moving. These transitions are the events that need evaluation in order to describe accuracy and tracking performance, as all other tracking details remain the same as described in [8].

2.2. Fragmented Regions of Interest

In our previous work [8], we showed that our method cannot accurately track objects moving faster than a certain speed. Here, we address this issue by proposing a new way to define the RoI and update its position and size with respect to the calculated motion-group parameters.

Originally the center coordinates ($X_C^i,Y_C^i)$ and the size $(L_x^i,L_Y^i)$ of the RoIs between frames $i$ and $i+1$are updated as shown in Eq. (2):

$$\begin{gathered} X_C^{i+1}=X_C^i+T_x^i;Y_C^{i+1}=Y_C^i+T_y^i; \\ {L}_x^{i+1}=L_x^i\ast \left(1+D^i\right);L_y^{i+1}=L_y^i\ast (1+D^i) \end{gathered}$$

(2)

Here $T_x^i,T_y^{i}and{D}^i$ are the values provided by the GLORIA [16] algorithm. $T_x^i,T_y^i$ show the translations in x and y direction, while $D^i$ is the dilatation (a scale parameter) showing a relative change in size. We suggest that the original RoI at frame $i$ gets divided in $n$ smaller rectangles with the same size (they may overlap, depending on the current size of the RoI):

Figure 3. Fragmented region of interest, containing n smaller RoIs with their centers and sizes.

Figure 3. Fragmented region of interest, containing n smaller RoIs with their centers and sizes.

Our new approach is to find the motion group parameters $\widetilde{{\mathit{T}}_{\mathit{x}}}=\left\{T_x^1,\dots {,T}_x^n\right\},\widetilde{{\mathit{T}}_{\mathit{y}}}=\left\{T_y^1,\dots {,T}_y^n\right\},\tilde{\mathit{D}}=\left\{D^1,\dots {,D}^n\right\}$ for each smaller RoI, and after that recalculate the motion-group parameters for the total RoI by taking the maxima over the fragmented RoIs:

$$\begin{gathered} D=\max \left(\tilde{\mathit{D}}\right) \\ {T}_x=\max \left(\widetilde{{\mathit{T}}_{\mathit{x}}}\right) \\ {T}_y=\max \left(\widetilde{{\mathit{T}}_{\mathit{y}}}\right) \end{gathered}$$

(3)

We have prepared comprehensive examples to show how this approach helps us to track faster moving objects.

2.3. Multiple Camera Tracking with Overlap

In certain applications multiple cameras track the patient at the same time. An example of this would be the scenario where the patient is at the door frame and two cameras are placed so that two RoIs track the patient. When such a situation arises, we can use the information from all sensors to calculate a more accurate position for the RoIs. The optic flow signal that all cameras register is caused by the same moving object or person and therefore the resulting RoIs will be related. First, we propose a linear model to describe this relationship:

$$\begin{gathered} RO{I}_{ij}\left(k\right)=W_{ij}\ast RO{I}_j\left(k\right)+R_i \\ i,j=\left\{1,\dots ,N\right\};i\ne j;N=number of cameras \end{gathered}$$

(4)

Without any loss of generality and for the sake of simplicity, for the rest of the paper, we will limit our analysis for N = 2. Equation (4) then can be expressed as:

$$\begin{gathered} RO{I}_1\left(k\right)=W_{12}\ast RO{I}_2\left(k\right)+R_1 \\ RO{I}_2\left(k\right)=W_{21}\ast RO{I}_1\left(k\right)+R_2 \end{gathered}$$

(5)

Here $W_{12}$ and $W_{21}$ are $3x3$ transitional matrices and $R_{\mathrm{1,2}}$ are $3x1$ offset vectors that describe how the RoI vectors $RO{I}_{\mathrm{1,2}}$ are linked. In Equations (4-5) and all subsequent equations where the explicit indexing is omitted, the symbol $\ast$ denotes inner matrix multiplication. We assume a dynamic relation that can be updated at any time instance. In order to find the parameters of the linear model, we form the following quadratic cost function:

$$\begin{gathered} C_{W,R}={\sum }_{k=t-n}^{t-1}{\left(RO{I}_1\left(k\right)-W_{12}\ast RO{I}_2\left(k\right)-R_1\right)}^2 \\ +{\sum }_{k=t-n}^{t-1}{\left(RO{I}_2\left(k\right)-W_{21}\ast RO{I}_1\left(k\right)-R_2\right)}^2 \end{gathered}$$

(6)

where $t$ is the current time point and summation is done over the previous $n$ frames. We then minimize the cost function by finding its partial derivatives ${\displaystyle \frac{\partial C_{W,R}}{\partial W}}$ and ${\displaystyle \frac{\partial C_{W,R}}{\partial R}}$ and setting them to zero. That yields the following solutions for the transitional matrices and the offset vectors:

$$\begin{gathered} W_{12}^{ij}={\sum }_{k=t-n}^{t-1}\left(RO{I}_1^i\left(k\right)-R_1^i\right)RO{I}_2^j\left(k\right)/{\sum }_{k=t-n}^{t-1}RO{I}_2^i\left(k\right)RO{I}_2^j\left(k\right) \\ {W}_{21}^{ij}={\sum }_{k=t-n}^{t-1}\left(RO{I}_2^i\left(k\right)-R_2^i\right)RO{I}_1^j\left(k\right)/{\sum }_{k=t-n}^{t-1}RO{I}_1^i\left(k\right)RO{I}_1^j\left(k\right) \\ {R}_1={\sum }_{k=t-n}^{t-1}(RO{I}_1\left(k\right)-W_{12}\ast RO{I}_2\left(k\right)) \\ {R}_2={\sum }_{k=t-n}^{t-1}\left(RO{I}_2\right(k)-W_{21}\ast RO{I}_1(k)) \end{gathered}$$

(7)

We would like to point out that we do not impose any restrictions or special requirements on camera placement besides the ones mention in the previous subchapter. There is no need for initial calibration or any other measurement or localization. We only need the cameras to record the same person at the same time. The model is trained by tracking a moving object in in both fields of view for some number of frames $n$. Generally, we want a higher value for $n$, although there are ways to reduce the number of frames needed for the learning stage.

After the training is complete and the values for $W_{12}^{ij}$,$W_{21}^{ij},R_1$ and $R_2$ acquired, we can use them to fuse the size and position of both regions of interest (as long as the cameras do not change location, if they do, we need to repeat the learning stage). We name the new regions of interest as $RO{I}_1^n$and $RO{I}_2^n$. They are obtained by minimization of the following quadratic cost function:

$$\begin{gathered} C_{ROI}={\left(RO{I}_1^n\left(t\right)-RO{I}_1\left(t\right)\right)}^2 \\ +{\left(RO{I}_2^n\left(t\right)-RO{I}_2\left(t\right)\right)}^2{+\mu \left(RO{I}_1^n\left(t\right)-W_{12}\ast RO{I}_2^n\left(t\right)-R_1\right)}^2 \\ {+\mu \left(RO{I}_2^n\left(t\right)-W_{21}\ast RO{I}_1^n\left(t\right)-R_2\right)}^2 \end{gathered}$$

(8)

In Equation (8) $RO{I}_{\mathrm{1,2}}$ are the RoIs derived at the current time point $t$ from the optical flow reconstruction as in Equation (2) applied for each camera independently, $\mu$ is a mixing weight constant, controlling how much we trust our linear model of inter-camera movement. We find the first derivatives with respect to $RO{I}_1^n$ and $RO{I}_2^n$ and set them to zero to arrive at the following system of equations with for the optimized regions of interest:

$$\begin{gathered} RO{I}_1=\left(1+\mu +{\mu W}_{21}^T\ast W_{21}\right)\ast RO{I}_1^n-\mu \left(W_{12}+W_{21}^T\right)\ast RO{I}_2^n-\mu R_1+\mu W_{21}^T{\ast R}_2 \\ RO{I}_2=\left(1+\mu +\mu W_{12}^T\ast W_{12}\right)\ast RO{I}_2^n-\mu \left(W_{21}+W_{12}^T\right)\ast RO{I}_1^n-\mu R_2+\mu W_{12}^T{\ast R}_1 \end{gathered}$$

(9)

We combine the two RoI vectors into one six-dimensional vector:

$$\mathit{R}\mathit{O}\mathit{I}\equiv \left(\begin{array}{c}RO{I}_1\\ RO{I}_2\end{array}\right);\mathit{R}\mathit{O}{\mathit{I}}^{\mathit{n}}\equiv \left(\begin{array}{c}RO{I}_1^n\\ RO{I}_2^n\end{array}\right)$$

(10)

Next, we introduce the following aggregated variables:

$$\begin{gathered} \mathsf{\Omega }\equiv \left(\begin{array}{cc}\left(1+\mu \right)\mathit{I}+{\mu W}_{21}^T\ast W_{21}& -\mu \left(W_{12}+W_{21}^T\right)\\ -\mu \left(W_{21}+W_{12}^T\right)& \left(1+\mu \right)\mathit{I}+{\mu W}_{12}^T\ast W_{12}\end{array}\right) \\ R\equiv \left(\begin{array}{c}-\mu R_1+\mu W_{21}^T{\ast R}_2\\ -\mu R_2+\mu W_{12}^T{\ast R}_1\end{array}\right) \end{gathered}$$

(11)

Here $\mathit{I} \mathrm{i}\mathrm{s} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{i}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{y} \mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x}$, $\mathsf{\Omega }$ is a 6x6 matrix and $R$ is a 6x1 vector. Equation (9) can be then presented in matrix form:

$$\mathsf{\Omega }\ast {\mathit{R}\mathit{O}\mathit{I}}^{\mathit{n}}+\mathit{R}=\mathit{R}\mathit{O}\mathit{I}$$

(12)

In order to find the values of the new RoIs, we must invert the full coefficient matrix $\mathsf{\Omega }:$

$$\mathit{R}\mathit{O}{\mathit{I}}^{\mathit{n}}={\mathsf{\Omega }}^{-1}\ast (\mathit{R}\mathit{O}\mathit{I}-\mathit{R})$$

(13)

Aiming to avoid local singularities such as a zero value for the eigenvalues, we perform a modified Tikhonov regularization:

$$\begin{gathered} {\mathsf{\Omega }}_{\rho }={\displaystyle \frac{e_{max}}{e_{max}+\eta }}\left(\mathsf{\Omega }+\mathsf{\eta }\mathbf{I}\right); \\ \eta =(\rho e_{max}-e_{min})\theta (\rho e_{max}-e_{min}) \end{gathered}$$

(14)

In Equation 11, $e_{max}and{e}_{min}$ are the maximum and minimum eigenvalues of $\mathsf{\Omega }$, $\rho$ is a constant of regularization $\mathbf{I}$ is the unit matrix, $\theta$ is the Heaviside step function. This procedure leads to a non-singular coefficient matrix that is safe to invert:

$$\mathit{R}\mathit{O}{\mathit{I}}^{\mathit{n}}={\mathsf{\Omega }}_{\rho }^{-1}\ast (\mathit{R}\mathit{O}\mathit{I}-\mathit{R})$$

(15)

2.4. Non-Linear Iterative Corrective Approach

In certain scenarios during tracking, the person or object of interest moves very slowly or does not move at all. In those cases, the region of interest may sometimes experience minor jittering due to camera noise or changing lighting conditions (which is a problem for all optical flow methods, and other non-optical flow as well). We have developed an approach that increases accuracy of our tracking and removes any (if present at all) jittering. We start by defining the 0^th iteration of our new regions of interest:

$$\begin{gathered} RO{I}_1^{n,0}\left(t\right)=RO{I}_1\left(t\right) \\ RO{I}_2^{n,0}\left(t\right)=RO{I}_2\left(t\right) \end{gathered}$$

(16)

In the RHS of Equation (16), $RO{I}_{\mathrm{1,2}}\left(t\right)$ are the RoIs derived from the RoI of the previous time step and the optical flow reconstruction for each camera independently as in equation (2).

The next iteration can be calculated by using the current one and the mean value of all ROIs that fulfill a certain criterion:

$$\begin{gathered} RO{I}_1^{n,a+1}\left(t\right)=\left(1-{\nu }_1\right)RO{I}_1^{n,a}\left(t\right)+{\nu }_1{〈RO{I}_1\left(k\right)〉}_{kϵK^a} \\ RO{I}_2^{n,a+1}\left(t\right)=\left(1-{\nu }_2\right)RO{I}_2^{n,a}\left(t\right)+{\nu }_2{〈RO{I}_2\left(j\right)〉}_{jϵJ^a} \end{gathered}$$

(17)

In Eq. (17) $a$ is the iteration index, $kandj$ are indices that belong to the following domains:

$$\begin{gathered} K^a\equiv \left\{k:t-n\le k<t,‖RO{I}_1\left(k\right)-RO{I}_1^{n,a}\left(t\right)‖\le \epsilon \right\} \\ {J}^a\equiv \left\{j:t-n\le j<t,‖RO{I}_2\left(j\right)-RO{I}_2^{n,a}\left(t\right)‖\le \epsilon \right\} \end{gathered}$$

(18)

Here $‖.‖$ denotes the Euclidean vector norm and $n$ shows how many frames back we wish to use for our estimation and $\epsilon$ is the desired threshold.

In Equation (18) ${\nu }_{1}and{\nu }_2$ are an asymmetric mixing weight functions defined as:

$$\begin{gathered} {\nu }_1={\displaystyle \frac{\left|K^a\right|}{\left|K^a\right|+1}} \\ {\nu }_2={\displaystyle \frac{\left|J^a\right|}{\left|J^a\right|+1}} \end{gathered}$$

(19)

Here, $\left|J^a\right|,\left|K^a\right|$is the number of elements in each domain. The process terminates when the series $RO{I}_{\mathrm{1,2}}^{n,a}\left(t\right)$ converges with iterations or reaches a maximum iteration count.

2.5. Parameters for Evaluation

$$\begin{gathered} {∆C}_x^i=\sqrt{{\left(C_{xTrue}^i-C_{xROI}^i\right)}^2};{∆C}_y^i=\sqrt{{\left(C_{yTrue}^i-C_{yROI}^i\right)}^2} \\ {∆L}_x^i=\sqrt{{\left(L_{xTrue}^i-L_{xROI}^i\right)}^2};{∆L}_y^i=\sqrt{{\left(L_{yTrue}^i-L_{yROI}^i\right)}^2} \\ ∆C^i=\sqrt{{∆C^i}_x^2+{∆C_y^i}^2};∆L^i=\sqrt{{{∆L}_x^i}^2+{{∆L}_y^i}^2} \\ {∆}^i=\sqrt{{∆C^i}^2+{∆L^i}^2} \end{gathered}$$

(20)

$$\begin{gathered} ∆C_x={\displaystyle \frac{1}{N}}\left(\sum _{i=1}^N{∆C}_x^i\right);∆C_y={\displaystyle \frac{1}{N}}\left(\sum _{i=1}^N{∆C}_y^i\right) \\ ∆L_x={\displaystyle \frac{1}{N}}\left(\sum _{i=1}^N{∆L}_x^i\right);∆L_y={\displaystyle \frac{1}{N}}\left(\sum _{i=1}^N{∆L}_y^i\right) \\ {∆}_{Total}=\sqrt{{∆L_y}^2+{∆L_x}^2+{∆C_x}^2+{∆C_y}^2} \end{gathered}$$

(21)

We propose the usage of several quantities to evaluate the performance of our method. Whenever ground truth information is available, we can calculate the absolute difference between the “true” center coordinates $\left(C_{xTrue}^i,C_{yTrue}^i\right)$ of the moving object and the calculated values of the center coordinates $\left(C_{xROI}^i,C_{yROI}^i\right)$ of the RoI, as well as the absolute differences between the true $L_{xTrue}^i$and calculated $L_{xROI}^i$ values of the dimensions of the RoI:

These values show mismatches between individual frames. We can use such an approach when labeled or synthetic data is present. We also introduce quantities to assess the mismatch for a full image sequence:

In Equation (21) $N$ is the total number of frames, the quantities in Equation (21) are the average values of the quantities in Equation (20) for $N$ frames. These values are used to evaluate the accuracy of the fusion RoI when multiple cameras are capturing the same moving object from more than one viewing position. They are also used to show how the fragmented region of interest helps track faster moving object than the standardly-defined region of interest. We also introduce a relative mismatch value D, to compare the mismatch of the standard RoI ${∆}_{Total}^{GLORIA}$ to the fusion one ${∆}_{Total}^{FUSION}$:

$$D=100\ast {\displaystyle \frac{{∆}_{Total}^{GLORIA}-{∆}_{Total}^{FUSION}}{{∆}_{Total}^{GLORIA}}}$$

(22)

In order to analyze the effectiveness of the proposed method when tracking people going from one living space to the next, we introduce the transition effectiveness $TE$:

$$TE=100\ast {\displaystyle \frac{N_{st}}{N_{tt}}}$$

(23)

Here $N_{tt}$ is the total number of transitions from one room to another, and $N_{st}$ are the number of transitions for which the RoI in the second room has successfully begun tracking the patient entering said room or living space. All other tracking limitations are the same as in our previous work. Here the transition between living spaces is the only new activity that needs to be described and evaluated.

We introduce the ratio $R$between object area $S_o$ and region of interest area $S_R$ as:

$$R=100\ast {\displaystyle \frac{S_o}{S_R}}$$

(24)

This measure is going to be used when we evaluate the effect of region of interest size to the transition effectiveness.

3. Results

3.1 Experimental Setup and Living Space Configuration

We use two Logitech® C170 webcams to capture video. A Lenovo® ThinkPad with an Intel® Core i5 CPU and 16Gb of DDR4 RAM is used to process the video, calculate movement and display RoIs. Synchronization of both cameras is achieved using the Image Acquisition Toolbox by MATLAB®. Algorithm realization is done in the MATLAB® environment as well. Our living space setting is presented in Figure 4:

3.1. Evaluation of Transition Efficiency for Two Camera Tracking with no Overlap

We work with a two-room setup to demonstrate the performance of our method, although scaling to 4 or 5 rooms is straightforward. The camera in the room to the right can move (PTZ) so that the whole room is covered by it. In our first test we capture a person moving from one room to the other. The size of the initial region of interest drawn at the entry of room two is varied and its effects (different area ratios - Equation (23) on transition efficiency, given by Equation (24), are examined. A total of one hundred transitions are made. Results are presented in Figure 5.

These results show us that the first step of RoI initialization is very important as only for certain values of R we get a transition efficiency of 1. This means that certain area ratios are more preferable than others and provide better tracking performance. When placing the initial RoIs, these findings should be taken into consideration.

3.3. Evaluation of Region of Interest Optimization for Areas with Camera Overlap

For our next test we examine the scenario in which both cameras capture a moving person at the same time from different positions. When this is the case, we can utilize the fusion model introduced in the previous chapter to calculate a better, more accurate region of interest. We have prepared two sets of tests – one on simulated data and one on real-world footage. Our simulated tests were made in the following order:

• Generate some N>50 number of points in 3D space randomly;
• Define two planes, whose normal vectors have some random angle between them;
• Project the points from 1) to both planes, considering distance between point and plane;
• In each plane, form the convex hull of the projected points;
• Everything inside the convex hull is set to value “1”, everything outside it is set to “0” – this is our object in both planes, respectively;
• Generate a motion pattern consisting of some number K points, that will be used to move the points from 1). Repeat 2-5 for each point of the K points.

This process allows us to generate a wide variety of video sequences containing moving objects. The objects have varying size and shape, move with varying speed, and occupy different area of the cameras’ fields of view. A snapshot of the simulation is presented on Figure 6.

Using this scheme 90 one-minute long videos were made and the relative mismatch D as defined in Equation (21) is examined. Results are presented in Figure 7.

We can see that the method is very beneficial for some videos in our set, while for others it does not improve the tracking quality very much. We have tried to find a link between object speed and other motion characteristics, but no correlation was found. This leads us to think that the difference in performance is related to the specific shape of the object in the different camera planes.

We have also prepared several labeled real-world datasets with a simple spherical object whose center and size are initially determined for each frame. Results are presented for both regions of interest, with the non-fusion approach present on the graph for comparison. We use one video from the dataset for acquiring the linear model parameters given by Equation (7) and the calculate the fusion RoIs using Equation (15). We then compare regions of interest by evaluating the total mismatch Δ as defined by Equation (21) with respect to the mixing coefficient $\mu$. We test on 5 different datasets to compare the performance of the new method to the single camera tracker (marked as GLORIA ROI in the figures) one and present the results on Figure 8.

We present the best results with regard to the mixing coefficient µ from the test presented in Figure 8 in a single graph to outline the optimal performance. The results are presented in Figure 9.

We also present the relative mismatch to better understand the effectiveness of the new fusion method. The results are given in Figure 10.

From the results presented in Figure 8, Figure 9 and Figure 10, we can see that, except for one dataset, the fusion RoI approach produces substantially more accurate results and is better for tracking than the standard one. This shows that using this model generally increases tracking accuracy and is preferable whenever camera overlap is present.

A similar set of tests is prepared for the non-linear iterative corrective approach. In a simulated example, an object moves slowly in a single direction. The object’s journey is three pixels long for a total of one thousand frames. We generate fifty-one different videos. Both true and “measured” positions of the object’s centre are available, and we use the measured ones to calculate using Equations (16-19) new RoI values. We also calculate RoI values using Equations (4-15) to show the effectiveness of the non-linear corrector approach. We show the total mismatch Δ, given by Equation (21) for each video. Results are available in Figure 11.

The mean value for the relative mismatch D (given by Equation (22)) for the linear fusion model is 1.55%, while the mean value for the non-linear approach is 14.75%. This figure shows us that the non-linear approach provides substantially better results when we track slow-moving objects than the linear one.

We also investigate the effectiveness of the non-linear algorithm for different object speeds. We create a number of video sets, each with a different object speed. For each different case, the total movement interval is divided into a different number of parts, or in other words, the object moves at different speeds. We evaluate the performance of this method based on object speed and see at what object velocity the method stops being beneficial. We analyse the behaviour of the relative mismatch D based on different object velocities. Results are available in Figure 12. If the movement interval is divided into 10 steps, the method does not increase accuracy. That would be equivalent to an object velocity of 0.3 pixels per frame. For slower speeds, the technique is beneficial, and it is advisable to use it.

3.4. Evaluation of Region of Interest Segmentation

We continue our analysis by demonstrating the usefulness of the proposed region of interest segmentation. We have prepared two sets of tests – one with synthetic data generated in the memory of a personal computer and one with regular real-world data. The first test consists of a simple circular moving object on a non-homogenous (in colour) background. There are fourteen videos in total, and in each one, the object moves faster than the previous one. The idea is to examine whether or not the approach with a segment region of interest keeps tracking the object at speeds for which the standard region of interest cannot. We divide the original RoI into 16 smaller ones. Results are presented in Figure 13.

We again observe how the total mismatch Δ, given by Equation (21), behaves depending on object speed. This test shows us that the use of a fragmented region of interest is recommended for higher object velocities. Moreover, at a speed of 42 pix/frame, the standard region of interest can no longer keep track of the object, whereas the fragmented RoI continues functioning. At lower velocities, the performance of both methods is comparable.

Our second test involves a person who is initially standing still but, after a brief period of time, moves in one direction with great speed. The purpose of the tests is to give an example of the responsiveness of each region of interest definition when tracking fast-moving objects in real-world scenarios. Results are presented in Figure 14.

We can see that in the period of 20 frames, which is exactly one second with the current camera setup, the standard region of interest (shown in green to the right) lags behind the moving person. The fragmented RoI (shown in blue to the left) is able to keep up with the sudden movement. This test confirms the results we acquired from the synthetic data test and definitely shows the usefulness of using a fragmented region of interest for tracking purposes.

3.5. Occlusion Tests

A big challenge in the field of video tracking is when the object becomes obscured by other objects in the scene. This causes it to be hidden from view and interrupts any continuous tracking. Much work is done to address this problem; here we will show how our method can help solve it. As stated in our previous work with the single-camera tracker, we need to develop ways to reposition the RoI if it, for some reason, stops surrounding the person of interest. Using our linear model in the case of overlapping fields of view, we will demonstrate how to reposition the region of interest in the case of an occluded object. We will use the regions of interest defined by Equation (5) to continue tracking the object correctly even when it is not visible by one of the cameras. An outline of this capability is presented in Figure 15. In this test, a person moves in a straight line. Cameras are positioned perpendicularly to one another. The person leaves the field of view of camera 1 after moving behind another object. At that moment, the region of interest associated with that camera stops tracking the person, as it registers no motion. The region of interest of camera 2 continues tracking (as the person is not occluded in the field of view of camera 2), and that RoI, alongside the transitional matrix and reminder $W_{12}and{R}_1$ can be used to correctly position an RoI in the field of view of camera 1. This property of the presented method can be used to reposition $RO{I}_1$ after it gets stuck. Further work is required to describe how to best reposition regions of interest if they stop following the person of interest.

4. Discussion

As with our previous work [8], we would like to point out that patient privacy is of the utmost importance in the current context. Other methods for video tracking may require the storage of sensitive data. As an extension of our previous method, this one does not have these requirements and can even function with cameras that capture vector fields (not intensities). This detail alone is a considerable privacy advantage.

Acquiring more frames will help when training the model, but they may not be needed. If clear translational and dilatational motions are captured during the learning stage, the number of training images for each camera can be reduced significantly.

In addition to simulated tests, we did manual labelling for some of our tests that examine the fusion region of interest placement. We chose that approach for several reasons, namely, a circular object with a mark on its center is easy to track with different methods and to determine its position with ease. This is a real-life test, so all the details regarding camera noise, synchronization, processing speed, illumination changes and others are present.

The simulated tests in the camera overlap case show worse performance than the real-world tests. This may be because they encompass rare camera positions in practical situations and skew the results because of their inter-camera behaviour.

Using a fragmented region of interest shows great promise, and so far, results tell us that it is advisable to use it. A key property of this approach is the number of smaller areas into which we divide the initial region of interest. If that number is not big enough, we may not get the benefits of using the approach. If it is too big, tracking may become inaccurate. We would also like to point out that dividing the region of interest in smaller RoIs does not make the overall tracking algorithm slower – processing time remains the same.

As it currently stands, the method is intended for tracking a single person (object). In the future, we will explore ways to track multiple people at the same time. Such a task faces more challenges on its own, such as object reidentification (also known as re-id), which will be an interesting problem to solve given the privacy context of the method’s field.

Another future extension is to apply the same fusion paradigm to the epileptic seizure detection method. Currently, the method uses a single camera to detect seizures. Based on the results presented here, we believe that if a patient is monitored by two or more cameras simultaneously, we can increase the sensitivity of the epileptic seizure detection technique.

Finally, we compare the results from the linear and the non-linear multi-sensor interaction models. It is clear from Figure 8 that the linear approach may give only a marginal increase in the tracking accuracy and can be critically sensitive to the choice of the mixing coefficient. It is worth noting, however, that although the non-linear approach proposed in this work demonstrates, as shown in Figure 11, clear performance advantages as compared to the linear sensor interaction model, it also has its limitations. It may require a much larger number of previous frames because it relies on re-entrance statistics around each time point, as seen from equation (18). The linear interaction model, on the other hand, tries to establish a universal linear mapping between the camera RoIs and, therefore, can be achieved with much fewer previous samples.

5. Conclusions

We have proposed an extension of an existing tracking method with applications in the prevention of life-threatening epileptic seizures. Our first work shows how we can detect epileptic seizures from video data, and our second work helps that method function better as it allows us to focus only on the person we want to monitor. This paper proposes various solutions to problems related to our patient tracker. The improvements presented in this work significantly increase the capabilities of our tracking technique and are a considerable upgrade. We can sum them up briefly:

• Extend the tracking method to cover multiple rooms;
• Introduce a model of inter-camera relationship that describes how movements in different fields of view are connected;
• Use the model to improve tracking accuracy for places of camera overlap;
• Showcase the usefulness of the same approach for tracking challenges such as occlusion events;
• Introduce a non-linear approach for low-velocity cases and demonstrate its use on different tests;
• Redefine the RoI to successfully track objects moving with higher speed.

These upgrades help make the method better, more robust and accurate. We intend to integrate these improvements in the existing system and validate our current findings with additional data from real patients in a future study.