Remote Object Tracking Based on Optical Flow Reconstruction of Motion-Group Parameters

1. Introduction

The method presented in the current paper addresses medical applications where the strategic goal is to provide better medical care through real-time detection, warning, prevention, diagnosis, and treatment. The specific task is to detect seizures in people who have epilepsy.

Epilepsy is a neurological disease whose symptoms are sudden transitions from normal to pathological behavioral states called epileptic seizures, often accompanied by rhythmic movements of body parts. Some of these seizures may lead to life-threatening conditions and ultimately cause Sudden Unexplained Death in Epilepsy (SUDEP).

Therefore, medical treatment involves continuous observation of individuals for long periods to obtain sufficient data for an adequate diagnosis and to plan therapeutic strategies. Some people, especially those with untreatable epileptic conditions, may need long-term care in specialist units to allow early intervention that prevents complications.

We have developed and implemented an earlier method that detects motor seizures in real time using remote sensing by video camera. Video human surveillance is used successfully for monitoring patients, but it poses certain societal burdens and costs and creates ethical issues related to privacy. Epilepsy patients are sick people, often stigmatized and sensitive to privacy. Therefore, it is necessary to use only remote sensing devices (i.e., no contact with the object) such as video cameras, recording not movies but vector fields.

The method was published [1] and is currently used in medical facilities with validated detection success. It uses an original, efficient algorithm to reconstruct global optical flow group velocities (GLORIA). These quantities also provide input for detecting falls and non-obstructive apnea [2,3]. However, the technique only works in a static situation, for example, during night observation when the patient lies in bed. When the patient changes position, for example, during daytime observation, the operator has to adjust either the camera field of view with PTZ (pen tilt zoom) field of view or the Region of Interest (ROI). Here, we address the challenge of developing and embedding patient-tracking functionality in our system. We approach this task using the same group velocities provided by the GLORIA algorithm. This way, we avoid introducing extra computational complexity that may prevent the system from being used in real-time. To the best of our knowledge, this is a novel video-tracking approach and constitutes the main contribution of this work. Below, we acknowledge other works related to video tracking but none of them can directly utilize the group velocities that provide also input for our convulsive seizure detection algorithm.

Real-time automated tracking of moving objects is a technique that has a wide range of applications in various research and commercial fields [4]. Examples are self-driving cars [5,6], automated surveillance for incident detection and criminal activity [7], traffic flow control [8,9], interface for human-computer interactions and inputs [10,11,12], and patient care, including intensive care unit monitoring and epilepsy seizure alerting [1,13].

Generally, there are different requirements and limitations when one attempts to track a moving object, such as noise in the video data, complex object movement, occlusions, illumination changes between frames, and processing complexity. Many object-tracking methods involve particular tracking strategies [14,15,16,17], including frame difference methods [18], background subtraction [19], image segmentation, static and deep learning algorithms [20,21], and optic flow methods. All the methods have advantages and disadvantages, and the choice depends on the specific task.

In the current work, we propose a method based on direct global motion parameter reconstruction from optical camera frame sequences. As noticed above, the main reason for focusing on such a method is that it uses already calculated quantities as part of the convulsive seizure detection system. The method avoids intensive pixel-level optic flow calculation. Such an approach provides thus computationally efficient and content-independent tracking capabilities. Tracking methods that use the standard pixel-based optic flow reconstruction [22,23] suffer from limitations caused by relatively high computational costs and ambiguities due to the absence of sufficient variation of the luminance in the frames. We propose a solution to these limitations using the multi-spectral direct group parameter reconstruction algorithm GLORIA, developed by our group and introduced in [24]. The GLORIA algorithm offers the following advantages compared to other optic flow-based tracking techniques: A) It calculates directly the rates of group transformations (such as but not limited to translations, rotation, dilatation, and shear) of the whole scene. Thus, it avoids calculating the velocity vector fields for each image point, thus lowering computational requirements. B) Unlike the standard intensity-based algorithms, we apply a multi-channel method directly using co-registered data from various sources, such as multi-spectral and thermal cameras. This early fusion of sensory data features increases accuracy and decreases possible ambiguities of the optic flow inverse problem.

Our objective for the particular application here is to track a single object that moves within the camera’s field of view. We introduce a rectangular Region of Interest (ROI with a specific size around the object we plan to track). This GLORIA method then further accurately estimates how the object moves. Subsequently, the novel algorithm proposed here adjusts the ROI accordingly. Using an ROI also helps reduce the size of the images for optical flow estimation and limits the adverse effects of any areas with high levels of brightness change outside the ROI. It also automatically disregards movements of no interest to us outside the specified area. In short, our approach reduces the tracking problem to a dynamic ROI steering algorithm.

The present work is part of a more extensive study on autonomous video surveillance of epilepsy patients. Tracking how a patient moves can further improve results related to seizure detection from video data [1]. The proposed method, however, is not limited to use only in health care but can successfully apply to other scenarios related to automated remote tracking.

The rest of the paper is organized as follows. The next section introduces the proposed original tracking method. Then, we present our results from both simulated and real-world image sequences. We also apply the novel method to a sequence from a publicly available dataset (LaSoT). The outcome from all the examples provides quantitative validation of the algorithm’s effectiveness and qualitative illustrations. Finally, in the Discussion, we comment on the features, possible extensions, and limitations of the proposed approach to tracking.

2. Methods

Optic Flow Reconstruction Problem

The algorithm presented in the current work uses motion information reconstructed from the optic flow in video sequences. Optical flow reconstruction is a general technique that enables determining the spatial velocities of a vector field from changes in luminance spectral intensities between sequential observed scenes (frames). Here, we briefly introduce optic flow methods, leaving the details to the dedicated literature [25,26,27,28,29,30,31,32,33].

We denote the pixel content in a multi-spectral image frame as $F^c(x,y,t)$, where $\left(x,y,t\right)$ are the spatial coordinates and the time, $c$ is the spectral index, most commonly labeling the R, G, and B channels. Assuming that all changes of the image content in time are due to scene deformation and defining the local vector velocity (rates of deformation) vector field as $v(x,y,t)$, the corresponding image transformation is:

$$\frac{d{F}^c}{dt}=-\nabla F^c·v\equiv {\nabla }_v{F}^c$$

(1)

In Eq. (1) ${\nabla }_v$ is the vector field operator, $\left(x,y\right)$ are the two-dimensional spatial coordinates in each frame, and $t$ is the time or frame number. The velocity field can determine a large variety of object motion properties such as translations, rotations, dilatations (expansions and contractions), etc. In the current work, however, we do not need to calculate the velocity vector field for each point, as we can directly reconstruct global features of the optic flow, considering only specific aggregated values associated with it. In particular, we are interested in the global two-dimensional linear non-homogeneous transformations consisting of translations, rotations, dilatations, and shear transformations. Therefore, we use the Global Optical-flow Reconstruction Iterative Algorithm “GLORIA,” which was developed previously by our group [3]. The vector field operator introduced in Eq. (1) takes the following form:

$${\nabla }_v\equiv v·\nabla \equiv \sum _k{v}_k{\nabla }_k; {\nabla }_k{F}^c\equiv \frac{\partial F^c}{\partial x_k}$$

(2)

The Eq. (2) representation can be helpful when decomposing the transformation field $v$ as a superposition of known transformations. If we denote the vector fields corresponding to each transformation generator within a group as $v^u$, and the corresponding parameters as $A^u$, then:

$$v\equiv \sum _u{A}^u{v}^u$$

(3)

With Eq. (3) one may define a set of differential operators for the group of transformations that form a Lie algebra:

$$G^u\equiv \sum _k{v}_k^u{\nabla }_k$$

(4)

As a particular case, we apply Eq. (4) to the group of three general linear non-homogeneous transformations in two-dimensional images that preserve the orientation of the axes and the ratio between their lengths:

$$\begin{array}{l}{G}^{translatio{n}_x}={\nabla }_x; \\ G^{translatio{n}_y}={\nabla }_y;\\ G^{dilation}=x{\nabla }_x+y{\nabla }_y;\end{array}$$

(5)

Region of Interest (ROI) Transformations

Using GLORIA, the amplitudes $A^u$ for each of the three transformation vector fields is the solution of the global transformation optic flow inverse problem. These amplitudes represent the rate of each type of movement as defined by Eq. (4). These amplitudes, along with the coordinates of the initial region R1 of interest (corner and center points) can be used to determine iteratively the coordinates of the subsequent region of interest R2. Figure 1 illustrates the changes in ROI due to the impact of each group transformation.

The movement of an object can be followed using the values of the group transformations reconstructed from the GLORIA algorithm by updating the ROI after each (or a set number of) frames. A diagram outlining the general process is presented in Figure 2.

The method in this work can be applied to any group of transformations. Our choice here is on the two translation rates and the dilatation (a global scale factor quantity) that are provided by the first three generators from Eq. (5). We mark them as $T_x^i$ and $T_y^i$ for the translations and $D^i$ for the dilatation, where $i$ indicates which two consecutive frames were used for the calculation. We restrict the current method to only these three transformations because we do not intend to rotate the ROI with the tracked object nor change the ratio between the ROI dimensions. In this way, our method is directly applicable to a situation where pan, tilt, and zoom (PTZ) hardware actuators are affecting the camera field of view that corresponds to the two translations (pen and tilt) and the dilatation (the zoom). Next, we define the values that will parametrize the extent of our ROI. These values, related to the rectangular ROI in Figure 3., are its width and length $(L_x^1, L_y^1)$.

They relate to the corner points of the ROI by Eq. (6):

$$\begin{gathered} L_x^1=X_2^1-X_1^1=X_4^1-X_3^1 \\ {L}_y^1=Y_4^1-Y_2^1=Y_3^1-Y_1^1 \end{gathered}$$

(6)

The first step of our algorithm then becomes the selection of the coordinates of the center $(X_C^1, Y_C^1$) and the width and length $(L_x^1, L_y^1)$ of a region of interest R1 from the initial frame in the video sequence or live feed.

Using the same region of interest in the next frame, we calculate by applying the GLORIA algorithm between the two frames the global rates of translation $(T_x^1, T_y^1)$ and the dilatation D¹. The center and sizes of the subsequent region of interest R2 are then defined by the elements of the initial frame, and the values are calculated by GLORIA according to the following set of equations:

$$\begin{gathered} X_C^2=X_C^1+T_x^1; Y_C^2=Y_C^1+T_y^1 \\ {L}_x^2=L_x^1*\left(1+D^1\right); L_y^2=L_y^1*(1+D^1) \end{gathered}$$

(7)

Eq. (7) allows us to track an object by adjusting the ROI containing the object with each following frame. In particular, if a PTZ steerable camera is used, the changes of the ROI’s center coordinates in the $X$ and $Y$ direction would require pan and tilt actions to re-center the scene and, changes of the ROI’s size due to the dilatation would translate in a zoom in or zoom out action. This process is repeated for all subsequent frames, updating the ROI’s elements along the way.

Evaluation of the ROI Tracking Performance

We introduce several quantities to assess the proposed method’s accuracy and working boundaries. The first one reflects a combination of the absolute difference between the center coordinates of the moving $\left(C_{xTrue}^i,C_{yTrue}^i\right)$ object and the calculated values of the center coordinates $\left(C_{xGloria}^i,C_{yGloria}^i\right)$ of the ROI, as well as the absolute differences between the true $L_{xTrue}^i$ and calculated $L_{xGloria}^i$ values of the dimensions of the ROI:

$$\begin{gathered} {∆C}_x^i=\sqrt{{\left(C_{xTrue}^i-C_{xGloria}^i\right)}^2}; {∆C}_y^i=\sqrt{{\left(C_{yTrue}^i-C_{yGloria}^i\right)}^2} \\ {∆L}_x^i=\sqrt{{\left(L_{xTrue}^i-L_{xGloria}^i\right)}^2}; {∆L}_y^i=\sqrt{{\left(L_{yTrue}^i-L_{yGloria}^i\right)}^2} \\ ∆C^i=\sqrt{{∆C^i}_x^2+{∆C_y^i}^2}; ∆L^i=\sqrt{{{∆L}_x^i}^2+{{∆L}_y^i}^2} \\ {∆}^i=\sqrt{{∆C^i}^2+{∆L^i}^2} \end{gathered}$$

(8)

We use these values to determine the maximum velocities of moving objects that can be registered with the method. They only apply when the ROI’s ground truth coordinates and sizes are known, for example, when dealing with synthetic test data. To assess the average deviation between the true position of the moving object and the detected one for a given tracking sequence, we define the following quantity:

$$\begin{gathered} ∆C_x=\frac{1}{F}\left( \sum _{i=1}^F{∆C}_x^i\right); ∆C_y=\frac{1}{F}\left( \sum _{i=1}^F{∆C}_y^i\right) \\ ∆L_x=\frac{1}{F}\left( \sum _{i=1}^F{∆L}_x^i\right); ∆L_y=\frac{1}{F}\left( \sum _{i=1}^F{∆L}_y^i\right) \\ ∆C=\frac{1}{F}\left( \sum _{i=1}^F∆C^i\right); ∆L=\frac{1}{F}\left( \sum _{i=1}^F∆L^i\right) \\ {∆}_{Total}=\sqrt{{∆L_y}^2+{∆L_x}^2+{∆C_x}^2+{∆C_y}^2} \end{gathered}$$

(9)

In eq. (9). $F$ is the total number of frames, and the values in brackets are the summed values of Eq. (8) for the corresponding number of frames, Eq. (9) represents the average values of Eq. (8) a $F$ frames. We apply the measure in Eq. (9) to explore the influence of the background image contrast on the accuracy of our tracking algorithm. Image contrast is defined as in [34] –the root-mean-square deviation of the pixel intensity from the mean pixel intensity for the whole frame, divided by the mean pixel intensity for the entire frame. Each color channel has a specific background contrast value. It can affect the optical flow reconstruction quality and, accordingly, the quality of the ROI tracking

Initial ROI placement also affects the accuracy of the method. To examine what the optimal size of the initial ROI is, we define the ratio $K$ between the ROI area $A_{RoI}$ and object area $A_{object}$ Eq. (10):

$$K=\frac{A_{object}}{A_{RoI}}$$

(10)

If one wants to verify that the ROI tracks the object accurately but does not have access to the true ROI center position dimensions (as is in Eq. (8)), we introduce the relative mismatch ${\delta }^i$:

$${\delta }^i=\frac{\sum _k{\left(I_i^k-I_0^k\right)}^2}{{\sum }_k{\left(I_i^k+I_0^k\right)}^2}\ast 100$$

(11)

In Eq. (11), $I_i$ is the image in the ROI in the $I^{th}$ frame, resampled to the pixel size of the initial ROI, $I_0$ is the initial image from the initial ROI in the first frame, $i$ is the index of the current frame, $k$ is a summation index over all the pixels of $I_i$ and $I_0$. In our tests, we will show that the quantities ${\delta }^i$ and ${∆}^i$ are highly correlated using two correlation measures – the Pearson correlation coefficient and the nonlinear association index $h^2$, developed in [35]. That would mean one might use the relative mismatch ${\delta }^i$ to give a qualitative measure of the accuracy of the method in real-world data where the true positions and sizes of moving objects are unknown.

The precision value is a measure used in the literature for tracking performance evaluation. It is defined as the ratio between the number of frames $N_T$ in which the center location is below some threshold $T$ and the total number of frames $N_{frames}$ in the sequence:

$${precision}_T=\frac{N_T}{N_{frames}}$$

(12)

Another measure is the success rate, which also considers the ROI box’s size and compares it to ground truth. It is defined as the relative number of frames $N_A/N_{frames}$ where the area of intersection between the tracked ROI with the ground truth bounding box divided by the area of the union between the two is larger than some threshold $A, (0\le A\le 1)$

$${success}_A=\frac{N_A}{N_{frames}}=\frac{1}{N}{\sum }_{i=1}^N{\mathit{f}}_{\mathit{A}}\left(s_i\right),$$

(13)

Here $s_i$ is the intersection-over-union between the tracking region of interest of the $i^{th}$ frame and the ground truth region of interest. The function ${\mathit{f}}_{\mathit{A}}\left(s_i\right)$ is an indicator which returns a value of “1” if $s_i$ is above the current threshold $A$, and “0” otherwise.

We note the last two quantifiers of tracking quality, just as the one defined by Eq. (8) and (9) depend on the existence of "unequivocal" ground truth. For a general assessment of tracking quality, the quantity introduced by Eq. (11) applies to complex objects and scenes.

3. Results

Tracking Capabilifties

To show the feasibility of our method, we started by creating simple test cases with only a single moving object. Initially, we made tests of movements comprised of only one of the primary generators in Eq. (5). The test methodology goes as follows:

• Generate an initial image, in our case, a Gaussian spot with a starting size and coordinates on a homogenous background
• Specify the coordinates and size of the first region of interest, R1
• Transform the initial image with any number $N$ of basic movement generators as described in Figure 1 to arrive at an image sequence
• Using the GLORIA algorithm, calculate the transformation parameters
• Update the ROI according to Eq. (7)
• Compare properties of regions of interest – coordinates and size

For translations in only the $X$ or $Y$ direction, the method proved to be very effective:

Figure 4 a) shows the Gaussian “blob” moving with a speed of 2 pixels per frame to the right, while in the second row, the object’s velocity is 1 pixel per frame. Figure 4 b) shows how much the calculated values deviated from the original. Although there is some spread, the final region of interest selection uses integer values for pixel coordinates, meaning that those computed values are rounded up. After rounding the values calculated by GLORIA and applying Eq. (8) and Eq. (11), both $∆$ and $\delta$ show a positive linear correlation with a Pearson coefficient value of 1. Therefore, the complete positive linear correlation between the two measures of tracking precision shows that $\delta$ can be used instead $∆$ for translational motion tracking assessment.

Further, we demonstrate the usefulness of the GLORIA algorithm when estimating the dilatational transformation rate Figure 5. In this test, the size of the observed object is increased by a fixed amount with each frame. The algorithm successfully detected the scaling of the object:

The mismatch values $∆$ and $\mathsf{\delta }$ are calculated again. However, this time, they only exhibit a partial linear correlation. We provide their scatter plot in Figure 6.

The nonlinear association index h² value shows that $∆$ and $\delta$ have a high nonlinear correlation. The variance of values obtained by Eq. (8) can be explained by the variance of values obtained by Eq. (11) for dilatational movements.

The next step is to show the tracking capabilities of our method when multiple types of movement are involved, as illustrated in Figure 7. We have prepared a test where both translations and dilatation are present.

We applied both Eq. (8) and Eq. (11) to this test to show that both measures are highly correlated, and the relative mismatch $\delta$ can be used for cases where no ground truth is available. The linear correlation between the measures $∆$ and $\delta$ is much lower than the nonlinear association index $h^2$ which accounts for arbitrary functional relations. The measured $h^2$ between the two mismatch measures $∆$, and $\delta$ is 0.8103. In other words, the variance of the values, given by Eq. (8), can be explained by the variance of the values, given by Eq. (11), and this fact, alongside the results from Figure 4 and Figure 6, allows using the relative mismatch $\delta$ for real-world data.

In Figure 8, we combined different types of movement and changed the scene’s background. We tested both low-contrast and high-contrast backgrounds. Our method works both with grayscale and RGB data.

After the initial tests, we tried applying it to various real-world tracking scenarios, which showed accurate tracking results as well. We started with a video sequence that contained only one moving party to track with a relatively static background Figure 9. The method successfully estimated a proper region of interest around the moving objects:

Finally, we tested a dynamic scene Figure 10 with multiple moving objects and a high-contrast background.

The results in Figure 10 show the benefits of the proposed method. It can easily track the person in the ROI throughout the frames. Although the background is complex and there are other moving objects, the ROI stays centered around the man and changes size accordingly based on the distance to the camera (can be observed in the last presented frame). The relative mismatch $\delta$ for the video sequence in Figure 10 is shown in Figure 11:

To test the performance of our method with test data provided for the evaluation of tracking methods [20], we have applied it to a sample of image sequences from the LaSOT dataset [36]. We provide the precision (Eq.12) and success (Eq.13) plots in Figure 12. The initial ROI for our method is the same as the RoI from the first image in the specific LaSOT image sequence:

Our method shows higher precision and success values in this example than the dataset’s averaged performance of any of the other tested methods [20]. This is certainly not a conclusive comparison, but it still indicates that the proposed technique provides promising tracking abilities.

Our method works significantly better when multi-spectral data is used. It is a consequence of the GLORIA algorithm, which provides an early fusion of all spectral components and reduces any contrast-related ambiguities for the group parameter reconstruction. We have prepared an example demonstrating the importance of multi-channel data (in our case, the use of colored image sequences). The test is presented in Figure 13. We have prepared a moving object (circle) on a shallow contrast background image. The moving object is not trackable in greyscale but is successfully tracked when the video has all three color channels:

Tracking Limitations

Several limitations apply when using the method presented in the current work. One is the maximum speed with which an object can move and be tracked by our method. To find the extent of this limitation, we made numerous simulations of a moving circular spot on a homogenous background with varying speeds. We use Eq. (8) to compare the method’s accuracy for varying object velocities. The means of the quantities of Eq. (8) were analyzed for movement spread out in twenty consecutive positions (frames), and a one-way analysis of variances test can summarize the results Figure 14:

The tests show that the tracking method becomes less reliable for velocities over seven pixels per frame, and some inaccuracies become apparent.

Another critical detail of our method is its applicability in real-time image sequences. It is limited by the processing time needed to update the ROI between frames. We investigated how fast our algorithm is on the following personal computer: Lenovo™ IdeaPad 130-15IKB laptop with an Intel® Core™ i5-8250 CPU and 20 GB of DDR4-2133 MHz SODIMM random access memory. Results for processing time depending on ROI size are presented in Figure 15.

The graph shows that real-time calculations between each frame can be done for a smaller ROI. However, our initial tests with a PTZ camera have shown that updating the ROI between each two frames is not always necessary, leaving even more room for real-time applications. Newer systems would also demonstrate significantly faster results.

The contrast of an image can also affect the accuracy of the presented method. We tested our method using a moving Gaussian spot on backgrounds with different contrasts Figure 16. The total error, as defined in Eq. (9), is given in the title of the background pictures.

These results show we can expect less reliable behavior when the background scene’s contrast is more significant. The reason is that higher background contrast within the ROI may interfere with the changes caused by the moving object and obscure the tracking. Specific actions such as selecting a smaller initial region of interest can reduce the deviation caused by higher contrast values.

When analyzing the effect that the initial ROI size has on the performance of the tracking algorithm, we devised three different sets of tests. The first test was varying the length and width of the rectangular region for a moving circular spot on a homogenous background. In the second test, various backgrounds were used, and the third analyzed real-world tracking scenarios. An example is presented in Figure 17.

We explore further the difference between the actual coordinates of ROI and the measured ones. For the case of non-simulated videos, we examined whether or not the object of interest remained within the tracking area. An example test that examines the lower and upper boundary for the area ratio K is given in Figure 17. After selecting the ROI size, we track the object’s movement for a set number of frames. We plot the values from Eq. (8) along the z-Axis depending on the object center’s initial ROI size and current frame position.

Our tests found that the mean lower boundary for the area ratio defined in Eq. (10) is 69%, while the mean upper boundary is 34%. For the proposed tracking algorithm’s future development, additional analysis on the effect of background contrast and autonomous initial ROI selection is underway.

4. Summary and Discussion

We propose a novel method for object tracking. It addresses the challenge of real-time object-tracking optic flow techniques. The method successfully applies to numerous tests and real-world data, showing its effectiveness with various examples. An essential feature of our approach is the reconstruction of global transformation parameters, mitigating the computational complexity associated with most pixel-based optical flow algorithms this way. The method can be helpful for virtual tracking by dynamically adjusting a region of interest in a static wide-angle video stream and tracking with a mechanically steerable PTZ camera.

Remote sensing and detecting adverse and potentially dangerous events is an ever-growing necessity. In certain situations, attached sensors are not the optimal or even not a possible solution. Video observation provides remote functionality, but in its commonly used operator-based form, it requires constant alertness of trained personnel. For this reason, we have established a broad program dedicated to automated remote sensing algorithms. One of the currently operational systems is dedicated to real-time detection of convulsive epileptic seizures. The results presented in this work intend to develop modules that deliver tracking capabilities and operate in conjunction with existing detection and alerting facilities.

One of the limitations of our approach is that we have used only group transformations that preserve the aspect ratio between the ROI axes, namely the translations and dilatations of the video image. As explained in the Methods, one argument for this is the potential application of PTZ cameras. A second argument related to operator-controlled settings is that standard, fixed aspect-ratio monitors render the video images. We will explore an extended version of our ROI adaptive control paradigm in a forthcoming study.

The comparison between the performance of the proposed ROI tracking method and that of other existing techniques for only one available data set is for reference only. We remind that our goal is to investigate the potential use of reconstructed optical flow group velocities for autonomous ROI tracking. To the best of our knowledge, no other published methodology provides such functionality. Even if other algorithms produce better tracking results in particular applications or according to some specific performance criteria, implementing them in our integrated system would require higher, additional computational resources. In our modular approach, the optical flow reconstruction is done for detecting epileptic motor fits, and applying it to other modules, such as ROI or PTZ tracking, involves minimal added complexity. This said, the illustrative comparison suggests that the proposed technique may be generally competitive with other tracking methods, especially if the required computational resources are considered.

Further limitations and restrictions of the method related to the velocity of the tracked object and the initial size of the region of interest are examined and listed earlier in this work. An open question that remains here is how to proceed if the algorithm "loses" the object of observation. One immediate solution is to detect the situation and alert an operator to intervene. Such an approach will, of course, undermine the autonomous operation of the system. Another possibility we are currently investigating is to introduce a dual-ROI concept where the algorithm keeps a broader observation margin that would allow for mitigating some of the limitations.

Our technique can also utilize adaptive features that provide performance reinforcement on the move while operating in real time. The adaptive extension is now considered on synthetic and real-life sequences and will be published elsewhere. Here, we note that it does not need large sets of pre-recorded training samples, as when using conventional learning techniques [14,20,21,37].

We would also point out that because the method proposed in our work is ROI-based, it allows parallel proliferation for simultaneous tracking of multiple separated objects if the computational resources permit. A typical application of such a technique would be using a wide-angle high-resolution static camera for multiple targets observation. If, however, the objects cross their positions in the camera’s field of view, rules of disambiguation should apply. This extension of the methodology goes beyond the scope of this report and is a subject of our further investigations.

Finally, we note that the method introduced in this work was primarily dedicated to the already running application for detecting adverse motion events in patients. This allowed us to restrict the optical flow reconstruction task to those transformations relevant to the PTZ camera control. The technique is, however, applicable to a broader set of situations, including applications where the camera is moving, as for example considered in [38]. In the clinical practice of patient observation, cameras moving on rails and/or poles are available. It is, however, the added complexity of manual control that limits their use. To achieve automated control in real-time, a larger set of group transformations, including rotations and shear, may be used to track and control the camera position and orientation.