Camera-Based Respiration Monitoring of Unconstrained Rodents

1. Introduction

Animal research has played a major role in many scientific breakthroughs for centuries, even though it has been a source of various ethical debates [1]. This caused governing bodies to implement laws and other regulatory means to safeguard animals in experimental settings. The European Union (EU) requires member states by its Directive 2010/63/EU [2] to apply the 3R principal proposed by Russell et al. [3] in 1959. These principals refer to Reduction, Refinement and Replacement, as a mean to minimize the use of animals in scientific studies while maximizing animal welfare. The term reduction refers to reducing the number of animals used in a study while still providing the scientific significance needed. Refinement refers to minimizing the pain, suffering, or distress introduced by animal trials. This can be achieved by using less invasive methods or improving the living conditions in terms of housing and care. Replacement refers to finding alternatives to animal testing which are similar or more effective, thus making the animal trial needless. Feasible alternatives could be using cell cultures, simulations, or human studies.

However, reality shows that not all experiments with living animals can be replaced. In 2019, the EU reported that 10.61 million animals were still used in animal trials [4], showing the great need for further refinement methods. Of these, 72% were used for research, 17% to satisfy regulatory requirements and another 6% for routine production. Most of the animals were used to enhance the understanding of the nervous system or finding treatments for diseases like cancer. Until today, researches have not been able to find adequate replacements for these kinds of animal testing, which makes the refinement and improvement of these experiments crucial.

Due to their high anatomical, physiological, and genetic similarity to humans while being small and easy to maintain, mice and other rodents are most used in research [5] and represent about half of all trial animals [4]. Cardiovascular, pharmacological, and toxicological research requires vital parameters such as heart rate (HR) or respiratory rate (RR) to assess a given theory. Currently, implanted radio transponder are the only methods to monitor these for unrestrained mice or rats [6]. Despite its ability to generate highly precise data, there are several significant drawbacks associated with this methodology. First, it requires an initial implantation surgery, which is invasive and time-consuming. Recovery time for animals to regain their normal circadian rhythms can take up to five to seven days, according to Braga and Burmeister [7]. Second, the implanted device may cause distress and discomfort, especially in small species. Braga and Burmeister also noted that the implanted device could have adverse physiological effects, such as increased volume in abdominal viscera, which can potentially compromise the movement of the diaphragm and alter breathing patterns in terms of depth and rhythm. Therefore, there is a great need for contactless and unobtrusive monitoring of techniques, which permit on the one hand to monitor continuously the laboratory animals and on the other hand to obtain objective parameters for welfare assessment.

Over the years, numerous researchers have explored monitoring RR remotely. Kunczik et al. [8] showed in 2019 that monitoring of mice and rats can be achieved using an RGB-camera while undergoing anesthesia. In this approach, RR is measured by tracking the movement of the abdominal areas, while HR is measured using a DistancePPG as proposed by Kumar et al. [9]. Another approach was presented by Takahashi et al. [10], using camera recordings of mice from below a see-through acrylic glass, monitoring and tracking hairless areas. Both approaches lack the possibility of long-term monitoring as we would like to see due to the animals being restrained or in a specialized cage with no possibility of litter or enrichment materials like nesting pads.

The current paper presents a novel approach for respiratory rate monitoring in rodents from by using visual imaging from above. In contrast to other publications that use videos of anaesthetized animals to estimate this vital parameter, our focus here is to demonstrate the capability of the presented algorithm in extracting this from moving animals.

2. Materials and Methods

The proposed algorithm is multi-step approach for monitoring respiration in RGB-video of unconstrained rats, as illustrated in Figure 1. This paragraph provides a brief overview of all steps, which will be described in detail in the following sections. During the first step segmentation masks of images are computed from video recordings using a deep-learning algorithm to detect the respiration- associated movement. In the second step preprocessing of the segmented regions is done. In the third step, the signal is extracted. Last, the actual computation of the respiratory rate is done. As a reference, respiration signals were extracted from electrocardiography (ECG) data and used to compare the camera-based signal.

Figure 1. Key stages involved in extracting the RR from RGB-videos of rats: Video preprocessing (segmentation, preprocessing), signal extraction, and RR calculation.

Figure 1. Key stages involved in extracting the RR from RGB-videos of rats: Video preprocessing (segmentation, preprocessing), signal extraction, and RR calculation.

2.1. Segmentation

For assessing the heart rate, first a target RoI must be defined. In contrast to previous works, which mostly monitored anesthetized animals using on the upper abdomen as the region for signal extraction [8], our goal was to monitor unconstrained animals. This means that the RoI must be detected and tracked over time. Thus, the RoI was set to cover the entire chest and abdomen and was bounded by the connecting line between both upper and lower legs, which can be recorded by cameras when they are mounted above the cage.

In 2019, Wu et al. [12] published the detectron2 framework for image segmentation and object detection, which was customized for segmenting the RoI in rats in this work. Such supervised deep learning approaches need annotated image data before the training process of the neural network can be started. Therefore, images from our study (described in detail in section 3) were selected, such that 50 images were automatically extracted from each of the 39 recorded videos, beginning with images with little to no movement and then randomly sampling until the required number (50) was reached. These images were annotated using LabelMe, a project created by the MIT Computer Science and Artificial Intelligence Laboratory (Cambridge, MA, United States), which provides an annotation tool to build image databases for computer vision research. An example of an annotation can be seen in Figure 2, which was applied in RGB images. Along with the detectron2 framework, Wu et al. [12] also published pretrained models on various datasets. To begin training our network, the Mask-RCNN-R50-FPN architecture was chosen, which was pretrained on the CoCo-Dataset [12] (referenced as model-ID: 137849600). Mask-R-CNN-R50-FPN references a deep learning model for instance segmentation. As a backbone a ResNet-50 is used, consisting of 50 convolutional-layers, to extract features from the input image. These features are then used in a Feature Pyramid Network (FPN) to build multi-scale feature pyramid for improved object detection and segmentation.

Figure 2. Annotated rat images: The red area corresponds to the desired RoI (thorax and abdomen), which should be automatically identified and segmented.

Figure 2. Annotated rat images: The red area corresponds to the desired RoI (thorax and abdomen), which should be automatically identified and segmented.

To adapt Mask-R-CNN R50 FPN to the current data, minor changes were made to its architecture. Appendix A provides a complete set of changed parameters of the model architecture. The feature extraction layers of the network were frozen, and the number of RoI-heads was set to 128 to enable a batch size of 8 during training. Training was performed using a GeForce RTX 2080 Super (NVIDIA Corporation, Santa Clara, California, USA). To evaluate the neural network properly, the dataset was divided into three parts (training, validation, test), with each part containing data from a single rat. For each rat, a network was trained on the 650 annotated images per rat, validated on a second rat, and tested on a third rat. This is done to ensure that the neural network had not been exposed to any images of the animals included in the test data and thus prevent any bias to the evaluation caused by any animal specific visible features. During training, several augmentations were applied (see Appendix B for a complete set of augmentations). Applying the segmentation network to each frame of the video, results in two different outputs: a binary mask and a certainty score between 0 and 1. Detections which are exceeding a score 0.99 were defined as valid segmentations.

2.2. Preprocessing and Signal Extraction

For RR assessment from the segmented images several steps of preprocessing were performed. Based upon the binary masks, from the segmentation step, the centers of mass were computed, and each image was cropped to the extent of the bounding box of the segmentation mask, after nullifying every pixel outside the segmented area. Subsequent to obtaining all masked images of a given video, the images were shifted so that the centers of mass are overlapping for each frame in a video. The preliminary respiration signal R was obtained by computing the area of the segmentation in each image. To extract the signal, R was denoised using a linear denoising algorithm according to Nowara et al. [13], which was originally developed for denoising remote photoplethysmography signals, but should be also applicable for respiration signals, due to a similar temporal profile.

The noise signals include the linear detrended center-of-mass coordinates over time for both X-and Y-coordinates, as well as their first derivatives. The algorithm uses the disturbed signal R projected onto the noise subspace Q to compute the denoised signal Z with $\mathit{Z}=\mathit{R}-\frac{\mathit{Q}{\mathit{Q}}^{\mathit{T}}}{{\mathit{Q}}^{\mathit{T}}\mathit{Q}}\mathit{R}$. Furthermore, the resulting signal was preprocessed with a second-order Butterworth bandpass filter with a lower and upper cutoff frequency of 1 Hz (60 breaths/min) and 3.3 Hz (200 breaths/min), respectively, and clipped wherever the gradient exceeded 1.5. The clipped values were then filled by interpolating the two neighboring values of the respiration signal.

2.3. RR-Computation

Once the filtered respiration signal has been acquired, a peak detection is done to determine both in- and exhale-cycles which can later be used to compute the RR. An algorithm developed for electrical impedance tomography (EIT) by Khodadad et al. [14] was adapted for this purpose. First, the signal was detrended by subtracting the mean of a best-fit line and zero crossings in the signal were found. Second, a separate search for extreme points at both rising and falling zero crossings was performed. Third, an outlier-detection algorithm was applied to identify valid peaks based on their distance from neighboring peaks. Once the peaks have been computed, the instantaneous RR (fRR) can be calculated as the inverse of the distance between two consecutive peaks, using the equation: fRR = 60/dpeak, where dpeak corresponds to the number of sampling points divided by the sampling rate and the respiration signal fRR is given in breaths per minute (breaths/min). Figure 3 illustrates the algorithm, showing two signals an ECG-derived-respiration signal at the top and the corresponding computed RR at the bottom.

2.4. ECG Analysis and ECG-Derived-Respiration

The results were validated using ECG as the ground truth, since the radio transponder employed in the animal trial allowed for the extraction of this parameter. ECG-derived-respiration (EDR) describes the process of extracting the respiration signal from a given ECG-signal. However, to obtain the EDR-signal of interest, processing of the raw ECG signal was required.

Several methods were proposed for peak detection in an ECG-signal, such as Pan et al. [15], Vuong et al. [16], Kalidas et al. [17], Koka et al. [18], and Makowski et al. [19]. Most of these methods focus on detecting the QRS complexes of a given ECG as it is the most prominent feature. The peak detection method used was proposed by Makowski et al. [19] uses the gradients' steepness to detect QRS complexes, followed by searching the local maxima within the detected region to find the R-peak. Customization was required to enable the computation of the HR of rats, as their ECGs have a morphology that is vastly different from that of humans. . The schematic ECG of a normal human is shown in Figure 4, along with the recorded ECG of a rat.

The customization involved filtering the signal with a Butterworth low pass filter with a cutoff at 4 Hz and discarding possible artifacts resulting from a 50 Hz powerline frequency. To apply the peak detection method to rats, the kernel size for smoothing and averaging was reduced by a factor of two and four (smoothwindow=0.05s; avgwindow=0.1875s), respectively. Additionally, the minimum delay between two different peaks was set to 0.1s. The threshold for discarding a QRS complex because it is too short was set to 0.1s. An exemplary detection of the resulting R-Peaks can be seen in Figure 5.

Many methods have been proposed to extract the EDR from an ECG-signal. Sarkar et al. [20], Charlton et al. [21], and van Gent et al. [22] used simple filtering to reconstruct the respiratory signal while Kontaxis et al. [23] computed the respiratory signal from the difference between the maximum and the minimum slopes in the QRS complex. Langley et al. [24], in turn, computed the EDR signal by applying Principal Component Analysis of the global amplitude variation of the QRS complex. To receive the respiratory signal from our data, the approach from van Gent et al. [22] was used, as it was most robust especially when used on noisy signals. An EDR-signal computed with this method can be seen in Figure 6 along with its respiratory rate. Figure 7, in turn, shows the spectrum of a processed ECG-spectrum, clearly showing the respiratory rate and the first harmonic.

3. Experimental Protocol

The data used in this work is part of a larger study that adhered to the 3R principle (replacement, refinement, and reduction) to ensure ethical treatment of animals. The study followed the approved experimental protocol of the governmental animal care and use institution "Regierung von Oberbayern” (Germany, ROB-55.2-2532.Vet_02-16-105) and was conducted in compliance with the German Animal Welfare Law. All animals received humane care in accordance with the principles outlined in the "Guide for the Care and Use of Laboratory Animals" (8th edition, NIH Publication, 2011, USA).

Three male Sprague Dawley rats (360–375 g; 9–11 weeks; Envigo, Horst, The Netherlands) were included in this study. They were subjected to an operation in which ECG- and EEG-transponder (DSI-HDX02, Data Sciences International, Inc., New Brighton, MN, USA) were implanted. A detailed description about the surgical procedure was already published in 2019 by Seiffert et al. [25]. Prior and following the operation, the rats were placed into an open glass cage measuring approximately 0.30m x 0.30m and recorded using two cameras (Cam1 and Cam2). The cameras were mounted above the cage on a tripod at about 1.5 m above the bottom of the cage. The distance was selected so that both cameras could acquire the complete bottom of the cage. The experimental setup is depicted in Figure 8.

Cam1 is a long-wave infrared thermal camera (Infratec VarioCAM HD head 820, InfraTec GmbH, Dresden, Germany) with a resolution of 640 x 480 pixels, a thermal resolution of up to 20 mK, a frame rate of 60 FPS, and a dynamic range of 16 bit. Cam2 is an RGB-camera (Allied Vision Mako G-223C, Allied Vision Technologies GmbH, Stadtrova, Germany) with a resolution of 1368 x 640 pixels and a framerate of 60 FPS.

The experiment was conducted over five consecutive days, as shown in the experiment schedule displayed in Figure 9. At each measurement time (MT), two 5-minute videos were recorded with a parallel ECG recording:

• Day 1: One video recording was done for establishing a baseline and let the rats acclimate to the environment. For this recording no ECG was recorded.
• Day 2: Surgery day where the EEG- and ECG-transponder were implanted. Two recordings with all three rats were carried out: the first directly after the surgical procedure and the second approximately two hours later.
• Days 3 to 5 followed a similar schedule, with recordings starting at 9 am, 11 am, 1 pm, and 3 pm. On day 5 only the two first video acquisitions were made.

For every recording, the ECG-transponder had to be activated using a magnetic switch. Shortly afterwards, the camera recordings were started simultaneously for both cameras. After 5 minutes of recording time, the cameras switched off automatically, followed by activating the magnetic switch again to turn off the transponder. This allowed the recording of 13 videos for each rat, with 5 minutes each, totaling to 39 videos (in total 195 minutes of video recordings). All videos were captured in raw format, without any compression.

After the experiment, the animals were euthanized with intraperitoneal sodium pentobarbital injection (600 mg/kg Narcoren®, Merial GmbH, Hallbergmoos, Germany).

4. Results

4.1. Reference Respiratory Rate

Figure 10 shows the RR derived from the ECG for each measurement time point, as well as a box plot diagram showing the variation of the ECG-derived RR for each animal. Looking at the results, it can be observed that the RR ranges from 79.08 breaths/min to 98.87 breaths/min. On average 92.09 breaths/min were recorded with a standard deviation of 4.23 breaths/min. A detailed list of respiratory rates for all measurement time points is reported in Table 1.

4.2. Segmentation

The neural networks were trained on the images of one rat each, over the time of 100,000 iterations. Thus, leaving the images of the other two rats for validation and testing. Throughout the training process, the weights of the neural network were saved periodically every 10,000 steps and validated on the validation set, as is shown in Figure 11. The figure is split into three parts, showing the validation losses, intersection over union (IoU) for the detected bounding boxes and IoU of the segmentation masks for each of the three trained networks over time. At the end of the training process the network-weights with the smallest validation loss were selected for the evaluation on the test set.

Intersection over union is defined as the area of overlap divided by the area of union IoU = Aintersection/Aunion. Overall, the segmentation on the test data resulted in an average IoU of 87.75% ± 5.04% for the segmentation masks and IoU of 82.52% ± 6.69% for the bounding boxes. Even though the networks were trained on different animals only small differences can be seen in the IoU scores. Table 2 shows the detailed results for the two IoUs, along with the subjective certainty score computed by the network, for all three rats, along with the average.

4.3. Respiratory Rate

In the left part of Figure 12 the EDR (blue) can be seen together with the RR computed from RGB-videos (orange) for each measurement time point. In turn, the right part is showing the variation of the EDR and camera-based RR for each animal. In addition, table 1 shows the RR for each video that was analyzed and average RR of the reference. As can be observed in the table, the relative error averaged 5.47% while the absolute error was 4.95 breaths/min.

5. Discussion

The aim of this research paper is to assess the feasibility, and accuracy of monitoring RR in unrestrained, awake laboratory rats using visible imaging. This is of particular interest considering that previous approaches have only been performed with sedated animals, which does not correspond to reality for most respiratory monitoring situations.

The results confirm the successful performance of the segmentation and tracking algorithm, it accurately identified the thorax and abdominal area as the RoI and effectively tracked them, achieving an IoU of the segmentation mask on average of 87.74%. Unfortunately, due to the absence of enrichments in the open glass cage, image occlusion testing could not be carried out. However, based on the inherent nature of the algorithm, we have strong confidence in its ability to perform effectively even when the animal is occluded and reappears in the image. The respiratory waveforms were extracted by leveraging the cyclical changes in the size of the area of the RoI caused by the expansion and contraction of the thorax during the respiratory cycle. Despite the presence of challenging conditions, such as motion artifacts caused by the animal's movement in the cage, the RR could still be extracted with a high degree of accuracy from the videos, with the absolute error averaging 4.95 breaths/min. Still, the error could be further minimized by reducing the overall coverage. In this work, all available video sequences were used for RR estimation and evaluation. Therefore, the animal movement lead to movement artifacts and thus to higher errors between reference and RR computed from visual imaging. Additionally, ECG-derived RR rate is not the best ground truth as it is very prone to motion artifacts. Varon et al. [26] also reported that EDR is quite prone to errors by noisy ECG signals. This is caused by faulty peak detection propagating into the respiration signal. Nonetheless, alternative gold standard methods, such as respiratory belt transducers, require the animal to be restrained during the RR measurements.

There are other studies in the literature that aimed to extract the respiratory waveform/RR from rats noninvasively. Wang et al. [27], Guan et al. [28] used humidity sensors to evaluate the RR of rodents, but both methods require the animal to be restrained. These studies primarily focus on describing the sensors themselves and the extracted respiratory waveform but lack comprehensive investigations and comparisons with a reference/ground truth. Esquivelzeta Rabell et al. [29] and Kurnikova et al. [30] used camera-based methods to monitor respiration, namely thermal and visual imaging. In these studies, the focus was not on the RR itself, but rather on the waveform of the respiratory curve extracted from the temperature variation around the nostrils to analyze exploratory sniffing. As a result, the parameter RR was not calculated further. The algorithms used required a close-up view of the animal's nostrils with minimal motion involved. In 2019, Kunczik et al. [8] extracted RR from six anesthetized laboratory rats. The results have demonstrated excellent algorithm performance, with a mean root-mean-square error of 0.32 breaths/min. It is worth highlighting that the animals were under anesthesia during the study, and thus the influence of motion artifacts on the algorithm performance was not tested. In a study by Anishchenko et al. [31], RR of laboratory rats during sleep was remotely measured using a radar, webcam, and thermal camera. Yet, no reference for validation purposes was acquired, which makes a direct comparison with the present approach unfeasible.

While the tests in this study were conducted on rats, the algorithm developed can potentially be applied to other rodents such as mice and hamsters. Even though, retraining of the tracking algorithm would be necessary, along with minor adjustments, such as modifying the parameters of the temporal filter to adapt to the expected RR range of the specific animal species.

In relation to the presented study there are some limitations that should be discussed as they may have influenced the results. First, the similar colors of animals and background (both white) might have impaired the algorithm and most probably decreased the overall accuracy, as the contrast between both is very low. Moreover, when considering the approach for denoising the respiration signal, it solely focuses on the general relative movements and does not consider movements such as scratching or sniffing during the denoising process, which could potentially affect the accuracy of the results. Inaccuracies of the tracking might have also contributed to more noise in the respiration signal and thus a smaller signal-to-noise ratio. To further enhance the results, a dynamic assessment of the exposure time setting for the camera depending on the illumination of the RoI could be beneficial. This assessment would involve adjusting the exposure time based on the RoI rather than the overall lighting environment. By tailoring the exposure time to the specific RoI, more accurate and precise measurements could be obtained, leading to improved outcomes. Another possibility to improve the results, and thus the overall accuracy, would be to decrease the coverage of the algorithm, i.e., by considering only those videos sequences in the extraction where no movement is present. However, this would implicate that continuous monitoring would no longer be possible. . In this context, the question arises whether continuous monitoring is really indispensable in laboratory research or whether fewer measurements, for example one measurement per hour, would be sufficient. Obtaining a short video sequence (e.g., 10-20 seconds) of motionless animals could potentially be adequate for this purpose. This could potentially minimize the monitoring burden while still providing sufficient data for analysis, depending on the specific research objectives and requirements. Further investigation and validation would be necessary to determine the optimal frequency and duration of measurements for the specific research context. Another potential limitation is that the current segmentation algorithm is not real-time capable, but this could be improved with a different architecture. If continuous monitoring is not necessary, then the algorithm does not necessarily need to be real-time capable.

Overall, the proposed algorithm can evaluate RR of unconstrained rodents properly. Further studies will focus, on the application of the developed methods in a home cage scenario, to assess the feasibility of continuous long-term monitoring and the robustness over a wider range of respiratory rates.

6. Conclusions

Until today it was not possible to replace animal research entirely in medical and biological science. Therefore, the need for further refinement of the experiments is significant. Vitals signs like the respiratory rate are mostly monitored by using ECG-implants. Until now camera-based methods only allow monitoring the respiratory rate in anesthetized, thus a new method was proposed for unconstrained and moving animals. The respiratory rate is analyzed by looking cyclical expansion and contraction of the rats’ thorax/abdominal region. Compared to the EDR a relative error of 5.47% could be achieved, while the IoU of the segmentation mask of the thorax region averaged to 87.74%.

Improvements and further experiments are still needed to evaluate the performance of the algorithm when animals are occluded, furthermore a higher range of respiratory rates is needed to evaluate the robustness of this approach. This could enable a fully automatic camera-based monitoring of rodents, reducing the need of implanted transmitters and thereby surgeries in animal experiments.