Heart Rate Estimation by Video-Based Reflec-tance Photoplethysmography

1. Introduction

This chapter establishes the aim and rationale of the research reported in this thesis, discusses the relevant theory and associated problems, and enumerates research objectives. It concludes with an outline of the remainder of the thesis.

1.1. Aim of Thesis

The primary aim of this thesis is to implement an algorithmic framework for video-based non-contact heart rate (HR) estimation using the principle of reflectance photoplethysmography (rPPG). It is intended as a foundation for further research devoted to the optimization of rPPG performance metrics for dark-skinned subjects (for which performance is notably poor).

1.2. Rationale for Non-Contact Heart Rate Monitoring

Non-invasive measurement and monitoring of HR is important in clinical settings, as it plays a critical role in diagnosing a range of health conditions and evaluating general well-being. At present, the gold standards for HR measurement are all based on electronic or optical sensors (e.g., ECG and pulse oximeter) which require skin contact (Kumar, Veeraraghavan, and Sabharwal 2015).

Apart from basic inconvenience and discomfort, contact sensors have proven problematic in certain scenarios – they cannot be used when mechanical isolation of the patient is imperative (burn victims, patients with shaky hands and feet), cause skin damage to premature babies in the ICU and increase the risk of spreading infections (Kumar, Veeraraghavan, and Sabharwal 2015). The pulse oximeter, a representative contact sensor (widely used in clinical applications for measuring HR and/or blood oxygen saturation), presents well-known problems. For example, its utility is significantly impaired for long-term monitoring outside the ICU, where motion artefacts lead to repeated false alerts (Tarassenko et al. 2014). The spring-loaded clips in conventional finger oximeters have also been shown to affect the photoplethysmography (PPG) waveform (Teng and Zhang 2004).

Considering these problems, a method for HR monitoring which obviates the need for electrodes or sensors attached to the patient is a long-awaited prospect. Non-contact methods for HR monitoring using a camera have recently emerged as a viable alternative (Verkruysse, Svaasand, and Nelson 2008, Poh, McDuff, and Picard 2010, Sahindrakar, de Haan, and Kirenko 2011). For example, it is now possible to determine the pulsatile blood volume variations associated with the cardiac cycle from facial videos of human subjects captured up to 2 metres away (using only ambient lighting) (Tarassenko et al. 2014). Variations in blood volume produce corresponding variations in the skin reflectance which can be extracted from video as a raw rPPG signal and processed to reveal HR.

Camera-based HR monitoring has numerous applications – from continuous monitoring of patients in the ICU to monitoring in more mundane scenarios like working in front of a computer (Kumar, Veeraraghavan, and Sabharwal 2015). It is also economically viable, with the cost of digital cameras progressively declining as the technology becomes more omnipresent.

HR monitoring via video camera is not without its problems. Present algorithms demand the restrictive condition that a person be effectively at rest facing a camera to ensure accurate measurements (Kumar, Veeraraghavan, and Sabharwal 2015). Accordingly, the technique is highly prone to motion-induced artefacts, particularly when using webcams in ambient light (Bal 2015). However, unlike the motion-based problems in contact monitoring, the use of a camera indicates when the subject is moving and thereby facilitates remedial action (Tarassenko et al. 2014). Another significant drawback is that the technique does not perform well for subjects with dark skin tones and/or under low lighting conditions (Kumar, Veeraraghavan, and Sabharwal 2015).

1.3. Importance of the Heart Rate

The heart is a muscular organ which acts as a pump that circulates oxygen and nutrient-carrying blood around the body and assists in metabolic waste removal. Accordingly, the HR is one of the most important physiological signals of the human body. It is defined as the number of times the heart contracts per unit time and is often expressed in beats per minute (bpm). The normal resting HR for an adult human is 50 – 90 bpm (Spodick 1993), though it is typically lower during sleep and significantly higher during exercise (> 160 bpm) (American Heart Association 2015). Heart rates below and above the normal range are referred to as bradycardia and tachycardia, respectively (Texas Heart Institute).

HR is closely linked to the of viability of the cardiovascular system and the medical status of a person and thus HR measurement has long been used in medicine for non-invasive evaluation of wellness (Mensink and Hoffmeister 1997). An elevated resting HR is correlated with increased risk of cardiovascular (Dyer et al. 1980, Kannel et al. 1987, Thaulow and Erikssen 1991) and all-cause mortality (Hjalmarson et al. 1990, Gillman et al. 1993), as well as cancer mortality (Persky et al. 1981).

Apart from illness, there are many other determinants of HR, both adjustable (e.g., physical activity, smoking, alcohol, stress, posture) and non-adjustable (e.g., age, sex, race) (Valentini and Parati 2009). HR differs significantly for different activities (e.g., sleeping versus exercising) and can change rapidly with postural changes (Borst et al. 1982) and involuntary events such as cardiac arrhythmia and cardiac arrest (National Heart , National Heart). It increases steadily during exercise and gradually returns to the resting value after exercise. It is a well-known indicator of fitness. Generally, the fitter a person, the lower the HR.

The gold standard for measuring HR is electrocardiography (ECG) (Phua et al. 2012), which involves measuring the electrical impulses generated by the heart muscles with electrodes attached to the skin. Other methods include photoplethysmography (used in this thesis), Doppler ultrasonography, use of a piezoelectric transducer, and ballistocardiography (Lindqvist and Lindelow 2016).

1.4. Fundamental Theory of Photoplethysmography

The fundamental theory of photoplethysmography – that pulsatile variations in tissue blood volume modulate the transmission or reflection of visible (or infra-red) light – has been known since the 1930s (Tarassenko et al. 2014). The blood-oxygen transport protein, haemoglobin, is strongly absorptive in visible and near-infrared light. Accordingly, changes in blood volume during the cardiac cycle alter the transmission (or reflection) of light in synchrony with the heartbeat. It is this variation in light transmission or reflection which constitutes the (raw) PPG signal. HR may be extracted from the PPG signal in either the time domain (by measuring the time between consecutive peaks or troughs of the PPG waveform) or the frequency domain (using spectral analysis techniques).

Figure 1.1. AC component of PPG signal compared to ECG. Allen (2007). Reproduced by permission from IOP Science. 

Figure 1.1. AC component of PPG signal compared to ECG. Allen (2007). Reproduced by permission from IOP Science. 

A PPG signal can be obtained from any vascular area of the skin using optical probes in either reflection or transmission mode (Hertzman and Spealman 1937, Nijboer, Dorlas, and Mahieu 1981). After the light is transmitted through or reflected from vascular tissue, it reaches a photodetector and is transduced into current variations which are then amplified and sampled to generate the PPG signal. The PPG signal is made up of two components – a pulsatile AC component and a slowly varying quasi-DC component (Severinghaus 2007, Challoner and Ramsay 1974). The AC component originates from the pulsatile arterial blood (synchronous with HR) while the quasi-DC component is related to non-pulsatile arterial blood, venous blood and other tissues (including bone) (Shao 2016, Saquib et al. 2015). The quasi-DC component also accounts for low frequency events such as respiration, vasometric activity and thermoregulation (Garcia 2013).

Figure 1. 2. AC and DC components of PPG signal.

Figure 1. 2. AC and DC components of PPG signal.

PPG is most commonly associated with pulse oximetry – a technique for estimating the arterial blood oxygen saturation, SaO₂.

1.5. Pulse Oximetry—An Example of Contact PPG

Pulse oximetry was developed in the 1970s as a non-invasive means of estimating SaO₂ by measuring the pulsatile variations in the intensity of light transmitted through tissue at two different wavelengths (Severinghaus and Honda 1987). Pulse oximeters operate using red and infrared wavelengths and are sensitive to changes in oxygen saturation because of measurable absorption differences between oxygenated and deoxygenated haemoglobin at these wavelengths (Njoum and Kyriacou 2013).

Most pulse oximeters are designed to operate in transmittance mode (Chan, Chan, and Chan 2013), with a probe attached to the finger or earlobe. Reflectance-mode pulse oximeters (with light source and detector on the same side) are less common and operate by detecting light that has back-scattered from the tissue. They are mainly used in forehead pulse oximetry (Agashe, Coakley, and Mannheimer 2006) for clinical applications involving peripheral shut-down, where transmittance-mode pulse oximeters malfunction due to reduced blood flow in the extremities (Tarassenko et al. 2014).

Figure 1. 3. Transmission (left) and reflection (right) mode contact pulse oximetry. Tamura et al. (2014).

Figure 1. 3. Transmission (left) and reflection (right) mode contact pulse oximetry. Tamura et al. (2014).

1.6. Non-Contact Photoplethysmography

In non-contact PPG, there is no mechanical coupling between sensor and subject. Non-contact PPG, like pulse oximetry, may operate in transmission or reflection mode. This thesis implements video-based non-contact reflectance photoplethysmography, with the camera operating as the sensor of reflected light. Figure 1.4 shows a representative setup for HR estimation by video-based rPPG.

Figure 1. 4. Representative setup for video-based rPPG. Poh et al. (2010).

Figure 1. 4. Representative setup for video-based rPPG. Poh et al. (2010).

1.7. Problems with Video-Based rPPG

The performance and applicability of video-based rPPG is challenged by three main problems:

• Intrinsically low signal strength
• Motion artefacts
• Noise due to fluctuations in illumination.

The low signal strength arises from the fact that the blood volume perfusing the skin is very small (only 2 – 5% of body total), with an even smaller volume (5% of skin volume) in sync with the HR (Kumar, Veeraraghavan, and Sabharwal 2015). Accordingly, the variations in subsurface skin reflectance due to the cardiac-synchronous fraction of blood volume is also very small. The result is changes in camera-recorded light intensity that are minute relative to changes in surface reflection caused by ordinary motion. Under life-like conditions, this represents patently low signal-to-noise ratio (SNR), which is further degraded for dark skin tones (poor reflectance due to increased melanin) and/or low lighting conditions (Kumar, Veeraraghavan, and Sabharwal 2015).

The motion artefacts and noise contamination are closely related to low signal strength. As discussed, small movements of a subject can lead to large variations in surface reflection which introduce high levels of noise into the weak rPPG signal. Background variations in illumination result in similar contamination. Thus, while relatively noiseless data (with artificially high accuracy) may be obtained under controlled settings, the established techniques significantly underperform in more realistic situations such as monitoring of patients on dialysis (Tarassenko et al. 2014). In such life-like settings, there may be sudden and significant variations in the intensity of ambient light, as well as rigid motions of the head and other less subtle motions (such as changes in facial expression) which significantly compromise the utility of rPPG.

Many investigations have attempted to counter the effects of motion. Poh et al., for example, attempted to reduce motion artefacts by using automatic face detection to track the face from frame to frame (2010). They, however, limited the motion to slow and uniform head swings. Still, the technique was fraught with difficulties in continuously detecting moving faces, with the researchers reporting a large number of false-negatives.

A comparable method computes 2D shifts in the location of the face between frames and uses image correlation to model the motion (Yu et al. 2011). This method also has its limitations as it can only capture the basic translational motion of the face and is unable to effectively compensate for more life-like motion such as the turning or tilting of the face, or the change in the contours of the face that come with smiling or talking.

1.8. Research Objectives

The primary research objectives of this thesis are to:

• Implement an algorithmic framework for HR estimation by video-based reflectance photoplethysmography
• Investigate the effect of the presence/absence of dedicated lighting on the SNR of rPPG signals
• Investigate the effect of choice/size of the region of interest (ROI) on the SNR of rPPG signals
• Compare the SNR of green rPPG signals with that of rPPG signals based on luminance
• Investigate the viability of the use of the summary autocorrelation function (SACF) as a technique for improving the SNR of rPPG signals
• Implement the HR estimation algorithm into a graphical user interface (GUI)
• Prepare a laboratory practical centred on HR estimation by video-based rPPG

1.9. Thesis Outline

This thesis is organised as follows:

Chapter 2 presents a literature review of previous work on various implementations of HR estimation by reflectance photoplethysmography and its precursors. Chapter 3 itemizes the materials used for carrying out this investigation and discusses the experimental setups. Chapter 4 continues the discussion on materials and methods in terms of the implementation workflow and algorithms. Chapter 5 presents the results of the study, while Chapter 6 discusses the results. Chapter 7 summarises the findings, draws conclusions and makes recommendations for future work.

2. Literature Review

This chapter reviews the literature relevant to an implementation of video-based rPPG.

2.1. Previous Work

The first non-contact HR estimation system was designed by Da Costa using two optical methods to record the cardiac-synchronous deflection of a human vein (1995). In the first method, the skin in the region of the vein is illuminated using a 2 mW HeNe laser and the reflected speckle pattern is recorded with a TV camera. The speckle contrast is then computed across frames and represented as a time series (from which HR can be determined). In the second method, the position of a light spot formed by laser light reflected from a small mirror glued to the skin is recorded by a CCD camera and plotted as a function of time. Since the displacement of the mirror is synchronized with that of the vein, HR can be extracted from the fluctuations of the spot position. The investigation, however, failed to report quantitative results or any correlation with reference ECG measurements (Kumar, Veeraraghavan, and Sabharwal 2015).

Following Da Costa’s initial attempt, further progress was moderate until Wieringa et al. showed, for the first time, that it is possible to derive rPPG signals from backscattered light detected by a monochrome CMOS camera (2005). The signals were acquired from the left inner arm (near the wrist) of seven volunteers (positioned 0.7 m away from the camera) by sequential illumination with non-coherent light at three different wavelengths. All signals contained a pulsatile component correlated with breathing and a smaller amplitude component at the cardiac frequency.

In a similar investigation, Humphreys et al., using a CMOS camera, obtained rPPG signals from light reflected off the inner arm of ten volunteers (2007). An array of LEDs (at two different wavelengths – 760 and 880 nm) served as the light source, allowing two multiplexed rPPG signals to be obtained concurrently at a rate of 16 frames per second (fps). HR estimates from the rPPG signals showed excellent agreement with reference pulse oximeter values.

Takano et al. also showed that it is possible to simultaneously extract HR and BR using a CCD camera (2007). They recorded images of a selected region of a subject’s skin and then determined the variations in the average intensity of the ROI over a period of 30 s. The intensity data was then processed using standard MATLAB functions for filtering and spectral analysis. Both HR and BR showed very high correlation with reference values.

Another noteworthy study reported a non-contact method for determining HR based on the analysis of thermal images (Garbey et al. 2007). The study revealed that the temperature of blood vessels is modulated by pulsatile blood flow, resulting in a thermal signal from which HR can be extracted after appropriate processing. The thermal signal also yields quantitative information about respiratory function and blood flow velocity.

Perhaps the most important recent development in camera-based rPPG is the ability to monitor vital signs using facial videos under ambient light (Verkruysse, Svaasand, and Nelson 2008, Poh, McDuff, and Picard 2010, 2011, Kwon, Kim, and Park 2012, Balakrishnan, Durand, and Guttag 2013). Verkruysse et al. showed, for the first time, that rPPG signals could be acquired from face videos in ambient illumination using a consumer grade digital camera positioned greater than 1 metre (2008). The video analysis involved the selection of ROIs in each frame of the face (usually the forehead) and the calculation of the mean pixel intensity of the ROIs to form the raw rPPG signal. The raw signal was bandpass filtered using a fourth-order Butterworth filter with cut-off frequencies of 0.8 – 6 Hz (corresponding to heart rates of 48 and 360 bpm). The filtered signals were then subjected to FFT over a 10 s window and the HR was determined from the frequency content.

The Verkruysse study found the face (particularly the forehead) to be the best surface for extracting rPPG signals because of improved SNR. The study also demonstrated that the green channel of an RGB camera outperforms the red and blue channels for detecting HR and BR. This is not surprising since it is known that the absorption spectra of haemoglobin and oxyhaemoglobin – the primary color agents in blood – have absorption peaks in the passband range of the green filters in color cameras (Kumar, Veeraraghavan, and Sabharwal 2015). A supplementary discovery is that cyan, orange and green (COG) channels work better than RGB (McDuff, Gontarek, and Picard 2014).

The work of Poh et al. (2010) represents a notable attempt to improve on the foundation laid by Verkruysse et al., particularly with respect to motion robustness. Using the webcam of a Macbook Pro, positioned about 0.5 m from the subject, the researchers simultaneously detected the HR, HRV and BR of 12 volunteers with varying complexion. The studies were conducted indoors at day-time, with varying intensities of ambient illumination. The improvement in motion robustness was achieved using continuous automatic face detection based on the Viola-Jones and Lienhart-Maydt face detectors (Viola and Jones 2001, Lienhart and Maydt 2002), and the JADE algorithm for independent component analysis (ICA) (Cardoso 1999) (subsequent investigations have reduced the computational cost of continuous face detection by performing initial face detection followed by the Kanade-Lucas-Tomasi (KLT) algorithm (Lucas and Kanade 1981) for motion tracking (Tarassenko et al. 2014, Li et al. 2014, Kumar, Veeraraghavan, and Sabharwal 2015, Rahman et al. 2016)). Following face detection, the average intensity of the ROI was computed across frames for each RGB channel. The RGB traces were then subjected to ICA to yield three independent signals. The rPPG signal was extracted from one of these independent signals and then Fourier-transformed to determine the peak of maximum amplitude within the range of 0.7 – 4 Hz. This was taken as the HR frequency. The results showed that ICA-decomposed signals can achieve higher accuracy for estimating HR compared to the green signal.

A later study, however, yielded results which contradicted that of Poh et al. with respect to the superiority of ICA separation (Kwon, Kim, and Park 2012). Comparing HR estimates extracted from green signals versus ICA separated signals (from face videos captured by a smartphone camera), the researchers found that ICA slightly underperformed. Poh et al. later designed an improved technique which achieved very accurate HR estimation. It involved the application of several temporal filters before and after ICA (2011).

Sahindrakar et al. (2011) proposed and assessed several improvements on various aspects of framework established by Poh et al. (2010, 2011) (ROI selection, tracking, signal pre-processing etc.), with an overall view to reduce motion artefacts. Though they succeeded in improving SNR, motion artefacts still remain an issue. Particularly interesting among the improvements was the demonstration that the three channels of the RGB space may be combined in various ways to yield rPPG traces of superior SNR compared to the green signal.

In terms of understanding the nature of the rPPG signal, the work of Balakrishnan et al. requires special mention (2013). According to their research, facial rPPG signals are due not only to volume changes in facial blood vessels (Poh, McDuff, and Picard 2010) or color changes in these vessels (Wu et al. 2012), but also involves a third component – ballistocardiographic changes. These are motion-induced changes hypothesized to originate from cyclical movement of blood from heart to head, giving rise to oscillatory motion of the head at the cardiac frequency.

Other noteworthy experiments using digital cameras for vital signs estimation include Yu et al. (2011), Lewandowska et al. (2011), Holton et al. (2013), and Shao et al. (2016).

A chief limitation of the systems discussed thus far is that they are not designed for real-time implementation and are consequently of limited utility in applications such as personal health care and telemedicine. Within the last five years, research has continued to improve techniques, with an increasing focus on making camera-based vital signs monitoring more suitable for real-time applications (Datcu et al. 2013, Zhao et al. 2013, Rahman et al. 2016).

A real-time application of considerable value is the vital signs monitoring of drivers (Rahman, Barua, and Begum 2015, Rahman, Begum, and Ahmed 2015). One implementation used an iSight webcam to capture facial images, separated the image data into three the RGB channels and performed further processing for HR and BR estimation (Zhang et al. 2014). A similar system was developed for continuous monitoring of the HRV of drivers under real-world driving conditions (Guo, Wang, and Shen 2014).

Another significant trend in camera-based vital signs monitoring is a turn towards integration with smartphone technologies (Papon et al. 2015). Apart from monitoring based on the finger in contact with the phone camera (Pelegris et al. 2010, Peng et al. 2015), smartphone systems have also been developed for remote monitoring via facial videos (Kwon, Kim, and Park 2012, Jiang et al. 2014).

The camera-based vital signs monitoring studies discussed thus far originate primarily from the United States. Paralleling work in the US is work by Phillips Research Laboratories in Europe. They have developed similar FFT-based methods for estimating HR from remotely derived PPG signals and BR from motion analysis. This culminated in the release of a software application for the iOS operating system in 2011 (Tarassenko et al. 2014).

There is also significant work at Philips devoted to minimizing the effect of motion artefacts – for example, by using multi-spectral illumination and adaptive filtering techniques (Cennini et al. 2010). There is also focus on issues of motion estimation and compensation (Schmitz 2011, Sahindrakar, de Haan, and Kirenko 2011), with a recent paper dealing with problems related to periodic motion during exercise (Haan and Jeanne 2013).

3. Materials and Methods 1: Experimental Setup

This chapter describes the materials and methods used in this thesis. It begins with an itemized description of materials, followed by a discussion of experimental setup. It also describes the various ROIs used for rPPG signal extraction and sets up a classification for these signals based on choice of ROI, color space component and spectral analysis technique (FFT or Welch periodogram). It concludes with definitions of the performance metrics of the rPPG signals – namely mean error, root-mean-square-error (RMSE) and mean SNR.

3.1. Research Materials

Dell Inspiron 14-3425 PC: for video capture via 0.9-megapixel built-in webcam

HP Pavilion 15 Notebook PC: for signal processing

MATLAB R2017: for coding signal processing algorithms and GUI

COHFACE dataset: an online dataset of 160 facial videos (from 40 healthy subjects under natural and studio lighting) and corresponding physiological signals (contact PPG and respiratory)

Pulse oximeter module: for ground truth HR measurements for lab exercise (research objective 6)

Arduino microcontroller: for interfacing pulse oximeter with computer

MATLAB support package for Arduino: for obtaining ground truth signal in MATLAB environment

Photo Studio LED light: 600 lumens dedicated lighting for self-captured videos

3.2. Experimental Setup

This investigation involved two experimental setups:

self-captured videos (of dark-skinned subject) and

COHFACE database videos

3.2.1. Setup 1—Self-Captured Videos

Fifty (50) videos (each) were shot under ambient and dedicated lighting, respectively. Ambient lighting was composed of a mixture of natural and fluorescent lighting, while dedicated lighting constituted ambient lighting plus an LED light focused on the subject’s face. All videos were shot in the same room to minimize variability in system performance introduced by external factors. The timing of video capture was also chosen to keep ambient light intensity approximately constant.

For every trial, the subject was seated 0.4 – 0.5 m from the camera. The subject was instructed to remain still while facing the camera, with left arm on an arm rest and hand fixed (in order to obtain reliable ground truth information). The distance between the subject and dedicated light source (600 lumens) was approximately 1 metre. Ground truth HR was obtained manually by a medical doctor (counting radial pulse). However, for the laboratory exercise (see research objective 6), the ground truth signal can be obtained from a pulse oximeter module interfaced with a computer (via an Arduino microcontroller). In this case, HR is determined by taking the FFT of the ground truth signal over the length of measurement.

Video capture was achieved using a 0.9-megapixel webcam (built into a Dell Inspiron 14-3452 PC) at 29 fps in most cases (30 fps in all other cases), with a resolution of 720 x 1280. All videos were automatically stored in the MP4 format without any compression, resulting in 24-bit RGB video. However, 12 videos failed facial detection and it was found that conversion to AVI format resolved the problem. Accordingly, these videos were processed for signal extraction in AVI format.

3.2.2. Setup 2—COHFACE Videos

Forty-eight (48) videos from 12 subjects (2 under natural lighting, 2 under studio lighting per subject) were taken from COHFACE – an online dataset of 160 facial videos and corresponding physiological signals from 40 healthy subjects (Heusch, Anjos, and Marcel 2017). The sampling of the 12 subjects involved simply selecting the first 6 males and 6 females in the dataset. The sample was composed of 10 (fair-skinned) Caucasians and 2 (dark-skinned) Indians.

Two illumination conditions were used:

• Natural (which approximates ambient for self-captured videos): for which all lights were turned off and the blinds were opened, and
• Studio (which approximates dedicated for self-captured videos): for which the blinds were closed to minimize natural light and extra light from a spot was used to illuminate the subject’s face.1

Each subject was asked to sit still while facing a webcam for four sessions, each lasting approximately 60 s. Physiological signals (contact PPG and respiratory) were taken by a BVP sensor and respiration belt from Thought Technologies at 256 Hz and 32 Hz (respectively) and stored in HDF5 format (which is accessible by built-in MATLAB functions). Both sensors were connected to a computer running Microsoft Windows via 2-channel USB-based acquisition. Data from the sensors, together with the video stream, were synchronized and recorded using Thought Technologies’ BioGraph Infiniti Software suite, version 5.

Videos were captured using a PC with built-in Logitech HD Webcam C525 at 20 fps, with a resolution of 640 x 480 pixels. All videos were stored in AVI format without any compression, resulting in 24-bit RGB video.

3.3. ROI Description and Classification for Associated rPPG Signals

Figure 3.1 shows the various ROIs from which a total of eight (8) different rPPG signals were extracted based on choice of ROI, color space component and spectral analysis technique (FFT or Welch periodogram). For example, two types of signals are extracted from ROI A based on color space – a green signal (RGB color space) and a luminance signal (YCbCr color space). These signals are labelled SIG 1 and SIG 2. They are further distinguished as SIG 1.1, SIG 1.2, SIG 2.1 and SIG 2.2 depending on whether they are processed using the FFT or Welch periodogram. A summarized description of the ROIs and their associated signals is provided in Table 3.1. These designations apply to signals from self-captured videos. Their equivalents for the COHFACE videos are distinguished by “prime” as in the case of SIG 1.1′.

Figure 3. 1. Various ROIs used for rPPG signal extraction. Clarification: ROI B represents the union of ROI A and the small rectangle above ROI A.

Figure 3. 1. Various ROIs used for rPPG signal extraction. Clarification: ROI B represents the union of ROI A and the small rectangle above ROI A.

Table 3. 1. ROI description and rPPG signal classification.

ROI Name	ROI Description*	Associated Signals	Description of Signal
ROI A	ROI centred on Forehead xmin=xmin_bbox+(1/3)*bbox_w ymin=ymin_bbox+(1/10)*bbox_h width = (1/3)*bbox_w height = (1/5)*bbox_h	SIGNAL 1.1	Green (RGB) signal (FFT)
		SIGNAL 1.2	Green (RGB) signal (Welch)
		SIGNAL 2.1	Luminance (YCbCr) signal (FFT)
		SIGNAL 2.2	Luminance (YCbCr) signal (Welch)
ROI B	Forehead ROI 50% larger than ROI A xmin= xmin_bbox+(1/3)*bbox_w ymin=ymin_bbox width = (1/3)*bbox_w height = (3/10)*bbox_h	SIGNAL 3.1	Green (RGB) signal (FFT)
		SIGNAL 3.2	Green (RGB) signal (Welch)
ROI C	Face ROI comprises central 60% (width) and central 80% (height) of bbox xmin=xmin_bbox+(1/5)*bbox_w ymin=ymin_bbox+(1/10)*bbox_h width = (3/5)*bbox_w height = (4/5)*bbox_h	SIGNAL 3.1	Green (RGB) signal (FFT)
		SIGNAL 3.2	Green (RGB) signal (Welch)

^*x_min and y_min refer to the minimum horizontal and minimum vertical pixel positions for the ROI, respectively; x_min_bbox and y_min_bbox are the x_min and y_min for the bounding box of the face detector, while bbox_w and bbox_h are its height and width, respectively.

Table 3. 1. ROI description and rPPG signal classification.

ROI Name	ROI Description*	Associated Signals	Description of Signal
ROI A	ROI centred on Forehead xmin=xmin_bbox+(1/3)*bbox_w ymin=ymin_bbox+(1/10)*bbox_h width = (1/3)*bbox_w height = (1/5)*bbox_h	SIGNAL 1.1	Green (RGB) signal (FFT)
		SIGNAL 1.2	Green (RGB) signal (Welch)
		SIGNAL 2.1	Luminance (YCbCr) signal (FFT)
		SIGNAL 2.2	Luminance (YCbCr) signal (Welch)
ROI B	Forehead ROI 50% larger than ROI A xmin= xmin_bbox+(1/3)*bbox_w ymin=ymin_bbox width = (1/3)*bbox_w height = (3/10)*bbox_h	SIGNAL 3.1	Green (RGB) signal (FFT)
		SIGNAL 3.2	Green (RGB) signal (Welch)
ROI C	Face ROI comprises central 60% (width) and central 80% (height) of bbox xmin=xmin_bbox+(1/5)*bbox_w ymin=ymin_bbox+(1/10)*bbox_h width = (3/5)*bbox_w height = (4/5)*bbox_h	SIGNAL 3.1	Green (RGB) signal (FFT)
		SIGNAL 3.2	Green (RGB) signal (Welch)

^*x_min and y_min refer to the minimum horizontal and minimum vertical pixel positions for the ROI, respectively; x_min_bbox and y_min_bbox are the x_min and y_min for the bounding box of the face detector, while bbox_w and bbox_h are its height and width, respectively.

3.4. Mean Error, RMSE and SNR Definitions

The performance of the rPPG signals were evaluated using two agreement-based metrics – mean error and RMSE – along with the mean SNR. Mean error, RMSE and SNR are mathematically defined as follows:

$$\mathrm{M}\mathrm{e}\mathrm{a}\mathrm{n} \mathrm{E}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r} = \frac{1}{m}\sum _{i = 1}^{m}abs\left({Est}_i - {Ref}_i\right)/{Ref}_i$$

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{1}{m}\sum _{i=1}^m{\left[{Est}_i-{Ref}_i)\right]}^2}$$

where Est_i and Ref_i represent the i^th estimated and reference HR, respectively and m represents the number of HR values

$$\mathrm{M}\mathrm{e}\mathrm{a}\mathrm{n} \mathrm{S}\mathrm{N}\mathrm{R}=\sum _{i=1}^m\frac{S_i}{T_i-S_i }$$

where S_i is the sum of spectral components contributing to the i^th signal and T_i is the total energy in a spectrum of that signal (obtained by summing all the spectral components); m is the number of signals.

Mean error and RMSE measures for the self-captured videos were based not only on the maximum amplitude frequency peak but also on the signal peak i.e., the peak associated with the HR. The use a signal-peak assessment in addition to the standard maximum-peak assessment was motivated by very low SNR coupled with the observations that:

• error is (negatively) correlated with SNR
• when the max peak did not correspond to the ground truth HR, a distinct inferior peak which matched the heart rate could most often be seen in the frequency spectrum

This secondary measure afforded a more useful assessment of mean error under low SNR as it approximated what mean error would be like for high SNR signals. There was, however, some uncertainty in signal peak identification in (a few) cases where there was no distinct signal peak. Nevertheless, since most signal peaks were easily identifiable, the advantage of the method outweighed its shortcoming.

A similar problem occurred in the calculation of SNR. While calculating T is straightforward, calculating S is ambiguous because of the stated uncertainty in the determination of the signal peak. There was also the problem of specifying the bandwidth of the signal peak. Generally, the bandwidth of a signal is defined in terms of the full width at half maximum (FWHM) of its main lobe. However, it was observed that the main lobe was often flanked by noisy peaks. In such a case, calculating S translates to summing significantly different frequency ranges, depending on the level of noise flanking the signal peak. To avoid this, a fixed range of ±10 bpm was used for the calculation of S. This is comparable to the method used by Sahindrakar et al. (2011).

Mean error and RMSE evaluation for the COHFACE videos were based exclusively on maximum peak estimates, obviating the need to resolve the problems discussed above for the self-captured videos. However, in 2 (out of 48 cases), SNR was very low and warranted the evaluation of SNR based on a crude estimation of the signal peak location. Like the self-captured videos, the SNR of signals extracted from the COHFACE videos were computed using a value of S summed over a fixed range of ± 10 bpm.

4. Materials And Methods 2: Workflow & Algorithms

This chapter begins with an overview of the thesis workflow and associated algorithms, followed by a step-by-step discussion of the same. It also considers how HR is determined from the processed rPPG signals and offers a simple description of color models and spaces, with particular reference to RGB and YCbCr (the color models used in this thesis).

4.1. Overview of Workflow

The workflow adopted in this thesis (Figure 4.1) is primarily based on that of Rahman et al. (2016).

Figure 4. 1. Implementation workflow.

Figure 4. 1. Implementation workflow.

In the first block, a facial video of the subject is captured by webcam. The video frames are then fed into the next block for facial detection, ROI selection and tracking. Facial detection is performed using the popular Viola-Jones algorithm while tracking employs the equally popular KLT algorithm. A face is detected only in the first frame and is tracked across frames. This is far more computationally efficient than detecting the face in each frame. Three ROIs were investigated in this thesis – forehead ROI A, forehead ROI B (50% larger than ROI A) and ROI C (which covers most of the face). ROI selection is crucial in optimizing SNR. The use of forehead ROIs is based on the discovery by Verkruysse et al. that the forehead yields the strongest rPPG signal (2008). For each frame, the mean pixel intensity of each ROI is calculated for the relevant color component (in RGB or YCbCr space). The mean pixel intensity constitutes a single point in a vector representing the raw rPPG signal. The raw signals are pre-processed in the fourth block via detrending, filtering and normalizing algorithms. The next block converts the pre-processed time-domain signals into the frequency domain using spectral analysis (FFT and Welch periodogram). The HR is taken as the frequency of the component with maximum amplitude.

All algorithms in the workflow were programmed in MathWorks MATLAB^® – a numerical computing programming language and integrated development (IDE) environment. By way of toolboxes such as Signal Processing, Image Processing and Computer Vision System, MATLAB is especially suited for signal and image processing.

The following sections of this chapter will delineate the techniques and algorithms in each block and their relevance in implementing research objectives.

4.2. Video Input

The data input for the generation of the raw rPPG signals consisted of the frames of 60-s videos recorded at 29 fps (for self-captured videos) and 20 fps (for COHFACE videos) with a resolution of 720 x 1280 pixels and 640 x 480 pixels, respectively. Videos were stored in MP4 format (for self-captured videos), and AVI format (for COHFACE videos), with 8-bit encoding per colour channel (i.e., 24-bit RGB). The videos were captured by webcam and viewed using the Microsoft Films and TV app. See Section 3.2 for a more comprehensive description of the circumstances of video capture.

4.3. ROI Selection and Tracking

The selection of an appropriate region of interest (ROI) within an input frame is an important preliminary step in reducing the noise in raw rPPG signals extracted from video. Ideally, a ROI contains only skin pixels since only these contain information about the cardiac-synchronous reflectance variations; all other pixels contribute to signal noise. Another advantage of ROI selection is its computational efficiency as it reduces the number of pixels that must be processed. Once a ROI has been selected in the initial frame, it can be tracked across frames – a method that is both efficient and useful in reducing motion artefact. The face has proven an ideal vascularized surface for rPPG signal extraction (Verkruysse, Svaasand, and Nelson 2008). The first step in ROI selection is face detection.

4.3.1. Face Detection

Face detection is a type of object-class detection. In object-class detection, the objective is to identify every instance of an object in an image belonging to a particular class. Typical objects include pedestrians and cars. Face-detection algorithms are most often designed for the detection of frontal human faces. A widely used algorithm is the Viola-Jones face classifier (Viola and Jones 2001), which is the default option for the Cascade Object Detector in MATLAB’s Computer Vision Toolbox. The Viola-Jones algorithm is known for its robustness (very high detection rate), efficiency and utility in real-time applications. It is made up of four main components:

• Haar-like feature selection
• Integral image creation
• AdaBoost training
• Cascading classifiers

Figure 4. 2. Bounding box of face detected by Viola-Jones algorithm.

Figure 4. 2. Bounding box of face detected by Viola-Jones algorithm.

Haar-like features are digital image features used in object detection. They are so named because of their resemblance to Haar wavelets – a series of rescaled square-shaped functions which together comprise a wavelet family.

Figure 4. 3. A Haar wavelet. Wikimedia Commons contributors / CC-BY-SA-3.0.

Figure 4. 3. A Haar wavelet. Wikimedia Commons contributors / CC-BY-SA-3.0.

A Haar-like feature can be imagined as a crude approximation of the intensity profile of an image or image sub-window and attempts to capture the essential appearance attributes of an object class. For example, all human faces contain some commonalities which may be matched using Haar-like features. These include:

• Eye region darker than upper cheeks
• Nose bridge region brighter than eyes

The use of Haar-like features to evaluate these commonalities is illustrated in Figures 4.4 and 4.5

Figure 4. 4. Haar-like feature that looks like bridge of nose applied to face.

Figure 4. 4. Haar-like feature that looks like bridge of nose applied to face.

Figure 4. 5. Haar-like feature that looks like eye region applied to face.

Figure 4. 5. Haar-like feature that looks like eye region applied to face.

Haar-like feature evaluation is based on the principle of computing the difference between the sum of pixels in adjacent white and black rectangles in an image i.e., value = Σ (pixels in black area) – Σ (pixels in white area). In the original implementation of the Viola-Jones algorithm, feature evaluation was performed on 24 x 24-pixel sub-windows of an image (Viola and Jones 2001). Though the Haar-like features were limited to four types (shown in Figure 4.6), at all possible scales and positions, these features required more than 160,000 evaluations over the sub-window (far more than the number of pixels, which is 24 x 24 = 576). At 160,000 evaluations per sub-window, the summing over pixels required for each feature evaluation is computationally very expensive and of limited use in real-time applications. Viola and Jones proposed an novel solution to this problem by introducing the integral image, more generically known as the summed-area table (Crow 1984).

Figure 4. 6. Haar-like features used in the original implementation of the Viola-Jones algorithm. Prmorgan / Wikimedia Commons / Public Domain.

Figure 4. 6. Haar-like features used in the original implementation of the Viola-Jones algorithm. Prmorgan / Wikimedia Commons / Public Domain.

The integral image is a data structure which allows rapid computation of the sum of pixels in an image or image sub-region in constant time. The pixel intensity I(x,y) at any point (x,y) in an integral image is the sum of the intensity i(x,y) of all pixels above and to the left of (x, y), inclusive, in the original image. This may be mathematically expressed as:

$$\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)=\sum _{x\text{'} \le x, y’ \le y}i\left(x^{\text{'}},y^{\text{'}}\right)$$

The integral image can be computed in a single pass over an image since the value I(x, y) in the integral at (x, y) is defined recursively as:

$$\mathrm{I}(\mathrm{x},\mathrm{y})=\mathrm{i}(\mathrm{x},\mathrm{y})+\mathrm{I}(\mathrm{x},\mathrm{y}-1)+\mathrm{I}(\mathrm{x}-1,\mathrm{y})-\mathrm{I}(\mathrm{x}-1,\mathrm{y}-1)$$

Once the integral image has been set up, computing the sum of pixels for any rectangular sub-region of an image requires only four references (independent of the size of the sub-region). For a rectangular sub-region defined by coordinates A, B, C, D, the sum of pixels is given by:

$$\sum _{x_o \le x\le x_{1,}y \le y\le y_{1,} }i(x,y) = I(A) + I(D) - I(B) - I(C)$$

where A=(x₀, y₀), B=(x₁, y₀), C=(x₀, y₁) and D=(x₁, y₁)

Figure 4. 7. Intensity values of an image and its integral counterpart. I(a) + I(d) – I(b) – I(c) = 20 + 74 – 39 – 33 = 22.

Figure 4. 7. Intensity values of an image and its integral counterpart. I(a) + I(d) – I(b) – I(c) = 20 + 74 – 39 – 33 = 22.

Even with the computational efficiency afforded by the integral image, evaluating 160,000+ Haar-like features for each 24 x 24 sub-window is still computationally prohibitive for real-time facial detection. Most Haar-like features are useless for a given object class. Therefore, a method for selecting the most useful Haar-like features could significantly improve the time complexity of the detector. Viola and Jones achieved this selectivity by using a variant of a pre-existing algorithm known as AdaBoost (Freund and Schapire 1997).

AdaBoost, short for Adaptive Boosting, is an example of a boosting algorithm – a class of machine-learning algorithms than can be used to improve the performance of other so-called weak learner algorithms. Within the Viola-Jones algorithm, Adaboost both identifies the weak learners and constructs a strong learner from a linear combination of weak learners. Each weak learner is based on one of a set of best-performing Haar-like features (i.e., those that yield consistently high detection rates when applied to a face). With a boosting algorithm, the individual weak learners need only perform slightly better than random guessing to converge to a strong learner. A weak learner h(x,f,p,θ) evaluated over a 24 x 24-pixel sub-window x consists of a feature (f), a threshold (θ, which decides whether x is classified as face or non-face) and a polarity (p, indicating the direction of the inequality). Both p and θ are determined during training.

$$\mathrm{h}(\mathrm{x},\mathrm{f},\mathrm{p},\mathsf{\theta })=\left\{\begin{array}{c}1 \\ 0\end{array}\right.\genfrac{}{}{0pt}{}{ \mathrm{i}\mathrm{f}pf(x)<p\theta }{\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}}$$

A strong learner H_T (a linear combination of weak learners) selected over T trials of testing may be mathematically defined as:

$${\mathrm{H}}_{\mathrm{T}}(\mathrm{x})=\sum _{i = 1}^T{h}_i(\mathrm{x})$$

where h_i represents the i^th weak learner in the T trials.

Apart from determining the best Haar-like features, the modified AdaBoost algorithm also sets the polarity and threshold for each weak classifier. There seems to be no satisfactory solution to this problem except brute force i.e., the determination of each new weak classifier in the T trials of testing involves evaluating each feature on all the training examples (at different thresholds and polarity). The best performing feature in a testing trial is the feature which minimizes the weighting error it produces. This weighting error is a function of the weights assigned to the training images. The weight of a correctly classified image is decreased, while that of a misclassified image is kept constant. As a result, the i^th feature added to the classifier is forced to “focus” harder on the test images misclassified by the (i – 1) previous features. The weighting system is, therefore, an integral aspect of AdaBoost.

Table 4. 1. Modified AdaBoost algorithm (Viola and Jones 2001).

Table 4. 1. Modified AdaBoost algorithm (Viola and Jones 2001).

With the integral image and modified AdaBoost, the Viola-Jones algorithm seems ready for facial detection. However, computations thus far have focused on a single 24 x 24-pixel sub-window. Most images have higher resolution, meaning that the Viola-Jones detector would have to evaluate every 24 x 24-pixel sub-window of that image. This is still computationally too expensive and also very inefficient as it is known in advance that most sub-windows contain no face. This calls for a method that can streamline evaluation by discarding sub-windows with no face with minimal computational effort and spending more time on potentially positive windows. Viola and Jones resolved this problem through yet another novel contribution known as the attentional cascade.

An attentional cascade or cascaded classifier is a multi-layered classifier architecture in which each level of the classifier consists of a strong classifier prepared by the modified AdaBoost. The objective at each layer is to determine whether a given sub-window in an image is not a face or possibly a face. A sub-window which fails at any layer is immediately discarded. A simple 2-feature classifier can be adjusted to achieve 100% detection rate with a 40% false positive rate, significantly reducing the number of times the cascade must be evaluated. This classifier can serve as the first layer (or attentional operator) of the cascade, followed by a more complex layer (made up of, say, 10 features) which can perform a stronger evaluation on potentially “harder” negative-windows which survived the first-layer evaluation, and so on. A classifier built on this logic achieves slightly lower detection rates with each successive layer but is compensated by a much faster reduction in false positive rate. This relationship can be evaluated through the detection rate and false positive rate equations below:

Given a trained cascade of K classifiers, the false positive rate F of the cascade is:

$$\mathrm{F}=\prod _{i = 1}^K{f}_i$$

where f_i is the false positive rate of the i^th classifier in the cascade on the examples that it is run on. The detection rate D is:

$$\mathrm{D}=\prod _{i = 1}^K{d}_i$$

where d_i is the detection rate of the i^th detector.

As an example, a detection rate of 0.9 and very low false positive rate of approximately 6 x 10⁻⁶ can be achieved by a 10-layer classifier if each layer can achieve a detection rate of 99% (since 0.9 ≈ 0.99¹⁰) and a false positive rate of about 30% (0.30¹⁰ ≈ 6 x 10⁻⁶).

Figure 4. 8. Schematic depiction of the attentional cascade.

Figure 4. 8. Schematic depiction of the attentional cascade.

Like the weak learners in the boosting stage, cascading classifiers are trained with hundreds of positive and negative samples of the object of interest. In the case of the face, the object of interest in this thesis, training was unnecessary as the MATLAB Cascade Object Detector is pre-trained on a large database of faces.

The process of optimal training of the classifiers in the cascade is not straightforward and requires the solution of an intractable optimization problem. Viola and Jones proposed, instead, a simple framework which nevertheless yields a highly efficient and effective classifier. Under this framework, the user selects the minimum acceptable detection rate and maximum acceptable false positive rate for each classifier. Each layer of the cascade is then trained by AdaBoost with the number of features increased until the target detection and false positive rates are met for that layer. A new layer is added to the cascade until the overall target false positive rate is achieved.

4.3.2. ROI Selection

A face detected by the Viola-Jones algorithm is bounded by a rectangular bounding box (bbox) which contains skin pixels (of the face) and non-skin pixels (in the space surrounding the face). The simplest method of ROI selection sets the modest objective of eliminating non-face pixels by cropping the image to a central fraction of the width and height of the bbox. One recommendation is 60% of the width and 80% of the height (Rahman et al. 2016).

Another method involves the use of a rectangular ROI centred on the forehead. The choice of the forehead as a ROI is based on the discovery by Verkruysse et al. that it provides the strongest rPPG signal (2008). It is also supported by the work of Lewandowska et al. which used thermal images from several subjects to establish the forehead as a particularly uniform region for rPPG signal extraction (2011). Both ROI selection methods were used in this thesis. The positional specifications of the ROIs implemented using the second method are based on the recommendation of Tofighi et al. (2014).

Figure 4. 9. Various ROIs for rPPG signal extraction.

Figure 4. 9. Various ROIs for rPPG signal extraction.

Smarter ROI selection methods such as those based on skin detection algorithms are also in use (Kumar, Veeraraghavan, and Sabharwal 2015). These algorithms attempt to take advantage of the higher signal strength associated with larger ROIs, while minimizing the noise contributed by non-skin pixels and the movement on non-rigid facial structures.

4.3.2. Motion Tracking

In the generation of the rPPG signal, it is imperative that the reflectance information be extracted from the same ROI. Inability to maintain this constancy leads to signal noise due to inhomogeneous spatial distribution of skin reflectance. The ideal way of attaining this constancy is by zero relative motion between the subject and camera. However, this is obviously unrealistic, even when attempted, as small variations in the location of the face associated with processes such as breathing and the ejection of blood from the heart (as well as small movements of the camera) invariably lead to error-inducing movements of the ROI. A natural technique for addressing this motional error is to track the movement of the face across frames so that relative motion between the ROI and face is minimized.

Motion tracking could easily be implemented by detecting the face in all frames. However, this is naïve not only because it is computationally inefficient but also because it is prone to significant errors when the face position in a frame is not properly detected.

More robust tracking systems may be designed by calculating the optical flow of image pixels from frame to frame. Optical flow is defined as the pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera (Miura et al. 2017). Optical flow is best represented by a 2D displacement vector field in which every vector points from a point in a given frame to an equivalent point in the next frame. Optical flow algorithms may be classed into two categories – dense methods which compute optical flow by considering every pixel and sparse methods which make use of only select pixels. While providing greater tracking accuracy, dense methods are computationally expensive and therefore not viable for real-time applications. A very popular sparse method is the Lucas-Kanade algorithm which limits the optical flow computation to a small subset of image points known as feature points (Lucas and Kanade 1981).

The Lucas-Kanade algorithm is based on three assumptions:

• brightness constancy – the brightness of each pixel is constant between two consecutive frames.
• temporal persistence – movements of image objects are small.
• spatial coherence – neighbouring pixels belonging to a certain surface move in a similar way.

These assumptions are used to construct a system of linear equations describing the optical flow.

Deviation from any one of the three assumptions naturally affects the performance of the algorithm. Large motions, for example, cannot be captured by the optical flow. This limitation has been addressed by Bouguet in a pyramidal implementation of the algorithm (2001). With the pyramids, small motions are eliminated, and large motions are reduced to small motions. The pyramid is constructed by recursive scaling of the resolution of consecutive image frames. The number of frames over which this scaling is done is typically no more than four. If the resolution of the first image is 640 x 480, for example, then that of the next four images will be 320 x 240, 160 x 120, 80 x 60 and 40 x 30.

Figure 4. 10. Schematic depiction of pyramidal implementation of Lucas-Kanade algorithm. Tarasenko et al. (2016). Reproduced with permission from Research India Publications.

Figure 4. 10. Schematic depiction of pyramidal implementation of Lucas-Kanade algorithm. Tarasenko et al. (2016). Reproduced with permission from Research India Publications.

Even with the motion correction by Bouguet’s pyramid, the Lucas-Kanade algorithm is still far from ideal. This means that the optical flow motion estimation will not always match the actual motion of the subject. This manifests as drift in the ROI position which introduces noise in the associated rPPG signals. Garcia attempted to reduce this error by detected the face every second and readjusting the ROI (2013). However, he also reported that this technique may introduce spurious frequency information in the signal and compromise the accuracy of HR estimation.

Figure 4. 11. Tracking points (white) of the Lucas-Kanade algorithm.

Figure 4. 11. Tracking points (white) of the Lucas-Kanade algorithm.

4.4. Signal Extraction

The raw rPPG signal is extracted by averaging the pixel intensity of the ROI for each RGB color channel (or luminance channel, YCbCr) in each input frame. Each mean pixel intensity constitutes one data point in the signal vector and therefore the length of the signal is equal to the number of frames. This rPPG signal is described as raw since it is contaminated with multiple sources of noise which must be reduced before spectral analysis. Noise reduction takes place in the pre-processing stage.

Figure 4. 12. Raw red, green, blue and luminance (Y component of YCbCr) rPPG signals.

Figure 4. 12. Raw red, green, blue and luminance (Y component of YCbCr) rPPG signals.

4.5. Signal Pre-Processing

The purpose of signal pre-processing is to separate out the signal of interest from noise sources in preparation for spectral analysis (or other analytical technique) and HR estimation. The process is not optimal and some noise unavoidably remains in the extracted signal. The first stage of pre-processing is detrending, followed by filtering and normalization.

4.5.1. Signal Detrending

Detrending is used to remove undesirable trends in a signal that may affect its stationarity. A signal is described as stationary if its long-term statistics (such as mean and variance) are relatively invariant in time. Signals typically develop trends through drifting and noising of different kinds. In the case of rPPG signals, this may be caused by changes in physical parameters such as ambient light intensity and temperature. Spectral analysis algorithms typically assume that input signals are at least weakly stationary. The stationarity of an rPPG signal may be significantly compromised in the presence of motional noise. In this research, detrending is performed using the highly recommended smoothness priors approach (SPA), with smoothing parameter λ = 10 and cut-off frequency f_c = 0.059 Hz (Tarvainen, Ranta-aho, and Karjalainen 2002). SPA works by removing the low frequency aperiodic trend component from an input signal, leaving behind a nearly stationary signal of interest.

Figure 4. 13. Detrended red, green, blue and luminance rPPG signals.

Figure 4. 13. Detrended red, green, blue and luminance rPPG signals.

4.5.2. Signal Filtering

Signal filtering was performed with a fourth-order Butterworth bandpass filter with cut-off frequencies of 0.8 and 6 Hz, following the method of Tarassenko et al. (2014). Signal filtering amounts to a denoising process as it eliminates all frequency components outside the bandpass interval.

Figure 4. 14. Filtered red, green, blue and luminance rPPG signals.

Figure 4. 14. Filtered red, green, blue and luminance rPPG signals.

4.5.3. Signal Normalization

The last stage of pre-processing normalizes all signals of interest to zero mean and unit variance according to the method mentioned by Cochran (1988). The normalized signal X_i given by:

$$X_i(t) = \frac{Y_i(t) - {\mu }_i}{{\delta }_i}$$

for each i = R, G and B signals where μ_𝑖 is the mean and 𝛿_𝑖 is the standard deviation of signal 𝑌_𝑖.

Apart from establishing signal comparability, normalization improves algorithmic stability and is a necessary preparatory step for signal processing by procedures such as PCA and correlation analysis.

Figure 4. 15. Normalized red, green, blue and luminance rPPG signals.

Figure 4. 15. Normalized red, green, blue and luminance rPPG signals.

4.6. Spectral Analysis

An intuitive approach for HR estimation would be counting the number of peaks in the time-domain rPPG signal. However, this method is error prone for several reasons, primarily:

• spurious peaks presented by the low frequency quasi-DC component of the rPPG signal (it is the AC component which contains the signal of interest)
• spurious peaks created by residual noise left after the denoising of the pre-processing stage

These problems are either eliminated or minimized in the frequency domain and therefore HR estimation is best handled by spectral analysis.

4.6.1. Fast Fourier Transform (FFT)

The Fourier Transform is a popular method for spectral analysis i.e., the conversion of a signal from the time domain to a representation in the frequency domain. The Fourier transform may be continuous or discrete. Since the rPPG represents a discrete signal, Fourier analysis of the rPPG signal employs the discrete Fourier Transform (DFT).

FFT refers to a class of algorithmic implementations of the DFT. While a naïve (direct from definition) implementation of the DFT has time complexity O(n²), FFT implementation has complexity O(nlogn) and is the most efficient known method for implementing the DFT (Cochran et al. 1967). The DFT is defined by the formula:

$$X_k=\sum _{n = 0}^{N-1}{x}_n{e}^{-i2\pi kn/N}$$

where X_k represents a set of complex numbers of cardinality k, x_n represents the input signal of length N, and k = 0, …, N – 1.

In this thesis, the FFT was implemented using the directly available MATLAB function.

Figure 4. 16. FFT of red, green, blue and luminance rPPG signals.

Figure 4. 16. FFT of red, green, blue and luminance rPPG signals.

The FFT operates on the assumption that the input signal is periodic and is therefore sensitive to discontinuities at signal end points. In the frequency domain, discontinuities translate to the problem of spectral leakage in which the power of the actual frequency is leaked to surrounding frequencies. Leakage can be reduced by minimizing discontinuities (while preserving frequency information). This can be achieved by windowing in which a function is applied to the signal to produce a smoothed redistribution of amplitude with end points equal to zero.

The window function employed in this thesis is the Hann window (depicted in Figure 4.18). This window combines acceptable frequency resolution with moderate spectral leakage. Alternative window functions such as the Blackmann and Blackmann-Harris, though better at reducing spectral leakage, do so at the cost of frequency resolution. The Hann window is highly recommended for general-purpose applications where the nature of the signal is unknown. The mathematical definition of the Hann window (of length N) is given by:

$$w(n) = \frac{1}{2}\left[1 -cos\left(\frac{2\pi n}{N - 1}\right) \right]$$

where n = 0, …, N – 1.

Figure 4. 17. Hann window function (left) and its Fourier transform (right). Olli Niemitalo / Wikipedia Commons / Public Domain.

Figure 4. 17. Hann window function (left) and its Fourier transform (right). Olli Niemitalo / Wikipedia Commons / Public Domain.

4.6.2. Welch Periodogram

A second spectral analysis technique employed in this thesis is the Welch periodogram. It is an implementation of the Fourier transform optimized for unevenly sampled signals and periodic signals of different shapes. The Welch periodogram is an improvement over the standard periodogram as well as Bartlett’s periodogram. Bartlett’s periodogram – also known as the method of averaged periodograms – operates by calculating the average of a series of periodograms derived from non-overlapping segments of an input signal. Its advantage over the standard periodogram is that it reduces signal noise at the cost of frequency resolution. Welch’s method modifies Bartlett’s by averaging over a set of periodograms obtained from overlapping segments of in input signal, yielding greater noise reduction. The procedure for the Welch periodogram involves:

• splitting of signal into overlapping segments
• windowing of overlapped segments in the time domain
• computing of periodograms for individual segments by FFT, followed by squaring of the magnitude
• averaging of individual periodograms (resulting in the reduction of the noise/variance in the power of the frequency components)

Figure 4. 18. Welch periodogram of red, green, blue and luminance rPPG signals.

Figure 4. 18. Welch periodogram of red, green, blue and luminance rPPG signals.

4.6.3. Summary Autocorrelation Function (SACF)

Autocorrelation, as the name implies, is the correlation of a signal with a delayed/lagged copy of itself as a function of the delay. The correlation measure used in autocorrelation is the Pearson correlation. Autocorrelation is important in signal processing for finding periodicities in noisy signals as correlation peaks occur at lags corresponding to signal harmonics.

Summary autocorrelation is a technique conventionally used in speech processing for identifying fundamental frequencies. The summary autocorrelation function (SACF) is the vector sum of the autocorrelation function of a signal across a range of frequencies (the bandpass in this case) (Bernstein and Oxenham 2005, Brown et al. 2006). The SACF conveniently exhibits peaks at the period of each fundamental frequency (Raghi and Lekshmi 2016).

In this thesis, it was hypothesized that the technique could produce a similar effect for rPPG signals (where harmonics may significantly contribute to signal noise).

4.7. Heart Rate Determination

In this thesis, HR is determined by spectral analysis of the rPPG signals using the FFT and Welch periodogram. Spectral analysis represents the time-domain signals in terms of their frequency components. The HR is taken as the frequency of the component with the highest power. HR (in bpm) is related to frequency (in Hz) by the simple relation:

$$HR \left(in bpm\right)=60\ast frequency (in Hz)$$

For this implementation, frequency is constrained to the interval [0.8, 6.0] Hz by a bandpass filter. This interval encompasses the range of realistic heart rates in humans.

4.8. Color Models and Spaces

This investigation employed two color models – the RGB model and (the luminance component of) the YCbCr model. A color model is an abstract formal/mathematical description of the representation of colors as tuples of typically three or four elements. The particularization of a color model to a specific interpretation of tuple components is known as a color space. For example, within the RGB color model, there are several color spaces (Adobe RGB, sRGB etc.)

4.8.1. RGB Color Model

The RGB color model is an additive color model in which the primary colors red, green and blue light are combined in various intensity weightings to yield a wide range of secondary colors. The RGB model is widely used for sensing, representing and displaying images in modern electronic devices such as TV and computers, though its theory – based on human color perception – long preceded modern electronics.

In this research, the green rPPG signal was of primary importance as seminal research by Verkruysse et al. (2008) divulged it to yield the strongest rPPG component. Signals derived from the other components – red and blue – were also stored for future analysis aimed at verifying the discovery that a signal X with higher SNR than the green signal can be obtained by combining the R, G and B traces in the mathematical relation: X = R – G + 2B (Sahindrakar, de Haan, and Kirenko 2011).

4.8.2. YCbCr Color Model

YCbCr is a family of color spaces commonly used as a color option in digital photography and video. Y stands for the luminance (approximately brightness) while C_b and C_r are the blue-difference and red-difference chroma components (chroma is approximately equivalent to the concept of hue) A key difference between the RGB and YCbCr models is that the YCbCr isolates brightness to the luminance component and hue to the chroma components while both hue and brightness are simultaneously stored in the in each of the three components of RGB.

In this thesis, in addition to traces from each of the three RGB components, a signal trace was generated from the Y component of YCbCr. The primary motivation for such a signal was to investigate whether there was any advantage in using a component based purely on brightness given that the rPPG signal is based primarily on light intensity variations.

The YCbCr color space may be mathematically derived from the RGB color space using the following transformations (Poynton 2012):

$$Y=16+\frac{65.481R}{256}+\frac{128.553G}{256}+\frac{24.966B}{256}$$

$$Cb=128-\frac{37.797R}{256}-\frac{74.203G}{256}+\frac{112B}{256}$$

$$Cr=128+\frac{112R}{256}-\frac{93.786G}{256}-\frac{18.214B}{256}$$

5. Results

This chapter presents the results of the rPPG investigation. Results are classified into two main groups:

• results for self-captured videos
• results for COHFACE videos

For self-captured videos, results are further classified based on whether videos were captured in ambient or dedicated lighting and whether HR determination was based on maximum peak or signal peak estimates. For the COHFACE videos, results are further classified based solely on whether videos were captured in ambient or dedicated lighting (since all HR estimates were based on maximum peaks).

5.1. Self-Captured Videos

Mean error (and RMSE)2 of HR estimates were generally poor for self-captured videos. The worst performance was obtained for max peak estimates under ambient lighting, with Table 5.1 showing a mean error range of 50% to 119%. Significant improvement was achieved for signal peak estimates (under the same lighting), with all eight (8) rPPG signals showing a mean error of 2% (Table 5.2).

Table 5. 1. Mean error and RMSE of HR estimates (max peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	1.13	0.95	0.75	0.63	1.13	1.19	0.62	0.50
RMSE	124.40	105.00	93.67	76.16	122.60	126.70	83.33	71.08

Table 5. 1. Mean error and RMSE of HR estimates (max peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	1.13	0.95	0.75	0.63	1.13	1.19	0.62	0.50
RMSE	124.40	105.00	93.67	76.16	122.60	126.70	83.33	71.08

Table 5. 2. Mean error and RMSE of HR estimates (signal peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02
RMSE	2.54	2.49	2.54	2.49	2.59	2.49	2.49	2.42

Table 5. 2. Mean error and RMSE of HR estimates (signal peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.02	0.02	0.02	0.02	0.02	0.02	0.02	0.02
RMSE	2.54	2.49	2.54	2.49	2.59	2.49	2.49	2.42

Under dedicated lighting, mean error of max peak estimates of HR ranged from 22% to 61% (Table 5.3). This represents an approximately two-fold improvement compared to the ambient lighting case. As with the case for ambient lighting, under dedicated lighting, signal peak estimates of HR showed marked improvement relative to max peak estimates, with all eight (8) rPPG signals showing a mean error of 3% (Table 5.4).

Table 5. 3. Mean error and RMSE of HR estimates (max peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.57	0.48	0.39	0.39	0.55	0.61	0.25	0.22
RMSE	78.55	69.55	61.24	60.59	77.46	84.30	49.28	45.45

Table 5. 3. Mean error and RMSE of HR estimates (max peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.57	0.48	0.39	0.39	0.55	0.61	0.25	0.22
RMSE	78.55	69.55	61.24	60.59	77.46	84.30	49.28	45.45

Table 5. 4. Mean error and RMSE of HR estimates (signal peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.03	0.03	0.03	0.03	0.03	0.03	0.03	0.03
RMSE	3.41	3.70	3.43	3.57	3.34	3.37	3.42	3.76

Table 5. 4. Mean error and RMSE of HR estimates (signal peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.03	0.03	0.03	0.03	0.03	0.03	0.03	0.03
RMSE	3.41	3.70	3.43	3.57	3.34	3.37	3.42	3.76

As a visual representation of agreement, Bland-Altman plots were prepared for signal 4.2 only. This signal displayed the lowest mean error and highest SNR (under both ambient and dedicated lighting) and was deemed most indicative of the success of the implementation of HR estimation by rPPG. The Bland-Altman plot is a popular graphical method for comparing two measurement techniques (in this case, HR measurement by rPPG and ground truth reference). In this method, the differences of values obtained using the two techniques are plotted against the average of the two values. The Bland-Altman plot is typically displayed with two sets of horizontal lines – one line representing the mean of the differences or systematic bias and two more representing the limits of agreement, defined as the mean difference ± 1.96 standard deviations. If these limits do not exceed the maximum allowed difference between methods, the two methods are considered in agreement. Bland-Altman analysis has significant advantages over correlation analysis as it represents the degree of agreement between methods and reveals both systematic and random errors. Correlation analysis, while it evaluates the strength of relationships between variables, does not evaluate agreement.

For max peak HR estimates under ambient lighting (Figure 5.1), the Bland-Altman plot reveals that rPPG systematically overestimate the HR by an average of approximately 50 bpm and that the variance of HR estimates increases with HR (as indicated by the greater spread). The plot also displays a proportional bias, whereby the degree of agreement between the two HR measures degrades with increasing HR. For signal peak HR estimates under ambient lighting (Figure 5.2), the Bland-Altman plot reveals a small bias of approximately -2 bpm i.e., rPPG underestimates the heart rate by an average of 2 bpm. The variance does not seem dependent on the magnitude of the heart rate, though there seems to be a bias towards underestimation at higher HRs.

Figure 5. 1. Bland-Altman plot for HR determined from rPPG signal 4.2 using max peak estimates (ambient lighting).

Figure 5. 1. Bland-Altman plot for HR determined from rPPG signal 4.2 using max peak estimates (ambient lighting).

Figure 5. 2. Bland-Altman plot for HR determined from rPPG signal 4.2 using signal peak estimates (ambient lighting).

Figure 5. 2. Bland-Altman plot for HR determined from rPPG signal 4.2 using signal peak estimates (ambient lighting).

The Bland-Altman plot for max peak HR estimates under dedicated lighting (Figure 5.3) also shows a systematic overestimation of the heart rate, though by a smaller amount (approximately 20 bpm) relative to the ambient lighting case. It also displays some degree of a proportional bias, with higher disagreement at higher HRs. The Bland-Altman plot for signal peak HR estimates under dedicated lighting (Figure 5.4) displays a small HR underestimation of about 1 bpm, no proportional bias, and a random distribution of points about the mean.

Figure 5. 3. Bland-Altman plot for HR determined from rPPG signal 4.2 using max peak estimates (dedicated lighting).

Figure 5. 3. Bland-Altman plot for HR determined from rPPG signal 4.2 using max peak estimates (dedicated lighting).

Figure 5. 4. Bland-Altman plot for HR determined from rPPG signal 4.2 using signal peak estimates (dedicated lighting).

Figure 5. 4. Bland-Altman plot for HR determined from rPPG signal 4.2 using signal peak estimates (dedicated lighting).

Referring to Table 5.5, mean SNR for the self-captured videos under ambient lighting ranged from 0.08 to 0.16. SNR improved significantly under dedicated lighting for each of the eight (8) rPPG signals (Table 5.6), with a minimum increase of 33% (for signal 1.2) and a maximum increase of 70% (for signal 2.1).

Table 5. 5. Mean SNR of rPPG signals (ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.08	0.09	0.09	0.11	0.08	0.09	0.14	0.16

Table 5. 5. Mean SNR of rPPG signals (ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.08	0.09	0.09	0.11	0.08	0.09	0.14	0.16

Table 5. 6. Mean SNR of rPPG signals (dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.12	0.12	0.16	0.16	0.12	0.13	0.22	0.23

Table 5. 6. Mean SNR of rPPG signals (dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.12	0.12	0.16	0.16	0.12	0.13	0.22	0.23

As a supplement to the evaluation of agreement, Pearson correlation analysis was performed to evaluate the systematic relationship between rPPG HR estimates and reference values. In correlation analysis, the correlation coefficient (r) quantifies the linear relationship between two variables while its corresponding p-value determines whether the correlation is statistically significant. The correlation coefficient has a value ranging from +1 to –1, with +1 indicating perfect positive correlation, 0 indicating no correlation, and –1 indicating perfect negative correlation. The threshold value for p is the standard p = 0.05, meaning that only p-values less than 0.05 indicate statistical significance.

Max peak HR estimates under ambient lighting (Figure 5.7) all show weak negative correlation with ground truth HR, with only two statistically significant r values (r = – 0.28 and a max r = – 0.35 for signals 3.1 and 3.2, respectively). Correlation coefficients significantly increased for signal peak estimates (Table 5.8), showing strong positive correlations with all r = 0.99 and all p = 0.00 (indicating statistical significance in all cases).

Table 5. 7. Correlation coefficients (r) and p-values for est. HR vs ref. HR (max peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	-0.15	-0.03	-0.09	-0.06	-0.28	-0.35	-0.22	-0.13
p	0.29	0.84	0.56	0.68	0.05	0.01	0.13	0.35

Table 5. 7. Correlation coefficients (r) and p-values for est. HR vs ref. HR (max peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	-0.15	-0.03	-0.09	-0.06	-0.28	-0.35	-0.22	-0.13
p	0.29	0.84	0.56	0.68	0.05	0.01	0.13	0.35

Table 5. 8. Correlation coefficients (r) and p-values for est. HR vs ref. HR (signal peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Table 5. 8. Correlation coefficients (r) and p-values for est. HR vs ref. HR (signal peak, ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Max peak HR estimates under dedicated lighting (Table 5.9) all show positive correlation with ground truth HR, with two signals (4.1 and 4.2) showing statistical significance (r = 0.36 and r = 0.47, respectively). As in the case for ambient lighting, correlation coefficients significantly increased for signal peak estimates (Table 5.10), showing strong positive correlations with all r = 0.98 or 0.99 and all p = 0.00.

Table 5. 9. Correlation coefficients (r) and p-values for est. HR vs ref. HR (max peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.03	0.09	0.08	0.20	0.18	0.03	0.36	0.47
p	0.82	0.56	0.59	0.16	0.22	0.84	0.01	0.00

Table 5. 9. Correlation coefficients (r) and p-values for est. HR vs ref. HR (max peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.03	0.09	0.08	0.20	0.18	0.03	0.36	0.47
p	0.82	0.56	0.59	0.16	0.22	0.84	0.01	0.00

Table 5. 10. Correlation coefficients (r) and p values for est. HR vs ref. HR (signal peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.99	0.98	0.99	0.98	0.99	0.99	0.99	0.98
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Table 5. 10. Correlation coefficients (r) and p values for est. HR vs ref. HR (signal peak, dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.99	0.98	0.99	0.98	0.99	0.99	0.99	0.98
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Correlation (scatter) plots were produced for signal 4.2. The plot for max peak estimates of HR under ambient lighting (Figure 5.5) shows a negative slop (negative correlation) and a general decrease in the variance of the estimated HR with increasing reference HR. This correlation is, however, weak (r = -0.13) and not statistically significant (p = 0.35). For the corresponding signal peak estimates (Figure 5.6), the correlation between HR estimates and reference values is visibly strong and positive (r = 0.99, p = 0.00).

Figure 5. 5. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (max peak, ambient lighting).

Figure 5. 5. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (max peak, ambient lighting).

Figure 5. 6. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (signal, ambient lighting).

Figure 5. 6. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (signal, ambient lighting).

For max peak estimates of HR under dedicated lighting (Figure 5.7), the correlation between estimates and reference values is positive, moderate (r = 0.47) and statistically significant (p = 0.00). The distribution of points also indicates significant variance in the estimated HR. However, unlike the ambient lighting case, this variance is fairly constant with increasing reference HR. For the corresponding signal peak estimates (Figure 5.8), as in the ambient lighting case, the correlation is strong and positive (r = 0.98, p = 0.00).

Figure 5. 7. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (max peak, dedicated lighting).

Figure 5. 7. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (max peak, dedicated lighting).

Figure 5. 8. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (signal peak, dedicated lighting).

Figure 5. 8. Scatter plot of est. HR versus ref. HR for rPPG signal 4.2 (signal peak, dedicated lighting).

A correlation analysis was also performed for error of HR estimates versus SNR to assess the impact of SNR on error. For the case of ambient lighting (Table 5.11), correlation coefficients ranged from -0.47 to -0.69 (moderate to strong correlation), with all p = 0.00. For the case of dedicated lighting (Table 5.12), the correlation coefficients ranged from -0.43 to -0.52 (moderate correlation), with all p = 0.00.

Table 5. 11. Correlation coefficients (r) and p-values for error versus SNR (ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	-0.50	-0.55	-0.47	-0.56	-0.54	-0.60	-0.63	-0.69
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Table 5. 11. Correlation coefficients (r) and p-values for error versus SNR (ambient lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	-0.50	-0.55	-0.47	-0.56	-0.54	-0.60	-0.63	-0.69
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Table 5. 12. Correlation coefficients (r) and p-values for error versus SNR (dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	-0.45	-0.43	-0.44	-0.46	-0.48	-0.43	-0.45	-0.52
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Table 5. 12. Correlation coefficients (r) and p-values for error versus SNR (dedicated lighting).

	SIG1		SIG2		SIG3		SIG4
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	-0.45	-0.43	-0.44	-0.46	-0.48	-0.43	-0.45	-0.52
p	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Correlation plots of error versus SNR were produced for signal 4.2. As indicated, correlations are negative for both the ambient lighting and dedicated lighting case (Figures 5.9 and 5.10), with the point distributions showing significant variance in both cases.

Figure 5. 9. Scatter plot of error versus SNR for rPPG signal 4.2 (ambient lighting).

Figure 5. 9. Scatter plot of error versus SNR for rPPG signal 4.2 (ambient lighting).

Figure 5. 10. Scatter plot of error versus SNR for rPPG signal 4.2 (dedicated lighting).

Figure 5. 10. Scatter plot of error versus SNR for rPPG signal 4.2 (dedicated lighting).

5.2. COHFACE Database Videos

Mean error (and RMSE) of HR estimates for COHFACE videos were far superior to that of self-captured videos. Referring to Tables 5.13 and 5.14, mean error of HR estimates ranged from 3% to 15% under ambient lighting and 0% to 9% under dedicated lighting. Interestingly, the worst case mean error for the COHFACE videos (15%, ambient lighting) significantly outperforms the best case mean error for the self-captured videos (22%, dedicated lighting).

Table 5. 13. Mean error and RMSE of HR estimates (ambient lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.08	0.15	0.15	0.12	0.03	0.09	0.08	0.10
RMSE	14.88	24.37	24.46	18.14	6.27	14.85	16.27	16.32

Table 5. 13. Mean error and RMSE of HR estimates (ambient lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.08	0.15	0.15	0.12	0.03	0.09	0.08	0.10
RMSE	14.88	24.37	24.46	18.14	6.27	14.85	16.27	16.32

Table 5. 14. Mean error and RMSE of HR estimates (dedicated lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.01	0.02	0.02	0.09	0.01	0.02	0.00	0.01
RMSE	1.71	3.12	5.41	13.66	2.19	1.73	0.32	1.05

Table 5. 14. Mean error and RMSE of HR estimates (dedicated lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean Error	0.01	0.02	0.02	0.09	0.01	0.02	0.00	0.01
RMSE	1.71	3.12	5.41	13.66	2.19	1.73	0.32	1.05

The high level of accuracy obtained for COHFACE videos is reflected in Bland-Altman plots drawn only for the worst performing (signal 2.1′) and best performing FFT signals (signal 3.1′) (to provide a visual display of the range of accuracy). In this context, determination of the best and worst performing signal was based on mean error of HR estimates under ambient lighting. All Bland-Altman plots (Figures 5.11, 5.12, 5.13 and 5.14) display a small or negligible bias in mean differences (with a maximum of approximately +10 bpm for signal 2.1′, ambient lighting) with no indication of systematic error. Perhaps a worthy observation revealed by the Bland-Altman plots is that significant difference between estimated and reference HR tends to involve an overestimation (as opposed to an underestimation), possibly indicating the observed tendency of strong noisy peaks in the higher frequency portion of the spectrum.

Figure 5. 11. Bland-Altman plot for HR determined from sig 2.1′ (ambient lighting).

Figure 5. 11. Bland-Altman plot for HR determined from sig 2.1′ (ambient lighting).

Figure 5. 12. Bland-Altman plot for HR determined from sig 3.1′ (ambient lighting).

Figure 5. 12. Bland-Altman plot for HR determined from sig 3.1′ (ambient lighting).

Figure 5. 13. Bland-Altman plot for HR determined from sig 2.1′ (dedicated lighting).

Figure 5. 13. Bland-Altman plot for HR determined from sig 2.1′ (dedicated lighting).

Figure 5. 14. Bland-Altman plot for HR determined from sig 3.1′ (dedicated lighting).

Figure 5. 14. Bland-Altman plot for HR determined from sig 3.1′ (dedicated lighting).

Mean SNR for COHFACE videos ranged from 0.26 to 0.36 under ambient lighting (Table 5.15) and 0.43 to 0.68 under dedicated lighting (Table 5.16). This represents marked improvement in mean SNR under dedicated lighting (for each of the 8 rPPG signals), with a minimum increase of 59% (for signal 2.2′) and a maximum increase of 104% (for signal 1.1′). A noteworthy observation (like that observed for mean error) is that the worst-case SNR for the COHFACE videos (0.26) is superior to the best-case SNR for the self-captured videos (0.23).

Table 5. 15. Mean SNR for rPPG signals (ambient lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.27	0.28	0.26	0.27	0.35	0.35	0.36	0.36

Table 5. 15. Mean SNR for rPPG signals (ambient lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.27	0.28	0.26	0.27	0.35	0.35	0.36	0.36

Table 5. 16. Mean SNR for rPPG signals (dedicated lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.55	0.55	0.43	0.43	0.68	0.68	0.63	0.62

Table 5. 16. Mean SNR for rPPG signals (dedicated lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
Mean SNR	0.55	0.55	0.43	0.43	0.68	0.68	0.63	0.62

Correlation coefficients (r) and corresponding p-values for estimated HR versus reference HR were calculated for the 8 rPPG signals under ambient and dedicated lighting, taken separately, and together. Table 5.17 shows correlation coefficients ranging from a weak 0.26 (p = 0.22, signal 2.2′) to a strong 0.90 (p = 0.00, signal 3.1′) for ambient lighting, with only three of the eight r values statistically significant. Referring to Table 5.18, correlation coefficients ranged from 0.55 (signal 2.2′) to 1.00 (signal 4.1′) (all but one r value < 0.9) for dedicated lighting, with all p-values indicating statistical significance (all except one equal to 0.00). For rPPG signals considered together (ambient + dedicated) (Table 5.19), correlation coefficients ranged from a weak 0.39 (signal 2.2′) to a very strong 0.93 (signal 3.1′), with the coefficients generally bearing a value between that obtained for their equivalents under ambient and dedicated lighting. The p-values exactly reproduced the results obtained for dedicated lighting.

Table 5. 17. Correlation coefficients (r) and p-values for est. HR vs ref. HR (ambient lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.39	0.69	0.39	0.26	0.90	0.53	0.39	0.33
p	0.06	0.00	0.06	0.22	0.00	0.01	0.06	0.12

Table 5. 17. Correlation coefficients (r) and p-values for est. HR vs ref. HR (ambient lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.39	0.69	0.39	0.26	0.90	0.53	0.39	0.33
p	0.06	0.00	0.06	0.22	0.00	0.01	0.06	0.12

Table 5. 18. Correlation coefficients (r) and p-values for est. HR vs ref. HR (dedicated lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.99	0.96	0.93	0.55	0.98	0.99	1.00	1.00
p	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.00

Table 5. 18. Correlation coefficients (r) and p-values for est. HR vs ref. HR (dedicated lighting).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.99	0.96	0.93	0.55	0.98	0.99	1.00	1.00
p	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.00

Table 5. 19. Correlation coefficients (r) and p-values for est. HR vs ref. HR (all videos).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.64	0.68	0.56	0.39	0.93	0.70	0.62	0.59
p	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.00

Table 5. 19. Correlation coefficients (r) and p-values for est. HR vs ref. HR (all videos).

	SIG1′		SIG2′		SIG3′		SIG4′
	FFT	Welch	FFT	Welch	FFT	Welch	FFT	Welch
r	0.64	0.68	0.56	0.39	0.93	0.70	0.62	0.59
p	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.00

Correlation (scatter) plots were also prepared for the worst and best signals (2.1′ and 3.1′, respectively) under ambient and dedicated lighting (Figures 5.15, 5.16, 5.17 and 5.18). All plots display strong positive correlation, with very few (≤ 2) significant deviations from the line of best fit (except in the case of signal 2.1′ under ambient lighting, where there are five (5) significant deviations).

Figure 5. 15. Scatter plot of est. HR versus ref. HR for sig 2.1′ (ambient lighting).

Figure 5. 15. Scatter plot of est. HR versus ref. HR for sig 2.1′ (ambient lighting).

Figure 5. 16. Scatter plot of est. HR versus ref. HR for sig 3.1′ (ambient lighting).

Figure 5. 16. Scatter plot of est. HR versus ref. HR for sig 3.1′ (ambient lighting).

Figure 5. 17. Scatter plot of est. HR versus ref. HR for sig 2.1′ (dedicated lighting).

Figure 5. 17. Scatter plot of est. HR versus ref. HR for sig 2.1′ (dedicated lighting).

Figure 5. 18. Scatter plot of est. HR versus ref. HR for sig 2.1′ (dedicated lighting).

Figure 5. 18. Scatter plot of est. HR versus ref. HR for sig 2.1′ (dedicated lighting).

6. Discussion

This chapter discusses the results of the preceding chapter in relation to the research objectives set forth in Chapter 1. For each objective, the results for the self-captured videos will be discussed first, followed by that for the COHFACE videos, which will make comparative references where relevant.

6.1. Implement an Algorithmic Framework for HR Estimation by Video-Based Reflectance Photoplethysmography

The primary objective of this thesis was to set up an algorithmic framework for the estimation of HR by the analysis of rPPG signals extracted from video. Accordingly, an immediate concern is whether the framework is set up correctly. Such validity may be evaluated through the level of agreement between rPPG HR estimates and reference (ground truth) values. Agreement was assessed using two statistical measures – mean error and root-mean-square error (RSME)¹ – as well as Bland-Altman plots. Correlation (scatter) plots, while not representing agreement, were also used to assess the systematic relationship between estimated and reference values. Another performance metric used in this investigation is the signal-to-noise ratio (SNR) – a primary measure of signal quality significantly correlated with the error in HR estimates.

SNR for rPPG signals extracted from the self-captured videos was generally low. This is primarily ascribed to the dark skin tone of the subject as it is known that rPPG SNR is particularly low for dark skin. This presented a significant drawback to the evaluation of agreement between rPPG HR estimates and reference values. In many cases, low SNR resulted in a peak of maximum amplitude at a frequency other than the HR (with the peak representing the HR still dominant). Figure 6.1 illustrates a typical instance of a noisy peak predominating the peak representing the HR (termed interchangeably signal peak or heart rate peak in this thesis). As discussed in Materials and Methods, this limitation was addressed by using two criteria for assessing agreement:

considering only maximum peaks in the HR estimation and

considering signal peaks, which may or may not be maximum peaks.

Figure 6. 1. Representative example of signal peak in relation to maximum peak.

Figure 6. 1. Representative example of signal peak in relation to maximum peak.

Tables 5.1 and 5.3 show mean error (based on max peak HR estimates) ranging from 50% to 119% for rPPG signals extracted under ambient lighting and 22% to 61% for signals extracted under dedicated lighting. The improvement in accuracy under dedicated lighting is imputed to increased SNR (mean SNR ranged from 0.08 to 0.16 and 0.12 to 0.23 under ambient and dedicated lighting, respectively). This immediately gives a sense of the impact of SNR on mean error, which drastically improved for signal peak estimates of HR (2% for every signal under ambient lighting and 3% for every signal under dedicated lighting).

In this investigation, there is a possibility that good accuracy values arose from systematic error rather than actual agreement between estimated and reference HR. For example, if a spurious peak consistently exists in the region of say 80 bpm and the investigated range of HR is, say, 78 – 82 bpm, then highly accurate HR estimates could be seemingly obtained even if the rPPG signal contains no HR information. To address this possibility a large range of HRs were obtained for self-captured videos (both resting and post-exercise) and a correlation analysis was performed between estimated and reference values. Max peak HR estimates under ambient lighting all displayed weak negative correlation with ground truth HR, with only two statistically significant correlations. This unexpectedly negative correlation may be explained in terms of the randomizing effect of noise on max peak HR estimates. Max peak HR estimates under dedicated lighting all displayed positive correlation with ground truth HR, with two signals (4.1 and 4.2) showing statistical significance. With the correlations bearing the expected sign (notwithstanding that only two are statistically significant), this may be deemed an improvement over ambient lighting case. This improvement is ascribed to a reduction in the effect of the randomization on the HR estimates due to improved SNR under dedicated lighting. Correlation coefficients for HR estimates versus ground truth significantly increased for signal peak HR estimates, showing strong positive and statistically significant correlations under ambient and dedicated lighting. This correlation analysis supported the view that, despite the interference caused by low SNR, HR estimates latently tracked ground truth values and can be considered legitimate.

To further support the proposition that poor accuracy stemmed from low SNR, a correlation analysis was performed for the error in HR estimates versus SNR for each of the 8 rPPG signals. For the case of ambient lighting, correlation coefficients ranged from -0.47 to -0.69 (moderate to strong correlation). For the case of dedicated lighting, coefficients ranged from -0.43 to -0.52, reflecting moderate correlation in the expected direction. Though correlations are too weak to ascribe all inaccuracy to low SNR, it does indicate that low SNR significantly accounts for loss of accuracy.

For the COHFACE videos, a resolution of the problems related to generally low mean SNR was obviated as the COHFACE rPPGs displayed comparatively high mean SNR for both the dedicated and ambient lighting case. As such, there was no need for a separate evaluation of measures of agreement based on signal peaks; in (virtually) every case, the signal peak was the maximum peak. The high SNR (and correspondingly low mean error and RMSE) also nullified the relevance of an assessment of the correlation between error and SNR (performed for the self-captured videos).

Mean error of HR estimates for COHFACE videos ranged from 3% to 15% under ambient lighting and 0% to 9% under dedicated lighting. Importantly, the worst case mean error for the COHFACE videos (15%, ambient lighting) significantly outperformed the best case mean error for the self-captured videos (22%, dedicated lighting). This highlights the impact of SNR on accuracy performance.

Correlation coefficients (r) and p-values for estimated HR versus reference HR were calculated for the eight (8) rPPG signals under ambient and dedicated lighting (Tables 5.17 and 5.18). Correlation coefficients ranged from a weak 0.26 to a strong 0.90 for ambient lighting, with p-values indicating only three (3) statistically significant correlations. These unexpectedly poor correlations, despite generally high corresponding mean accuracies, reflect the sensitivity of the correlation measure to outliers. The strength and significance of correlations, however, showed marked improvement under dedicated lighting, with coefficients ranging from 0.55 to 1.00 (all but one r value < 0.9), with all p-values negligible. This could easily be explained in terms of the increase in accuracy (and reduced number of outliers) associated with the improvement in SNR afforded by dedicated lighting.

Mean SNR for rPPG signals extracted from COHFACE videos ranged from 0.26 to 0.36 under ambient lighting (Table 5.15) and 0.43 to 0.68 under dedicated lighting (Table 5.16). A significant observation (like that observed for mean error) is that the worst-case SNR for the COHFACE videos is superior to the best-case SNR for the self-captured videos (0.23), giving an indication of the strength of the impact of skin tone on SNR performance.

6.2. Investigate the Effect of Presence/Absence of Dedicated Lighting on the SNR of rPPG Signals

Apart from setting up a correct algorithmic framework for rPPG extraction and analysis, another major point of investigation was the effect of the presence/absence of dedicated lighting on the mean SNR of rPPG signals.

For the self-captured videos (Tables 5.5 and 5.6), mean SNR improved significantly under dedicated lighting (relative to ambient lighting) for each of the eight (8) rPPG signals, with a minimum increase of 33% in the case of signal 1.2 and a maximum increase of 70% in the case of signal 2.1. Mean SNR also improved significantly under dedicated lighting (Table 5.15 and 5.16) for the signals extracted from the COHFACE videos, with a minimum increase of 59% (for signal 2.2′) and a maximum increase of 104% (for signal 1.1′). This increase in SNR is consistent with theoretical expectation as the greater the light intensity, the greater the reflected component in relation to noisy fluctuations arising from motion and/or background illumination.

It is also important to note that mean SNR for the rPPG signals extracted from the COHFACE videos, even under ambient lighting, surpassed that for the self-captured videos under dedicated lighting. This is indicative of the extent to which dark skin tone (self-captured videos) can degrade SNR performance.

6.3. Investigate the Effect of Choice/Size of Region of Interest (ROI) on the SNR of rPPG Signals

Another objective of this thesis was to investigate the effect of the choice/size of the ROI for rPPG signal extraction on the SNR of the signals. Three ROIs were investigated. The description of each ROI – termed A, B and C – is provided in Table 3.1. ROI B was expected to yield a higher SNR than ROI A since their only significant difference is that ROI B is 50% larger than ROI A (they are both centred on the forehead and contain only skin pixels). All things being equal, the SNR of two images scales as the square root of the number of pixels. It was also hypothesized that ROI C would be inferior in SNR relative to ROI B (and possibly also ROI A), notwithstanding its superior size, since it contains a large percentage of non-skin pixels and non-rigid structures (such as the eyes and mouth) which contribute to signal noise.

For the self-captured videos under ambient lighting (Table 5.5), ROI A yielded a mean SNR of 0.08 for the FFT signal and 0.09 for the Welch periodogram (Welch) signal. The corresponding values for ROI B were also 0.08 and 0.09 while that for ROI C were 0.14 and 0.16. While the SNRs for ROI A and B are equivalent, that for ROI C (invalidating hypothesis) is approximately twice as large (relative to ROI A or B). The difference may be explained in terms of the large number of skin pixels contained in ROI C coupled with minimal motion of the eyes and mouth by the subject. It is not clear why ROI B did not yield a higher mean SNR relative to ROI A. A possible explanation is that the size difference in the ROIs was not significant enough to produce a statistically significant difference in the SNR.

For the self-captured videos under dedicated lighting (Table 5.6), ROI A yielded a mean SNR of 0.12 for both the FFT and Welch signal. ROI B performed comparably, yielding a mean SNR of 0.12 for the FFT and 0.130 for the Welch. Just as in the case of ambient lighting, ROI C displayed a significantly higher mean SNR at 0.22 for the FFT and 0.23 for the Welch.

For the COHFACE videos under ambient lighting (Table 5.15), the mean SNRs for ROIs A, B and C were 0.27, 0.35 and 0.36 (respectively) for the FFT, and 0.28, 0.35 and 0.36 for the Welch. Under dedicated lighting (Table 5.16), corresponding values were 0.55, 0.68 and 0.63 for the FFT and 0.55, 0.68 and 0.62 for the Welch. ROIs B and C may be described as comparable in SNR performance (with C slightly superior to B under ambient lighting, and B superior to C under dedicated lighting) and superior to ROI A. The commonality between the results for the self-captured videos and that for the COHFACE videos is that they both indicate superiority of ROI C. However, while comparability exists between ROIs A and B for the self-captured videos, it exists between ROIs B and C for the COHFACE videos (with ROI B significantly outperforming ROI A). This difference may be explained by the hypothesis that the SNR difference (due to size difference) between ROIs B and A becomes more significant for all signals at higher SNR (as is the case for the COHFACE videos with a mean SNR under ambient lighting greater than that of the self-captured videos under dedicated lighting). As with the self-captured videos, the high SNR of ROI C is believed to be a direct consequence of its superior size. Again, minimal motion of the eyes and mouth may explain why noise did not override the size advantage and degrade the SNR of ROI C.

This analysis reveals ROI C as a simple high SNR ROI, perhaps useful in the case of optimizing SNR for dark skin and/or low lighting conditions. However, the SNR performance of ROI C may not hold under significant motion, and for this reason ROI B may prove a more robust option (given its comparability to ROI C in SNR performance and status as a forehead ROI containing only skin pixels and no non-rigid structures). An even better ROI may be designed using a skin mask with the eyes and mouth regions removed (to maximize the number of useful pixels and minimize the motion-induced noise).

6.4. Compare the SNR of Green rPPG Signals with That of rPPG Signals Based on Luminance

Given that reflectance photoplethysmography is based on variations in the intensity of light reflected from the skin, an evaluation of the SNR of a luminance-based rPPG signal relative to that of a color-based signal seemed appropriate. For the RGB color space, both hue and light intensity information are coded in the same channel. However, for color spaces containing luminance components (such as YCbCr and HSV), light intensity (luminance) information is coded independently of hue information. In this investigation, the YCbCr color space was used to provide the luminance component Y.

Referring to Tables 5.5 and 5.6, for the self-captured videos, mean SNR for the luminance signal (signal 2) under ambient lighting is 0.09 for the FFT and 0.11 for the Welch periodogram compared to 0.08 and 0.09 for its green counterpart (signal 1). Under dedicated lighting, the SNR for the luminance signal is 0.16 for both the FFT and Welch compared to 0.12 for the green signal (both FFT and Welch). This indicates that the luminance signal slightly outperforms its green counterpart in terms of SNR. A secondary comparison of mean error (based on maximum peaks) shows that the luminance signal also outperforms its green equivalent with a mean error of 75% (FFT) and 63% (Welch) for ambient lighting and 39% (FFT and Welch) under dedicated lighting, compared to the green signal’s 113% and 95%, and 57% and 48%, respectively.

For the COHFACE videos, Tables 5.15 and 5.16 show that the mean SNR for the luminance signal (signal 2′) under ambient lighting is 0.26 (FFT) and 0.27 (Welch) compared to 0.27 (FFT) and 0.28 (Welch) for its green equivalent (signal 1′). Under dedicated lighting, the mean SNRs are 0.43 for the luminance signal (both FFT and Welch) and 0.55 for its green counterpart (both FFT and Welch). These results indicate that the green signal outperforms the luminance signal in SNR (though only slightly in the case of ambient lighting) and inverts the finding for the self-captured videos. However, since the mean parameters for the COHFACE rPPG signals were based on a larger sample size (12 subjects), the superiority of the green signal over the luminance signal is a safer conclusion. Still, the contradictory results may warrant an investigation into the SNR performance of green versus luminance signals for dark-skinned subjects compared to fair-skinned subjects. It may very well be that the luminance signal is superior in SNR for dark tones but inferior for fair tones. A secondary consideration of mean error for the luminance and green signal reveals the green signal to be superior in every case except one (the case of the Welch signals under ambient lighting).

6.5. Investigate the Viability of the Summary Autocorrelation Function (SACF) for Improving the SNR of rPPG Signals

Summary autocorrelation is a technique conventionally used in speech processing for identifying fundamental frequencies. In this thesis, it was hypothesized that the technique could be used to improve the SNR of the rPPG signals. However, preliminary results for HR estimates using the SACF for spectral analysis did not bear any correlation with reference values. Assuming the technique can in fact accomplish the stated objective, the troubleshooting of the problems with the implementation was deemed too rudimentary to warrant a reporting of results. However, the theoretical soundness of the idea warrants further investigation. Accordingly, the technique is given consideration under further work.

6.6. Implement the HR Estimation Algorithm into a GUI

In keeping with objective 6 of this thesis, the HR estimation algorithm was implemented within a graphical user interface (GUI) to facilitate user-friendly interaction. The GUI was programmed using MATLAB GUIDE (GUI development environment) which provides tools for graphical design of user interfaces for custom applications. GUIDE automatically generates the MATLAB code for constructing the GUI, which can then be modified to program the behavior of the application.

There are two GUI components depending on the level of interaction desired by the user. The first GUI component (GUI1) is designed to facilitate a high level of user interaction with the data to provide more direct exposure to the analysis of signals in the frequency space. The second GUI component (GUI2), on the other hand, is structured to simply output HR estimates and matching ground truth values (in real time) for monitoring and/or recording, without any presentation of rPPG signals and analysis in the frequency space.

GUI1 contains a pushbutton at the top of the of GUI window for browsing and selecting a pre-recorded video (SELECT VIDEO) and an adjacent pushbutton for initiating video processing (labelled PROCESS VIDEO). After processing, the FFTs of signals 1 – 4 automatically appear in labelled panels. The Welch periodogram of these signals are also accessible by pressing the pushbutton labelled WELCH at the bottom of the GUI window. Return to FFT can be achieved by pressing the same button, which is automatically relabelled FFT. By identifying the frequency peaks of the FFTs and the Welch periodograms, users can estimate the HR associated with their input videos. A fuller explanation of the functionality of GUI1 will be provided in Appendix C where the GUI will serve as the interface for a laboratory experiment centred on HR estimation by video-based rPPG.

GUI2 is displayed in Figure 6.2. It contains a START pushbutton for synchronized initiation of three events:

• live video capture and processing
• a timer and,
• a continuous display of the ground truth pulse oximeter signal

Figure 6. 2. GUI2.

Figure 6. 2. GUI2.

The live video stream (with processing features such as the bounding boxes of ROIs) is displayed in the largest panel. During processing (i.e., running of HR estimation algorithm), an HR estimate is outputted into a static text field every 15 s. The corresponding ground truth HR is also displayed in a static text field. Pressing START reconfigures the START pushbutton into a STOP pushbutton for synchronized termination of video capture and processing, the timer object and the continuous display of the ground truth signal. The RESET button resets the system for another video capture and processing event. The EXIT button performs garbage collection and closes the GUI figure. There is also a static text field for date and time of the video capture. All HR estimates and corresponding ground truths are automatically outputted to a file named by date and time of video capture. A fuller description of the functionality of GUI2 will be provided in Appendix C where the GUI will serve as the interface for a laboratory experiment centred on HR estimation by video-based rPPG.

6.7. Prepare a Laboratory Practical Centred on HR Estimation by rPPG

The GUIs discussed in Section 6.6 will serve as the interface for an experiment centred on HR estimation by rPPG. The objectives of this experiment are to:

• To provide students an introduction to the MATLAB IDE
• To provide students experience with extracting a clinically relevant biosignal (rPPG) using a video-based method
• To provide students an introduction to signal processing and signal processing algorithms such as the FFT
• To provide students the opportunity to perform simple statistical analyses on data

A full description of the laboratory exercise is provided in Appendix C.

7. Conclusions and Further Work

7.1. Conclusions

An algorithmic framework for HR estimation by rPPG was successfully implemented in MATLAB. Though the results from the self-captured videos were generally poor (best case mean error of 22%), they were good enough to validate the algorithmic implementation.

The COHFACE videos yielded far superior results with mean error ranging from 3% to 15% for ambient lighting and 0% to 9% for dedicated lighting. Overall for COHFACE videos, ROI B and C performed comparably in terms of mean SNR, with both outperforming ROI A. ROI B is, however, likely to prove more robust than ROI C under significant motion. The luminance signal proved inferior to its green equivalent in terms of both mean SNR and mean error. While its mean SNR values (FFT) were 0.26 (ambient lighting) and 0.43 (dedicated lighting), corresponding values for its green counterpart were 0.27 and 0.55. Similarly, while the mean error values (FFT) for the luminance signal were 15% (ambient lighting) and 2% (dedicated lighting), corresponding values for the green signal were 8% and 1%. Performance metrics of rPPG signals were far superior under dedicated lighting (relative to ambient) with mean error reductions ranging from 25% to 100% and mean SNR increases ranging from to 59% to 104%. In keeping with research objectives 6 and 7, the HR estimation algorithm was successfully implemented within a GUI; it serves as the interface for an rPPG lab exercise detailed in Appendix C.

7.2. Further Work

This research project sets a foundation for further investigation. Possible future work includes:

Better ROI selection: ROI selection based on a skin mask from which non-rigid structures of the face (mouth, eyes) have been removed. This should lead to an improvement in SNR as it maximizes the number of skin pixels and reduces motion-based noise.

Better motion tracking: periodic re-detection of the face to minimize effect of ROI drift due to suboptimal tracking by KLT algorithm

Use of cleanest signal: for SNR and accuracy comparison with original signal; this can be accomplished by taking the 30 s sub-trace of original signal with minimum outliers

Use of PCA or ICA: to assist with denoising

Evaluation of extent of noise due to fluctuations in background lighting

Prospective further correction for suboptimal tracking: by the development of a technique based on evaluating the time series of the coordinates of the face detector bounding box

Test of comparative SNR performance of various orthogonal signals in RGB and other color spaces: based on the findings of Sahindrakar et al. that signal X = R – G + 2B outperforms the green RGB signal in SNR (2011)

Pooling of best practices for SNR optimization: to improve accuracy of HR estimation for dark-skinned subjects

Further investigation of the viability of the summary autocorrelation function (SACF): for improving the SNR of rPPG signals