Validity and Inter-Device Reliability of the OTBeat Burn^TM Monitor to Estimate Heart Rate During Exercise

1. Introduction

Commercially available fitness monitors have become an incredibly popular form of wearable technology for tracking health and performance at rest and during exercise [1,2,3,4]. In fact, the American College of Sports Medicine has named “wearable technology” as the top fitness trend for seven of the last nine years [1] and the global industry of these products is currently valued at approximately $100 billion [4] with an estimated 100 million wearables sold annually [5,6]. These wearable devices are used to estimate numerous cardiovascular (heart rate, heart rate variability, percent oxygen saturation, irregular rhythms), pulmonary (breathing rate), metabolic (oxygen uptake, energy expenditure, lactate threshold, body temperature), performance (step count, distance covered, duration of exercise, training volume, pace, acceleration), recovery, sleep, and stress variables [1]. Of these features, the monitoring of heart rate is often highly beneficial due to its utility in prescribing optimal exercise intensity zones specifically designed for cardiorespiratory fitness, weight loss, and rehabilitation strategies in athletic and clinical populations [1,7,8,9,10]. Previous studies, however, have established the validity of measuring heart rate in these devices can be influenced by the mode and intensity of exercise, location of placement on the body, sensor technology, and other factors [6,11,12,13]. Moreover, wide-ranging accuracy values have been reported with some devices exhibiting measurement error exceeding 20% or up to 50 beats per minute compared to criterion methods (i.e. electrocardiogram, ECG) [6,8,12,14,15,16,17]. Concerns over these issues continue to be reported [3,4,12] and many authors have proposed more research should be completed on the accuracy as well as reliability of these devices [1,16,17,18,19]. Thus, the importance of determining the validity metrics of specific heart rate monitors is highly necessary based on their overall popularity, diverse capabilities, practical applications, and overall cost.

Orangetheory® Fitness is a commercial group fitness franchise with over 1,500 locations and one million clients worldwide [20]. Their studios offer high-intensity workout classes based on heart rate responses to ensure patrons are within the ideal exercise intensity zones to achieve their individual goals. The OTBeat Burn^TM heart rate monitor is the device used by Orangetheory® Fitness, which is recommended to be worn on the forearm or upper arm [21]. This device estimates heart rate through a noninvasive technique called photoplethysmography (PPG) that utilizes LED optical sensors to detect changes in blood volume through the emission, absorption, and reflection of infrared light in the underlying vessels [7,22]. Despite being a simple and cost-effective technique found in many heart rate monitors on the market, PPG has shown limitations in measurements at higher exercise intensities that may be due to motion artifact or device location on the body [6,14,15,23]. To our knowledge, the OTBeat Burn^TM monitor has yet to be validated across a heart rate range consistent with high-intensity exercise. In addition, the inter-device reliability for this monitor has not been established in the different placement locations (i.e. forearm versus upper arm). Therefore, the purpose of this study was to determine the accuracy and reliability of the OTBeat Burn^TM heart rate monitor during an incremental test to exhaustion on a cycle ergometer.

2. Materials and Methods

2.1. Subjects

Twenty college-aged males (mean ± SD age = 21.1 ± 1.9 years; body mass = 84.7 ± 15.0 kg; height = 181.2 ± 7.5 cm) who participated in recreational training (aerobic training = 1.8 ± 1.9 hr·wk^-1; resistance training = 6.2 ± 2.7 hr·wk^-1) volunteered for a single laboratory visit. Inclusion criteria included ages of 19-29 years, and not reporting any of the following that could influence the outcome of the study: history of cardiovascular disease, chest pain, loss of balance due to dizziness, unconsciousness, taking medications for blood pressure or heart condition, musculoskeletal, metabolic, respiratory, renal, immune, hematological, or neurological diseases or any other reason that would prevent them from safely completing the protocol. Prior to their visit, the subjects were encouraged to consume a typical meal approximately three hours beforehand and instructed to avoid strenuous exercise and supplement ingestion for 24 hours. All subjects completed a health history questionnaire, physical activity readiness questionnaire, and signed an informed consent prior to testing. The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of Northern Illinois University (IRB #HS24-0455, August 20, 2024).

2.2. Exercise Protocol

Subjects completed an incremental test to exhaustion on an electronically-braked cycle ergometer (Lode Corival CPET, Groningen, The Netherlands). Seat height was adjusted to the greater trochanter of the subject while standing next to the cycle ergometer to ensure their legs were near full extension during each pedal revolution. Feet were secured to the pedals through foot straps to maintain pedal contact during the test. The test began at 80 watts with power output increasing 30 watts every two minutes until volitional fatigue or when 70 revolutions per minute could not be sustained despite strong verbal encouragement. During the test, subjects were instructed to pedal at a cadence of 70-80 revolutions per minute and strongly encouraged to continue through maximal exertion. Heart rate was recorded every 30 seconds through an ECG measurement and two OTBeat Burn^TM monitors placed on the right forearm and upper arm. Rating of perceived exertion was measured at the end of each two-minute stage using the Borg scale (6-20). Criteria for defining maximal intensity involved the subject achieving heart rate values within 10 bpm of their age-predicted maximum and rating of perceived exertion ≥ 18.

2.3. Measurement of Heart Rate

A standard 12-lead ECG (Marquette Case Plus Stress Test Monitor, General Electric, Boston, Massachusetts, USA) with a sampling frequency of 4000 Hz was utilized as the criterion method for heart rate. Each subject was positioned on an examination table with their shirt removed. Prior to the visit, subjects were instructed to have their chest shaved within 24 hours of their scheduled time to ensure optimal skin-to-electrode contact and signal quality. The skin was lightly abraded with gauze and cleaned with isopropyl alcohol prior to placement of the electrodes (Skintact Electrodes, Leonhard Lang GmbH, Innsbruck, Austria) using the Mason-Likar configuration.

OTBeat Burn^TM heart rate monitors (Orangetheory Fitness®, Boca Raton, Florida, USA) were placed according to the manufacturer recommendations on the right forearm and upper arm of the subject [21] (Figure 1). Specifically, the devices were tightly secured to the comfort level of the subject approximately two finger-widths below the elbow for the forearm placement and two finger-widths above the elbow for the upper arm placement [21] (Figure 1). The monitors were then rotated until the sensor flashed green indicating heart rate was detected [21]. Both heart rate monitors were subsequently paired via Bluetooth to separate mobile devices using the Orangetheory Mobile App and at-home indoor cycling option. The sensor lens of each device was cleaned with a soft cloth after completion of each test.

2.4. Data Analyses

Data were organized in Microsoft Excel (Microsoft Corp, Redmond, WA) and transferred to SPSS (Version 29.0, SPSS Inc. Armonk, NY: IBM Corp.) for subsequent analysis. Instantaneous heart rate values from the ECG and OTBeat Burn^TM monitors were recorded simultaneously every 30 seconds [17,18]. The four heart rate values associated with each two-minute stage were then averaged for each device. Accuracy of the OTBeat Burn^TM monitors was assessed through: 1) calculation of the mean absolute percent error (MAPE) to determine the average error associated with each measurement, 2) simple linear regression to determine the Pearson correlation coefficient (r) and standard error of estimate (SEE), 3) Bland-Altman plots with associated calculation of constant error (CE = mean difference for ECG – OTBeat Burn^TM heart rate values) and 4) Lin’s Concordance Correlation Coefficient (CCC) (r_c) to measure the agreement for each OTBeat Burn^TM monitor with ECG (i.e. the degree the paired observations fall on the identity line) [8]. For the Bland-Altman plots, limits of agreement (LoA) were calculated as:

upper limit = (ECG heart rate – OTBeat Burn heart rate) + (SD of difference x 1.96)

lower limit = (ECG heart rate – OTBeat Burn heart rate) - (SD of difference x 1.96)

Absolute error (AE) was calculated as: $\left|\mathrm{ECG}\mathrm{heart}\mathrm{rate}–\mathrm{OTBeat}\mathrm{Burn}\mathrm{heart}\mathrm{rate}\right|$Absolute percent error (APE) was calculated as:

${\displaystyle \frac{\mathrm{AE}}{\mathrm{ECG}\mathrm{heart}\mathrm{rate}}}\mathrm{x}100$MAPE was calculated as the average of all APE values. Acceptable accuracy was determined using the combined criteria of Navalta et al. [12] and Jo et al. [16] that required all the following to be met: MAPE < 5%, CE < 3 bpm, SEE < 5 bpm, and Lin’s CCC ≥ 0.9. Simple linear regression and intraclass correlation coefficient (ICC, R) (two-way mixed model with absolute agreement) were utilized to assess inter-device reliability and interpreted according to Hopkins [24]: very poor (r = 0.45 to 0.69), poor (r = 0.70 to 0.84), good (r = 0.85 to 0.94), very good (r = 0.95 to 0.994) and excellent (r ≥ 0.995). Statistical significance was set a priori at an alpha level of 0.05.

3. Results

The MAPE (± SD) values were 0.9 (± 0.6)% and 0.8 (± 0.5)% for the forearm and the upper arm OTBeat Burn^TM heart rate monitors, respectively. The linear regression analyses for ECG vs. forearm OTBeat Burn^TM heart rate resulted in an excellent correlation (r = 0.998) and SEE value of 1.2 bpm, whereas the relationship for ECG vs. upper arm OTBeat Burn^TM heart rate resulted in an excellent correlation (r = 0.999) and SEE value of 1.0 bpm. The mean (± SD) CE (actual ECG heart rate – predicted OTBeat Burn^TM heart rate) values were 1.0 (± 1.2) bpm and 0.8 (± 1.1) bpm for the forearm and upper arm, respectively. The Bland-Altman regression analyses (CE vs. ECG heart rate) resulted in very poor correlations for the forearm (r = 0.107, p = 0.268) (Figure 2) and upper arm (r = 0.303, p < 0.001) (Figure 3). Lin’s CCCs were r_c = 0.998 and 0.998 for the forearm and upper arm, respectively. The Pearson correlation coefficient and ICC for inter-device reliability between the forearm and upper arm OTBeat Burn^TM monitors were r = 0.999 and R = 0.999, respectively (Figure 4).

4. Discussion

This is the first study to examine the validity and reliability of the OTBeat Burn^TM heart rate monitor from Orangetheory® Fitness. The main findings indicated the OTBeat Burn^TM monitor provided highly accurate and reliable heart rate values when compared to a 12-lead ECG from low to maximal exercise intensities during cycle ergometry. All the validity criteria for heart rate monitors proposed by Navalta et al. [12] and Jo et al. [16] were achieved in both locations: MAPE < 5%, CE < 3 bpm, SEE < 5 bpm, and Lin’s CCC ≥ 0.9. Specifically, both the forearm and upper arm placements of the OTBeat Burn had acceptable MAPE (0.9 and 0.8%), SEE (1.2 and 1.0 bpm), CE (1.0 and 0.8 bpm), and Lin’s CCC (r_c = 0.998 and 0.998) values, respectively. Other studies have shown a wide range of MAPE values (0.6-25.7%) depending on the specific device, mode and intensity of exercise, and location of placement on the body [6,12,13,14,15,25]. For example, Gillinov et al. [6] reported varying MAPE values for the Polar Chest Strap (0.6-1.1%), Scosche Rhythm+ (4.8-13.1%), Apple Watch (3.2-6.5%), Fitbit Blaze (10.4-15.9%), Garmin Forerunner 235 (4.6-13.7%), and TomTom Spark Cardio (5.9-6.7%) during incremental treadmill, cycle ergometer, and elliptical (with and without arms) exercise. Of the four exercise modes, the authors [6] found cycle ergometry resulted in the most MAPE values achieving “acceptable” criteria (i.e. < 5%) in four of the six devices (Polar Chest Strap, Scosche Rhythm+, Apple Watch, and Garmin Forerunner 235). In terms of exercise intensity, conflicting findings have been reported by previous studies. At higher exercise intensities, a number of investigations have shown improved accuracy, likely due to improved blood perfusion [11,13]. Other findings [6,14,15,23,25], however, indicated reduced accuracy at higher intensities, potentially due to increased motion artifact. In the present investigation, the Bland-Altman results showed accuracy remained unchanged across heart rate range (r = 0.107) for the forearm placement (Figure 2), whereas accuracy slightly improved with intensity at the upper arm placement (r = 0.303) (Figure 3). Although every heart rate estimation from the upper arm was approximately within three bpm of the ECG, visible inspection of the Bland-Altman (Figure 3) illustrated there was marginal overestimation at lower intensities and underestimation at higher intensities. Consistent with our findings, Hettiarachchi et al. [11] reported highly accurate heart rate values for the Polar OH1 monitor also placed on the forearm and upper arm (CE = -0.15 to 0.55 bpm) across light to vigorous exercise intensities on the treadmill (ICC = 0.962 – 0.993) and cycle ergometer (ICC = 0.990 – 0.995). Collectively, the findings of present investigation and those of Hettiarachchi et al. [11] suggested arm-based heart rate monitors such as the OTBeat Burn^TM provides acceptable accuracy from low to maximal intensities during cycle ergometry as well as other exercise modes (i.e. treadmill) with higher potential for motion artifact. Despite the well-documented accuracy of heart rate chest straps, arm-based monitors may potentially provide better comfort, especially with the female population, due to its location and convenience on the body [11].

The available data on heart rate monitor reliability are very limited, indicating the importance of conducting more comprehensive studies on inter-device (i.e. location of placement) and intra-device (test-retest) precision. Determining strong inter-device reliability provides the user with options of device placement at different locations or body side based on personal preference and comfortability while measuring heart rate consistently. Previous studies [26,27,28,29,30] have shown reliability is influenced by the same factors as accuracy (mode and intensity of exercise, location of placement on the body). The current findings illustrated the OTBeat Burn^TM provided excellent inter-device reliability between the forearm and upper arm locations (ICC, R = 0.999, Figure 4). Khushhal et al. [29] and Abt et al. [26] reported the inter-device reliability (left vs. right wrist) of the Apple Watch during incremental treadmill exercise to have ICCs of R = 0.91-0.99 and 0.84, respectively. In addition, the Xiaomi Mi Band 2 exhibited improved inter-device reliability (left vs. right wrist) at higher intensities with ICC ranges of R = 0.47-0.71 [30], whereas the Mio Smart Bracelet provided highly variable reliability (ICCs, R = -0.05 – 0.959) based on differences in physical activity level of the subjects and exercise intensity [27]. Although test-retest reliability was not assessed in the present study, Climstein et al. [28] and Cai et al. [27] found intra-device ICCs of R = 0.58-0.92 and 0.53-0.70 during separate incremental treadmill tests to exhaustion, respectively. When compared to our findings, it can be concluded that heart rate devices designed for the wrist can influence inter-device reliability, potentially due to a relationship between motion artifact and distal placement of the device. Reliability was higher the more proximal the device was on the body, as seen with the OTBeat Burn^TM having better inter-device reliability at the forearm and upper arm locations when compared to wrist-based monitors (left vs. right) [26,29]. Furthermore, the high level of reliability of the OTBeat Burn^TM was likely due to less motion artifact associated with cycle ergometry versus treadmill running and potentially greater blood perfusion from gripping the ergometer handles, thereby creating a more stable environment for the assessment of heart rate.

There were a number of limitations in the present study. First, our sample consisted of apparently healthy, college-aged males. Therefore, it is unknown if our findings can be extended to females, older individuals, or clinical populations due to differences that may exist in cardiovascular function (e.g. cardiac output, maximal heart rate, endothelial function), body composition (e.g. amount of subcutaneous tissue between device and blood vessel), and other factors. Furthermore, we only examined heart rate responses during stationary cycle ergometry. Given that the OTBeat Burn^TM monitor is arm-based, it is possible more dynamic exercise (e.g. running, rowing, resistance training) could lead to greater motion artifact, thereby influencing the validity and reliability of this device.

5. Conclusions

In summary, the findings of the present study indicated the OTBeat Burn^TM exhibited highly acceptable validity metrics (i.e. MAPE < 5%, CE < 3 bpm, SEE < 5 bpm, and Lin’s CCC ≥ 0.9) during incremental cycle ergometry in healthy, college-aged males. Furthermore, there was a high degree of inter-device reliability (ICC, R = 0.999) between the forearm and upper arm placements across the heart rate range consistent with high-intensity exercise. Thus, the OTBeat Burn^TM is an acceptable heart rate monitor for Orangetheory® Fitness members that provides accurate and reliable values and can be useful for assisting in the maintenance of heart rate zones for achieving personal goals. Future studies should examine the OTBeat Burn^TM monitor in other modes of exercise due to its high accuracy and precision during cycle ergometry.