Assessing Area Under the Curve (AUC) with a Larger Number of Repeated Measures: A Simulated Mediation Analysis Comparing AUC Calculated with the Trapezoid-Rule and Definite Integrals with 30 and 100 Repeated Measures

1. Introduction

Researchers are beginning to employ Area Under the Curve (AUC) to assess change across time [1,2,3,4,5,6]. Utilizing an equation based on the trapezoid rule [7], researchers have demonstrated the validity of Area Under the Curve (AUC) as an efficacious alternative to longitudinal data analysis methods such as Latent Growth Curve Modeling (LGCM) and Generalized Estimating Equations (GEE) [8,9,10]. These results provide researchers interested in panel studies a simple alternative to multivariate statistical methods that often require purchasing expensive software, and additional study to understand the intricacies of these methods and the software. To date, however, studies exploring the efficacy of AUC in panel studies have focused on repeated measures data involving relatively few time points (e.g., three or six repeated measures). Although these numbers of repeated measurements are typical of traditional cohort studies in the social sciences and the health professions, evolving data collection technology often captures seemingly countless data points. For instance, wearable technology and apps such as Fitbit and Strava which are employed to assess physical fitness and athletic performance, can generate a staggering amount of data points for individuals even during a single exercise bout. Indeed, in a typical 30-day month, one may have thirty data points at minimum, if the data are aggregated daily across smaller units of time (e.g., seconds, minutes, & hours). This of course is higher if one focuses on workout parameters collected multiple times per minute. Examples are steps, cadence (revolutions per minute pedaling in cycling), or heart rate. With so much data and their exercise-related fluctuations, focusing on linear components such as trends and their shape may not be of primary interest to researchers, nor are they easily interpretable. Thus, novel technology necessitates assessing different methods for analyzing variables that transcend the most common approaches. AUC is one such method.

The purpose of this study was to assess AUC as a possible data analysis method to organize large numbers of data points into single variables for analysis. To achieve this goal, we generated two repeated-measures data sets from a cross-sectional sample of publicly available data on bicycling performance using the popular Strava app [11]. The data were taken from a user-defined segment to estimate time to completion in minutes, average speed in miles per hour (mph), and power measured in watts. Using actual performance data to inform our estimates, we generated simulated datasets of 30 and 100 repeated measures for each of 21 hypothetical cyclists on the three variables. Using these data, we compared three different AUC calculations for each hypothetical cyclist calculating AUC using the trapezoid rule, and definite integrals for the best fitting polynomial and linear equation. We hypothesized that the three methods would perform equally well in calculating AUC, and in a bivariate correlation analysis and multiple linear regression analysis, achieving the same pattern of results. Further, we used a mediation analysis to explore differences in performance between AUC with 30 versus 100 repeated measures. We hypothesize that the AUC results with 100 versus 30 repeated measures would differ due to the greater amount of performance data with 100 than 30 repetitions, although both resulting in a significant indirect effect from power to time through speed.

2. Materials and Methods

This study employed simulated data based on actual results from a Strava segment. Strava is a popular app that tracks athlete performance, such as a cyclist’s performance during a ride [11]. Cyclists, for instance, can compare their performance to the performance of other cyclists riding specific segments. Segments are user-defined portions of routes, such as a quarter mile sprint, a half-mile hill climb, or a 10-mile loop. For the current study, we used data on performance time, average MPH, and power (watts) to inform our data generation. Watts is an estimate of the average power in joules per second transferred from the cyclist to the bicycle through pedaling during the segment [12,13,14]. The segment employed here is an 8.17-mile (13.15 kilometers) time trial loop used by a local bicycle racing group and includes rolling hills with a total elevation gain of 522 feet (159.11 meters). As our data were estimated from a single performance (cross-sectional data), we estimated the most likely standard deviation values for the simulations from the author’s cycling experience using the Strava app on the selected segment, with one mile per hour for MPH (1.61 KPH), and two seconds for time. For watts standard deviation, we used 7 watts as it represented approximately 2% of the average watts for the segment, a value consistent with prior research findings on cycling performance [14]. With these data, we generated samples of 30 and 100 repeated measures (representing bicycle rides) for each of our 21-sample cyclists for all three variables: time, MPH, and power. We used SAS 9.4 statistical software to generate the simulated dataset. The code employed to generate the data, including the MPH, time, and power values, is included in the appendix.

Once the simulated data were generated, they were transferred to Microsoft Excel to identify the best fitting line and higher-order polynomial curve. For the polynomial curve, we attempted models with powers of two through six, as six is the highest power available in Excel using the graphing options. We used Excel as it is readily available software, and identifying the best fitting model is relatively simple. This was done by generating a scatter plot, adding a trend line, setting the trendline to polynomial, and then testing the various powers. To select the optimal power, we used the R² statistic, as it indicates the proportion of variance accounted for by the model [15]. To simplify our approach, we settled on a six-degree (sextic) polynomial as it was the best fitting curve for most repeated-measures data. For the linear model, we fitted the best line with no adjustment. Figure 1 presents the plots, equations, and R² values for the first cyclist for MPH (A), time (C), and watts (E) with the sextic equations on the left half and MPH (B), time (D), and watts (F) with the linear equations on the right half for 30 repeated measures data set. Figure 2 presents the same plots with equations and R² values for the 100 repeated measures data set. The remaining figures are available from the author upon request.

In addition to graphing the data and identifying the best-fitting curve for each hypothetical cyclist, we used the Excel function linest [16] to provide each cyclist’s equation for integration within Excel. The equations provided by linest are identical to the equations obtained with the graphing process. We next calculated the definite integral from 1 to 30, for the first to 30th observation, providing the area under the curve (AUC) between those two points (Equation (1)) [17]. We did this for the sextic model (Equation (2)) and the linear model (Equation (3)). We repeated the process for the dataset with 100 repeated measures. As such, we had one AUC for time, one for MPH, and one for watts for the linear models, and one each for the polynomial models, for a total of six AUC variables. These data were then added to the final dataset with the 30 estimated MPH, 30 estimated time, and 30 estimated watts values for each of the 21 simulated cyclists and uploaded the data to SAS for final analysis. We did the same for the dataset with 100 repeated measures. All data are available from the authors upon request.

$$AUC={\int }_a^{b}f\left(x\right)dx={\left[F\left(x\right)\right]}_a^b=F\left(b\right)-F\left(a\right)$$

(1)

$$AUC={\int }_1^{30}{a}_1{x}^6+a_2{x}^5+a_3{x}^4+a_4{x}^3+a_5{x}^2+a_{6}x+a_{7}dx$$

(2)

$${=\left[{\displaystyle \frac{a_1{x}^7}{7}}+{\displaystyle \frac{a_2{x}^6}{6}}+{\displaystyle \frac{a_3{x}^5}{5}}+{\displaystyle \frac{a_4{x}^4}{4}}+{\displaystyle \frac{a_5{x}^3}{3}}+{\displaystyle \frac{a_6{x}^2}{2}}+a_{7}x\right]}_1^{30}=F\left(30\right)-F(1)$$

$$AUC={\int }_1^{30}{a}_{1}x+a_{2}dx{=\left[{\displaystyle \frac{a_1{x}^2}{2}}+a_{27}x\right]}_1^{30}=F\left(30\right)-F\left(1\right)$$

(3)

For the standard AUC calculation (AUC with respect to the ground, AUC_g), we used the trapezoid rule, Equation (4) [7]. For sake of simplicity, we assumed that the time difference (x_n+1 – x_n) is equal to 1, suggesting equal spacing between consecutive rides. This reduced Equation (4) to Equation (5) [8]. These calculations were conducted within SAS. SAS code is available from the author upon request.

$${AUC}_g=\left[{\displaystyle \frac{y_2-y_1}{2}}\times \left(x_2-x_1\right)\right]+\left[{\displaystyle \frac{y_3-y_2}{2}}\times \left(x_3-x_2\right)\right]+\dots +\left[{\displaystyle \frac{y_T-y_{T-1}}{2}}\times \left(x_T-x_{T-1}\right)\right],$$

(4)

$$where{y}_{T}representsthevariablevalueattimetandxrepresentsthetimepointt,$$

$$withtrangingfrom1toT$$

$${AUC}_g={\displaystyle \sum _{i=1}^{T-1}}{\displaystyle \frac{t}{2}}\left(y_{i+1}+y_i\right),wherei=timepointt,whichrangesfrom1toT$$

(5)

To compare the relative efficacy of the three AUC calculations, we used bivariate correlation analysis and multiple regression analysis. In the multiple regression models, time was the criterion (outcome) variable, and watts (referred to as power hereafter) was the predictor variable. MPH served as a confounding variable, a variable related to both the predictor and outcome variable [8]. However, instead of simply employing MPH as a control variable, we also assessed its role as a possible mediator between power and time. As such, for our assessment of mediation, the research question was whether greater power is associated with lower time, and whether the effect if any is channeled through MPH. We used both methods (bivariate correlations and multiple regression analysis) as each allows for a unique assessment of the efficacy of the trapezoid rule and definite integral calculation methods and the effects with 30 versus 100 repeated measures. We provide R² and Root Mean Square Error (RMSE) [18] to facilitate comparison among the different regression models with 30 and 100 repeated measures.

To assess our mediation hypothesis, we employed the Baron and Kenny approach and bootstrapped standard errors [19,20]. Although the use of the Baron and Kenny approach is widely considered obsolete given what is now known about the behavior of mediated effects [20], we believe that it still provides an informative starting point for mediation analysis. We use Figure 3 to facilitate the explanation of this process. According to the Baron and Kenny approach, for mediation to be plausible, the path from the putative predictor variable (power) to the outcome variable (time; path a) must be significant. Further, the path from the putative predictor variable to the presumed mediator variable (MPH; path b) must also be significant. Finally, the path from the presumed mediator to the outcome variable (path c) must be significant. The last path is assessed in the full multiple regression model including both the potential predictor and mediator. These prerequisites present, one can conclude mediation if the inclusion of the presumed mediator in the regression model results in a non-significant direct effect (path a).

We followed the Baron and Kenney analysis with the calculation of the indirect effect (paths b and c) and its 95% confidence interval employing the Process macro with bootstrapped standard errors. This is a critical step as the Baron and Kenney approach implies, although not made explicit, the existence of a normally distributed and significant indirect effect when the correct conditions apply. The indirect effect is the product of the two constituent paths (a*b). However, this product is not necessarily normally distributed, affecting the validity of the standard errors which can result in increased likelihood of an inaccurate conclusion [15,21,22]. To deal with this important issue, Hayes’s process macro [23] uses bootstrapping to calculate standard errors without assuming a normally distributed indirect effect, thereby allowing us to verify the validity of a presumed mediated effect found using the Baron and Kenney approach by means of confidence intervals, along with testing whether the indirect effect supports a conclusion of complete or partial mediation.

3. Results

Table 1 presents the means, standard deviations, and minima and maxima for the three AUC calculations for MPH, time, and power for the 30 and then 100 repeated measures (rides). The values are similar for all three calculations for each facet in Table 1. Table 2 presents the bivariate correlations among the nine AUC variables, with 30 rides. All correlations were significant, with the correlations involving the power AUC calculations being the lowest, especially those involving MPH, p=0.03. As expected, the correlations between time and MPH were negative and highly significant, p<0.0001. Table 3 presents the bivariate correlations among the nine AUC variables, with 100 rides. Like the results for the 30 rides, the lowest correlations were those involving power and MPH, p=0.01.

Table 4 presents the three multiple regression analysis results for models employing AUC calculations with the trapezoid rule (Model 1), definite integral for the linear model (Model 2), and the definite integral for the sextic model (Model 3), with 30 simulated rides per cyclist in the top half. For the models with 30 rides, all three AUC calculations performed equally well with RMSE for model 1, model 2, and model 3 being 8.802, 8.95, and 8.914, respectively. Both MPH and power were significantly and negatively related to time. The residual plot diagrams for the three models appear in Figure 4. The bottom half of Table 4 presents the three multiple regression analysis results for models employing AUC calculations with the trapezoid rule (Model 1), definite integral for the linear model (Model 2), and the definite integral for the sextic model (Model 3), with 100 simulated rides per cyclist. All three AUC calculations performed equally well with RMSE for model 1, model 2, and model 3 being 28.959, 28.678, and 29.029, respectively. Unlike the model with 30 simulated rides, the effect of power on time was not significant, despite the identical pattern of effects as with the 30 simulated rides. The residual plot diagrams for the three models in the bottom half of Table 4 appear in Figure 5. The residuals are more disperse with the 100 simulated rides than with the 30 simulated rides. This is consistent with the RMSE being over three times larger with the 100 simulated rides compared to the 30 simulated rides.

The difference in the results in Table 4 between the two simulated rides datasets suggest we may have partial mediation with the 30 rides and full mediation with the 100 rides. Table 5 presents the direct and indirect effects for the impact of power on time using the Process macro.

To assess mediation using the Baron and Kenny approach we conducted two separate simple regression analyses to supplement the data in Table 3, Table 4 and Table 5, as we needed to assess the direct effect of power on time without MPH in the model, along with the direct effect of power on MPH. Focusing on the results from the trapezoid rule calculation and the sample with 30 rides, the direct effect of power on time was significant (b=-0.0116, t=-3.40, p=0.0030). Moreover, the direct effect of power on MPH was also significant (b=0.0093, t=2.2966, p=0.0332). These results when taken together with the effects of MPH on time in Table 4 (b=-0.7757, t=-9.9756, p<0.0001), meet the requirements to assess mediation using the Baron and Kenny approach. However, given that the effect of power on time remained significant after the inclusion of MPH (Table 4 and repeated in Table 5 for convenience), mediation if present is partial, not full. The bootstrap results (Table 5) do indeed support a significant indirect effect from power to time through MPH (b=-0.0072, 95%CI= -0.0162, -0.0011) given that the 95%CI does not cross 0, the null population mean. To determine the percent mediated, we divided the indirect effect by the total effect and get,

$$\mathit{\%}\mathit{m}\mathit{e}\mathit{d}\mathit{i}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}={\displaystyle \frac{\mathit{s}\mathit{p}\mathit{e}\mathit{c}\mathit{i}\mathit{f}\mathit{i}\mathit{c}\mathit{i}\mathit{n}\mathit{d}\mathit{i}\mathit{r}\mathit{e}\mathit{c}\mathit{t}\mathit{e}\mathit{f}\mathit{f}\mathit{e}\mathit{c}\mathit{t}}{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}\mathit{e}\mathit{f}\mathit{f}\mathit{e}\mathit{c}\mathit{t}(\mathit{i}\mathit{n}\mathit{d}\mathit{i}\mathit{r}\mathit{e}\mathit{c}\mathit{t}+\mathit{d}\mathit{i}\mathit{r}\mathit{e}\mathit{c}\mathit{t})}}={\displaystyle \frac{-0.0072}{(-0.0072+-0.0044)}}=0.621~62\mathit{\%}$$

As such, we can conclude that 62% of the effect of power on time is channeled through MPH.

The remaining results for the 30 repeated measures (i.e., sextic and linear models) are identical and are therefore not presented here.

Moving to the analysis with 100 rides and once again focusing on AUC calculated with the trapezoid rule, all criteria are fulfilled to support mediation using the Baron and Kenny approach. The effect of power on time was significant (b=-0.01128, t=-3.31, p=0.0037). Moreover, the effect of power on MPH was also significant (b=0.0105, t=2.8604, p=0.01). This taken together with the effects of MPH on time in Table 4 (b=-0.8608, t=-10.3511, p<0.0001), support the ability to assess and perhaps conclude mediation using the Baron and Kenny approach. Unlike the results with 30 rides, however, the effect of power on time (Table 5) was no longer significant after the inclusion of MPH in the multiple regression model (b=-0.0023, t=-1.4221, p=0.1721), suggesting full mediation. The bootstrap results do indeed support this conclusion, with a significant indirect effect from power to time through MPH (b=-0.0090, 95%CI= -0.0169, -0.0032) given that the 95%CI does not cross 0, the null population mean.

4. Discussion

Recently, there has been a proliferation of data sources aimed at helping individuals better control their health, including wearable devices and fitness apps [24,25,26]. For instance, heart rate data, steps, and blood pressure streamed live on smart watches and apps allow individuals to make lifestyle adjustments that can improve health. Athletes use this information to better assess performance. Researchers, however, must identify optimal statistical methods to deal with the large amount of data produced to test their hypotheses. Although there are many valuable methods for working with repeated measures data, not all were designed specifically for high-frequency repeated measurements that can span 100s of data points.

One option that may be useful in the assessment of high-frequency repeated measures is Area Under the Curve (AUC). Although past studies have validated its use in calculating variables to capture panel data, most have focused on relatively few repeated measurements [8,9,10]. As such, the aim of the present study was to begin to understand how AUC performs in studies involving a larger number of repeated measures. Toward that aim, we employed three different calculations of repeated measures of speed, time, and power in a real-world scenario to explore the relative efficacy of each in bivariate correlations and a mediation analysis. Our results suggest that whether calculating AUC using definite integrals or the trapezoid rule, there is little discrepancy in the results.

One interesting finding was the difference in the results of the mediation analysis between the analysis with 30 repeated measures (simulated bicycle rides) and that with 100 rides. While there were significant indirect effects in both analyses regardless of how AUC was calculated, total mediation was found with 100 but not 30 rides. This result may speak to the importance of collecting larger amounts of data to better understand individual performance, as there is a greater ability to assess an individual performance across diverse situations. For instance, in a sport like cycling where conditions (e.g., temperature, humidity, and wind) vary across different days, a greater number of repeated measures may be more sensitive to deviations in performance parameters.

Another key observation is how easily AUC can summarize a vast amount of data on multiple variables such as 100 repeated measures for each participant for three variables in this study. This provides a tremendous amount of flexibility for researchers desiring to assess complicated theoretical models using methods such as regression analysis or structural equation modeling (SEM). For instance, conducting this same mediation analysis with methods such as LGCM is feasible but would involve 100 measured variables for each of the three processes before even including control variables. We only needed three using AUC.

Regarding the best choice for calculating AUC, there is no evidence to support any one calculation method over the other. Given the simplicity of a linear model and the calculation of AUC integrating with a linear equation, it may be a preferred choice for some. However, we did not provide any evidence here against employing the trapezoid-rule based equation either. A drawback, however, is its calculation given the number of polygons required when faced with a continual stream of measurements. A recursive program could however alleviate this issue. One could also argue that finding the best fitting polynomial for everyone in a within participant design is daunting, although well-designed software would make the task manageable.

There are several limitations to this study. First, the data were generated by simulation based on data gathered from an app along with the author’s experience cycling a selected segment. As such, the estimates of variance may not accurately reflect the variation in a real dataset. Second, we used equal spacing for the calculation of AUC employing the trapezoid rule. This is the most likely scenario when assessing data accrued per minute or hour, for instance. However, future studies should vary the distance between repeated measures and compare the different methods under those constraints as there will inevitably be circumstances when data are not acquired with equal spacing such as when weather affects one’s ability to exercise outdoors. Finally, the author only included three variables in this analysis. It would be interesting to test more complicated models involving a greater number of variables and using methods such as SEM to test hypotheses.

5. Conclusions

The limitations notwithstanding, the results of this study provide initial insights into the relative efficacy of three different methods to calculate Area Under the Curve when dealing with many repeated measures. Future studies should employ these methods with even larger numbers of data points and assess other equations to calculate AUC. Further, studies should be conducted using real data instead of simulations.