Comparative Efficacy of Statistical Learning and Ensemble Methods for Multivariate Obesity Condition Classification

1. Introduction

Obesity is a major health problem in the world and is strongly associated with chronic diseases like type 2 diabetes, cardiovascular diseases and various forms of cancer. Over the past decades, there has been a significant increase in obesity rates in some Latin American countries, including Mexico, Peru and Colombia, which is due to the changes in their dietary habits and increased sedentary lifestyle.

The primary research question in this study is to determine the best statistical learning model for accurately and robustly classify obesity level and efficiency of modern ensemble methods compared to the traditional statistical methods.

This study employs a dataset that was collected and published by [6] and made publicly available by the UCI Machine Learning Repository (UCIMLR) [9]. It has 2111 records, and 17 attributes. The variable to be classified is the categorization NObeyesdad with 7 categories: Insufficient Weight, Obesity Type I, Obesity Type II, Obesity Type III. The 23% medical data is collected directly from users via a web platform in Mexico, Peru and Colombia, as data of this scale and balance is not easy to obtain, the remaining 77% was synthetically generated using the Weka SMOTE (Synthetic Minority Over-sampling Technique) filter to balance the classes.

2. Related Works

Machine learning methods have increasingly been found useful in recent years in healthcare, particularly for predictive modeling and risk classification. It has been shown previously that algorithms like Random Forests and Support Vector Machines (SVM) have generally better performance in classification problems with complex, non-linear relationships than traditional statistical methods [1,2]. Random Forest can be used to deal with variance and overfitting by performing bootstrap aggregating, whereas SVM can be used to optimize classification by finding hyperplanes in high dimensional spaces. In some cases multi-class health data is noisy and highly imbalanced and SVMs may not work on such data.

Concerning obesity prediction, [5] found that the ensemble models had the highest accuracy of 89.68%, as they were able to capture the complicated relationships between different types of lifestyle factors. Similarly, [3] showed that structured tabular data generally performs well with gradient boosting models like XGBoost, which improve iteratively by decreasing the error of classification, compared to single-model methods, while also showing higher accuracy and robustness.

A solution to the issue of class imbalance still needs to be developed for these medical datasets. One such technique that has been widely adopted is the Synthetic Minority Over-sampling Technique (SMOTE) [4] which has been widely used to improve the classification accuracy by generating synthetic observations in the minority classes.

There is, however, limited of literature available which studies these predictive models independently or critically investigates the interaction between class imbalance techniques and model selection. To address this, the study will compare five models systematically in the same stringent cross validation setting, which are parametric and non-parametric models, and select the best model architecture for the multivariate obesity classification.

3. Methods

3.1. Data and Preprocessing

There are 17 attributes, and 2111 participants in the data. The population is highly balanced by gender, consisting of 1,043 females (49.4%) and 1,068 males (50.6%). The attributes are continuous variables such as the Age, Height, Weight, and categorical variables, which are related to lifestyle choices, for example, the Frequency of Consumption of Vegetables [FCVC], Number of main Meals [NCP] or the Physical activity Frequency [FAF].

There were several pre-processing steps preceeding the training of the models. One-hot encoding with no reference category dropped and with the option fullRank = TRUE in the dummyVars function from the caret package, was used to encode categorical predictor variables, with no intention of dropping any variables to prevent multicollinearity. Second, the predictors showing a negligible variance were identified and dropped from the dataset with the function nearZeroVar which identifies predictors with insufficient discriminatory information. Third, the features that were highly collinear (i.e. with correlation between the two features greater than 0.95) were removed using a Pearson correlation filter. The last feature matrix was filtered and then standardized within the folds, without leakage data into the folds by only using the statistics for the training fold in each fold.

3.2. Statistical and Machine Learning Methods

Five machine learning algorithms were developed to classify obesity level. Each model was chosen to give a range of comparative baseline models from the rigid parametric assumption through to the very flexible non-linear mapping. Suppose that $\mathbf{x}\in {\mathbb{R}}^p$ is the standardized feature vector for a particular observation and that $K=7$ is the number of classes of obesity.

XGBoost (eXtreme Gradient Boosting)

XGBoost trains a collection of T regression trees to minimize the following cost function: a differentiable loss function over all N training data points, added to a regularization term on the complexity of all the trees [3]:

$$\mathcal{L}\left(\varphi \right)={\displaystyle \sum _{i=1}^n}\ell \left({\widehat{y}}_i^{\left(t\right)},y_i\right)+{\displaystyle \sum _{t=1}^T}\Omega \left(f_t\right),\Omega \left(f_t\right)=\gamma {\mathcal{T}}_t+\frac{1}{2}\lambda {\parallel w_t\parallel }^2,$$

(1)

where $\ell (·)$ is the multiclass log-loss (mlogloss), ${\mathcal{T}}_t$ is the number of leaves in tree t, $w_t$ is the vector of leaf weights, and $\gamma$, $\lambda$ are regularization constants. The ensemble prediction is the sum of all tree outputs:

$${\widehat{y}}_i={\displaystyle \sum _{t=1}^T}{f}_t\left({\mathbf{x}}_i\right).$$

(2)

Class membership is recovered via the multi:softmax objective. Hyperparameters: max_depth = 6, eta = 0.1, nrounds = 100.

Assumptions: The observations are independent of each other. There are no distributional assumptions for the predictors. Features can have arbitrary non-linear relationships with the outcome. The following trees are scaled to the negative gradient The (pseudo-residuals) of the loss with respect to the current ensemble prediction.

Random Forest

Random Forest is a combination of B classification trees, where each tree is built on a random subset of the training data (bootstrap sample) [1]. The final class prediction is the one with the largest number of votes:

$$\widehat{y}=\underset{k}{argmax}{\displaystyle \sum _{b=1}^B}\mathbf{1}\left[h_b\left(\mathbf{x}\right)=k\right],$$

(3)

where $h_b\left(\mathbf{x}\right)$ is the class predicted by the b-th tree. At each node split, only a random subset of $\mathtt{mtry}=\lfloor \sqrt{p}\rfloor$ features is considered, reducing tree correlation. Node purity is measured by the Gini impurity:

$$G=1-{\displaystyle \sum _{k=1}^K}{\widehat{p}}_k^2,$$

(4)

where ${\widehat{p}}_k$ is the proportion of class-k observations at the node. Hyperparameters: ntree = 500, mtry = $\lfloor \sqrt{p}\rfloor$.

Assumptions: Observations are independent. No distributional assumptions are placed on the predictors. Enough trees are provided if there are enough trees for the shelter they require. is obliged to bring the variability of the vote to a standstill; by the law of large numbers, it is guaranteed that convergence of the ensemble error as $B\to \infty$.

Multinomial Logistic Regression

Multinomial logistic regression models the conditional probability of class k as a softmax function of linear combination of the predictors:

$$P(Y=k\mid \mathbf{x})={\displaystyle \frac{exp\left({\mathbf{x}}^{\top }{\beta }_k\right)}{{\displaystyle {\displaystyle \sum _{j=1}^K}exp\left({\mathbf{x}}^{\top }{\beta }_j\right)}}},k=1,\dots ,K,$$

(5)

where ${\beta }_k\in {\mathbb{R}}^p$ is the coefficient vector for class k. Parameters are estimated by maximizing the penalized log-likelihood with weight decay regularization ($\lambda =0.01$):

$${\ell }_{\mathrm{pen}}\left(\beta \right)={\displaystyle \sum _{i=1}^n}logP(Y=y_i\mid {\mathbf{x}}_i)-\lambda {\displaystyle \sum _{k=1}^K}{\parallel {\beta }_k\parallel }^2.$$

(6)

Assumptions: (1) Independence of irrelevant alternatives (IIA): The log-odds for any two classes are independent of the other classes. (2) Linear in the log-odds: The log-odds $log\left[P\right(Y=k)/P(Y=K\left)\right]$ are a Function that is linear in $\mathbf{x}$. (3) Observations are independent. (4) No perfect multicollinearity among predictors. These assumptions are frequently not met by High dimensional, non-normally distributed health data.

Linear Discriminant Analysis (LDA)

LDA uses linear combinations of the features to maximally separate the K classes. Using the class conditional densities that are Gaussian, with a common covariance matrix $\Sigma$, Bayes’ theorem yields the linear discriminant function:

$${\delta }_k\left(\mathbf{x}\right)={\mathbf{x}}^{\top }{\Sigma }^{-1}{\mu }_k-\frac{1}{2}{\mu }_k^{\top }{\Sigma }^{-1}{\mu }_k+log{\pi }_k,$$

(7)

where ${\mu }_k$ is the class-k mean vector, ${\pi }_k$ is the class prior, and $\Sigma$ is the pooled within-class covariance matrix, estimated as:

$$\widehat{\Sigma }={\displaystyle \frac{1}{n-K}}{\displaystyle \sum _{k=1}^K}\sum _{i:y_i=k}({\mathbf{x}}_i-{\widehat{\mu }}_k){({\mathbf{x}}_i-{\widehat{\mu }}_k)}^{\top }.$$

(8)

An observation is assigned to the class $k^{*}=arg{max}_k{\delta }_k\left(\mathbf{x}\right)$.

Assumptions: (1) Multivariate normality: the predictors Class data are assumed to be normally distributed (multivariate Gaussian distribution) within each class. (2) Homoscedasticity: the covariance matrix is the same for each of the K classes, $\Sigma$. (3) Observations are independent. Violations of assumptions (1) and (2) - both expected in real-world health data - directly inflate classification error.

Support Vector Machine (SVM) with RBF Kernel

The SVM looks for the hyperplane with the largest margin in the feature space induced by the kernel. It is a solution to the following equation in the soft-margin formulation:

$$\underset{\mathbf{w},b,\xi }{min}{\displaystyle \frac{1}{2}}{\parallel \mathbf{w}\parallel }^2+C{\displaystyle \sum _{i=1}^n}{\xi }_i\mathrm{subject}\mathrm{to}{y}_i\left({\mathbf{w}}^{\top }\varphi \left({\mathbf{x}}_i\right)+b\right)\ge 1-{\xi }_i,{\xi }_i\ge 0,$$

(9)

where $\varphi (·)$ is the mapping of the inputs to a higher-dimensional Hilbert space, while C is the regularization parameter, where cost = 1 and ${\xi }_i\ge 0$ is a slack variable for soft-margin constraints. The inner product in feature space is implicit in the radial basis function (RBF) kernel:

$$K({\mathbf{x}}_i,{\mathbf{x}}_j)=exp\left(-\gamma \parallel {\mathbf{x}}_i-{\mathbf{x}}_j{\parallel }^2\right),$$

(10)

where $\gamma >0$ controls the width of the Gaussian kernel. Multi-class classification using one-vs-one (OVO) decomposition, fitting $\left({\displaystyle \genfrac{}{}{0pt}{}{K}{2}}\right)=21$ binary classifiers and assigning each observation to the class with the most votes.

Assumptions: (1) No parametric distributional form is assumed for the predictors. (2) The margin-maximization principle requires that the classes be separable, if not in a soft-margin sense, in the space induced by the kernel. (3) Observations are independent. (4) The RBF kernel assumes implicitly smooth class boundaries and locally Euclidean class structures, which may be problematic when classes are very overlapping in the original feature space.

3.3. Model Evaluation

All models were tested in 10-fold cross validation framework for out-of-sample reliability and to prevent over fitting. The dataset was split into $K_{\mathrm{cv}}=10$ stratified folds, with each fold k used for test set and the rest used for train set. The mean and standard deviation were used as standardization parameters only in the training folds, and not in the held out fold, thus avoiding data leakage.

The performance was quantitatively measured using four macro-averaged metrics calculated by averaging across the $K=7$ obesity classes. Assume that there are k classes and that the true positive rates, false positive rates and false negative rates are denoted by ${\mathrm{TP}}_k,{\mathrm{FP}}_k,{\mathrm{FN}}_k$ respectively.

$$\begin{array}{cc}\hfill \mathrm{Accuracy}& ={\displaystyle \frac{{\displaystyle {\displaystyle \sum _{k=1}^K}{\mathrm{TP}}_k}}{{\displaystyle {\displaystyle \sum _{k=1}^K}\left({\mathrm{TP}}_k+{\mathrm{FP}}_k+{\mathrm{FN}}_k\right)}}},\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \mathrm{Precision}& ={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle \frac{{\mathrm{TP}}_k}{{\mathrm{TP}}_k+{\mathrm{FP}}_k}},\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill \mathrm{Recall}& ={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle \frac{{\mathrm{TP}}_k}{{\mathrm{TP}}_k+{\mathrm{FN}}_k}},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill F_1& ={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle \frac{2·{\mathrm{Precision}}_k·{\mathrm{Recall}}_k}{{\mathrm{Precision}}_k+{\mathrm{Recall}}_k}}.\hfill \end{array}$$

(14)

McNemar’s test [7] was used to determine the statistical significance of any pair-wise differences in performances. If b is the number of cases that are classified as disagreements between the best model and the alternative model and c is the number of cases that are classified as disagreements between the alternative model and the best model, then the test statistic with continuity correction is:

$${\chi }^2={\displaystyle \frac{{\left(|b-c|-1\right)}^2}{b+c}},$$

(15)

which is distributed as a ${\chi }^2$ with one degree of freedom under the null hypothesis of equal error rates. The significance level used was set as $\alpha =0.05$.

4. Results

4.1. Descriptive Summaries and Feature Analysis

The distribution of the data is very uniform within each of the seven obesity groups. As seen in Figure 1, the populations were split into five classifications, ranging from 12.9% to 16.6% of the total population.

Table 1 provides a descriptive summary of the primary physical attributes stratified by gender. While the mean age is similar between females ($M=24.00$, $SD=6.41$) and males ($M=24.62$, $SD=6.27$), males in the dataset present with higher average height and weight. Notably, the weight variance for females ($SD=29.72$) is substantially higher than for males ($SD=21.41$), indicating a wider spread of extreme weight values in the female cohort.

The feature importance analysis based on Gini impurity (see Figure 2) shows that Weight and Height have the highest Gini scores of 560.8 and 166.7, respectively, suggesting that they are the most prominent features in the classification process. There are also high predictive importance for certain other variables, such as Age (150.3) and FCVC (vegetable consumption, 142.7). The strong signal of lifestyle factors (FCVC) proves that obesity classification is not solely based on the anthropometric values.

The separability of obesity classes in a reduced dimensional space was evaluated by a principal component analysis (PCA) on the standardized feature matrix (Figure 3). Together, the first two principal components explain 31.7% of the total variance, with the first principal component accounting for 19.0%, and the second for 12.7%. The resulting biplot shows largely separate clusters for the seven obesity categories, and the separation between Insufficient Weight and the higher levels of Obesity are clearly seen, which is consistent with the feature importance results displayed in Figure 2 showing that Weight is the most dominant feature.

4.2. Model Performance

The results of 10-fold cross validation are summarized in Table 2. XGBoost achieved the highest performance across all metrics, with an accuracy of 97.54% and an F1-Score of 0.9747. Random Forest was very close behind with an accuracy of 95.64%. The results in Figure 4 show that XGBoost was consistent across all 10 folds.

The confusion matrix for Figure 4 is shown in Table 3. They obtained a highly accurate classification with the XGBoost model, achieving an overall accuracy of 97.54%. The most common misclassifications are between the neighboring weight categories (e.g. Normal Weight and Overweight Level I), which is physiologically predictable because the boundaries between weight categories are continuous. The pairwise significance tests using McNemar’s method showed that XGBoost’s superiority is statistically significant when compared to all other models being analysed ($p<0.0001$;Table 4).

Interpretation: These two models, XGBoost and Random Forest, have shown to be more effective compared to other models, highlighting that obesity indicators are very non-linear. The complex real-world interactions between the physical and lifestyle variables made the classic models, such as LDA, which have strict parametric assumptions (that is, equal covariance matrices for the seven obesity classes) inadequate. The iterative corrections made by the previous trees in XGBoost helped the algorithm be better able to capture these complex interactions between features than SVM or Logistic Regression.

5. Discussion

The results show that ensemble learning methods such as XGBoost gave a very accurate prediction of obesity level. Both of these predictors (Weight and Height) are important in defining body mass, confirming the central role of these measures in any good obesity classification system, and the contribution of the six other lifestyle variables (such as Frequency of consumption of vegetables (FCVC)) reinforces the need to include the behavioral context in any sound obesity classification system.

The results obtained from this methodological approach indicate that classical statistical models may not be suitable for high-dimensional health data, given its non-normal distribution. It is better to go with ensemble approaches as they can be more flexible in their structure and don’t require any distributional assumptions.

Limitations and Bias: The accuracy of the study is very high (97.54%), but the disadvantage is the high percentage of synthetic data used in this work. The machine learning models have the potential to have learned fake distribution structures instead of real-world variance in the dataset, since 77% of the dataset was created using SMOTE before the evaluation stage. Due to their tightly clustered and unnatural patterns, synthetic oversampled data can be exploited by models such as XGBoost. As a result, these very good performance ratings are likely to be over-optimistic and have serious external validity issues. It is likely to be much less accurate in the real world, using a non-synthetic patient population. In addition, the sample size is limited to three Latin American countries (Mexico, Peru, Colombia) and results may not be applicable to other countries and communities in Latin American countries regarding dietary patterns and healthcare systems.

Implications and Future Work: Although there are limitations on the synthetic data, the results are applicable to public health. Basic physical features can be used together with behavioural survey data in a gradient boosting model, which have the potential to accurately predict the obesity stage and to prevent clinical interventions at higher risk of obesity stages. Future studies could use the following directions: (1) expanding the authentic, non-synthetic component of the data set by using ongoing data collection via web platforms, to validate the effectiveness of the ensemble models in predicting real populations; (2) using post-hoc interpretability methods such as SHAP (SHapley Additive exPlanations) values to generate clinically-actionable feature-level explanations of individual predictions; (3) testing model performance in external, geographically-distributed validation cohorts, to assess cross-cultural generalizability; and (4) investigating other ensemble methods such as stacking or voting classifiers, and determining their performance and whether further accuracy gains are achievable beyond XGBoost.

AI Tools Disclosure: This manuscript was not prepared with the aid of AI writing or language tools. The authors shall be solely responsible for any errors of fact or omissions in or resulting from the use of this information.