2.1. Data
We utilised a dataset from randomised controlled trials assessing the effectiveness of mobile messaging psychosocial interventions for depression and subthreshold depression among older adults in resource-limited settings in Brazil [
13]. Specifically, we focused on the groups that did not receive the psychosocial interventions, which comprised two subgroups: one with 305 individuals identified with depressive symptomatology (9-item Patient Health Questionnaire (PHQ-9) scores≥ 10) and another with 231 individuals experiencing subthreshold depressive symptomatology (9-item Patient Health Questionnaire (PHQ-9) scores between 5 and 9, inclusive).
The data included baseline information on socio-demographics (gender, age, marital status, race), self-reported general health (hypertension, diabetes, depression, balance), depressive symptomatology (PHQ-9), anxiety symptomatology (GAD-7), loneliness (3-item UCLA), health-related quality of life (EQ-5D-5L), and capability well-being (ICECAP-O).
Additionally, the dataset provided information on two follow-up periods: the first at three months after receiving a single message (follow-up 1) and the second at five months after baseline (follow-up 2). For these follow-ups, the only data included were related to depression (PHQ-9) and anxiety (GAD-7).
2.3. Bayesian Network Model
To comprehend the relationships between the baseline data and our outcome, we developed a Bayesian network model. A Bayesian Network is a directed acyclic graph comprising nodes and edges, utilised to construct an approximation of the joint probability distribution over all variables and the outcome of interest. This joint probability distribution provides all necessary information for making probabilistic inferences on one variable given knowledge of the other variables in the distribution. Nodes represent variables, while directed edges, depicted by arrows, elucidate relationships among these variables [
14].
In our approach, we learned the BN from data [
15,
16]. We utilised a bootstrap approach [
17] to generate 1,000 samples of the BN using a constraint-based algorithm, named Incremental Association Markov Blanket (IAMB) [
18] . The IAMB algorithm effectively constructs Bayesian networks by incrementally including variables based on mutual information [
19]. The latter which quantifies the statistical relationship between variables. This approach helps to identify relevant variables and construct a network that precisely represents the relationships found in the data.
A bootstrap approach ensures that the results are robust and helps minimise false patterns. This method outputs a summary table, illustrated in
Table 1, that shows pairs of variables (called "features") and their relationships. The first two columns, "From" and "To," show the features associated with the outcome. The last two columns, "Strength" and "Direction," indicate the likelihood of a connection between the pairs and the direction of this connection. This process helps us understand which variables are related to the outcome, how strong their relationships are, and the direction of these relationships [
20].
We only selected feature pairs where our outcome was present in either the "From" column or the "To" column. This selection was made because we are specifically interested in understanding the relationship between the outcome and the features at baseline. Additionally, we filtered only those directions with a probability greater than 50% and then ordered the summary table in descending order based on strength.
Next, we began constructing the first BN using only the first row of the table. As depicted in
Table 1, this initial BN would consist of two nodes, with the "feature 1" node having an arrow pointing towards the "outcome" node (
Figure 1 (A)). We then proceeded to build another BN using the first two rows. In this case, our outcome node receives arrows from both the "feature 1" and "feature 2" nodes, as illustrated in
Figure 1 (B). We continued this process until all the connections from the bootstrap output had been used.
For each Bayesian network defined by the incremental addition of nodes ordered by strength, we computed the conditional probability distributions (CPD) associated with each node. This process, known as parameter learning, involves applying the Bayesian method [
21]. Next, predictions for the outcome were made using exact inference [
14], where the posterior probability is calculated based on a set of events. These events consist of all possible values of the nodes connected to our outcome.
Specifically, for cases (A), (B), and (C) in
Figure 1, the joint probability distributions can be described by equations
1,
2, and
3, respectively.
where
,
,
are the probability distributions of feature 1, 2 and 3.
represents the conditional probability distribution of the outcome given feature 1.
denotes the conditional probability distribution of the outcome given feature 1 and 2, and
describes the conditional probability distribution of the feature 3 given the
outcome.
We define features 1 and 2 as “parent” node of the “child” node
outcome, while feature 3 is the “child” of its “parent” node
outcome. Therefore, we can generalise the joint probability distribution for all other Bayesian network structures as shown in equation
4.
where
denotes the parent of node
. The conditional probability for a node without parents is simply its prior probability
.
To develop the Bayesian Network model, we utilised the bnlearn package in R [
22].
2.4. Model Evaluation and Inference
To rigorously evaluate our data-driven approach statistically, for each BN constructed as explained in the previous section, we employed repeated cross-validation with four folds, repeated 25 times using different random samples from the training data. In each of the 25 iterations, the dataset was divided into subsets, allowing the models to be trained on three folds and validated on the remaining fold. This process was conducted solely with the training data, which represents 70% of our total dataset to avoid any bias when evaluating the performance of the model on the unseen test data.
For each fold, using the predictions described in the previous section, we recorded the Area Under the receiver operating Characteristic curve (AUC), F1-score, and the threshold that maximised both sensitivity and specificity simultaneously, as determined by Youden’s index, in both the training folds (three folds) and the validation fold (one fold).
Then, we selected the BN that achieved the highest AUC value in the validation part to be tested on the remaining 30% of the data that had not been used during training. Using the BN that maximises the AUC, we learned the parameters on the entire training dataset, enabling us to generate new predictions for the test data using exact inference and the optimal threshold identified in the repeated cross-validation. Additionally, we calculated AUC and F1-score values on the test data for comparison with the cross-validation results.
Utilising the chosen BN and the learned parameters, we proceeded with an inference analysis by calculating the conditional probability tables of our outcome given the features individually. Moreover, we investigated the combinations of the features, assessing how their different values would collectively impact the outcome. We then analysed the individual contributions of features and also their interactions, elucidating potential synergistic or antagonistic effects on the outcome variable. This analysis was conducted by calculating probabilities, marginalising the selected features in each scenario.
2.5. Machine Learning Models Comparison
The Bayesian network model applied in this study was compared with three other models: Logistic Regression, the SGDClassifier [
23], and XGBoost [
24]. To ensure a fair comparison, we maintained identical splits of training and testing data across all four models.
We used Recursive Feature Elimination with Cross Validation (RFECV) to select the features for developing the three models. RFECV [
25] is a feature selection method that iteratively eliminates the least important features from the dataset, while evaluating the model’s performance through cross-validation. This process uses a supervised learning estimator that provides information regarding feature importance. In this study, we used Logistic Regression as the estimator. After selecting the features, we used the same selected features for all three models.
It is common practice to fine-tune the parameters when developing a model, a process known as hyperparameter optimisation. This typically enhances the model’s performance. To fine-tune the hyperparameters and evaluate the model’s performance in a robust and unbiased manner, we applied nested cross-validation [
26]. Nested cross-validation involves two main loops: an outer loop and an inner loop. In the outer loop, the dataset is divided into multiple folds. In each iteration of the outer loop, one fold is reserved as the test set, and the remaining folds are used as the training set. The outer loop is dedicated to evaluating the model.
In the inner loop, nested within the outer loop, hyperparameter tuning is performed. Here, the training set is further divided into folds, with one fold held out as a validation set, while the rest are used for training. Multiple models are trained and evaluated within this loop to find the best-performing set of hyperparameters.
After the inner loop completes, the set of hyperparameters that provided the best performance on the training folds is used on the test dataset reserved in the outer loop. The best performance is usually measured by maximising a defined metric, such as AUC or F1-score. The performance metrics obtained from each iteration of the outer loop are summarised to provide an overall assessment of the model’s performance.
In this study, we used nested cross-validation with four folds for both the inner and outer loops and 25 repetitions, resulting in 100 evaluations, similar to the repeated cross-validation used in the Bayesian network model. To fine-tune the hyperparameters, we employed Bayesian hyperparameter optimisation maximising the AUC metric [
27].
Furthermore, we recorded the AUC metric, including the minimum, mean, maximum, and standard deviation from the outer loop iterations. Next, we selected the hyperparameters that resulted in the highest AUC value across the 100 evaluations and fitted the models with these hyperparameter settings on the test data.
Finally, we compared the AUC metrics of Logistic Regression, Stochastic Gradient Descent Classifier, and XGBoost with the AUC metric of the Bayesian network model.