Classification of Return on Equity Using Machine Learning Techniques

1. Introduction

Every investor needs accurate information before investing in the capital market. Incorrect information can mislead investors in making sound investment decisions and result in losses for them [28,29]. The Stock Exchange provides fair and transparent pricing for listed companies to limit the risk of share trading and has strict regulations that all listed companies must comply with [23]. The largest stock exchange in Africa is the Johannesburg Stock Exchange (JSE), also known as the Johannesburg Securities Exchange [9]. The main aim of every investor is to maximize profit and minimize loss; hence, comparing different companies using the right tools is very important before investing.The best way to compare different companies in the financial market is through financial ratios [4]. It is also the most popular and widely used financial analysis tool [5,16]. Profitability ratios, the most widely used financial ratios, are primarily used to determine whether a company will succeed or fail. Return on Equity (ROE) is a financial ratio used by investors to check whether their investments are profitable (Josue, 2015) [18]. From an investor’s perspective, a higher ROE indicates a higher rate of return on investment [31]. ROE is calculated as

$$\mathrm{ROE}={\displaystyle \frac{\mathrm{Net}\mathrm{income}\mathrm{after}\mathrm{tax}}{\mathrm{Total}\mathrm{Equity}}}\mathrm{x}100.$$

ROE is used to check if the company is heading in the right direction financially, however ROE must be used with other ratios to be reliable [22]. The primary objective of this study is to find determinants of ROE, using a binary dependent variable ROE.

1.1. Motivation and Objectives

Many studies have been conducted in the past to identify financial ratios that influence ROE, but all use a continuous dependent variable. Various methods, such as the DuPont formula, Correlation analysis, and multiple linear regression (MLR), have been used to identify determinants of ROE. However, all those methods are limited in one way or another. Both the DuPont formula and correlation analysis are limited in that they cannot be used to find a collective influence of ROE, nor can they be used to predict ROE. The limitation of MLR is its stringent assumptions, which are often not met, leading to biased and inconsistent results [19]. According to [10], negative ROE communicates a loss to shareholders. [27] stated that a positive ROE indicates shareholders are gaining, while a negative ROE indicates shareholders are incurring losses on their investment. Since ROE can be either positive or negative, using continuous dependent variable neglects the negative part of ROE or loss of information, when negative ROE is converted into positive ROE. Hence we chose binary dependent variable which combines both positive and negative ROE. For binary purposes, ROE was coded as follows

$$ROE=\left\{\begin{array}{cc}1\hfill & \mathrm{to}\mathrm{represent}\mathrm{a}\mathrm{positive}\mathrm{Return}\mathrm{on}\mathrm{Equity}\hfill \\ 0\hfill & \mathrm{to}\mathrm{represent}\mathrm{a}\mathrm{negative}\mathrm{Return}\mathrm{on}\mathrm{Equity}\hfill \end{array}\right.$$

The primary objective of this study is to fit a logistic regression (LR), random forest (RF), k-nearest neighbour (KNN), and naive bayes (NB) to the data using the binary dependent variable ROE, to find the determinants of ROE, while the secondary objective is to identify the best machine learning (ML) technique that can predict ROE. The limitation of the model is that, no value will be produced by the model but can only predict whether ROE will be positive or negative.

This study will help investors identify companies likely to deliver better returns on their investments. Investment managers will also benefit from this research, since with ratios that influence ROE at their disposal they will be able to come up with strategy to enhance ROE of their company. It will also provide answers as to which machine learning technique can be used to predict ROE for companies listed on the Johannesburg Stock Exchange and other Stock Exchanges.

1.2. Literature Review

Many researchers have studied financial ratios in the capital market that may yield dividends to an investor. Also, there have been many studies on identifying the best Machine learning (ML) to adopt across many fields of endeavour.

1.2.1. Return on Equity

[24] investigated the impact of total asset turnover and DTE ratio on ROE for listed companies on the Indonesian Stock Exchange. The companies selected were from the automotive industry, and their components were sampled with a sample size of 50. Ten companies were selected. The statistical technique used to analyze the data was multiple linear regression. They concluded that the Debt-to-Equity ratio negatively affects ROE, while Total Asset Turnover positively affects ROE. It was also found that DTE and Total Asset Turnover combined do affect ROE.

[5] conducted a study to examine the influence of the Quick ratio (QR), DTE ratio, and Total Asset Turnover on ROE for companies listed on the Indonesia Stock Exchange for the period 2012 to 2019. The test of the classical assumption was conducted in SPSS version 25. The relationship between ROE and the Quick ratio was found to be positive but insignificant. ROE and Total Asset Turnover also showed a positive, but insignificant, relationship. It was the Debt-to-Equity ratio that had a negative but insignificant relationship. The three independent variables, taken together, showed a negative association with ROE, but the effect was not significant.

[12] conducted a study to examine the impact of capital structure and financial characteristics on profitability among Jordanian service-sector companies listed on the Amman Stock Exchange. The independent variables used were capital structure, consisting of Debt to Equity (DTE) and Debt to Assets (DTA). The financial characteristics consisted of business risk, growth, size, and tangible assets. Their findings were that size and growth positively impact profitability, namely, EBIT-TA and ROE.

Kim21 used Oaxaca decomposition and regression analysis to examine the diversity of financial performance among listed food product manufacturers in Vietnam for the period 2014 to 2019. The independent variables used were Quick ratios, total assets turnover ratios, leverage ratios, company size, sales growth, and CPI. They concluded that growth in sales and total assets turnover does impact ROE and Return on sales. It was also discovered that leverage ratios negatively impact Return on sales.

[34] used panel data regression analysis to identify financial indicators that statistically influence ROA and ROE in the energy sector in Turkey. The companies targeted were gas, oil, electricity, and renewable energy producers and distributors in Turkey that are listed on the Borsa Istanbul Stock Exchange. The period targeted was 2010, Quarter 1, to 2019, Quarter 4. The two dependent variables used were ROE and ROA. The predictors were firm size, receivable turnover ratio, interest cover ratio, receivable turnover in days, price-to-book, current ratio, asset turnover ratio, leverage ratio, quick ratio, price-to-earnings, inventory turnover ratio, cost of sales ratio and EBIT margin. Sixteen companies were identified, and the sample size was 640 observations. The variables found to influence ROE were quick ratio, leverage ratio, receivable turnover ratio, inventory turnover ratio, firm size, and cost of sales ratio. In contrast, ROA was influenced by the receivables turnover ratio, cost of sales ratio, quick ratio, and price-to-book ratio.

[36] breaks down ROA into Net Profit Margin and Assets Turnover. To find determinants of ROE, they used ROA, Net Profit Margin, and Total Assets as independent variables. The listed companies on the Thai Stock Exchange for the period 2015 to 2019 were used to collect secondary data, with two sample sizes of 104 and 93. The combination of Total Assets and Net Profit Margin outperformed ROA in predicting ROE.

All the research above used a continuous dependent variable, ROE. Most of them were analyzed using Multiple Linear Regression (MLR), which requires many model assumptions to yield valid and reliable results. These assumptions are often not met; hence, in this study, we seek to use a binary dependent variable for ROE.

1.2.2. Machine Learning Methods

Tutcu24 used ML methods to determine whether the firms of IT sector listed at Borsa Istanbul Stock Exchange are profitable. They used financial data sourced from technology sector which consisted of 13 firms listed for the period 2000 to 2023. The two dependent variables that were estimated were ROE and ROA using ML methods in the form of Decision tree (DT), Multiple Linear Regression (MLR) and Artificial Neural Networks (ANN). It was found that the ML methods that were effective in predicting ROE and ROA were ANN and MLR.

[17] constructed models to predict ROA and ROE using retail companies located in Vietnamese. The ML methods used were Multilayer Perceptron XGBoost and Random Forest. The data was collected from financial statements of listed retail companies and the period used were from 2010 to 2024. The best predicting model was found to be RF with RMSE of 0.0926 followed by XGboost.

[37] implemented seven ML techniques — Decision Tree (DT), LR, Support Vector Machine (SVM), Neural Networks, KNN, NB, and RF — to predict heart disease. The research used health care data from the University of California, Irvine ML repository, with a sample size of 303:139 were positive and 164 were negative. The highest classification accuracy was achieved by Artificial Neural Networks (92.30%), while RF, LR, KNN, and NB achieved 85.71%, 81.71%, 80.21%, and 57.14%, respectively.

To detect fraud in financial accounting for Turkish Small and Medium Enterprises (SME) from different sectors, [14] compared ML techniques in the form of NB, KNN, LR, Support Vector Machine, Artificial Neural Network, RF and Bagging. The data was collected from one of the leading credit provider bank in Turkey and the number of firms used was 341 Turkish SME for the period 2013 to 2017. The sample size used was a highly unbalanced dataset of 1705, consisting of 321 fraudulent cases and 1384 non-fraudulent cases. Their finding was Random Forest outperformed the rest with an accuracy of 93.74%, meanwhile KNN, LR and NB had an accuracy of 89.92%, 88.08% and 58.81% respectively.

[6] used Google Colab to compare LR, KNN, SVM, RF, DT, and NB for breast cancer classification. The dataset was obtained from Wisconsin public records and comprises 569 cases (212 benign and 357 malignant). The number of features used, sourced from biopsy images, was ten (10), consisting of symmetry fractal, concave points, compactness, area, texture, radius, perimeter, smoothness, concavity, and dimension. The top classifier was logistic regression, with an accuracy of 98.5%. Meanwhile, KNN, RF, and NB had classification accuracies of 96.97%, 94.99%, and 93.47%, respectively.

To compare ML techniques in the classification of breast cancer, [3] used thirteen machine learning techniques, which are Bayes Net K2 search, Bayes Net Tab search, Lazy-KStar, Bayes TAN search, NB, LR, Lazy-IBK, Rules Zero R, Trees Decision stump, Lazy LWY, Trees J48, RF, and Trees Random trees. The dataset was analyzed using Waikato Environment for Knowledge Analysis (WEKA) software with a sample size of 699 and 10 independent variables. Lazy-IBK and K Nearest Neighbor achieved the highest classification with an accuracy of 98.2%, while RF, NB, LR, and NB achieved 96.4%, 96.4%, and 92.8%, respectively.

In breast cancer classification, [6] found that LR outperformed other ML techniques, while [3] found that Random Forest outperformed them. This suggests that more research using different datasets is needed. It is also unclear which ML method is best for classification. In this research, we aim to extend the comparison of the four ML models to the Stock Exchange Market.

2. Materials and Methods

2.1. The Dataset, Research Variables and missing values

The data was sourced from the Integrated Real-time Equity System (IRESS) website. IRESS contains a database of financial ratios for companies listed on the JSE from 1970. The dataset was collected using simple random sampling, selecting all companies listed on the JSE for the year 2019. We did not display Company names to protect the privacy of each company. The number of predictors before checking for multicollinearity among predictors was ten (10). One predictor was removed due to multicollinearity. The sample comprised 323 companies: 236 with positive ROE and 87 with negative ROE. To address the issue of the imbalanced dataset, the following methods were used:

• Firstly, the 2019 original imbalance dataset (dataset 1) was analysed using metrics such as precision, recall, ROC-AUC score, and F1 score, which are known to cope with imbalanced datasets. We also included accuracy to see how it would be affected by the imbalanced dataset.
• Secondly, the 2019 dataset was balanced using observations from previous years (dataset 2) while ensuring independence of observations. The observations to balance the dataset were selected from companies listed between 2000 and 2018. For each selected company, we selected one observation without missing values. The resulting sample size was 1074.
• Thirdly, SMOTE (synthetic minority oversampling technique) was utilized to correct the issue of an imbalanced dataset. In SMOTE, new data points are generated specifically for the minority class, while the majority class remains unchanged [32]. Using SMOTE, 87 new observations were generated for the minority class. The resulting sample comprised 410 companies: 174 with negative ROE and 236 with positive ROE.
• Lastly, the ROSE (Random oversampling examples). ROSE is an oversampling technique that uses bootstrap-based methods and assists in binary classification of an imbalanced dataset. Using ROSE oversampling, 149 new observations were generated for the minority class, yielding a final sample size of 472. The dataset was now balanced with 236 companies with positive ROE and 236 with negative ROE.

In this study, the binary ROE is used as the dependent variable. In contrast, the metric independent variables include Quick Ratio (QR), Debt to Assets ratio (DTA), Earning yield (EY), Price per earning (PPE), Interest cover (IC), Net profit margin (NPM), Debt to equity ratio (DTE), Asset per capital employed (APCE), and earning per share (EPS). For validation purposes, the data were partitioned into 60% for training and 40% for validation.

It should be noted that we did not have many missing values, since most of our observations were obtained using oversampling techniques and we also used observations from previous years without missing values. The few missing values that we had were replaced by the mean of the dataset. The missingness was below 5% of the dataset and just missing completely at random. It was also not related to any unobserved or observed variables. Outliers were also removed from the observations hence missing completely at random method was selected.

2.2. Statistical Models

The four statistical models used are Logistic Regression (LR), K-Nearest Neighbors (KNN), Naive Bayes (NB), and Random Forests (RF). A brief description of each of them follows.

2.2.1. Logistic Regression (LR)

The model for logistic regression is known as the Logit and is given by

$$log\left({\displaystyle \frac{{\pi }_i}{1-{\pi }_i}}\right)={\beta }_0+{\beta }_1{X}_{1i}+\dots +{\beta }_p{X}_{pi}\mathrm{for}i=1,\dots ,n.$$

(1)

where ${\beta }_0,{\beta }_1,\dots ,{\beta }_p$ are parameters to be estimated and $X_1,\dots ,X_p$ are metric independent variables. The following model assumptions should be observed:

• Binary outcomes for the dependent variable.
• Observations from duplicated measurements or those that came from matched data.
• The absence of multicollinearity.
• The sample size requirements (augmented using oversampling methods).

Model fit in LR uses Deviance, Likelihood ratio test, Hosmer-Lemeshow, Classification Table, and the two Pseudo $R^2$. The Wald statistic is an appropriate test for determining the suitability of including each predictor variable in the model [11,15]. Predictors were included in the LR model if the Wald p-value was < 0.05.

2.2.2. Random Forest (RF)

In random forest, for the $n^{th}$ tree, a randomly selected vector ${\theta }_n$ is constructed, which is entirely free from ${\theta }_1,\dots ,{\theta }_{n-1}$, previous random vectors, but equally distributed. The training dataset and a random vector ${\theta }_n$ are utilized in the growing of the tree. The resultant classifier becomes $h(X;{\theta }_n)$, where X stands for input vector. The most frequent class is voted after most of the trees have been generated, and the whole procedure is called Random Forest (RF) [8].

In RF, the two most used measures of variable importance are accuracy-based importance (also known as mean decrease accuracy) and Gini-based importance (also known as mean decrease impurity). The Gini-based importance was used for variable selection. The Gini-based importance method uses the Gini impurity to select a predictor for splitting a node. A predictor with a higher Gini importance was considered more important for predicting ROE.

2.2.3. Naive Bayes (NB)

A set of classification algorithms constructed using Bayes’ theorem is called Naive Bayes. It uses the assumptions of feature independence and conditional independence to estimate the class of Y given a set of features. It is used to predict the values of $Y=y\in (0,1)$ given a new instance $X=(X_1,\dots ,X_p)$ and it uses the training data set to estimate the values of $P(Y=y)$ and $P\left(X_i\right|Y=y)$. We write Naive Bayes Classifier as

$$\widehat{y}=arg\underset{y\in (0,1)}{max}\left({\displaystyle \frac{P(Y=y){\prod }_{i=1}^{p}P\left(X_i\right|Y=y)}{{\sum }_{j=1}^{1}P(Y=j){\prod }_{i=1}^{p}P\left(X_i\right|Y=j)}}\right).$$

(2)

Naive Bayes in SPSS uses forward selection for variable selection. Independent variables were entered into the model based on average log-likelihood, with predictors having the smallest average log- likelihood first. The best subset of predictors selected was the one that produced the highest classification accuracy.

2.2.4. K-Nearest Neighbor

KNN is defined by the arrangement of the dataset $D=\{\left(X_1{Y}_1\right),\left(X_2{Y}_2\right),\dots ,\left(X_n{Y}_n\right)\}$, in increasing order of magnitude for the purpose of accommodating the Euclidean distance. The ordered data set becomes $D=\{\left(X_{\left(1\right)}{Y}_{\left(1\right)}\right),(X_{\left(2\right)}{Y}_{\left(2\right)},\dots ,\left(X_{\left(n\right)}{Y}_{\left(n\right)}\right)\}$ and according to , the K-NN is defined as

$$h_n\left(X\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{k}}}{\mathrm{w}}_{\mathrm{i}}\left({\mathrm{Y}}_{\left(\mathrm{i}\right)}\left(\mathrm{X}\right)\right)=0<{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{k}}}{\mathrm{w}}_{\mathrm{i}}\left({\mathrm{Y}}_{\left(\mathrm{i}\right)}\left(\mathrm{X}\right)\right)=1\hfill \\ 0\hfill & \mathrm{elsewhere}\hfill \end{array}\right.$$

(3)

where the letter n represents the sample size and X the test instance. $Y_{\left(i\right)}$ is the same as $Y_{\left(i\right)}\left(X\right)$ the corresponding coordinate of $X_{\left(i\right)}$ in the $i^{th}$ nearest neighbour of X. $w_i$ are the weights of KNN used to weight the dataset. When selecting variables, KNN uses forward selection. At each step, a variable that minimises the error rate is included in the model. It uses a variable importance chart or a predictor selection log to display the relative importance of each selected predictor. The predictor selection log is used to identify the important features.

2.3. Classification Table

The model’s prediction accuracy can be measured using a classification table [25,26]. Hence, to compare the four machine learning techniques, metrics calculated from the classification table were used.

In Table 1, TP refers to observations classified as positive, and TN to those classified as negative. FN refers to observations classified as false negatives, while FP stands for observations classified as positive but are negative [7]. Using the classification table, the values of sensitivity, specificity, precision, F1 score, and accuracy were calculated for Logistic Regression, K-nearest neighbour, Naive Bayes, and Random Forest.

3. Results

We present the model assumptions’tests, followed by the comparisons of the four ML techniques using an imbalanced dataset, SMOTE, ROSE, and a balanced dataset. We note that ML methods — RF, KNN, and NB — do not require many model assumptions, except that observations are independent. LR assumes an additional assumption: the absence of multicollinearity. The dependent variable was originally metric, allowing VIF and tolerance to be calculated. Figure 1 (a) and (b) show the tolerance and VIF for dataset 2, after removing the current ratio (CR) variable.

3.1. Model Comparison Using Dataset 1

The four statistical models were compared using the following evaluation metrics: Sensitivity, Specificity, Precision, F1 Score, Area under the ROC curve (AUC), and Accuracy. According to [33], AUC is better suited to an imbalanced dataset, while Accuracy is better suited to a balanced dataset. Table 2 shows the average performance of the four ML models. (The colours blue and red represent the best- and worst-performing models, respectively). AUC was used in Table 2 since dataset was imbalance, for all other tables AUC was not reported since dataset was balanced.

From Table 2, the worst-performing model was KNN, and Naive Bayes achieved the worst AUC. The best performing models were LR and RF, with the highest specificity and precision of 99.2% and 97.6%, respectively. RF dominated Sensitivity (Recall), F1 Score, Accuracy, and AUC, with values of 96.6%, 96.6%, 98.2%, and 99.4%, respectively.

3.2. Model Comparison for Dataset 1 Using the ROC Curve

In Figure 2, the ROC curve for RF was more to the top left region of the graph than for LR, NB, and KNN. The AUC for the RF model also exceeded that of any other model. The RF graph was followed by the LR model, which also covered a larger area in the ROC curve. The NB model covered less area in the ROC curve, indicating it was the least performing model when using an imbalanced dataset. The KNN was the least-performing model across all evaluation metrics, except for AUC.

3.3. Data Analysis Using Dataset 2

Variable selection, using all statistical models, was conducted first because of the larger sample size. All statistical models selected NPM, IC, EY, EPS, and PPE as the best predictors of ROE. RF was the only statistical model to validate the selected predictor using the training and test datasets. The results are shown in Table 3 and Table 4, and Figure 3 and Figure 4.

3.3.1. Model Comparison Using Dataset 2

The model’s parameters must be considered when selecting the best ML model for a given dataset [13]. Before variable selection, there were nine predictors; after selection, five were identified as the best predictors of ROE. We compared the four statistical models both before and after variable selection.

In Table 5, with all predictors, NB was the worst-performing model in terms of sensitivity, F1 Score, and Accuracy, while KNN performed poorly in terms of specificity and precision. LR was the second- best model. RF was the best-performing model, with sensitivities, specificities, precisions, F1 scores, and Accuracies of 99.1%, 98.4%, 94.4%, 98.8%, and 98.8%, respectively.

From Table 6, NB was the worst-performing model, with sensitivities, specificities, precisions, F1 scores, and Accuracies of 82%, 93.1%, 92.7%, 86.7%, and 87.4%, respectively. KNN with five predictors or after variable selection improved significantly. LR remained the second-best model and did not improve after variable selection. RF was the best-performing model, with sensitivity, specificity, precision, F1-score, and Accuracy of 99.3%, 98.6%, 98.6%, 98.9%, and 98.9%, respectively.

3.4. Model Comparison Using the SMOTE Dataset

Table 7 and Table 8 show the results for the SMOTE dataset, before and after variable selection. Improvements were observed, especially in average specificity, precision, F1 score, and accuracy, with values of 99.6%, 99.6%, 98.4%, and 98.7%, respectively for LR. The performance of NB also improved after SMOTE oversampling, producing average sensitivity, specificity, precision, F1 score, and accuracy of 85%, 97.3%, 96%, 90.1%, and 92.1%, respectively. The KNN model was the worst performing model before and after SMOTE oversampling. After variable selection, KNN outperforms NB on important metrics, including sensitivity, F1 Score, and Accuracy, with values of 90.6%, 91.2%, and 92.6%, respectively. RF performed better than other methods in terms of model performance before and after variable selection, but it produced the best model when using SMOTE oversampling without variable selection.

3.5. Data Analysis Using ROSE Dataset

For the ROSE dataset, the average sensitivity, specificity, precision, F1 score, and accuracy for KNN without variable selection were 95.0%, 89.0%, 89.8%, 92.3% and 92.0% respectively, while after variable selection, it was 96.4%, 93.3%, 93.5%, 94.9% and 94.9% respectively. When comparing NB before and after variable selection, we noticed that NB with all predictors outperformed NB with variable selection. It was observed that LR after variable selection performed worse than LR before variable selection. The best model was produced by RF without feature selection, with average sensitivity, specificity, precision, F1 score, and accuracy of 100%, 99.1%, 99.2%, 96.6% and 96.6% respectively. Tables 10 and 11 show these results.

Table 9. Average model comparison (Training and Test) without feature selection using ROSE dataset.

	Logistic Regresion	Naive Bayes	K Nearest Neighbour	Random Forest
Sensitivity	100	86.5	95.0	100
Specificity	97.9	97.9	89.0	99.1
Precision	98.2	97.6	89.8	99.2
F1 Score	99.1	91.7	92.3	99.6
Accuracy	99.0	92.3	92.0	99.6

Table 9. Average model comparison (Training and Test) without feature selection using ROSE dataset.

	Logistic Regresion	Naive Bayes	K Nearest Neighbour	Random Forest
Sensitivity	100	86.5	95.0	100
Specificity	97.9	97.9	89.0	99.1
Precision	98.2	97.6	89.8	99.2
F1 Score	99.1	91.7	92.3	99.6
Accuracy	99.0	92.3	92.0	99.6

Table 10. Average model comparison (Training and Test) with feature selection using ROSE dataset.

	Logistic Regresion	Naive Bayes	K Nearest Neighbour	Random Forest
Sensitivity	97.1	85.7	96.4	100
Specificity	98.8	97.7	93.3	98.4
Precision	98.7	97.6	93.5	98.6
F1 Score	97.9	91.2	94.9	99.3
Accuracy	97.9	91.8	94.9	99.3

Table 10. Average model comparison (Training and Test) with feature selection using ROSE dataset.

	Logistic Regresion	Naive Bayes	K Nearest Neighbour	Random Forest
Sensitivity	97.1	85.7	96.4	100
Specificity	98.8	97.7	93.3	98.4
Precision	98.7	97.6	93.5	98.6
F1 Score	97.9	91.2	94.9	99.3
Accuracy	97.9	91.8	94.9	99.3

3.6. Variable Selection Using Shrinkage Methods

We evaluate how shrinkage methods perform in feature selection. The three shrinkage methods used were Lasso, Elastic Net, and Ridge Regression. The results are shown in Table 11 and Figure 5, Figure 6, and Figure 7.

Using the test dataset, the best predictors of ROE were NPM, EY, IC, APCE, and PPE. It was concluded that Logistic Regression, Lasso, Elastic Net, and Ridge regression were unable to validate their predictors using the training and test datasets fully.

Examining Figure 5 and Table 11, it is evident that NPM was the last predictor to reach zero, followed by EY, EPS, IC, PPE, and DTE, while APCE, DTA, and QR were eliminated from the regression model because their coefficients were zero. According to Lasso, the best predictors of ROE were NPM, EY, EPS, IC, PPE, and DTE. However, we noticed in Table 11 that the coefficient for DTE was insignificant at -0.0994. This was supported by Logistic regression in Table 11, where the p-value for DTE was insignificant at 0.438. Hence, it was concluded that DTE must be removed from the regression model. The final predictors of ROE were NPM, EY, EPS, IC, and PPE.

In Figure 6 and Figure 7 and Table 11, we observed that NPM, EY, IC, EPS, PPE, and DTE took time to reach zero. It was the confirmation of results found when Lasso Regression was used. The coefficients for NPM, EY, IC, EPS, PPE, and DTE when Elastic Net Regression was used were 6.8909, 0.6678, 0.6233, 0.5381, 0.2606, and -0.1730, respectively, while for Ridge Regression were 9.086, 0.5670, 0.5340, 0.4834, 0.2427, and -0.1532, respectively.

4. Discussion

In this study, the four statistical models identified NPM, IC, EPS, EY, and PPE as the best predictors of ROE, with positive slopes of 0.166, 0.011, 0.001, 0.016, and 0.006, respectively. This means, for example, that a one unit increase in NPM will increase ROE by 16.6%. [30] found that EPS and PPE were the best predictors of ROE when using a dataset collected from the Istanbul Stock Exchange for the period 2006 to 2016. [36] used a dataset collected from the Thai Stock Exchange and showed that NPM was the best predictor of ROE. In this research, NPM was a highly significant predictor identified by the four ML models. This study also confirms findings by [1], who found that EY was a predictor of ROE. The results of this study are also supported by the DuPont formula, which shows that NPM and IC were among the drivers of ROE.

It was also found that APCE, DTE, DTA, and QR were not significant drivers of ROE. Similar results were reported by [5], who found that QR and DTE were not significant in predicting ROE. [12] also excluded DTA and DTE as predictors of ROE.

In dataset 2, we found that RF with feature selection outperformed RF without it. This was expected since model performance typically improves with variable selection. RF had the best model performance, followed by LR, KNN, and NB. The results confirm the findings of [37], who found that RF outperformed LR, KNN, and NB.

For dataset 1, before oversampling, LR was affected by an imbalanced dataset since its specificity and sensitivity differ by (99.2-94.2) = 5%. NB and KNN were also affected by the imbalanced dataset. RF achieved the highest classification with sensitivity and specificity of 96.6% and 98.8% respectively, and was not affected by an imbalanced dataset. Hamal and Senvar (2021) found that RF, LR, KNN, and NB were all affected by an imbalanced dataset. It is suspected that the difference in results was due to the dataset, as its minority class was more imbalanced at 18.83% compared to ours at 26.93%. [33] stated that AUC is suitable for an imbalanced dataset than Accuracy. In this study, for models affected by an imbalanced dataset (KNN, NB, and LR), and using F1-score as a benchmark, we noticed that the F1-score was closer to AUC than to Accuracy.

From dataset 1 with SMOTE oversampling, RF without feature selection outperformed RF with feature selection. It was a surprise because this contradicts the results we found in dataset 2, where RF with feature selection outperformed RF without feature selection. NB was one of the models that did not improve after feature selection. Similar results were reported by [14] using SMOTE oversampling and financial ratios to detect financial accounting fraud. They also discovered that RF without feature selection outperformed RF with feature selection. We noticed the signs of over-fitting in the SMOTE dataset. A sign of over-fitting is evident when an ML model improves performance by adding more predictors. Overfitting normally occurs when predictors are highly correlated with one another. According to Alkhawaldeh23, SMOTE generates new instances that are similar to the minority class, leading to overfitting.

In dataset 1 with ROSE oversampling, RF without feature selection outperformed RF with feature selection, as observed with SMOTE oversampling. Over-fitting was also noticed with the ROSE dataset. The other two classifiers that could not improve their performances after feature selection, especially in terms of sensitivity, F1 score, and accuracy, were LR and NB. [14] found similar results when using SMOTE and financial ratios; none of their ML classifiers improved after feature selection in their dataset.

The model that highly benefited from feature selection in dataset 2 was KNN. Its sensitivity improved from 84.3% to 91%, the most significant improvement among all classifiers. It also improved tremendously from SMOTE and ROSE oversampling. [20] confirms our results. They stated that feature selection in KNN reduces the dataset’s dimensionality. It also reduces the number of variables used in KNN distance calculations, thereby lowering the algorithm’s complexity. They further stated that calculating distances between samples can affect the KNN algorithm when dealing with an imbalanced dataset and can also degrade performance.

5. Conclusion

Many studies in the literature used ROE as a continuous dependent variable. To avoid the stringent model assumptions of MLR, we identified determinants of ROE from the metric independent variables Quick Ratio (QR), Debt to Assets ratio (DTA), Earning yield (EY), Price per earning (PPE), Interest cover (IC), Net profit margin (NPM), Debt to equity ratio (DTE), Asset per capital employed (APCE), and earning per share (EPS). Four datasets were used, and models were constructed and compared using model evaluation metrics, sensitivity, specificity, precision, F1 score, and accuracy. The predictors were selected using LR, RF, NB, Lasso, Ridge, Elastic Net regression and KNN feature selection methods.

RF was found to be superior in terms of feature selection since it outperforms LR, Lasso, Ridge, Elastic Net regression, KNN and NB. It was the only statistical model to validate selected predictors (NPM, IC, EPS, EY, and PPE) using training and test datasets in all datasets (dataset1. dataset 2, SMOTE and ROSE dataset). For the SMOTE and ROSE datasets, only RF was used for variable selection, which successfully selected the five predictors (NPM, IC, EPS, EY, and PPE), and they were validated on the training and test datasets. RF outperformed all other models across all datasets, and LR ranked second. KNN was the only statistical model to improve significantly after feature selection and oversampling across all datasets.

In dataset 1, only the RF was not affected by the imbalance. This study confirmed that the F1-score and AUC are appropriate metrics for evaluating models on imbalanced data. Using dataset 2, the four statistical models and the three shrinkage methods—Lasso, Elastic Net, and Ridge Regression identified NPM, IC, EY, EPS, and PPE as the best predictors of ROE. Therefore, this study recommends that investors look for JSE companies with high NPM, high IC, high EPS, high EY, and low PPE when making investment decisions.

It is known that model performance improves with feature selection. We found out in this study that all models improved after feature selection when using dataset 2. However, the opposite occurred with the SMOTE and ROSE datasets; performance decreased after feature selection. Over-fitting by the SMOTE and ROSE datasets was suspected. When it comes to oversampling methods, the study recommends the use of both over sampling methods (SMOTE and ROSE) with caution, since both methods were prone to over fitting. In research with larger sample size, under sampling of majority class is more recommended than oversampling of minority class. Dataset 2 produced excellent and reliable results with no signs of over fitting, hence where possible, the use of original dataset through further sampling is recommended.

After oversampling, it was not easy to choose between the SMOTE and ROSE datasets because RF performed very well on both, with key metrics above 99%. In fact, the ROSE dataset with RF produced a model that was good at predicting negative ROE, with average values after feature selection of 100% sensitivity, 99.3% F1-score, and 99.3% accuracy. The SMOTE dataset with RF also produced a good model with high precision. Dataset 2 (balanced using observations from previous years) produced excellent results without overfitting.

RF not only dominated ROE prediction but also feature selection. Generally, RF performs well at predicting the response variable. However, in this work, only RF validated its selected predictors across all datasets using both training and test datasets. Hence, this study recommends using RF for feature selection and ROE prediction in companies listed on the Johannesburg Stock Exchange.