Developing Optimized Machine Learning Models For Timely Prediction and Prevention of College Dropout

2.1. Dataset

This paper used a comprehensive dataset of 4,424 students enrolled in an undergraduate degree course in a postsecondary education institution [10]. It includes data on each student’s graduation outcome (“Target”) in addition to key characteristics both before and after entering college. Specifically, for each student, the dataset reports 34 features in addition to the graduation outcome. It includes demographic data, socio-economic factors and academic performance data giving key information about each student prior to enrolling in college as well as information on courses and grades in college. Additionally, the dataset reports macroeconomic indicators such as unemployment rate, inflation rate and GDP from the region, these variables can correlate with the probability of graduating, staying enrolled or dropping out. For example, a study finds that dropout probability decreased as a result of the recession as the lower opportunity cost of education due to a weak job-market encouraged students to continue studying in colleges [11]. Importantly, this dataset includes a target variable (graduation outcome) that indicates whether the student graduated, is still enrolled or dropped out. This is the variable we will try to predict using the demographic, socioeconomic and educational features. Please refer to Figure 2 and Figure 3 for a full list of features.

2.2. Models

The data was split into training and testing samples — the training data was used to train alternative machine learning models while the testing data was used to test out-of-sample performance of these models. The testing-to-training ratio was varied and we used three alternative ratios: 0.4, 0.25, 0.10. These ratios were chosen as these present three disparate estimation-testing sample sizes. To determine the optimal set of features, we used three alternative machine learning models – logistic regression, decision tree and random forest. Starting with the features that are more highly correlated with the graduation outcome, model performance was assessed as the number of features used to train these models were varied to identify the optimal set of features. The performances of seven different machine learning models were then optimized using a grid search optimization algorithm trained on the optimized set of features with 3-fold, 5-fold and 10-fold cross-validations. Furthermore, we compared the outcome from a binary classification approach to a multi-class classification approach by investigating a multi-class classification problem. The purpose was to identify the best performing machine learning model based on a set of performance metrics. This paper utilized the following machine learning classification models both for binary and multi-class classifications: logistic regression, decision tree, random forest, artificial neural network (ANN), support vector machine (SVM), XGBoost and K-nearest neighbor (KNN).

3.1. Exploratory Data Analysis

We first assess the distribution of our outcome variable “Target”, the graduation outcome. Target is a categorical variable — it is formulated as a three-category classification task (dropout, enrolled, and graduate). The histogram shown in Figure 1 depicts the distribution of the three-class graduation outcome. The dataset includes 4,424 students. As depicted in the histogram in Figure 1, 1,421 students were dropouts, 2,209 students were graduates, and 794 students continued to be enrolled at the end of the normal duration of the course.

Next, to begin to identify the key features that would be incorporated in the model to predict dropout, we compute correlation coefficient of target with each of the other variables. Apart from target, the dataset contains 34 features. The horizontal bar chart in Figure 2 shows the correlations. Each bar shows the correlation coefficient of that feature with the target variable, the graduation outcome. The chart below shows that the probability of graduation is positively correlated with curricular units in second and first semesters (approved, grade, enrolled), whether the student’s tuition and fees are up-to-date, whether the student is a scholarship holder, while it is negatively correlated with whether the student is older, a debtor, married, female. Table A1 in the Appendix A includes the exact values of the correlations.

In the horizontal bar chart in Figure 3, the correlation coefficients are arranged in order of magnitude to understand which features are more strongly correlated with target, regardless of the direction. The feature that is most tightly correlated with Target is curricular units in 2nd semester (approved), followed by curricular units 2nd semester (grade), curricular units 1st semester (approved), curricular units second semester (grade), whether tuition and fees are up-to-date, whether the student is a scholarship holder and so on. Table A2 in the Appendix A includes the absolute values of the correlations.

As the initial point of feature screening, the top 10 features according to the magnitude of correlation coefficient between the graduation outcome and the various features, were considered. The top ten bars in Figure 3 above show the correlation coefficients of the ten features that are most strongly correlated with our target variable, the graduation outcome. Each of these correlations is higher than 0.2.

In Table 1, summary statistics of ten features that are most highly correlated with target are presented. We see that the mean number of curricular units that are approved in the second semester is 4.44, with a standard deviation of 3, a minimum of 0 and maximum of 20. These numbers are respectively 4.7, 3.1, 0 and 26 for the first semester. The grade average for the second semester is 10.2, with a standard deviation of 5.2, a min of 0 and a maximum of 18.6. These numbers for the first semester respectively are 10.6, 4.8, 0 and 18.9. We find below that 88% of the students are up-to-date with tuition and fees, 11% are debtors, 35% are male students and that the average age at enrollment is approximately 23.

Calculating pairwise correlations, we find that (a) curricular units 2nd semester (approved) and curricular units 2nd semester (grade) are very highly correlated, with a correlation coefficient of 0.761 and (b) curricular units 1st semester (approved) and curricular units 1st semester (grade) are very highly correlated with a correlation coefficient of 0.696. Therefore, to avoid multicollinearity issues, we include curricular units 2nd semester (approved) and curricular units 1st semester (approved) in the list of features instead of including all four. These two features will capture the features curricular units 2nd semester (grade) and curricular units 1st semester (grade) adequately.

We start with a broader, binary classification approach looking at students who dropped out and who did not. In this two-class classification, we classify students into two categories – students who dropped out (“dropout”) and students who graduated or were enrolled (“non-dropout”) at the end of the normal duration of the course. Figure 4 below shows the distribution of this binary variable. As depicted in Figure 4, 1,421 students were dropouts, 3,003 students were non-dropouts. To address this class imbalance, hyperparameters for class weighting and classification thresholds were empirically optimized in the grid-search optimization models in Section 3.3.

3.2. Feature Optimization and Model Training

We started with a logistic model as a baseline for feature engineering purposes. Table 2 below reports the features we use in the different logistic models, where the features are chosen in order of their correlation with the dropout variable. For simplicity and ease of reference, we call these sets of features F1-F8. Recall from section 3.1 that we exclude “curricular units 2nd semester (grade)” and “curricular units 1st semester (grade)” for multicollinearity concerns as they highly correlate respectively with “curricular units 2nd semester (approved)” and “curricular units 1st semester (approved)”.

Table 3 reports the results from logistic models with features F1-F8 and different testing to training ratios. The objective of this feature engineering and optimization exercise is to let the data decide which sets of features are most important in predicting dropout. For this purpose, we compare the accuracy and confusion matrix of the logistic models below with different sets of features F1-F8.

As shown in Table 3, models with features F3, F5 and F6 perform the best in terms of accuracy and the confusion matrix. Since there is not one set of features for which the logistic model dominates the models with other sets of features, we next conduct a more expansive analysis to screen the optimal set of features.

Specifically, as a next step in finalizing the optimum set of features, while considering features F3, F5, F6: (a) we examined a more comprehensive set of machine learning models (logistic, decision tree, random forest) (b) we considered three different testing to training ratios rather than two: 40%, 25% and 10% and (c) we investigated a wider set of performance metrics (accuracy, sensitivity, specificity, precision, AUC-ROC).

Table 4 presents results from logistic model, decision tree and random forest models using features F3, F5 and F6 respectively and three testing-to-training ratios (40%, 25%, 10%). As shown in Table 4, all models provided an acceptable performance. Notably, a training to testing ratio of 75:25% provided a tradeoff between data availability, avoiding overfitting the model, and achieving high classification performance. That said, the models performed well even at a lower training to testing ratio (60:40%) showing the successful training approaches implemented across all models. Additionally, while the 90:10% training-to-testing ratio can lead to overfitting, the performance metrics remain comparable across the various ratios, indicating that the models are learning the patterns rather than memorizing the data. This signifies that the models are robust and the performance does not worsen when we use a lower percentage of training data. In terms of the various performance metrics considered, the logistic and random forest models perform better. An exception is that the decision tree model dominates in terms of specificity except two cases when the logistic model has a higher specificity. The random forest model dominates in terms of sensitivity. The logistic and random forest models do better in terms of accuracy and the confusion matrix diagonal elements. All models have comparable AUC-ROC. Focusing on logistic and random forest models (as they perform better), and comparing models across the different sets of features, the models with the set of features denoted by F6 perform the best, even though the performance metrics remain similar across models with different features attesting to the robustness of the models. Moving forward, we focus on the set of features denoted by F6, which also subsumes F3 and F5. We consider training-to-testing ratio of 75:25% to strike a balance as discussed above.

3.3. Grid Search Optimization

Next, leveraging a comprehensive suite of machine learning models, grid search optimization was applied and validated with 3-fold, 5-fold and 10-fold cross validation. The machine learning models we utilize below are: logistic, decision tree, random forest, XGBoost, Artificial Neural Network (ANN), K-Nearest Neighbors (KNN) and Support Vector Machine (SVM). The set of features we use here is F6. We continue to use 25:75% testing-to-training ratio for reasons mentioned above. Since we are using k-fold cross validation, the performance metrics reported represent means across all the folds (as indicated by “macro” in the top row of Table 5 and Table 6). In subsection 3.3.1, A1 (“dropout” and “non-dropout”). In sub-section 3.3.2, we consider a multi-class classification of the graduation outcome (“dropout”, “enrolled”, “graduate”).

3.3.1. Binary Classification

As seen in Table 5, the grid-search optimized random forest model dominated in terms of most of the performance metrics. In terms of accuracy, specificity, precision and AUC-ROC, it is the best performing model. It is noteworthy that the ANN has the same values for specificity, precision and AUC-ROC but lower values of accuracy and sensitivity. Logistic has the same values of sensitivity and AUC-ROC but has lower values of accuracy, specificity and precision. XGBoost has higher sensitivity than Random Forest, but has lower specificity, precision and AUC-ROC. SVM has the same specificity as random forest but has lower values of accuracy, sensitivity, precision and AUC-ROC. Random Forest performs better than Decision Tree and KNN in terms of all the measures below. To summarize, using grid search optimized machine learning models with k-fold cross-validation in Table 5, we find that the model that performs the best in predicting dropout is the grid-search optimized random forest model.

Next, using the grid-search optimized random forest model, we examine which features predict dropout rate the best. This is an important exercise as with this knowledge administrators and students can use corresponding measures to reduce the probability of dropout. In Figure 5 below, we plot the importance of the features in predicting dropout from the binary grid-search optimized random forest model. We observe that the number of curricular units that are approved in the second semester is the most important factor and by far exceeds the importance of the other factors. The next in importance is the number of curricular units that are approved in the first semester. In terms of descending order of importance, the other features are whether the tuition and fees are up-to-date, the age at enrollment, the application mode, whether the student is a debtor, a scholarship holder and gender. This analysis suggests that higher number of approved curricular units can increase the probability of not dropping out. Being up-to-date in terms of tuition and fees, which may be positively correlated with parental resources, also lowers the risk of dropout. On the other hand, a higher age at enrollment increases the chance of college dropout.

3.3.2. Three-Class Classification

While so far we have considered a Two-Class Classification where the outcome of interest is classified into two-categories: dropout and non-dropout, in the analysis below, we move to a three-way classification, where the outcome of interest is classified into three categories: dropout, enrolled and graduate. In Table 6 below, we report findings from grid-search optimized machine learning models for three-classification. Once again, grid-search optimized logistic, decision tree, random forest, XG Boost, ANN, KNN and SVM models were estimated, and validated with 3-fold, 5-fold and 10-fold cross validation. Comparing the performance metrics of this model to the performance metrics of the grid search optimized binary model, we clearly see that the two-class models were considerably higher performing than the three-class models. As Table 6 shows, the grid-search optimized random forest models dominates the other models. It is the best performing model in terms of accuracy, sensitivity, and AUC-ROC. In terms of specificity and precision, the XGBoost model dominates the random forest model, but the difference is marginal.

Next using the 3-class classification grid-search optimized model, we analyze which features are most important for predicting dropout probability. The results are presented in Figure 6. The features, in order of importance, are the same between the 3-class and 2-class grid search models, even though the magnitudes are slightly different. As in the case of the binary classification, number of curricular units approved for the second semester is the strongest predictor, followed by number of curricular units approved for the first semester, and then whether tuition and fees are up-to-date. This implies once again that students who have a higher number of approved curricular units are less likely to dropout, indicating that educational investment and quality matter. Students who are up-to-date in paying tuition and fees are also more likely to graduate, suggesting that parental resources also help graduation.

4. Discussion

Dropping out of college has strong adverse consequences on labor market outcomes such as employment and earnings. Hence it is (a) important to understand the correlates of dropping out of college and (b) to develop a model that can optimally predict dropout probability based on the student information. To this end, this paper aimed to develop an optimal model of dropout probability. Such a model, along with an accurate understanding of the predictors of dropout, would enable policymakers, administrators, students and families to get an early indication of dropout risk of students. This, in turn, would enable them to devise measures and interventions and channel resources to prevent dropouts. Building on the existing literature, this paper expanded on the literature by using a broader suite of machine learning models and evaluation metrics, by conducting extensive feature optimization and by using a dataset that captures, in addition to other attributes, course information and grades, financial information, and macroeconomic data that are not present in the existing studies.

Using a dataset of 4,424 students that contains a comprehensive set of features, this paper conducted feature optimization, performed a comprehensive model evaluation, identified the best performing machine learning model comparing seven grid search optimized models, and identified the most important predictors of dropout risk. Additionally, we considered several training-to-testing ratios and finally considered a ratio that avoids the hazard of over/under-fitting while allowing an appreciable split between by training and testing samples. Grid search optimized versions of seven ML models with k-fold cross-validation were utilized and compared – logistic regression, decision tree, random forest, artificial neural network, support vector machine, XGBoost and K-nearest neighbor – to find the optimum model for predicting the likelihood of dropout. Models with a binary classification as well multi-class classification were estimated and compared.

Grid search optimized random forest model dominated the other models in terms of most of the performance metrics, for both binary and multi-class classifications. It was the best performing model in terms of accuracy, specificity, precision and AUC-ROC with 0.85 accuracy, 0.72 sensitivity, 0.92 specificity, 0.82 precision, and an AUC-ROC of 0.89 in the binary classification model. XGBoost provided a comparable performance, providing a higher sensitivity than random forest but lower specificity, precision, and AUC-ROC. Examining the predictors of dropout risk using the grid search optimal random forest model, we found that in order of importance, these are: the number of curricular units approved in the second semester, the number of curricular units approved in the first semester and whether the tuition and fees were paid up-to-date, signifying that both educational quality and parental resources play important roles in student success, consistent with the results from the correlation analysis.

To make this model broadly accessible, we are using the best performing grid search random forest model for developing (and launching) a web application that students and administrators can use to input students’ data to understand dropout probability. Additionally, college administrators and students can undertake countervailing measures to mitigate dropout risk based on the web app findings. Future work could expand on this research by collecting and utilizing a larger dataset extending across several colleges. Moreover, additional features such as high school scores, health status of students, special education and limited-English-proficient status might improve model performance and its predictive power. Furthermore, hybrid machine learning models that integrate multiple machine learning techniques to draw on their individual strengths may lead to improved performance and accuracy. These improvements could potentially advance our knowledge by increasing representativeness and predictability while providing deeper insights into the mechanisms behind college success, which could, in turn, be used by the colleges and students to further limit college dropout.