Recommender System for University Degree Selection: A Socioeconomic and Standardized Test Data Approach

1. Introduction

Quality education is vital to the sustainable development goals set by the United Nations for 2030. Specifically, the goal is to eliminate gender disparities in education and ensure equal access to all levels of education. To achieve this, objective tools are needed to aid students in making strategic and operational decisions regarding their education. Our study is committed to this goal, and we believe our machine-learning approach can significantly contribute to its achievement.

Choosing what to study is a pivotal decision in a person's life. Paradoxically, few objective frameworks are available to support this critical decision, and often, career choices are made based on subjective elements. Factors influencing career choices in college students include monetary and nonmonetary costs and benefits, cultural and social capital measures, and major differences between STEM and non-STEM graduates [1]. Various studies have shown that poor career choices can lead to significant issues, such as student dropout, poor academic performance, financial problems, and personal dissatisfaction [2,3].

Moreover, career indecision in college students is predicted by greater identity moratorium and diffusion, less maternal acceptance, and fewer years in college [4]. Consequently, a student satisfied with their professional choice is likely to perform better academically, thereby improving the efficiency of the study group. Data generated by former students regarding their career choices and academic performance can be used to create a decision-making framework that incorporates intrinsic factors of the learning process, such as standardised test results, and extrinsic factors, such as the type of school and parent's education and occupation.

In this context, our study proposes a machine-learning model to recommend university degrees for new students. By applying data science techniques to standardised high school and college test results and incorporating socioeconomic variables, the model predicts students' academic performance on standardised tests at the end of college. This allows for recommendations on the career paths that will likely result in better performance. This work has practical applications for any student deciding which career to pursue and offers universities an advancement in student counselling by using records from former students to generate helpful information for future professionals.

2. Related Work

In recent years, recommendation systems in the educational field have gained significant popularity, encompassing various stages of the student's relationship with academic institutions. From a high-level perspective, there are systems designed for university enrollment recommendations. These educational recommender systems aim to enhance the relevance of proposed study programs to students’ interests. Specifically, their objectives include mapping students' interests to a broad range of available courses to ensure they find courses that match their learning needs [5]. With the recent developments in the availability of online content, there is a growing need for systems that perform data analysis to find the best match for students based on their unique interests and capabilities [6]. Researchers have conducted numerous studies to develop systems that fill this gap using intelligent information processing techniques, such as recommender systems.

Throughout the literature, various approaches leverage recommender system techniques, including collaborative filtering [7], content-based filtering, and hybrid recommender systems based on fuzzy systems [8] and complex machine learning models [9]. Each approach collects and analyses students' historical behaviours, interests, or performance measurements to provide personalised recommendations. For example, collaborative filtering assumes that measuring the degree of similarity between users’ experiences can help identify items of interest. Suppose two students tend to enrol in similar courses. In that case, it is likely that if student A receives a good grade in a particular course, student B will also perform well, as they have demonstrated similar academic behaviours in past records [10].

Conversely, content-based approaches incorporate textual data from users' previous histories. Unlike collaborative filtering, content-based systems do not suffer from issues related to data matrix sparsity. However, these approaches require mechanisms to transform text features such as tags, keywords, and topics into vectors [11]. For instance, a content-based recommender system effectively assists prospective students in Bangladesh choose top-K private universities based on several parameters, achieving high accuracy [12]. After creating vector representations, content-based approaches use cosine similarity to determine the degree of similarity between users.

Moreover, hybrid frameworks combine complex models to generate personalised recommendations. Machine learning applications incorporating deep neural networks have produced robust and accurate recommendations. Natural language processing techniques, such as latent Dirichlet allocation (LDA) and latent semantic analysis (LSA), are also used to process extensive textual profiles to understand the semantics of the information used in the recommendation process [13]. Despite their advantages, textual features often contain inherent biases and uncertainties. Researchers have widely adopted fuzzy logic systems to handle these issues, which model vagueness logically and provide heuristic rules [14].

In educational recommender systems, fuzzy logic consistently mitigates vagueness by using linguistic variables and can be combined with other approaches, such as clustering, to find concrete user similarities. For instance, fuzzy rules can better interpret user ratings on specific items, offering greater flexibility in handling demographic data [15].

However, collaborative filtering does not perform well in specific domains, particularly in educational contexts where students have diverse capabilities, learning curves, and core strengths. The sparsity of data matrices indicating users' collaboration degrees further degrades the performance of collaborative filtering approaches in educational recommender systems [16]. To address this, researchers have developed hybrid recommendation frameworks that combine multiple methods to enhance robustness and accuracy. For example, a fuzzy logic-based intelligent recommender system measures students' skills using fuzzy rules and a Mamdani fuzzy inference system to identify their interests [17].

A context-aware recommender system based on fuzzy logic was proposed to generate personalised recommendations by assessing each user's performance [18]. This model calculates the required contexts for applying fuzzy logic based on the learners' assessment scores and the time to complete the assessment. Additionally, novel fuzzy-based recommender systems leverage family tree-based fuzzy inference systems to capture critical concepts in user profiles and available courses, improving neighbours' semantic and content similarities for personalised recommendations [19].

Ontology-based approaches have also been explored to develop recommendation engines. These approaches transform human-generated information into machine-readable language, which is then used in machine learning-based solutions. For instance, ontology-based recommender systems have been proposed to better guide students in higher education systems by assessing their strengths, weaknesses, interests, and abilities. These systems collect data explicitly from user profiles and implicitly from surveys conducted with graduate students. Machine learning methods create and cluster profile models of graduate students, which are then used in hybrid recommendation engines [20].

Furthermore, deep learning-based recommendation frameworks have been developed for e-learning platforms. These models use K-Nearest Neighbor (KNN) and collaborative filtering approaches to find similar students with comparable learning capabilities. Other frameworks incorporate sentimental and cognitive analysis and learning style categorisation to generate personalised course recommendations [21].

Overall, intelligent [22] personalised course advising models and hybrid recommendation systems combining collaborative and content-based filtering, optimised with genetic algorithms, have significantly improved recommendation reliability and performance. These advancements in recommendation system technologies offer practical applications for students and educational institutions, enhancing the guidance provided to students in their course registration processes (Tilahun & Sekeroglu, 2020).

3. Theoretical Framework

Academic processes generate much contextual data, such as academic performance and demographic and financial information. Thus, universities and colleges have tutors who carry out accompaniment processes to advise students on the most relevant careers according to their academic profile. However, a tutor can't manage all the information of the thousands of previous students about whom there is information about their chosen career and academic performance; in addition, tutors can generate a bias based on their relationship with the students or their own experiences and beliefs.

For the present study, we articulate a dataset where the academic and sociodemographic information of the students in the final stage of secondary school and university is related. The dataset comprises the results of standardised tests administered to students at the end of their high school and college education. Therefore, a supervised learning model was employed to train various algorithms to predict university exam results based on data from the high school stage. In addition, the key variables that determine the results in the standardised examination of the university stage were determined. To facilitate the application of the model, the results are provided as recommendations that students, tutors, or universities can utilise. This information is intended to aid in making one of the most critical decisions in a person's life: selecting a university career.

The vector {X₁, X₂, … X_n} gives the data structure, representing the input variables for the model, and "y" means the student's average result on the university standardised test. The output “y” is a continuous vector from 0 to 300. Hence, the proposed recommendation system can be structured as a forecasting model, wherein the implicit relationship between the model's input and response variables is articulated in Equation 1.

y=f(x₁, x₂,…,x_n)+ epsilon,

(1)

Where “f” is an algorithm associated with a supervised machine learning model used to make predictions based on the training dataset, and epsilon represents the error arising from the discrepancy between the actual value and the value predicted by the supervised model. Thus, the recommendation of studies is associated with the degree that presents a higher value of Ypred for a new student “Xnew” due to the function Ypred = f(Xnew).

In our study, a novel approach is given to structure a recommendation system for students, using longitudinal information of the student from his high school stage until the completion of the university. Consequently, the paper answers the following problem questions:

-

How to generate recommendations of careers to study at the university to new students based on academic and socio-demographic data

-

What features have the most significant impact on the recommendation of which career to study?

-

Which machine learning model offers the best prediction performance

4. Materials and Methods

It is difficult to anticipate which supervised model will be the one that will achieve a better fit to the data a priori; this condition is known as the No Free Lunch theorem. Since the algorithms with greater theoretical prediction capacity have less or no capacity to explain the relationships between the input and output variables. Thus, considering the difference in the performance of the different algorithms, five algorithms will be used: Random Forest, XGBoost, GLMNET, Linear Regression and Decision Trees.

4.1. Proposed Modelling Structure

This study proposes the creation of five machine learning models to generate suggestions on which professional path a student should pursue. The method is determined by dividing the first data set into 80% training and 20% testing. The training technique was designed using a 10-fold cross-validation scheme, in which the data set is split randomly into ten equal halves. Nine are utilised as the training set, with the remaining one serving as the testing set. The mean value of the predictive outcomes from the ten rounds is used to evaluate the algorithm.

Figure 1. Machine learning development.

Figure 1. Machine learning development.

The model's input variables represent student information. They are classified into three categories: information on standardised test results, socioeconomic information in high school, and information on standardised test results in college. Table 1 shows the structure, category, and description of the variables.

In operationalising the model, a new student enters their academic results and social variables into the model and thus receives the recommendation of the careers for which a better performance is projected in the Saber Pro university exam results. Figure 2 represents the operational flow in which the algorithm recommends degree options.

4.2. Data

Data were collected from the Colombian Education Institute (ICFES) [22,23] and included the records of undergraduate students who took the Saber11 (high school) and SaberPro (University) exams from 2006 to 2019. These data are made publicly available by ICFES for research and analysis. The comprehensive database contains 921,041 results from high school and university standardised tests. The dataset encompasses 102 focus options covering various categories such as engineering, literature, science, and art. Additionally, it includes socioeconomic information such as parent's education and occupation, type of school, academic calendar, gender, and department of residence. Consequently, the recommender system will be represented by fourteen variables, as outlined in Table 1. Eight variables are categorical and derived from demographic information and the student’s background. This information was sourced from the ICFES repository, with the standardised test result designated as the target variable at the university level.

4.2. Descriptive Data Analysis

Table 2 shows the statistical summary for the 921,041 records adjusted by gender in the database for the fourteen years of the study. As shown in Table 2, the gender distribution is 58% female and 42% male. Besides, the median and mean values for all modules evaluated are higher for men than women.

4.3. Performance Metrics

Metrics and comparison structures are needed to identify improvement patterns in the regression model that presents the best performance in different situations. These metrics are called performance indicators; they will give an idea of ​​the general performance of the prediction models of the standardised exam results—a well-fitted regression model results in predicted values ​​close to the observed data values. The base model, which uses the mean for each predicted value, would generally be used without informative predictor variables. Therefore, the fit of a proposed regression model should be better than the fit of the base model to be considered as a model capable of generating information beyond the average value.

4.3.1. Root Mean Square Value

The RMSE (Root Mean Square Error) is the square root of the residuals' variance, indicating the model's absolute fit to the data by measuring how close the observed data points are to the model's predicted values. As an absolute measure of fit, RMSE can be interpreted similarly to the standard deviation of the unexplained variance, sharing the same units as the response variable. Lower RMSE values denote a better fit. RMSE is an effective measure of the model's predictive accuracy and is the most critical criterion for assessing fit when prediction is the model's primary objective. Generally, to evaluate the quality of the fit, the model's RMSE value should be significantly lower than the standard deviation of the response variable. The RMSE is calculated by Equation (2).

$$\mathit{R}\mathit{M}\mathit{S}\mathit{E}=\sqrt{{\displaystyle \frac{1}{\mathit{n}}}\sum _{\mathit{i}=1}^{\mathit{n}}{\left({\widehat{\mathit{y}}}_{\mathit{i}}-{\mathit{y}}_{\mathit{i}}\right)}^2}$$

(2)

5. Results

In this study, a recommender system based on machine learning algorithms was built using the R programming language within the Caret package. Caret is a sophisticated library that develops supervised learning modelling with tools for preprocessing and cross-validation of data. Random Forest, XGBoost, GLMNET and KNN supported the recommender system. To train the proposed model, ten folds were created to experiment with the predictions and evaluate each algorithm's performance; the training stage results are listed in Table 3. The average value of the ten folds was calculated as the algorithm's overall prediction performance. As shown in Table 3, the mean RMSE values for XGBoost, RF, GLMNET and KNN are 30.1, 45.0, 41.1 and 51.1, respectively.

Therefore, considering that the standard deviation of the output variable is 70.4 and comparing the RMSE results shown in Table 3, the model with the best performance in the training stage is the XGBOOST with an RMSE value of 30.1, followed by the GLMNET algorithm with a RMSE value of 41.1. Consequently, it is essential to analyse the model's consistency; again, the XGBoost is the algorithm with the lowest standard deviation value, followed by the GLMNET.

The predictions of the standardised tests at the university in this study are used to recommend a degree to study; therefore, a high capacity for prediction generates confidence that the recommendations created serve as objective support for decision-making.

5.2. Model Testing

The materials and methods section explained that 30% of the data was reserved for the supervised learning models' validation phase. Table 2 shows the RMSE results for data not previously known by the algorithms. In comparing the models, XGBoost has the best performance, with an RMSE value of 30.1.

The RMSE values for the five algorithms implemented in the recommender system are all lower than the standard deviation of the testing dataset used for the respective models. These results demonstrate that these four models perform well in predicting standardised test results. However, the XGBoost algorithm surpasses the other models (XGBoost vs. GLMNET, RF, KNN) in terms of performance.

6. Discussion

Developing and evaluating a recommender system for predicting standardised test results and recommending suitable undergraduate degrees based on academic and sociodemographic data present significant advancements in educational technology. Our study leverages a combination of machine learning algorithms to enhance students' decision-making processes. This discussion compares our findings with those of previous studies and highlights the unique contributions of our research.

To begin with, Bhumichitr implemented collaborative filtering and ALS for elective course recommendations, achieving an accuracy of 86%. While their approach focused on course selection, our system extends to degree recommendations using a broader set of variables, including sociodemographic data, enhancing the recommendations' contextual relevance [24].

Similarly, Khanam and Alkhaldi used a random forest regression model to classify universities based on student preferences and voting data. Although their system focused on graduate admissions, it parallels our use of random forest in predicting undergraduate degree suitability, highlighting the versatility and effectiveness of this algorithm [25].

Moreover, Esteban developed a hybrid recommendation system combining collaborative and content-based filtering with genetic optimisation to recommend courses. Their success in integrating multiple criteria underscores the importance of using diverse data points, a strategy we also employed to capture a comprehensive view of student profiles [26].

In addition, Samin and Azim applied probabilistic topic models for course and supervisor recommendations, focusing on the faculty’s past publications and research interests. Our system similarly uses historical academic data but extends it to include sociodemographic factors, thus broadening the scope and potential impact of the recommendations [27].

Furthermore, Slim proposed a framework for recommending academic programs, courses, and instructors using collaborative filtering and non-negative matrix factorisation. While their system focused on optimising course schedules, our approach aimed at predicting standardised test results and recommending degrees, thereby supporting long-term academic planning [28].

Likewise, Ognjanovic used institutional data and the Analytical Hierarchy Process (AHP) to predict course selections. Similarly, our study utilises institutional data but expands its use to recommend undergraduate degrees, thus providing more holistic support for academic decisions [29].

Additionally, George and Lal employed a personalised approach to course recommendation using ontology and sequential pattern mining. Our system also emphasises personalisation by incorporating sociodemographic factors, ensuring that recommendations align closely with individual student profiles [30].

Moreover, Zayed investigated supervised machine learning techniques for recommending undergraduate programs using academic history and job market data. Our study similarly uses machine learning but focuses on standardised test predictions and degree recommendations, providing a comprehensive tool for student support [5].

In summary, our study stands out by integrating both academic and sociodemographic data to provide holistic recommendations. We achieved a robust prediction model by employing advanced machine learning algorithms such as XGBoost, Random Forest, GLMNET, and KNN. Including variables such as parental education, school type, and regional data enhances the accuracy and relevance of our recommendations, offering a significant improvement over systems that rely solely on academic performance data.

7. Limitations and Future Research

Firstly, while comprehensive, the dataset used in this study may not fully capture the diversity of student populations across different regions, genders and educational systems. The sociodemographic variables included, such as parental education and school type, are limited to available data and may not encompass all factors influencing students' academic performance and degree choices. The dataset utilized for this research only includes two gender options: Male and Female. This binary classification does not encompass the full spectrum of gender identities, which could lead to an incomplete understanding of gender-related dynamics in the context of our study. Future studies should aim to include a broader and more diverse dataset to enhance the generalizability of the findings.

Secondly, although robust, the prediction models employed in this study depend on the quality and accuracy of the input data. Any inconsistencies or inaccuracies in the standardised test results or sociodemographic information can affect the recommender system's performance. Thus, ensuring high-quality data collection and preprocessing is crucial for the system's reliability.

Thirdly, while the machine learning algorithms (XGBoost, Random Forest, GLMNET, and KNN) have shown solid predictive capabilities, they also require significant computational resources and expertise to implement and maintain. This may pose a challenge for some educational institutions with limited technical resources. Future research should explore more efficient algorithms and methods to reduce the computational burden without compromising accuracy.

Additionally, the temporal scope of the data, which spans from 2006 to 2019, presents another limitation. To articulate the dataset, it is necessary to trace the students' academic performance from their high school years through to their university education. This process requires an average of at least five years to obtain a high school student's university standardised exam results. Consequently, there is an inherent delay in the availability of comprehensive data, which may impact the timeliness and relevance of the study’s findings. Future studies should consider strategies to mitigate this delay, such as employing longitudinal data collection methods, to ensure more timely and applicable insights.

8. Conclusions

This study presents a significant advancement in educational technology by developing and evaluating a recommender system designed to predict standardised test results and recommend suitable undergraduate degrees based on a combination of academic and sociodemographic data. Our approach leverages advanced machine learning algorithms, specifically XGBoost, Random Forest, GLMNET, and KNN, to enhance students' decision-making.

Firstly, integrating academic and sociodemographic variables, such as parental education, school type, and regional data, proved crucial in improving the accuracy and contextual relevance of the recommendations. This holistic approach addresses the limitations of previous systems that relied solely on academic performance data, offering a more comprehensive tool for student support.

Our findings demonstrate that machine learning models, particularly XGBoost and Random Forest, exhibit robust performance in predicting standardised test outcomes and recommending undergraduate degrees. These models significantly outperform simpler algorithms, highlighting the importance of employing sophisticated techniques to achieve high accuracy in educational recommendations.

Furthermore, the comparative analysis with prior studies underscores the unique contributions of our research. While previous works focused on course recommendations or graduate admissions, our system extends its applicability to undergraduate degree recommendations, providing long-term academic planning support. Incorporating diverse data points and utilising hybrid recommendation frameworks enhance the system's effectiveness and reliability.

In conclusion, the proposed recommender system supports students in making informed academic decisions and aligns with the broader educational objectives of reducing gender disparities and ensuring equal access to education. Future research can explore the integration of real-time data and the continuous improvement of recommendation algorithms to refine the system's predictive capabilities further. Expanding the system to include personalised learning paths and career counselling features could offer more excellent value to students and educational institutions.

Overall, this study contributes to the ongoing efforts to harness the power of data science and machine learning in education, aiming to improve educational outcomes and promote equity in access to quality education.