Beyond Model Complexity: A Reproducible Comparison of Classical Machine Learning, Matrix Factorization, Graph Embeddings, and LightGCN for Recommendation

Recommender systems increasingly incorporate graph embeddings and graph neural networks to capture high-order relationships between users and items. However, the additional complexity of these approaches does not necessarily guarantee better recommendation quality than strong classical and latent-factor baselines. This study presents a reproducible comparison of six recommendation models representing four methodological families: Logistic Regression, Random Forest, Matrix Factorization with Bayesian Personalized Ranking, DeepWalk, node2vec, and LightGCN. The experiments were conducted on the MovieLens 1M dataset using a per-user temporal split. For each user, the most recent positive interaction was assigned to testing, the preceding interaction to validation, and all earlier positive interactions to training. All models were evaluated using identical candidate sets containing one held-out positive movie and 99 sampled unobserved movies. Performance was measured using Recall, Precision, Hit Rate, and NDCG at multiple cutoffs, complemented by bootstrap confidence intervals, paired statistical tests, computational-efficiency measurements, and analyses by user activity and movie popularity. Matrix Factorization achieved the best overall performance, reaching a Recall@10 of 0.7458 and an NDCG@10 of 0.4558, representing an approximately 56% improvement in NDCG@10 over Random Forest, the strongest classical baseline. LightGCN did not significantly outperform Logistic Regression and remained below Random Forest, despite its higher computational cost. DeepWalk and node2vec obtained similar and substantially lower aggregate results. Popularity-based analysis further revealed that classical models and LightGCN achieved substantially higher ranking effectiveness for popular movies, whereas Matrix Factorization maintained comparatively stronger performance for less-popular items. These findings demonstrate that model complexity alone is not a reliable indicator of recommendation effectiveness and highlight the importance of strong baselines, standardized evaluation, and reproducible experimental protocols.