Optimizing E-Commerce with Multi-Objective Recommendations Using Ensemble Learning

1. Introduction

The rapid evolution of e-commerce necessitates advanced recommendation systems to enhance user experience and increase sales. Traditional systems focus on a single objective, such as click prediction. However, real-world online shopping involves multiple actions, including clicks, cart additions, and orders. Accurately predicting these actions is crucial for online retailers to optimize their marketing strategies and sales funnels.

User behavior in online shopping is complex, influenced by factors like product attributes, user preferences, and contextual information. Traditional single-objective systems often fail to capture this complexity, leading to suboptimal performance. Our research introduces a multi-objective recommendation system that predicts clicks, cart additions, and orders by leveraging the sequential nature of user interactions.

E-commerce data presents significant challenges, particularly the imbalance between positive and negative samples. Click events occur more frequently than order events, creating a skewed data distribution that can bias model training. Additionally, many user sessions are short, complicating the extraction of meaningful behavior patterns. Another challenge is the varying length of user sessions. Training sessions in our dataset are, on average, four times longer than test sessions, with some sessions extending up to 500 interactions. This discrepancy can lead to overfitting, where the model performs well on training data but fails to generalize to shorter sessions.

To address these challenges, we propose a model integrating LightGBM, XgBoost, and Recbole GRU4Rec through ensemble learning. LightGBM and XgBoost are gradient boosting frameworks known for handling structured data and capturing complex non-linear relationships. These models manage class imbalances effectively with techniques like weighted loss functions and balanced sampling. Recbole GRU4Rec, a recurrent neural network variant, excels at capturing temporal dependencies in user interactions, making it suitable for modeling the sequential nature of e-commerce sessions.

Our methodology includes extensive data preprocessing to address sample bias and sampling issues, such as oversampling underrepresented events and undersampling overrepresented events. Feature selection identifies the most relevant features for predicting each type of user action, reducing model complexity and improving performance.

Our model has demonstrated superior performance in evaluations, making it a valuable tool for enhancing e-commerce sales strategies. By accurately predicting user behaviors such as clicks, cart additions, and orders, online retailers can optimize their marketing efforts, personalize user experiences, and ultimately increase sales. This research advances the field of e-commerce recommendation systems by addressing the limitations of traditional single-objective models and presenting a comprehensive, multi-objective approach that improves prediction performance and practical applicability.

2. Related Work

The development of recommendation systems has been a focal point in e-commerce research, with various models proposed to enhance prediction accuracy and user satisfaction. Neural Collaborative Filtering (NCF) employs neural networks to model latent features, improving recommendation performance compared to traditional methods. He et al. [1] demonstrated that NCF could effectively capture complex user-item interactions, resulting in more accurate and personalized recommendations.

Session-based recommendations have advanced with recurrent neural networks (RNNs), which capture sequential dependencies in user behavior. Hidasi et al. [2] introduced an RNN model for session-based recommendations, showing significant improvements in handling user sessions and predicting the next interaction. However, RNNs can struggle with long-term dependencies and require substantial computational resources, making them challenging to deploy at scale.

Gradient boosting frameworks like Xgboost and LightGBM have proven effective in handling structured data and capturing complex non-linear relationships. Chen and Guestrin [3] highlighted Xgboost’s scalability and efficiency in large-scale data processing, while Ke et al. [4] demonstrated LightGBM’s superior performance in terms of speed and memory usage. These models are particularly effective in managing class imbalances and handling missing values, but they may not effectively capture sequential data dependencies.

Deep learning techniques have further enhanced recommendation systems. Covington et al. [5] developed deep neural networks for YouTube recommendations, showcasing the power of deep learning in handling large-scale recommendation tasks. This model significantly improved the relevance of video recommendations by leveraging vast amounts of user interaction data. Similarly, Sun et al. [6] proposed BERT4Rec, a model leveraging bidirectional transformers to capture sequential patterns in user behavior, yielding state-of-the-art results in sequential recommendation tasks.

Content-based approaches, often integrated with collaborative filtering, have addressed some limitations of each method. Zhang et al. [7] provided a comprehensive survey on deep learning-based recommender systems, emphasizing the integration of user and item features through neural networks. This method allows for a more nuanced understanding of user preferences and item characteristics, improving recommendation accuracy. However, it requires extensive feature engineering and computational resources, which can be a limitation in practice.

Hybrid models combining collaborative filtering and content-based approaches have shown promise in leveraging the strengths of both methods. Koren and Bell [8] discussed advancements in collaborative filtering, highlighting the benefits of incorporating user and item attributes. Hybrid approaches can provide more accurate and personalized recommendations by utilizing a broader range of data sources. Zhao et al. [9] explored the integration of social media data into e-commerce recommendations, addressing the cold-start problem by leveraging microblogging information. This approach enhances the system’s ability to recommend items to new users or recommend new items more effectively.

Trust-based models have also been investigated to enhance recommendation accuracy. Guo et al. [10] proposed TrustSVD, a model that incorporates both explicit and implicit user trust information into the recommendation process, demonstrating improved performance over traditional collaborative filtering methods. Trust-based models can enhance recommendation accuracy by considering the reliability of user interactions and ratings.

Graph-based approaches have gained traction for their ability to model complex relationships in recommendation data. Wang et al. [11] introduced KGAT, a knowledge graph attention network that enhances recommendations by leveraging relational data from knowledge graphs. This model outperforms traditional approaches by capturing higher-order connectivity patterns, providing more accurate and contextually relevant recommendations.

Sequential recommendation models continue to evolve, with Ma et al. [12] proposing hierarchical gating networks that dynamically capture user preferences at multiple levels. This approach improves the accuracy of sequential predictions by considering the hierarchical structure of user preferences. Zheng et al. [13] combined user and item reviews using deep learning to jointly model user preferences and item characteristics, achieving superior performance in personalized recommendations.

Visual information has also been integrated into recommendation systems. He and McAuley [14] presented VBPR, a model that integrates visual features from product images into the recommendation process, enhancing the accuracy of predictions based on implicit feedback. This approach is particularly useful for e-commerce platforms where visual appeal is a critical factor in user decision-making.

Explainable recommendation systems have gained attention for their potential to increase user trust and satisfaction. Zhang and Chen [15] conducted a comprehensive survey on explainable recommendations, highlighting the importance of transparency in recommendation systems. Explainable recommendations provide insights into why certain items are recommended, which can enhance user engagement and trust.

In summary, significant progress has been made in recommendation systems, but challenges like sample imbalance, varying session lengths, and dynamic user behavior remain. Our proposed multi-objective recommendation system addresses these issues by combining state-of-the-art models and ensemble learning, resulting in enhanced prediction performance and practical applicability for online retailers.

3. Methodology

Our modeling strategy is a comprehensive approach to session-based recommendations using graph-based models and gradient boosting methods. Session-based recommendation systems aim to predict users’ next actions based on their current session data. We explores the implementation of Graph-SAGE, LightGBM, and GRU4Rec models to enhance recommendation accuracy. We describe the preprocessing techniques, model architectures, evaluation metrics, and experimental results in detail. The whole model ensemble pipeline is shown in Figure 1

3.1. LightGBM and Xgboost

LightGBM and XGBoost are powerful ensemble learning methods that leverage gradient boosting algorithms. These models are particularly effective for structured data and have shown superior performance in various machine learning tasks.

3.1.1. Gradient Boosting Framework

Both LightGBM and XGBoost build an ensemble of trees, where each tree is trained to correct the errors of its predecessor. The objective function combines the loss function l and a regularization term $\mathsf{\Omega }$ to prevent overfitting:

$$L\left(\mathsf{\Theta }\right)=\sum _{i=1}^{n}l(y_i,{\widehat{y}}_i)+\sum _{k=1}^K\mathsf{\Omega }\left(f_k\right)$$

(1)

where $\mathsf{\Theta }$ represents the model parameters, $y_i$ are the true labels, ${\widehat{y}}_i$ are the predicted labels, and $f_k$ represents the k-th tree in the ensemble.

3.1.2. Tree Structure

Each tree in the ensemble is constructed by optimizing the split points to maximize information gain. The leaf nodes of the trees contain the final predictions, and the model’s prediction is the sum of the predictions from all trees.

$${\widehat{y}}_i=\sum _{k=1}^K{f}_k\left(x_i\right)$$

(2)

3.1.3. Regularization

Both LightGBM and XGBoost incorporate regularization to control the complexity of the model. The regularization term $\mathsf{\Omega }$ penalizes large weights and complex tree structures:

$$\mathsf{\Omega }\left(f_k\right)=\gamma T+{\displaystyle \frac{1}{2}}\lambda \sum _{j=1}^T{w}_j^2$$

(3)

where T is the number of leaves in the tree, $w_j$ are the leaf weights, $\gamma$ is the regularization parameter for the number of leaves, and $\lambda$ is the regularization parameter for the leaf weights.

3.1.4. Optimization

The training process involves minimizing the objective function using gradient descent. For LightGBM, a histogram-based algorithm is used to speed up the training process by approximating the continuous feature values with discrete bins.

3.1.5. Feature Importance

Both models provide feature importance scores, which indicate the contribution of each feature to the prediction. This is useful for feature selection and model interpretation.

3.1.6. Handling Missing Values

LightGBM and XGBoost can handle missing values by automatically learning the best way to split the data even when some feature values are missing. By combining the strengths of these methods, we can build robust and accurate models for session-based recommendations.

3.2. GRU4Rec

GRU4Rec leverages Gated Recurrent Units (GRUs) for sequence modeling. It captures temporal dependencies in user behavior sequences. The GRU update equations are:

$$\begin{array}{cc}\hfill {\mathbf{z}}_t& =\sigma ({\mathbf{W}}_z{\mathbf{x}}_t+{\mathbf{U}}_z{\mathbf{h}}_{t-1})\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\mathbf{r}}_t& =\sigma ({\mathbf{W}}_r{\mathbf{x}}_t+{\mathbf{U}}_r{\mathbf{h}}_{t-1})\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{h}}}_t& =tanh(\mathbf{W}{\mathbf{x}}_t+\mathbf{U}({\mathbf{r}}_t\odot {\mathbf{h}}_{t-1}))\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathbf{h}}_t& =(1-{\mathbf{z}}_t)\odot {\mathbf{h}}_{t-1}+{\mathbf{z}}_t\odot {\tilde{\mathbf{h}}}_t\hfill \end{array}$$

(7)

3.3. Graph-SAGE

Graph-SAGE is a method for inductive node embedding. Unlike traditional methods that learn a unique embedding for each node, Graph-SAGE generates embedding by sampling and aggregating features from a node’s local neighborhood. This enables the model to generalize to unseen nodes, making it scalable and versatile.

Figure 2. Graph-SAGE Pipeline.

Figure 2. Graph-SAGE Pipeline.

3.3.1. Neighbor Sampling

For each node v, a fixed-size set of neighbors $\mathcal{N}\left(v\right)$ is sampled. This reduces the computational complexity by considering a limited neighborhood size.

$${\mathcal{N}}_k\left(v\right)=\{u_1,u_2,\dots ,u_k\}$$

(8)

3.3.2. Feature Aggregation

The sampled neighbors’ features are aggregated to form the node’s embedding. The aggregation function can vary, with common choices being mean, sum, or max pooling. The node embedding at layer k, ${\mathbf{h}}_v^{\left(k\right)}$, is computed as follows:

$${\mathbf{h}}_v^{\left(k\right)}=\sigma \left({\mathbf{W}}^{\left(k\right)}·{\mathrm{AGG}}^{\left(k\right)}\left(\left\{{\mathbf{h}}_u^{(k-1)},\forall u\in \mathcal{N}\left(v\right)\right\}\right)\right)$$

(9)

where ${\mathbf{W}}^{\left(k\right)}$ is a trainable weight matrix, $\sigma$ is a nonlinear activation function (e.g. ReLU), and AGG is a function that combines the features of the neighbors.

3.3.3. Multi-Layer Structure

Graph-SAGE employs multiple layers to capture higher-order dependencies. The embedding of a node at the final layer K is influenced by its K-hop neighborhood. For instance, a two-layer Graph-SAGE model would capture information from a node’s neighbors and the neighbors of those neighbors.

$${\mathbf{h}}_v^{\left(K\right)}={\mathrm{AGG}}^{\left(K\right)}\left({\mathbf{h}}_v^{(K-1)},\left\{{\mathbf{h}}_u^{(K-1)},\forall u\in \mathcal{N}\left(v\right)\right\}\right)$$

(10)

3.3.4. Unsupervised Learning

For unsupervised node representation learning, Graph-SAGE uses a loss function that maximizes the similarity between the embedding of adjacent nodes while minimizing the similarity between the embedding of randomly sampled nodes. The loss function is defined as:

$$L=-\sum _{(v,u)\in E}log\sigma ({\mathbf{h}}_v·{\mathbf{h}}_u)-Q·{\mathbb{E}}_{u_n\sim P_n\left(v\right)}log\sigma (-{\mathbf{h}}_v·{\mathbf{h}}_{u_n})$$

(11)

where E is the set of edges, Q is the number of negative samples, and $P_n\left(v\right)$ is a negative sampling distribution.

3.4. Model Embedding

To further enhance the prediction accuracy, we employed an ensemble strategy combining the outputs of GraphSAGE, LightGBM, and XGBoost. The ensemble method leverages the strengths of each model, providing a more robust and generalized solution.

3.4.1. Model Integration

The outputs from each model are integrated using a weighted averaging approach. The ensemble prediction $\widehat{y}$ is given by:

$$\widehat{y}=\alpha {\widehat{y}}_{GraphSAGE}+\beta {\widehat{y}}_{LightGBM}+\gamma {\widehat{y}}_{XGBoost}$$

(12)

where $\alpha$, $\beta$, and $\gamma$ are the weights assigned to each model’s prediction. These weights are determined through cross-validation to optimize the overall performance.

3.4.2. Optimization of Weights

The weights are fine-tuned using the Optuna library, which performs hyperparameter optimization. The objective is to minimize the ensemble’s validation error, thereby selecting the best combination of model predictions.

4. Experiments and Results

In the experiment, we focus on the training loss index and the AUC index. The integrated model achieves stable training effect after 350 epochs of training in Figure 3 and Figure 4

4.1. Evaluation Metrics

To evaluate the performance of our models, we used several metrics that capture different aspects of recommendation quality.

4.1.1. NDCG

NDCG measures the ranking quality by considering the position of relevant items. It is defined as:

$$\mathrm{NDCG}={\displaystyle \frac{1}{\mathrm{IDCG}}}\sum _{i=1}^p{\displaystyle \frac{2^{re{l}_i}-1}{{log}_2(i+1)}}$$

(13)

where $re{l}_i$ is the relevance score of the item at position i, and IDCG is the ideal DCG, which is the maximum possible DCG for the given set of items.

4.1.2. Area Under the Curve

AUC measures the model’s ability to distinguish between positive and negative samples. It is defined as the probability that a randomly chosen positive sample ranks higher than a randomly chosen negative sample. Mathematically, AUC is given by:

$$\mathrm{AUC}={\displaystyle \frac{1}{N^{+}{N}^{-}}}\sum _{i=1}^{N^{+}}\sum _{j=1}^{N^{-}}\mathbf{1}(s_i>s_j)$$

(14)

where $N^{+}$ and $N^{-}$ are the number of positive and negative samples, respectively, and $\mathbf{1}(s_i>s_j)$ is an indicator function that is 1 if the score of the positive sample $s_i$ is greater than the score of the negative sample $s_j$, and 0 otherwise.

4.1.3. Accuracy

Accuracy measures the proportion of correct predictions among the total number of predictions. It is defined as:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{Correct}\mathrm{Predictions}}{\mathrm{Total}\mathrm{Number}\mathrm{of}\mathrm{Predictions}}}$$

(15)

The models were evaluated on a public and private test set, with performance measured by the metric we mentioned before. The results are summarizedin Table 1:

5. Conclusion

In conclusion, Our research shows that combining graph-based models with gradient boosting and sequence modeling techniques can significantly enhance session-based recommendation systems. Sampling LightGBM + XgBoost for feature selection and preprocessing, and GRU4Rec for processing sequence features, the proposed preprocessing method and model architecture can effectively capture complex user interactions, thereby improving recommendation accuracy.