Hybrid Model Integration of LightGBM, DeepFM, and DIN for Enhanced Purchase Prediction on the Elo Dataset

1. Introduction

With big data, personalized recommendation systems are essential for optimizing user experiences and improving business outcomes, particularly in e-commerce and finance. These systems rely on analyzing vast amounts of data to predict user preferences and behaviors, ultimately providing tailored recommendations that meet individual needs. However, as data grows in volume and complexity, traditional recommendation models face significant challenges in effectively modeling user behavior, especially in environments where transaction data is both high-dimensional and temporally dynamic.

Collaborative filtering and matrix factorization have long been the backbone of recommendation systems, offering a solid foundation for predicting user preferences based on historical interactions. However, collaborative filtering often struggles with data sparsity and lacks scalability, making it less effective when applied to large, sparse datasets typical of financial transactions. While matrix factorization reduces dimensionality, it struggles to capture complex, non-linear relationships between users and items, especially as interactions change over time.

To address these limitations, this research proposes a hybrid model combining LightGBM, DeepFM, and DIN to predict user purchasing behavior using the Elo payment dataset. LightGBM, a highly efficient gradient boosting framework, is well-suited for handling large-scale datasets and complex feature interactions due to its tree-based structure. Its ability to process large volumes of data quickly and accurately makes it a powerful tool for baseline predictions. However, LightGBM’s tree-based approach is inherently limited in capturing intricate higher-order interactions, which are essential for understanding nuanced user behaviors and improving recommendation accuracy.

To bridge this gap, we integrate DeepFM, which combines factorization machines (FM) with deep neural networks (DNN). This approach enables DeepFM to capture both low-order linear interactions through FM and complex non-linear relationships via DNN. This dual capability enables DeepFM to better understand and predict user preferences, making it particularly effective for tasks involving personalized recommendations.

Moreover, to enhance the model’s ability to handle sequential data, we integrate the Deep Interest Network (DIN). DIN introduces an attention mechanism that selectively activates relevant user behaviors based on the specific context of each prediction. By focusing on the most pertinent aspects of a user’s historical actions, DIN significantly enhances the model’s predictive power, especially in scenarios where understanding the sequence and context of user actions is crucial. However, the sensitivity of DIN to noisy or sparse data necessitates careful model training to avoid overfitting and ensure robust predictions.

The significance of this research lies in its ability to integrate these advanced techniques into a cohesive hybrid model that addresses the inherent limitations of traditional recommendation systems. By combining LightGBM, DeepFM, and DIN, and employing sophisticated training methods, we achieve a model that not only improves prediction accuracy but also offers scalability and adaptability for real-world applications. This approach signifies a significant step forward in personalized recommendation systems, especially in the context of payment data, where precision and reliability are essential.

2. Related Work

The development of personalized recommendation systems has advanced significantly, driven by both traditional machine learning and deep learning techniques.Early methods like collaborative filtering and matrix factorization provided a foundation but faced challenges with scalability and data sparsity. He et al. [1] addressed these issues by introducing Neural Collaborative Filtering (NCF), which utilized neural networks to capture non-linear user-item interactions, though it required extensive hyperparameter tuning.

Deep learning models, including Wide Deep [2] and DeepFM [3] , marked a major shift in the field. Wide Deep combines linear and deep neural models to balance memorization and generalization, while DeepFM integrates factorization machines with deep learning to automate feature interaction modeling. Despite their success, these models increase computational complexity and training time.

Attention mechanisms have further enhanced recommendation systems. The Deep Interest Evolution Network (DIEN) by Li et al. [4] builds on this by modeling interest evolution over time, addressing some of DIN’s limitations.

He et al. [5] provide optimization techniques that we use to enhance clustering and temporal feature extraction, improving our model’s preprocessing efficiency and prediction accuracy.

Yu et al. [6] discuss fine-tuning strategies for domain-specific models, which inform our use of pseudo-labeling to improve the model’s learning from unlabeled data, boosting AUC and NDCG metrics.

Integrating external knowledge has also proven effective. Wang et al. [7] developed the Deep Knowledge-Aware Network (DKN), merging knowledge graphs with deep learning to improve the precision of news recommendation systems. However, this approach introduces additional complexity.

Gradient boosting machines like LightGBM [8] are popular for their efficiency in handling large datasets, though they struggle with higher-order interactions.

Robust training strategies, including adversarial training [9] and adversarial weight perturbation [10], have been critical in enhancing model generalization. Additionally, pseudo-labeling, as discussed by

Tang et al. [11] highlight the growing impact of deep learning on recommendation systems, noting the need for ongoing research to tackle scalability and interpretability challenges. Further work by Zhang et al. [12] and He et al. [13] demonstrate the effectiveness of deep learning and matrix factorization in improving recommendation accuracy.

Finally, large-scale applications of deep learning in recommendation systems have proven highly effective. Covington et al. [14] showcased the power of deep neural networks for personalized video recommendations on YouTube, managing vast amounts of user data to generate precise and scalable recommendations. This work exemplifies the success of deep learning in handling real-world recommendation tasks, paving the way for future innovations in the field.

Despite these advancements, challenges remain in developing models that are not only accurate but also interpretable and efficient. As personalized recommendation systems continue to evolve, integrating more sophisticated models with robust training strategies will be crucial. This involves exploring hybrid models and developing methods to ensure transparency and fairness in recommendations. The ongoing research in this domain underscores the importance of balancing innovation with practical considerations to build systems that can meet the diverse needs of users in increasingly complex environments.

3. Data Preprocessing

We will create a personalized recommendation system in this section, utilizing the dataset from Elo, one of Brazil’s major payment brands, and its associated merchants. In the recommendation system, the features determine the upper limit of the effect. Data preprocessing is essential to address missing values and enhance data richness.

3.1. Feature Clustering

We clustered user behaviors and imputed missing values with the cluster’s mean or median. Historical and new data were sorted by timestamps to create various datasets for different features.

$$x^{\prime }={\displaystyle \frac{x-min\left(x\right)}{max\left(x\right)-min\left(x\right)}}$$

(1)

This normalization technique ensures all features are on a similar scale.

3.2. Temporal Features

Temporal features are critical in understanding user behavior patterns over time. One of the primary temporal features we consider is the time gap between consecutive purchases. This feature measures the time gap, in days, between the current purchase date and the next one. By studying these gaps, we can uncover patterns in user purchasing cycles, enabling us to predict future buying behavior more accurately. To compute the time gap feature, we use the following equation:

$$\mathrm{Time}{\mathrm{Gap}}_i=\mathrm{Purchase}{\mathrm{Date}}_{i+1}-\mathrm{Purchase}{\mathrm{Date}}_i$$

(2)

where $\mathrm{Time}{\mathrm{Gap}}_i$ represents the gap between the i-th and $(i+1)$-th purchase dates. This feature helps in identifying patterns such as frequent buyers, seasonal buyers, and occasional buyers. Additionally, we create aggregated statistics of these time gaps, such as the mean, median, standard deviation, and maximum time gap for each user. These aggregated features provide a summary of the user’s purchasing behavior over the observed period.

$$\mathrm{MTG}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\mathrm{TG}}_i$$

(3)

$$\mathrm{SVD}\mathrm{of}\mathrm{TG}=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{({\mathrm{TG}}_i-\mathrm{MTG})}^2}$$

(4)

By incorporating these temporal features into our models, we enhance the predictive power by capturing the temporal dynamics of user behavior.

3.3. Dimensionality Reduction

Dimensionality reduction techniques are essential for managing high-dimensional data and improving model performance. This study uses Singular Value Decomposition (SVD) to reduce the dimensionality of merchant transaction sequences for each card ID. SVD transforms high-dimensional transaction data into a lower-dimensional space, retaining most of the variance. The SVD process is mathematically expressed as:

$$X=U\Sigma V^T$$

(5)

where X is the original data matrix, U and V are orthogonal matrices, and $\Sigma$ is a diagonal matrix containing the singular values. By selecting the top k singular values, we can approximate X with a reduced representation:

$$X_k=U_k{\Sigma }_k{V}_k^T$$

(6)

In our case, we reduce the transaction sequences to a 5-dimensional space. This reduction simplifies the complexity of the data and enhances the efficiency of the subsequent machine learning models. To validate the effectiveness of SVD, we perform cross-validation and monitor performance metrics such as RMSE and MAE. The reduced dimensions not only improve computational efficiency but also mitigate the risk of overfitting by removing noise and redundant information.

3.4. Word2Vec Embeddings

Word2Vec embeddings are a powerful tool for capturing semantic relationships between entities in a dataset. In this study, we apply Word2Vec to create dense vector representations for merchant IDs, merchant category IDs, and purchase dates. These embeddings are useful for identifying contextual similarities between entities, based on how frequently they co-occur in transaction sequences. The Word2Vec model is trained using Skip-gram to predict context words from a target word. The objective function is:

$$max{\displaystyle \frac{1}{T}}\sum _{t=1}^T\sum _{-c\le j\le c,j\ne 0}logp\left(w_{t+j}\right|w_t)$$

(7)

where T is the total number of words in the corpus, c is the context window size, $w_t$ is the target word, and $w_{t+j}$ are the context words. After training the Word2Vec model, we obtain embeddings for each entity. These embeddings are then aggregated to create feature vectors for each card ID. Aggregation methods include calculating the minimum, maximum, average, and standard deviation of the embeddings for all transactions associated with a card ID.

$${\mathbf{v}}_{card}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\mathbf{v}}_{transactio{n}_i}$$

(8)

where ${\mathbf{v}}_{transactio{n}_i}$ is the embedding vector of the i-th transaction, and n is the total number of transactions for the card ID. The resulting feature vectors capture the inherent similarities and differences between different entities, enhancing the predictive power of our models. By leveraging Word2Vec embeddings, we effectively transform categorical data into a continuous vector space, enabling more sophisticated analysis and modeling.

4. Model Architecture and Methodology

The primary models utilized in this study are LightGBM, DeepFM, and DIN, each contributing unique strengths to the predictive capabilities. The complete model ensemble pipeline is illustrated in Figure 1.

4.1. Lightgbm

LightGBM is a gradient boosting framework that uses tree-based learning algorithms. It is designed to be distributed and efficient, capable of handling large-scale data.

$$\mathrm{Objective}\mathrm{function}=\sum _{i=1}^{n}l(y_i,{\widehat{y}}_i)+\sum _{j=1}^m\Omega \left(f_j\right)$$

(9)

where l is the loss function, $\Omega$ is the regularization term, $y_i$ is the true value, and ${\widehat{y}}_i$ is the predicted value.

4.2. DeepFM

DeepFM combines factorization machines with deep neural networks to capture high-order feature interactions for recommendation tasks. DeepFM’s architecture combines two components: a factorization machine (FM) for low-order feature interactions and a deep neural network (DNN).

$$\mathrm{FM}\left(\mathbf{x}\right)=w_0+\sum _{i=1}^n{w}_i{x}_i+\sum _{i=1}^n\sum _{j=i+1}^n\langle {\mathbf{v}}_i,{\mathbf{v}}_j\rangle x_i{x}_j$$

(10)

where $w_0$ is the global bias, $w_i$ are the weights of the first-order features, ${\mathbf{v}}_i$ are the latent vectors representing the features, and $\langle {\mathbf{v}}_i,{\mathbf{v}}_j\rangle$ denotes the dot product between the latent vectors. The DNN component captures the high-order feature interactions. The input to the DNN is the concatenation of the embeddings of the sparse features:

$${\mathbf{z}}_0=[{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n]$$

(11)

The DNN layers are defined as follows:

$${\mathbf{z}}_l=\sigma ({\mathbf{W}}_l{\mathbf{z}}_{l-1}+{\mathbf{b}}_l)$$

(12)

where ${\mathbf{z}}_l$ is the output of the l-th layer, ${\mathbf{W}}_l$ and ${\mathbf{b}}_l$ are the weights and biases, and $\sigma$ is the activation function. The outputs of the DNN and FM components are combined for the final prediction:

$$\mathrm{y}=\sigma (\mathrm{FM}\left(\mathbf{x}\right)+\mathrm{DNN}\left({\mathbf{z}}_0\right))$$

(13)

4.3. DIN

The Deep Interest Network (DIN) uses attention mechanisms to dynamically capture user interests from relevant past behaviors. The core idea is to represent user interests as an aggregation of their historical behaviors, weighted by their relevance to the current context. The attention mechanism in DIN can be formulated as:

$${\alpha }_i={\displaystyle \frac{exp\left(\mathrm{score}({\mathbf{h}}_i,\mathbf{e})\right)}{{\sum }_{j=1}^{n}exp\left(\mathrm{score}({\mathbf{h}}_j,\mathbf{e})\right)}}$$

(14)

where ${\mathbf{h}}_i$ represents the hidden state of the i-th historical behavior, $\mathbf{e}$ is the embedding of the current context, and score is a function that measures the relevance between ${\mathbf{h}}_i$ and $\mathbf{e}$.

The weighted sum of historical behaviors forms the final representation of user interest.

$$\mathbf{u}=\sum _{i=1}^n{\alpha }_i{\mathbf{h}}_i$$

(15)

After combining with the current context embedding, this representation is passed to a fully connected network for the final prediction.

$$\mathrm{y}=\sigma ({\mathbf{W}}_o[\mathbf{u},\mathbf{e}]+{\mathbf{b}}_o)$$

(16)

where ${\mathbf{W}}_o$ and ${\mathbf{b}}_o$ are the weights and biases of the output layer.

4.4. Loss

To optimize the models, we employ different loss functions tailored to each model’s architecture and objective. For the DeepFM model, We utilize the binary cross-entropy loss function, which is ideal for binary classification tasks.

$${\mathcal{L}}_{\mathrm{DeepFM}}=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n[y_{i}log\left({\widehat{y}}_i\right)+(1-y_i)log(1-{\widehat{y}}_i)]$$

(17)

For the DIN model, we use a similar binary cross-entropy loss but also experiment with other loss functions to capture sequence-specific information better:

$${\mathcal{L}}_{\mathrm{DIN}}=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n[y_{i}log\left({\widehat{y}}_i\right)+(1-y_i)log(1-{\widehat{y}}_i)]$$

(18)

4.5. Model Ensemble

To further enhance the predictive performance, we combine the predictions of multiple models using an ensemble approach. The ensemble method integrates the strengths of each model, reducing the overall variance and improving generalization. We apply a weighted average ensemble, where the final prediction is the weighted sum of individual model predictions.

$$\widehat{y}=\sum _{k=1}^K{w}_k{\widehat{y}}_k$$

(19)

where K is the number of models, ${\widehat{y}}_k$ is the prediction of the k-th model, and $w_k$ is the weight assigned to the k-th model. The weights $w_k$ are determined through cross-validation, ensuring that the ensemble model leverages the best-performing models more heavily. This approach allows us to capitalize on the diverse strengths of LightGBM, DeepFM, and DIN, leading to robust and accurate predictions. By combining these advanced models and techniques, we achieve a comprehensive solution for predictive analysis, capable of handling complex datasets and delivering high-performance results.

5. Experiments and Results

5.1. Metrics

To assess the impact of the personalized recommendation, we use AUC, recall, and NDCG as the key evaluation indicators.

5.1.1. AUC

AUC measures the model’s ability to distinguish between positive and negative classes, with higher AUC indicating better performance. It is calculated as the area under the ROC curve, which plots the true positive rate against the false positive rate.

$$\mathrm{AUC}={\int }_0^1\mathrm{TPR}\left(t\right)d\mathrm{FPR}\left(t\right)$$

(20)

where $\mathrm{TPR}\left(t\right)$ is the true positive rate and $\mathrm{FPR}\left(t\right)$ is the false positive rate at threshold t.

5.1.2. NDCG

NDCG measures the quality of the model’s ranking, giving higher scores for correct predictions that appear earlier in the list. The formula for Discounted Cumulative Gain (DCG) at position p is defined as:

$${\mathrm{DCG}}_p=\sum _{i=1}^p{\displaystyle \frac{2^{{\mathrm{rel}}_i}-1}{{log}_2(i+1)}}$$

(21)

where ${\mathrm{rel}}_i$ is the relevance score of the item at position i. The Ideal Discounted Cumulative Gain (IDCG) is the maximum possible DCG for a given set of items. NDCG is then given by:

$${\mathrm{NDCG}}_p={\displaystyle \frac{{\mathrm{DCG}}_p}{{\mathrm{IDCG}}_p}}$$

(22)

Ranging from 0 to 1, NDCG values reflect better ranking performance as they increase. By using these diverse metrics, we provide a comprehensive evaluation of the models’ performance, capturing both prediction accuracy and ranking quality.

5.2. Performance

The train result is depicted in Figure 2 and in Figure 3

The models were evaluated on a public test set, with performance measured by the metric we mentioned before. The results are summarizedin Table 1:

6. Conclusions

In conclusion, we illustrates the importance of comprehensive data preprocessing and feature engineering in predictive modeling. By integrating LightGBM, DeepFM, and DIN, we achieved significant improvements in prediction accuracy. Future work will explore further optimization and real-world application of these methodologies.