Advanced Large Language Model Ensemble for Multimodal Customer Identification in Banking Marketing

1. Introduction

The banking industry has changed a lot in recent years because of digital technologies. Financial institutions now focus on identifying high-value customers to offer personalized services. Traditional methods, based on demographic or transactional data, do not fully capture customer behavior. With big data and machine learning, banks can now analyze both structured and unstructured data, giving better insights into customer preferences.

Talaat et al. [1] introduced a hybrid model integrating deep learning, explainable AI, and RFM analysis for customer segmentation, enhancing targeting and behavioral insights crucial for financial regulation.

Potluri et al. [2] utilized machine learning for personalized marketing in finance, demonstrating that improved segmentation enhances targeting, retention, and revenue.

Tian Jin [3] introduced attention-based temporal convolutional networks and reinforcement learning for supply chain delay prediction, which can also be used for customer segmentation in banking. This method improves decision-making, helping better resource allocation and customer relationship management.

The proposed ALIMN combines both structured and unstructured data. It uses Large Language Models (LLMs) to analyze complex financial data, like transaction histories and financial documents. By applying domain-specific fine-tuning and attention mechanisms, ALIMN improves the understanding of customer data and provides more accurate targeting. The multi-branch deep ensemble structure further increases prediction accuracy, offering a more comprehensive solution for customer segmentation in banking.

2. Related Work

Recent studies show that machine learning is increasingly used in customer segmentation for banking. Pandey et al. [4] demonstrated that machine learning models, such as clustering and supervised learning, provide more precise insights than traditional methods, improving targeted marketing.Tang et al. [5] introduced Box Adjuster, a reinforcement learning-based method that enhances OCR accuracy by optimizing text bounding boxes.Feng et al. [6] proposed DocPedia, a large multimodal model leveraging frequency-domain processing for superior OCR-free document understanding.Tang et al. [7] proposed a transcription-only text spotting method using query-based learning and audio annotations to reduce annotation costs.

Lu et al. [8] present an automated image-based method for extracting pavement texture and predicting MTD with high accuracy (R² = 0.9858).Tang et al. [9] introduced a transformer-based scene text detection method using feature sampling and grouping to enhance efficiency and eliminate post-processing, achieving state-of-the-art performance.Liu et al. [10] introduced SPTS v2, a single-point annotation-based scene text spotting framework that enhances efficiency and achieves state-of-the-art performance.Zhao et al. [11] proposed TextHarmony, a multimodal generative model leveraging Slide-LoRA for unified visual text comprehension and generation.

Julian and Hariprasath [12] emphasized the role of clustering algorithms in optimizing customer segmentation and marketing outcomes. Dan et al. [13] develop a deep learning-based multiview stereo method for asphalt pavement texture evaluation, achieving stable accuracy (IoU = 0.77) as a lightweight alternative to traditional techniques. Tian Jin [14] applied ensemble models for sales forecasting, enhancing segmentation accuracy in banking.Tang et al. [15] introduced MTVQA, a multilingual TEC-VQA benchmark addressing visual-text misalignment and performance gaps in state-of-the-art models.

Lastly, Dan et al. [16] propose an image-driven system for predicting pavement aggregate gradation, integrating deep learning and interactive processing, achieving high accuracy $R^2>0.99$ for quality assessment.Tang et al. [17] introduced TextSquare and Square-10M, a large-scale dataset that enhances text-centric VQA, surpassing state-of-the-art models. Zhao et al. [18] proposed E2STR, a multi-modal in-context learning model for scene text recognition, enabling training-free adaptation with state-of-the-art performance.Tang et al. [19] organized the first character recognition competition for street view shop signs, detailing tasks, datasets, and winning solutions.

3. Methodology

We propose the LLM Integrated Multi-Modal Network (ALIMN). This framework combines large language model (LLM) features with a multi-branch deep ensemble structure. It captures complex patterns in different types of financial data. ALIMN uses LLMs for extracting semantic features. It includes a contextual fusion module and a multi-model ensemble. These parts process structured and unstructured data together. The design applies special transformation layers and equations. This improves prediction accuracy and generalization in banking campaign response tasks. The system pipeline is shown in Figure 1.

3.1. Extended LLM Integration

Beyond the basic transformer-based embedding, the following advanced strategies are introduced to further enrich the LLM Feature Extractor and adapt it to the financial domain:

3.1.1. Domain-Adaptive Fine-Tuning

The pre-trained LLM is powerful. However, financial and banking texts have unique terms and meanings. To handle this, we fine-tune the model using banking documents, such as product descriptions and financial news. This process updates ${\Theta }_{\mathrm{LLM}}$ to learn financial terms and relationships. For each token $t_i$ in a financial text $\mathcal{D}$, we minimize:

$${\mathcal{L}}_{\mathrm{DA}}=-\sum _{i=1}^{N}logp\left(t_i\mid t_{<i};{\Theta }_{\mathrm{LLM}}\right),$$

(1)

where $t_{<i}$ includes all tokens before $t_i$, and N is the total number of tokens in the text.

3.1.2. Prompt Engineering and Instruction Tuning

When $\mathbf{x}$ has short text segments, such as customer notes, we use prompt engineering to help the LLM create more relevant embeddings. Instruction tuning makes the LLM respond better to financial prompts, like “Identify campaign-related interests.” If ${\mathbf{x}}_{\mathrm{prompt}}$ is the input text with a prompt r, we write:

$${\mathbf{e}}_{\mathrm{prompt}}=\mathrm{LLM}\left([\mathbf{x};r];{\Theta }_{\mathrm{LLM}}\right),$$

(2)

which gives a special embedding for later tasks.

3.1.3. Multi-Phase Training with Knowledge Distillation

We do not use a single fixed LLM checkpoint. Instead, we apply a multi-phase training method. First, a large teacher model, such as T5 or DeBERTa-v3, creates high-quality embeddings or intermediate outputs $z_i^{\left(\mathrm{T}\right)}$ for each input i. Then, a smaller student LLM generates $z_i^{\left(\mathrm{S}\right)}$ by copying these outputs. We minimize the Kullback-Leibler (KL) divergence:

$${\mathcal{L}}_{\mathrm{KD}}=\sum _{i=1}^N\mathrm{KL}\left(p_{\mathrm{teacher}}\left(z_i^{\left(\mathrm{T}\right)}\right),p_{\mathrm{student}}\left(z_i^{\left(\mathrm{S}\right)}\right)\right).$$

(3)

Here, $p_{\mathrm{teacher}}$ and $p_{\mathrm{student}}$ are the outputs of the teacher and student models. The pipeline of the teacher-student model is shown in Figure 2.

3.1.4. Adapter-Based Modularization

To process different data sources, such as regulatory text, marketing messages, and product descriptions, lightweight adapter modules are added to Transformer blocks. Let $h_l$ be the output of the l-th Transformer layer. The adapter ${\mathrm{Adpt}}_l(·)$ is applied as:

$$h_l^{\prime }=h_l+{\mathrm{Adpt}}_l\left(h_l;{\Theta }_{\mathrm{adpt}}\right),$$

(4)

where ${\Theta }_{\mathrm{adpt}}$ represents the adapter-specific parameters. During inference, the model activates the right adapters based on domain tags linked to the input data. The pipeline of the adapter-based approach is shown in Figure 3.

3.1.5. Hierarchical Attention for Hybrid Inputs

When $\mathbf{x}$ includes both text segments, such as tokenized sentences, and structured metadata, such as demographic codes, we use a hierarchical attention mechanism. Let $H_{\mathrm{text}}\in {\mathbb{R}}^{T\times d}$ be the hidden states of text from LLM self-attention. Let $H_{\mathrm{meta}}\in {\mathbb{R}}^{M\times d}$ be the embedding of M structured features. We compute:

$${\mathbf{A}}_{\mathrm{cross}}=\mathrm{softmax}\left({\displaystyle \frac{H_{\mathrm{text}}{H}_{\mathrm{meta}}^T}{\sqrt{d}}}\right),H_{\mathrm{fusion}}={\mathbf{A}}_{\mathrm{cross}}{H}_{\mathrm{meta}},$$

(5)

which captures cross-modal relationships. The fused representation $H_{\mathrm{fusion}}$ is then combined with $H_{\mathrm{text}}$ to create the final representation.

3.2. Contextual Fusion Module

This module combines semantic embeddings from the LLM with structured features $\mathbf{s}\in {\mathbb{R}}^p$. A fusion mechanism is used. A gating function learns the best combination:

$$\mathbf{z}=\sigma \left({\mathbf{W}}_f[\mathbf{e};\mathbf{s}]+{\mathbf{b}}_f\right)\odot \mathbf{e}+\left(1-\sigma \left({\mathbf{W}}_f[\mathbf{e};\mathbf{s}]+{\mathbf{b}}_f\right)\right)\odot \mathbf{s},$$

(6)

where $[\mathbf{e};\mathbf{s}]$ is the concatenation of $\mathbf{e}$ and $\mathbf{s}$. ${\mathbf{W}}_f\in {\mathbb{R}}^{q\times (k+p)}$, ${\mathbf{b}}_f\in {\mathbb{R}}^q$, $\sigma (·)$ is the sigmoid activation function, and ⊙ is element-wise multiplication. The gating function controls how features are mixed based on the input.

3.3. Multi-Branch Ensemble

The fused representation $\mathbf{z}$ is sent to a multi-branch ensemble. This structure captures different parts of the data using separate networks. There are M branches, each processing $\mathbf{z}$ with its own transformation:

$${\mathbf{h}}_m=f_m\left({\mathbf{W}}_m\mathbf{z}+{\mathbf{b}}_m\right),m=1,2,\dots ,M,$$

(7)

where $f_m(·)$ is the activation function for branch m, and ${\mathbf{W}}_m$, ${\mathbf{b}}_m$ are the weights and biases for that branch.

The outputs from all branches are combined using a weighted sum to get the final prediction:

$$\widehat{y}=\sigma \left(\sum _{m=1}^M{\alpha }_m{\mathbf{w}}_m^T{\mathbf{h}}_m+b_{\mathrm{out}}\right),$$

(8)

where ${\alpha }_m\in [0,1]$ is the weight for branch m, with ${\sum }_{m=1}^M{\alpha }_m=1$. ${\mathbf{w}}_m$ is a projection vector for branch m, and $b_{\mathrm{out}}$ is the output bias. The activation function $\sigma (·)$ (such as sigmoid) scales the output for binary classification.

3.4. Regularization and Residual Connections

Residual connections are added to each branch to improve training stability:

$${\mathbf{h}}_m=f_m\left({\mathbf{W}}_m\mathbf{z}+{\mathbf{b}}_m+\mathbf{z}\right),$$

(9)

and an $L_2$ regularization term is applied to all learnable parameters $\Theta$:

$$\mathcal{R}(\Theta )=\lambda \sum _i{\parallel {\Theta }_i\parallel }_2^2.$$

(10)

3.5. Loss Function

The training objective of ALIMN is to minimize a composite loss function that accounts for classification error, regularization, and auxiliary penalties to ensure robust learning. The overall loss is formulated as:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{BCE}}+{\mathcal{L}}_{\mathrm{reg}}+{\mathcal{L}}_{\mathrm{aux}},$$

(11)

where each term is detailed as follows.

3.5.1. Binary Cross-Entropy Loss

For binary classification, the primary loss function is the binary cross-entropy (BCE) loss, defined over a dataset of N samples:

$${\mathcal{L}}_{\mathrm{BCE}}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_{i}log{\widehat{y}}_i+(1-y_i)log\left(1-{\widehat{y}}_i\right)\right],$$

(12)

where $y_i\in \{0,1\}$ represents the true label and ${\widehat{y}}_i$ is the predicted probability obtained from the ensemble output in (8).

3.5.2. Regularization Loss

To prevent overfitting and promote generalization, an $L_2$ regularization term is added:

$${\mathcal{L}}_{\mathrm{reg}}=\lambda \sum _j{\parallel {\Theta }_j\parallel }_2^2,$$

(13)

where ${\Theta }_j$ denotes the j-th parameter group in the network and $\lambda$ is the regularization coefficient.

3.5.3. Auxiliary Loss

An auxiliary loss is added to improve feature learning in the multi-branch ensemble. It aligns intermediate representations with the final prediction. If ${\mathbf{h}}_m$ is the output from the m-th branch, we define:

$${\mathcal{L}}_{\mathrm{aux}}={\displaystyle \frac{1}{M}}\sum _{m=1}^M{∥{\mathbf{h}}_m-\mathbf{z}∥}_2^2,$$

(14)

which helps each branch stay consistent with the fused representation $\mathbf{z}$ from the Contextual Fusion Module.

3.6. Data Preprocessing

Optimized data preprocessing enhances ALIMN’s performance, involving key steps:

3.6.1. Normalization

Continuous features are standardized to zero mean and unit variance for consistency across different scales.

3.6.2. Categorical Encoding

Categorical variables are transformed using one-hot encoding to facilitate model compatibility.

3.6.3. Feature Augmentation

LLM-based feature extraction enriches input features, generating embeddings that are combined with structured data to form an enhanced feature representation.

3.6.4. Dimensionality Reduction

To mitigate feature redundancy and improve computational efficiency, PCA is applied to project augmented features into a lower-dimensional space while retaining key information.

These preprocessing steps refine data representation, balancing structure and semantics. Figure 4 illustrates PCA results and normalized feature distributions.

3.7. Evaluation Metrics

The ALIMN framework is evaluated using standard metrics.

Accuracy measures overall correctness:

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}}.$$

(15)

Precision quantifies the correctness of positive predictions:

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}.$$

(16)

Recall assesses sensitivity to positive cases:

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}.$$

(17)

F1-score balances precision and recall:

$$\mathrm{F}1-\mathrm{score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}.$$

(18)

ROC AUC evaluates class separability:

$$\mathrm{AUC}={\int }_0^1\mathrm{TPR}\left({\mathrm{FPR}}^{-1}\left(x\right)\right)dx.$$

(19)

4. Experiment Results

We conducted extensive experiments to evaluate the performance of the ALIMN framework and its variants, as well as to compare it with several mainstream models. Table 1 summarizes the performance of these models on a test dataset using the evaluation metrics defined previously. The changes in model training indicators are shown in Figure 5.

Table 2 presents the performance comparison between ALIMN and these mainstream models.

5. Conclusions

In conclusion, the proposed ALIMN framework, through its innovative integration of LLM-based semantic feature extraction, contextual fusion, and a multi-branch ensemble architecture, achieves superior predictive performance compared to conventional deep learning and traditional machine learning models. The ablation studies and comparisons with mainstream models validate its effectiveness and robustness, highlighting its potential as a promising solution for complex predictive tasks in the financial domain.