A Knowledge-Enhanced Multi-Task Learning Model for Domain-Specific Question Answering

1. Introduction

The financial industry generates complex, dynamic data, requiring advanced analysis. As queries grow more intricate, specialized NLP models become essential. General models like BERT and GPT-3 struggle with domain-specific tasks due to a lack of financial knowledge. This limitation affects performance in financial document understanding, question answering, and term extraction, reducing real-world applicability.

Tian et al. [1] showed the importance of domain-specific knowledge in improving NLP models for specific industries. They suggest using models that integrate external knowledge sources, such as financial metrics and industry-specific terminology, to improve task performance. Similarly, Yang et al. [2] introduced FinBERT, a pre-trained model for financial communication, highlighting the need for domain-specific models to better handle financial documents. However, there is still a gap in designing and training models that can handle many tasks in the financial sector. Sun et al. [3] (2018) proposed a relation classification model integrating coarse- and fine-grained networks with SDP-supervised keyword selection and an opposite loss function, achieving state-of-the-art performance on SemEval-2010 Task 8.

We propose FinGLM-6B, a financial QA system using multi-task learning (MTL). It integrates dynamic document attention, query refinement, and knowledge injection. The system directs attention to key financial document sections, refining queries for better accuracy. The knowledge injection layer enhances financial concept comprehension with external data. These improvements strengthen financial QA, document understanding, and term extraction, surpassing traditional NLP models.

2. Related Work

Recent advancements in language understanding, particularly transformer-based models, have significantly improved NLP. BERT [4] pre-trains deep bidirectional transformers, enhancing tasks like question answering and sentiment analysis. It learns context from both directions. XLNet [5] further improves performance using a more general pretraining technique to capture bidirectional context.J. Lu’s [6] study integrates Decision Tree, TF-IDF, and BERTopic to improve chatbot user satisfaction prediction using the Chatbot Arena dataset.

Tian Jin [7] discussed improving retail sales forecasting using an ensemble model enhanced by Particle Swarm Optimization (PSO), combining LightGBM, XGBoost, and Deep Neural Networks. This work highlights the importance of ensemble methods to improve prediction accuracy, which is similar to forecasting and risk prediction in finance. Dai et al. [8] proposed a contrastive learning-based KWS approach that enhances robustness in low-resource settings, aligning with our dynamic document attention for domain-specific QA. Their architecture optimization informs our efficiency improvements.

In financial data understanding, Douaioui et al. [9] reviewed machine learning and deep learning models for demand forecasting in supply chain management, showing how these models can be adapted for financial data tasks, like predicting market trends. Cao et al. [10] and Chen et al. [11] highlighted the importance of using attention mechanisms in specialized tasks, such as action segmentation and leukocyte classification, which can inspire financial models to improve detailed financial analysis.

3. Methodology

This paper presents a multi-task hybrid architecture for financial question answering based on ChatGLM-6B. The proposed model, FinGLM-6B, integrates a dynamic document attention mechanism (DDAM), a query refinement layer (QRL), and a knowledge injection layer (KIL). It is trained using a multi-task learning (MTL) framework, covering financial question answering, document comprehension, and term extraction.

DDAM adjusts attention based on query context. QRL iteratively refines question representation for better accuracy. KIL enhances understanding by incorporating external financial knowledge. Experiments on a large financial dataset show superior performance in handling financial queries. The pipeline is shown in Figure 1.

3.1. Dynamic Document Attention Mechanism

Traditional transformers assign fixed attention weights across all layers, disregarding token relevance to the query. In financial QA, varying importance of financial tables, statements, and footnotes necessitates a dynamic approach. We propose the Dynamic Document Attention Mechanism, which adjusts attention weights based on query content:

$$A_q(w_i,D)={\displaystyle \frac{exp\left(\mathrm{QueryAlignment}(q,w_i)\right)}{{\sum }_{j=1}^{\left|D\right|}exp\left(\mathrm{QueryAlignment}(q,w_j)\right)}}$$

(1)

where query alignment is computed as:

$$\mathrm{QueryAlignment}(q,w_i)=\mathbf{P}{\left(\mathrm{Embedding}\left(q\right)\right)}^T·\mathbf{P}\left(\mathrm{Embedding}\left(w_i\right)\right)$$

(2)

with $\mathbf{P}$ as a learned projection matrix. This mechanism enhances focus on critical document parts, improving accuracy in financial QA.

3.2. Query Refinement Layer

The Query Refinement Layer iteratively refines the query by leveraging document content and dynamic attention. The process follows:

$$q_t=\mathrm{QRL}(q_{t-1},A_q,D)$$

(3)

where $q_t$ is updated using attention scores and document context, ensuring alignment with relevant financial information.

3.3. Knowledge Injection Layer

Financial texts contain specialized terminology often overlooked by standard transformers. The Knowledge Injection Layer (KIL) integrates domain-specific knowledge, represented as $K_{\mathrm{fin}}$, into the decoding process:

$$h_{\mathrm{decoder}}=\mathrm{Decoder}(h_{\mathrm{encoder}},A_q,K_{\mathrm{fin}})$$

(4)

where $h_{\mathrm{encoder}}$ is the encoder state and $A_q$ is the attention matrix. This fusion of external knowledge enhances financial text comprehension.

3.4. Multi-Task Learning Framework

FinGLM-6B adopts a multi-task learning framework. It integrates gated and expert networks to process financial tasks, including question answering (QA), document comprehension (DC), and term extraction (FTE). The framework enables knowledge sharing while preserving task-specific features.

3.4.1. Task-Specific Gated Networks

Each task shares a common backbone architecture but uses a task-specific gating mechanism $g_t$ to control the flow of information. The gated output for task t is computed as:

$${\mathbf{h}}_t^{\mathrm{out}}=g_t\odot {\mathbf{h}}_t+(1-g_t)\odot {\mathbf{h}}_t^{\mathrm{aux}}$$

(5)

where $g_t$ is the gating vector learned through:

$$g_t=\sigma (W_t{\mathbf{h}}_t^{\mathrm{shared}}+b_t)$$

(6)

This ensures the model adapts its parameters for each task while maintaining shared knowledge.

3.4.2. Expert Networks for Task Specialization

Each task has an expert network that generates specialized representations. The output is a weighted sum of expert outputs ${\mathbf{e}}_t^{\left(i\right)}$, as follows:

$${\mathbf{h}}_t^{\exp }=\sum _{i=1}^K{\alpha }_t^{\left(i\right)}{\mathbf{e}}_t^{\left(i\right)}$$

(7)

The weights ${\alpha }_t^{\left(i\right)}$ are dynamically computed via:

$${\alpha }_t^{\left(i\right)}={\displaystyle \frac{exp\left(W_t^{\left(i\right)}{\mathbf{h}}_t^{\mathrm{shared}}\right)}{{\sum }_{i=1}^{K}exp\left(W_t^{\left(i\right)}{\mathbf{h}}_t^{\mathrm{shared}}\right)}}$$

(8)

This allows the model to specialize by focusing on relevant features for each task.

3.4.3. Joint Training of Shared and Task-Specific Modules

The FinGLM-6B model jointly optimizes the shared backbone and task-specific components with task-specific loss functions. For task t, the loss is defined as:

$$L_t={\mathbb{L}}_t({\mathbf{h}}_t^{\mathrm{out}},y_t)$$

(9)

where ${\mathbb{L}}_t$ is the task-specific loss, and $y_t$ is the ground truth for task t.

3.4.4. Task-Aware Attention Mechanism

A task-aware attention mechanism refines the shared representation by attending to both task-specific and query contexts. The attention weights $A_t$ are computed as:

$$A_t=\mathrm{Softmax}\left({\mathbf{Q}}_t{\mathbf{K}}_t^T\right)$$

(10)

This allows the model to focus on the relevant information for each task, enhancing task-specific performance.

3.4.5. Adaptive Task Allocation During Inference

During inference, the model dynamically allocates resources based on the task type. Expert networks and gating mechanisms work together to ensure efficient and accurate task-specific processing.

3.5. Loss Function

The FinGLM-6B model employs a composite loss function to optimize multi-task learning across various financial tasks. The overall loss combines task-specific losses, each targeting a distinct financial task such as question answering, document comprehension, and financial term extraction.

3.5.1. Task-Specific Loss Functions

Each task t has its own loss function, guiding task-specific learning:

- For QA, we use cross-entropy loss:

$$L_{\mathrm{QA}}=-\sum _i{y}_{\mathrm{QA},i}log\left(p_{\mathrm{QA},i}\right)$$

(11)

- For Document Comprehension, a contrastive loss is used:

$$L_{\mathrm{DC}}=\sum _{i}max(0,1-\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_i^{+})+\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_i^{-}))$$

(12)

- For Financial Term Extraction (FTE), a regression loss is applied:

$$L_{\mathrm{FTE}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|y_{\mathrm{FTE},i}-{\widehat{y}}_{\mathrm{FTE},i}\right|$$

(13)

3.5.2. Composite Loss Function

The overall loss is a weighted sum of the individual losses, where ${\lambda }_t$ represents the task-specific weight:

$$L_{\mathrm{total}}=\sum _t{\lambda }_t{L}_t$$

(14)

This allows the model to balance task-specific optimization during training.

3.5.3. Gradient Flow for Multi-Task Optimization

The gradients for shared parameters ${\theta }_s$ and task-specific parameters ${\theta }_t$ are computed based on the total loss:

$${\nabla }_{{\theta }_s}{L}_{\mathrm{total}}=\sum _t{\lambda }_t{\nabla }_{{\theta }_s}{L}_t$$

(15)

$${\nabla }_{{\theta }_t}{L}_{\mathrm{total}}={\lambda }_t{\nabla }_{{\theta }_t}{L}_t$$

(16)

These gradients ensure that both shared and specialized components are updated during backpropagation.

3.6. Data Preprocessing

The preprocessing of FinGLM-6B data involves cleaning, tokenization, and embedding to ensure structured input for training.

3.6.1. Data Cleaning and Normalization

Raw financial text and numerical data undergo cleaning:

• Text Cleaning: Removing headers and disclaimers.
• Numerical Normalization: Applying standardization:

$$x_{\mathrm{norm}}={\displaystyle \frac{x-\mu }{\sigma }}$$

(17)

where x is a value, $\mu$ the mean, and $\sigma$ the standard deviation.

3.6.2. Tokenization and Embedding

Text is tokenized via BPE or WordPiece and mapped to dense vectors:

$$\mathbf{e}\left(t\right)=\mathrm{Embedding}\left(t\right)$$

(18)

Figure 2 illustrates raw and processed data distributions.

3.6.3. Handling Missing Data and Outliers

Strategies include:

• Imputation: Filling missing values with mean or median.
• Outlier Detection: Using z-score:

$$\mathrm{z}-\mathrm{score}\left(x\right)={\displaystyle \frac{x-\mu }{\sigma }}$$

(19)

Extreme values are capped or removed based on thresholds.

4. Evaluation Metrics

To evaluate the performance of the FinGLM-6B model, the following metrics are used:

Accuracy measures the percentage of correct answers in the QA task:

$$A_t={\displaystyle \frac{\mathrm{Correct}\mathrm{Predictions}}{\mathrm{Total}\mathrm{Predictions}}}\times 100$$

(20)

Mean Squared Error (MSE) is used for financial term extraction (FTE) to measure prediction error:

$$\mathrm{MSE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_i-{\widehat{y}}_i\right)}^2$$

(21)

Precision, Recall, and F1-Score evaluate relevance and completeness for document comprehension or term extraction:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(22)

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(23)

$$\mathrm{F}1-\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(24)

Mean Average Precision (MAP) evaluates retrieval performance for document comprehension (DC):

$$\mathrm{MAP}={\displaystyle \frac{1}{Q}}\sum _{q=1}^Q{\displaystyle \frac{1}{R_q}}\sum _{r=1}^{R_q}\mathrm{Precision}\text{@}\mathrm{r}$$

(25)

5. Experiment Results

We compare the performance of FinGLM-6B with other models such as BERT and GPT-3 on QA, FTE, and DC tasks. The results are shown in Table 1:

The following table, Table 2, presents the results of the ablation study, where different components of the model were removed. The Figure 3 shows the training curves for four model variants over 50 epochs. In each subplot, the blue and red curves (QA Accuracy and DC MAP ×100) gradually increase and plateau with small fluctuations, while the green dashed curve (FTE MSE ×1000) decreases toward its optimum.

6. Conclusion

The FinGLM-6B model demonstrates strong performance in financial NLP tasks, surpassing other models like BERT and GPT-3. The ablation study highlights the importance of multi-task learning and modular components, contributing significantly to the model’s superior performance.