FinGPT-Agent: An Advanced Framework for Multimodal Research Report Generation with Task-Adaptive Optimization and Hierarchical Attention

1. Introduction

The growing complexity of financial markets and the variety of financial data, such as numbers, text, and images, create difficulties for automated financial report generation. Current methods often struggle to handle the diverse data types and real-time needs of financial tasks. Large Language Models (LLMs) have shown promise in natural language processing, but they often lack flexibility and cannot easily combine multiple types of data for specialized tasks.

To solve these issues, we propose **FinGPT-Agent**, a framework that improves financial report generation. The framework uses dynamic multimodal fusion, which combines different data types through specialized encoders and attention-based weighting. Task-specific optimization is achieved using Low-Rank Adaptation (LoRA), which allows efficient fine-tuning for financial tasks.

The framework also uses retrieval-augmented generation with contrastive learning to provide more accurate and relevant content. Reinforcement learning with human feedback (RLHF) further refines the quality of generated reports, balancing informativeness and compliance. To manage long financial documents, a hierarchical attention mechanism enhances summarization capabilities.

FinGPT-Agent offers a complete solution for improving financial report generation, addressing limitations in current methods. This work pushes the application of LLMs in finance forward and provides a benchmark for future research in this field.

2. Related Work

Recent advances in Large Language Models (LLMs) and multimodal data processing have improved their ability to solve complex problems. Many studies focus on methods to enhance model performance and usefulness.

Yin et al. [1] proposed a framework to improve code translation by using corrective techniques in LLMs. This method increases accuracy, especially for complex coding tasks.Jin [2] pioneered PSO-enhanced ensemble learning, influencing FinGPT-Agent’s multimodal fusion and task-adaptive optimization, improving model selection and fine-tuning efficiency.

In recommendation systems, Jiaxin Lu [3] combined LightGBM, DeepFM, and DIN to improve purchase predictions. This approach highlights the benefits of mixing tree-based models with deep learning. Yin et al. [4] also developed a system for generating unit tests, showing how LLMs can assist in software testing. Lu [5] further developed a framework to increase chatbot user satisfaction by combining decision trees and text analysis techniques.

For multimodal integration, Yang [6] proposed a high-capacity data hiding method in binary image mixed regions, reducing visual distortion with efficient encoding and enabling real-time applications. Li [7] used multimodal data and multi-recall strategies to improve product recommendations, showing how combining different data types leads to better results.Sun et al. [8] propose a multi-objective recommender system using ensemble reranking to optimize consumer behaviors, improving recall and recommendation accuracy.

Jin [2] pioneered ATCN with reinforcement learning, enhancing time-series modeling and influencing FinGPT-Agent’s hierarchical attention mechanism. The study also inspired FinGPT-Agent’s task-adaptive optimization, where RLHF refines report generation dynamically. Li and Zhou [9] introduced a dual-agent model to improve reasoning for complex decision-making tasks.

Shen[10] demonstrates that MEC-based computation offloading reduces latency and enhances real-time trading analysis on mobile devices.Xu and Wang [11] propose a multimodal LLMs-based MOE framework that improves healthcare recommendation accuracy and personalization by integrating diverse data sources.

Finally, Li [12] demonstrated how integrating external tools with LLMs improves mathematical problem-solving.Wang et al.[13]introduce LVMTL to enhance RUL prediction by modeling machine dependency and heterogeneity with QOIEM optimization.

These studies show progress in making LLMs better for specific tasks, handling diverse data, and reasoning. However, challenges like scalability, real-time use, and integrating different data types remain. FinGPT-Agent addresses these issues and provides an improved solution for financial tasks.

3. Methodology

The application of large language models (LLMs) in the financial domain faces challenges such as high data heterogeneity, timeliness, and complex task requirements. To address these, we propose a hybrid financial model training framework that integrates multi-modal fusion, dynamic task optimization, and retrieval-enhanced generation. By employing advanced techniques such as transformer architectures, contrastive learning, reinforcement learning with human feedback (RLHF), and multi-agent coordination, our framework demonstrates superior performance in generating comprehensive, timely, and actionable financial research reports. This work contributes to both the theoretical understanding and practical implementation of LLMs in financial analytics. The pipline of model is shown in Figure 1.

3.1. Model Network

Our proposed framework is an end-to-end pipeline that integrates financial data processing, dynamic modeling, and multi-task learning to generate high-quality financial research reports. The network’s core components and their interdependencies are described as follows:

3.2. Dynamic Multi-Modal Fusion Layer

Financial data spans multiple modalities, including numerical time-series data, textual news reports, and visual charts. To handle this heterogeneity, we introduce a dynamic multi-modal fusion layer that learns modality-specific embeddings and combines them into a unified representation:

$$\mathbf{z}=\mathrm{Concat}({\mathbf{z}}_{\mathrm{num}},{\mathbf{z}}_{\mathrm{text}},{\mathbf{z}}_{\mathrm{img}}),$$

(1)

where ${\mathbf{z}}_{\mathrm{num}},{\mathbf{z}}_{\mathrm{text}},{\mathbf{z}}_{\mathrm{img}}$ are the modality-specific embeddings for numerical, textual, and image data, respectively.

Each modality is processed using dedicated encoders:

$$\begin{array}{cc}\hfill {\mathbf{z}}_{\mathrm{num}}& =\mathrm{MLP}\left({\mathbf{x}}_{\mathrm{num}}\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\mathbf{z}}_{\mathrm{text}}& =\mathrm{Transformer}\left({\mathbf{x}}_{\mathrm{text}}\right),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathbf{z}}_{\mathrm{img}}& =\mathrm{CNN}\left({\mathbf{x}}_{\mathrm{img}}\right),\hfill \end{array}$$

(4)

where MLP denotes a multi-layer perceptron, Transformer represents a self-attention-based model, and CNN refers to a convolutional neural network. The concatenated embedding $\mathbf{z}$ is further passed through a gated attention mechanism:

$${\mathbf{z}}_{\mathrm{fusion}}=\mathrm{Attention}(\mathbf{z},\mathbf{w}),$$

(5)

where $\mathbf{w}$ are learnable weights that dynamically adjust the importance of each modality.

3.3. Task-Specific Adaptation via LoRA

To enable efficient domain adaptation, we employ Low-Rank Adaptation (LoRA) for fine-tuning large models on financial tasks. The weights of the pre-trained transformer layers are decomposed as:

$$\mathbf{W}={\mathbf{W}}_0+\Delta \mathbf{W},\Delta \mathbf{W}=\mathbf{A}\mathbf{B},$$

(6)

where ${\mathbf{W}}_0$ are the frozen pre-trained weights, and $\Delta \mathbf{W}$ is the task-specific adaptation, parameterized by matrices $\mathbf{A}\in {\mathbb{R}}^{d\times r}$ and $\mathbf{B}\in {\mathbb{R}}^{r\times d}$, with $r\ll d$ to reduce computational overhead.

3.4. Contrastive Retrieval-Enhanced Generation

To enhance the generation of contextually relevant content, we integrate contrastive retrieval into the training process. Documents are encoded into embeddings using a shared encoder $\varphi$, and the similarity score is computed as:

$$\mathrm{sim}({\mathbf{d}}_i,\mathbf{q})={\displaystyle \frac{\varphi \left({\mathbf{d}}_i\right)·\varphi \left(\mathbf{q}\right)}{\parallel \varphi \left({\mathbf{d}}_i\right)\parallel \parallel \varphi \left(\mathbf{q}\right)\parallel }},$$

(7)

where ${\mathbf{d}}_i$ is a document embedding and $\mathbf{q}$ is the query embedding. A contrastive loss is employed to maximize the similarity of relevant pairs and minimize that of irrelevant pairs:

$${\mathcal{L}}_{\mathrm{contrastive}}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log{\displaystyle \frac{exp\left(\mathrm{sim}({\mathbf{d}}_i^{+},{\mathbf{q}}_i)\right)}{{\sum }_{j=1}^{M}exp\left(\mathrm{sim}({\mathbf{d}}_j,{\mathbf{q}}_i)\right)}},$$

(8)

where ${\mathbf{d}}_i^{+}$ is a relevant document, and M is the number of candidate documents.

3.5. Reinforcement Learning with Human Feedback (RLHF)

To align model outputs with user preferences, we incorporate RLHF. The reward function $\mathcal{R}$ is designed to evaluate the quality of generated financial reports based on metrics such as informativeness, conciseness, and compliance:

$$\mathcal{R}(\mathbf{y},{\mathbf{y}}^{*})={\lambda }_1{\mathrm{Score}}_{\mathrm{info}}+{\lambda }_2{\mathrm{Score}}_{\mathrm{conc}}+{\lambda }_3{\mathrm{Score}}_{\mathrm{comp}},$$

(9)

where $\mathbf{y}$ is the generated report, ${\mathbf{y}}^{*}$ is the reference report, and ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are weighting factors. The policy ${\pi }_{\theta }$ is updated via proximal policy optimization (PPO):

$${\mathcal{L}}_{\mathrm{PPO}}={\mathbb{E}}_t\left[min\left(r_t\left(\theta \right)A_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ)A_t\right)\right],$$

(10)

where $r_t\left(\theta \right)$ is the ratio of the new and old policies, $A_t$ is the advantage, and $ϵ$ is the clipping parameter.

3.6. Hierarchical Attention Mechanism for Long-Context Modeling

To handle long-text summarization in financial reports, we introduce a hierarchical attention mechanism. First, document-level attention aggregates contextual embeddings:

$${\mathbf{h}}_i=\mathrm{Attention}({\mathbf{h}}_{i-1},{\mathbf{c}}_i),$$

(11)

where ${\mathbf{c}}_i$ is the context embedding of the ith segment. Sentence-level attention is then applied:

$${\mathbf{s}}_j=\mathrm{Attention}({\mathbf{s}}_{j-1},{\mathbf{w}}_j),$$

(12)

where ${\mathbf{w}}_j$ represents word embeddings within a sentence. The final representation integrates both levels:

$$\mathbf{r}=\mathrm{Concat}({\mathbf{h}}_i,{\mathbf{s}}_j).$$

(13)

This architecture ensures efficient summarization of long financial documents. The overall structure can be seen in Figure 2.

4. Loss Function

The training of the proposed framework incorporates a composite loss function designed to optimize both retrieval and generation tasks while ensuring alignment with user preferences. The overall loss is a weighted combination of task-specific objectives:

$$\mathcal{L}=\alpha {\mathcal{L}}_{\mathrm{contrastive}}+\beta {\mathcal{L}}_{\mathrm{PPO}}+\gamma {\mathcal{L}}_{\mathrm{summarization}},$$

(14)

where $\alpha$, $\beta$, and $\gamma$ are hyperparameters controlling the contribution of each component.

4.1. Contrastive Loss for Retrieval

$${\mathcal{L}}_{\mathrm{contrastive}}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log{\displaystyle \frac{exp\left(\mathrm{sim}({\mathbf{d}}_i^{+},{\mathbf{q}}_i)\right)}{{\sum }_{j=1}^{M}exp\left(\mathrm{sim}({\mathbf{d}}_j,{\mathbf{q}}_i)\right)}}.$$

(15)

4.2. PPO Loss for RLHF

$${\mathcal{L}}_{\mathrm{PPO}}={\mathbb{E}}_t\left[min\left(r_t\left(\theta \right)A_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ)A_t\right)\right].$$

(16)

5. Data Preprocessing

The preprocessing pipeline transforms raw multi-modal financial data into structured, model-compatible formats, ensuring data quality and consistency.

5.1. Text Data

Textual data from financial reports, news articles, and transcripts are tokenized using a domain-specific tokenizer fine-tuned on financial vocabulary. Stopwords are removed, and embeddings are generated using a pre-trained financial language model:

$${\mathbf{x}}_{\mathrm{text}}=\mathrm{Embed}\left(\mathrm{Tokenizer}\left(\mathrm{Text}\right)\right).$$

(17)

5.2. Numerical Data

Numerical time-series data are scaled using min-max normalization and augmented with technical indicators (e.g., moving averages):

$$x_t^{\prime }={\displaystyle \frac{x_t-min\left(x\right)}{max\left(x\right)-min\left(x\right)}}.$$

(18)

5.3. Image Data

Charts and visual data are resized and converted into feature maps using a convolutional neural network (CNN) encoder:

$${\mathbf{x}}_{\mathrm{img}}=\mathrm{CNN}\left(\mathrm{Resize}\left(\mathrm{Image}\right)\right).$$

(19)

5.4. Data Augmentation

Random masking and perturbation are applied to textual and numerical data to enhance robustness.

6. Evaluation Metrics

The evaluation of the proposed framework involves four metrics to comprehensively measure retrieval, generation, and prediction performance:

6.1. Normalized Discounted Cumulative Gain (nDCG

$$\mathrm{nDCG}={\displaystyle \frac{{\mathrm{DCG}}_p}{{\mathrm{IDCG}}_p}},{\mathrm{DCG}}_p=\sum _{i=1}^p{\displaystyle \frac{2^{r_i}-1}{{log}_2(i+1)}},$$

(20)

where $r_i$ is the relevance score, and p is the cutoff rank.

6.2. BLEU Score for Text Generation

$$\mathrm{BLEU}=exp\left(\sum _{n=1}^N{w}_{n}log{P}_n\right),$$

(21)

where $P_n$ is the precision of n-grams.

6.3. Mean Squared Error (MSE) for Predictions

$$\mathrm{MSE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{({\widehat{y}}_i-y_i)}^2.$$

(22)

6.4. Human Evaluation Score (HES)

A weighted average of informativeness, conciseness, and relevance rated by domain experts.

7. Experiment Results

We conducted extensive experiments to evaluate the performance of the proposed framework, referred to as FinGPT-Agent. Ablation studies were performed by systematically removing components. The changes in model training indicators are shown in Figure 3.

The results are shown in Table 1.

The full model achieves superior performance across all metrics, demonstrating the effectiveness of LoRA, RLHF, and multi-modal fusion. The ablation study highlights the significant impact of each component.

8. Conclusion

The proposed FinGPT-Agent framework successfully integrates advanced techniques such as LoRA, RLHF, and multi-modal fusion to address challenges in financial data modeling. Through comprehensive evaluation and ablation studies, the framework demonstrates robust performance, advancing the state-of-the-art in financial analytics and report generation.