Structure-Aware Multi-Stage Adaptation of Qwen-72B for Code Generation

1. Introduction

Code generation is an important application of large language models (LLMs). It can automate software development tasks and speed up programming workflows. Ultra-large models like Qwen-72B have shown success, but problems remain in producing consistent code quality. Full-parameter fine-tuning is effective but costly. Lightweight methods reduce cost but fail to keep deep semantics and structural integrity needed for executable code. Instruction alignment through supervised learning or reinforcement learning from human feedback improves task conformity, but it does not use execution semantics and structural fidelity. Current methods also depend too much on token-level accuracy, which ignores deeper program dependencies, and they provide little control over style and redundancy. These problems show the need for a framework that joins semantic, syntactic, and structural optimization with controllable generation and adaptive context use. CodeFusion-Qwen72B is built to solve these problems. It uses progressive adaptation, hybrid optimization, multimodal integration, and structure-preserving objectives. This work takes a technical approach to make Qwen-72B more adaptable and effective in code generation, aiming for outputs that are functionally correct and structurally coherent.

2. Related Work

Recent work in code generation stresses structural modeling and efficiency. Tipirneni et al. [1] proposed StructCoder, a Transformer that uses AST and data-flow information to improve structural consistency. Sirbu and Czibula [2] applied syntax-based encoding to malware detection tasks. Gong et al. [3] presented AST-T5, a structure-aware pretraining model that uses syntactic segmentation to enhance code generation and understanding.

Work on parameter-efficient adaptation has also grown. Wang et al. [4] proposed LoRA-GA with gradient approximation to improve optimization. Hayou et al. [5] introduced LoRA+ with different learning rates for better convergence. Hounie et al. [6] extended this with LoRTA, which uses tensor adaptation. Zhang et al. [7] presented AutoLoRA, which tunes matrix ranks with meta learning. Chen et al. [8] combined structure-aware signals with parameter-efficient tuning to reach results close to full fine-tuning.

Li et al. [9] showed that large code models work well as few-shot information extractors. Guan [10] used logistic regression and decision trees for medical claim denial prediction, proving that interpretable models are still useful in specialized tasks.

3. Methodology

Enhancing the code generation capabilities of ultra-large language models requires not only parameter-efficient fine-tuning but also structure-aware and context-rich optimization. We propose CodeFusion-Qwen72B, a multi-stage fine-tuning framework for the Qwen-72B model that integrates Progressive Low-Rank Adaptation (PLoRA), Hybrid Instruction Optimization (HIO), Multi-Context Fusion (MCF), Structure-Preserving Hybrid Loss (SPHL), Controllable Generation Decoder (CGD), and Adaptive Prompt Retrieval (APR). PLoRA progressively adapts model layers to domain-specific semantics, while HIO combines supervised fine-tuning with reinforcement learning from human feedback to align both functionality and readability. MCF enriches inputs with problem descriptions, code comments, and execution traces, and SPHL enforces token, syntactic, and semantic coherence. CGD and syntax-constrained decoding enable controllable, compilable generation, and APR provides dynamic retrieval of relevant code snippets. We evaluate CodeFusion-Qwen72B on three benchmarks: HumanEval, MBPP, and CodeContests. Compared to a fully fine-tuned Qwen-72B baseline, our framework improves pass@1 accuracy from 53.8% to 65.4% (+11.6), BLEU from 71.2 to 78.9 (+7.7), and execution success rate from 68.3% to 81.5% (+13.2). Ablation studies confirm that MCF and SPHL contribute most to execution gains, while PLoRA and HIO drive significant improvements in syntactic and semantic alignment. These results demonstrate that multi-stage, multi-context, and structure-aware adaptation can substantially advance the state of the art in LLM-based code generation. The pipline of MHST-GB is shown in Figure 1

4. Algorithm and Model

The proposed CodeFusion-Qwen72B is a multi-stage, multi-objective fine-tuning framework for enhancing the code generation capability of ultra-large language models, built upon the Qwen-72B architecture. This section introduces the key components of the framework, their technical motivations, and implementation details.

4.1. Qwen-72B Backbone Architecture

Qwen-72B adopts a Transformer-based autoregressive decoder architecture with L stacked layers, each containing multi-head self-attention (MHSA) and feed-forward networks (FFN). The standard attention mechanism computes:

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}+M\right)V$$

(1)

where M is an attention mask ensuring causal dependency. While this formulation captures long-range dependencies, it does not explicitly enforce programming-specific constraints such as variable scope or syntactic validity, motivating the introduction of structure-aware adaptations.

4.2. Progressive Low-Rank Adaptation (PLoRA)

Full-parameter fine-tuning of Qwen-72B is computationally prohibitive and risks overfitting to domain-specific patterns. We employ Progressive LoRA (PLoRA), a staged adaptation strategy where LoRA updates are first applied to high-level decoder layers to capture semantic patterns, then gradually extended to lower layers to refine syntactic generation. The parameter update is:

$$W^{\prime }=W+BA,B\in {\mathbb{R}}^{d\times r},A\in {\mathbb{R}}^{r\times d},r\ll d$$

(2)

This gradual adaptation mitigates catastrophic forgetting and allows stable convergence. Figure 2 details the Progressive LoRA adaptation mechanism, which strategically fine-tunes the 72-layer transformer architecture in three stages.

4.3. Hybrid Instruction Optimization (HIO)

Code generation quality depends heavily on alignment with user instructions. HIO combines supervised fine-tuning with reinforcement learning from human feedback (RLHF). The RLHF loss is:

$${\mathcal{L}}_{\mathrm{RLHF}}=-{\mathbb{E}}_{y\sim {\pi }_{\theta }}\left[r_{\mathrm{human}}\left(y\right)+\eta ·r_{\mathrm{auto}}\left(y\right)\right]$$

(3)

where $r_{\mathrm{human}}$ measures preference alignment and $r_{\mathrm{auto}}$ scores code functionality and readability. This hybrid approach ensures the model learns both semantic compliance and executable correctness.

4.4. Multi-Context Fusion (MCF)

High-quality code generation requires integrating multiple context sources beyond the textual problem description. MCF fuses three modalities: natural language descriptions, code comments, and execution traces. Each is encoded separately:

$$h_{\mathrm{fused}}={\varphi }_{\mathrm{fusion}}\left(E_{\mathrm{text}}\left(x_{\mathrm{text}}\right),E_{\mathrm{comment}}\left(x_{\mathrm{comment}}\right),E_{\mathrm{trace}}\left(x_{\mathrm{trace}}\right)\right)$$

(4)

The fusion is implemented via gated attention:

$${\varphi }_{\mathrm{fusion}}(h_1,h_2,h_3)=\sum _{m=1}^3\sigma \left(w_m^{\top }{h}_m\right)·h_m$$

(5)

This ensures that execution semantics and natural language semantics are jointly considered during generation.

4.5. Structure-Preserving Hybrid Loss (SPHL)

Token-level accuracy is insufficient for functional code correctness. SPHL integrates token-level cross-entropy, AST-based structure regularization, and semantic similarity:

$${\mathcal{L}}_{\mathrm{SPHL}}={\lambda }_t{\mathcal{L}}_{\mathrm{CE}}+{\lambda }_a{\mathcal{L}}_{\mathrm{AST}}+{\lambda }_s{\mathcal{L}}_{\mathrm{sem}}$$

(6)

where ${\mathcal{L}}_{\mathrm{AST}}$ is computed via tree edit distance between predicted and reference ASTs, and ${\mathcal{L}}_{\mathrm{sem}}$ measures cosine similarity between program dependency graph embeddings to preserve logical flow. Figure 3 demonstrates the Multi-Context Fusion module’s gated attention mechanism for integrating heterogeneous code-related inputs and the Structure-Preserving Hybrid Loss formulation.

4.6. Controllable Generation Decoder (CGD)

The CGD module enables control over the style and verbosity of generated code by injecting a control vector into the decoder:

$$h_t^{\prime }=h_t+W_c{v}_{\mathrm{ctrl}}$$

(7)

This mechanism supports different generation modes, such as minimal implementation or detailed, readable code, while maintaining syntax validity through constrained decoding:

$$P_{\mathrm{masked}}\left(t\right)=\left\{\begin{array}{cc}P\left(t\right)\hfill & \mathrm{if}\mathrm{syntactically}\mathrm{valid}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

4.7. Adaptive Prompt Retrieval (APR)

APR dynamically augments prompts with semantically similar code snippets retrieved from an indexed corpus:

$$s_i={\displaystyle \frac{\langle e_q,e_i\rangle }{\parallel e_q\parallel ·\parallel e_i\parallel }},{\mathrm{Prompt}}_{\mathrm{aug}}=\left\{\mathrm{top}\text{-}k\mathrm{snippets}\mathrm{by}{s}_i\right\}$$

(9)

These retrieved examples provide explicit patterns for the model to adapt, improving zero-shot and few-shot performance.

5. Data Preprocessing

The quality and consistency of training data are critical in adapting Qwen-72B for code generation. Our preprocessing pipeline ensures syntactic correctness, semantic diversity, and domain alignment before fine-tuning.

5.1. Code Deduplication and Normalization

Large-scale code corpora often contain near-duplicate functions, which can cause overfitting and reduce model generalization. We apply a MinHash-based similarity filtering mechanism to remove code pairs with Jaccard similarity above $\tau =0.85$:

$$\mathrm{Sim}(A,B)={\displaystyle \frac{\left|S\right(A)\cap S(B\left)\right|}{\left|S\right(A)\cup S(B\left)\right|}}$$

(10)

where $S(·)$ denotes the set of n-grams extracted from the code snippet. Normalization steps such as consistent indentation, removal of trailing whitespaces, and unification of naming conventions are also applied to reduce variability from irrelevant factors.

5.2. Syntactic Validation and Semantic Tagging

We use language-specific parsers to validate the syntactic correctness of each code snippet. Invalid code is automatically repaired using a lightweight sequence-to-sequence correction model. Additionally, we apply semantic tagging by extracting function names, API calls, and language constructs, and encoding them as auxiliary features $c_{\mathrm{tag}}$:

$$x_{\mathrm{aug}}=[x_{\mathrm{code}};c_{\mathrm{tag}}]$$

(11)

This tagging improves the model’s ability to attend to function-level semantics during fine-tuning.

6. Prompt Engineering Techniques

Effective prompt design significantly impacts the code generation quality of LLMs. In CodeFusion-Qwen72B, we integrate retrieval-augmented prompting and progressive prompt construction to enhance both accuracy and controllability.

6.1. Retrieval-Augmented Prompt Construction

We maintain an indexed database of high-quality code exemplars. Given a new task query q, we compute its embedding $e_q$ and retrieve the top-k similar examples based on cosine similarity:

$$s_i={\displaystyle \frac{\langle e_q,e_i\rangle }{\parallel e_q\parallel ·\parallel e_i\parallel }}$$

(12)

These retrieved examples are concatenated into the prompt before the target instruction. This method injects explicit patterns for the model to imitate, improving few-shot and zero-shot code generation performance.

6.2. Progressive Prompt Structuring

Instead of providing the entire problem description at once, we adopt a progressive reveal strategy, where the prompt is expanded in stages:

• Present a high-level task description.
• Include a partial code skeleton or function signature.
• Gradually add constraints, examples, and edge cases.

Formally, let $p_1,p_2,\dots ,p_m$ denote progressive prompt segments; the model receives:

$$\mathrm{Prompt}=\left[p_1\right]\to [p_1;p_2]\to \dots \to [p_1;p_2;\dots ;p_m]$$

(13)

This technique conditions the model’s attention sequentially, improving coherence and reducing hallucination in generated code.

7. Evaluation Metrics

To comprehensively evaluate the performance of CodeFusion-Qwen72B and baseline models, we adopt six complementary metrics that assess syntactic correctness, semantic fidelity, and functional accuracy.

7.1. Pass@1 Accuracy

Pass@1 measures the percentage of tasks where the first generated solution passes all unit tests:

$$\mathrm{Pass}\text{@}1={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}(\mathrm{Exec}\left({\widehat{y}}_i\right)=\mathrm{Exec}\left(y_i\right))$$

(14)

7.2. BLEU Score

BLEU evaluates n-gram overlap between generated and reference code:

$$\mathrm{BLEU}=\mathrm{BP}·exp\left(\sum _{n=1}^4{w}_{n}log{p}_n\right)$$

(15)

where BP is the brevity penalty and $p_n$ is the n-gram precision.

7.3. Code Execution Success Rate (CESR)

CESR measures the proportion of compilable and runnable code snippets:

$$\mathrm{CESR}={\displaystyle \frac{{\sum }_{i=1}^N\mathbb{I}(\mathrm{Compiles}\left({\widehat{y}}_i\right)=\mathrm{True})}{N}}$$

(16)

7.4. Abstract Syntax Tree Similarity (ASTSim)

ASTSim measures structural similarity using normalized tree edit distance:

$$\mathrm{ASTSim}=1-{\displaystyle \frac{\mathrm{TED}(\mathrm{AST}\left(\widehat{y}\right),\mathrm{AST}\left(y\right))}{max\left(\right|\mathrm{AST}\left(\widehat{y}\right)|,|\mathrm{AST}\left(y\right)\left|\right)}}$$

(17)

7.5. Semantic Similarity (SemSim)

SemSim is computed as cosine similarity between program dependency graph (PDG) embeddings:

$$\mathrm{SemSim}={\displaystyle \frac{\langle e_{\widehat{y}},e_y\rangle }{\parallel e_{\widehat{y}}\parallel ·\parallel e_y\parallel }}$$

(18)

7.6. Code Readability Score (CRS)

CRS measures readability using a learned regression model $f_{\mathrm{read}}$ trained on human-rated code samples:

$$\mathrm{CRS}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{f}_{\mathrm{read}}\left({\widehat{y}}_i\right)$$

(19)

8. Experiment Results

8.1. Experimental Setup

We evaluate CodeFusion-Qwen72B and baselines on three code generation benchmarks: HumanEval, MBPP, and CodeContests. Each model generates solutions with temperature $T=0.2$ and beam size $b=5$. For fairness, all methods use the same training data and tokenization scheme. We compare against Qwen-72B Full FT, Qwen-72B LoRA, and CodeGen-16B. We also conduct ablation experiments by removing key components of CodeFusion-Qwen72B.

8.2. Overall and Ablation Results

Table 1 presents the overall performance and ablation results. The top section compares different models across all datasets, while the lower section shows the impact of removing each module from CodeFusion-Qwen72B. Six metrics are reported: Pass@1, BLEU, CESR, ASTSim, SemSim, and CRS. And the changes in model training indicators are shown in Figure 4.

9. Conclusion

We proposed CodeFusion-Qwen72B, a multi-stage fine-tuning framework for Qwen-72B incorporating progressive LoRA, multi-context fusion, structure-preserving loss, and adaptive prompt retrieval. Experiments on HumanEval, MBPP, and CodeContests demonstrate consistent improvements across six metrics, with Pass@1 gains of over 11% on average compared to full-parameter fine-tuning. The integrated ablation analysis confirms that context fusion and structure-aware objectives are key contributors to performance gains.

The approach has practical constraints. First, progressive adaptation with RLHF requires substantial compute; our experiments consumed thousands of GPU-hours which may not be reproducible for smaller teams. Second, reliance on retrieval corpora and language-specific AST losses can reduce out-of-distribution and cross-language generalization. Third, AST-based structure penalties require robust parsers per language, increasing engineering complexity. Finally, syntax-constrained decoding and control vectors add inference latency that may hinder real-time interactive use.