Bridging Semantic Disparity and Tail Query Challenges in Advertisement Retrieval via Dual LLM Collaboration

1. Introduction

Advertising retrieval aligns user queries with ads at scale but faces noisy landing pages, code-mixed queries, and extreme click imbalance, which impair semantic matching and long-tail recall.

Prior work improves embeddings, features, or hybrid retrieval, yet colloquial or abbreviated queries and rare ads remain difficult. LLM-based augmentation widens semantic coverage but increases latency and can induce distribution shift at scale.

Practical systems require three properties jointly: strong semantic generalization, reliable long-tail coverage, and low inference latency. Single-model pipelines rarely achieve all three.

We present BRIDGE, a dual-LLM orchestration: (1) a generative LLM (Qwen-7B) synthesizes semantically rich queries; (2) a lightweight LLM (Gemma2-2B) enables high-throughput keyword extraction and coarse retrieval; and (3) a gating module fuses both via cross-attention knowledge transfer into an enhanced Unimo-text-large encoder. An interleaved multi-task schedule and hierarchical negative sampling target the long tail while limiting catastrophic forgetting. On an industrial-scale ad dataset, BRIDGE increases Recall@10 and long-tail coverage with sub-15 ms latency.

Section II reviews related work; Section III details architecture and training; Section IV describes data and setup; Section V reports results and ablations; Section VI concludes.

2. Related Work

We group prior work into (A) parameter-efficient tuning and LLM adaptations, (B) query understanding and augmentation, and (C) long-tail sampling, to situate BRIDGE.

(A) Parameter-efficient tuning and LLM adaptations. Adapters such as LoRA enable low-cost, stable specialization. Zhu et al. [1] apply LoRA-style tuning to named entities. We similarly fine-tune Gemma2-2B for keywording and distill Qwen-7B into the retrieval encoder.Luo et al. [2] used graph structures with language models to help reasoning. Guan [3] used simple models like logistic regression in health tasks. These are easy to use, but they do not work well in open tasks.

(B) Query understanding and augmentation. LLMs enrich queries with intent and semantic variants. [4] Related work also studies augmentation with lightweight models. [5] We couple generative expansion with a compact keyword model and fuse both signals during retrieval.Liu [6] introduces a hybrid LLM with Transformer-XL and GAT that combines prompt-based semantic search, session modeling, and cross-attention re-ranking to address cold-start sparsity. This motivates our dual-LLM, two-stage retrieval with cross-attention fusion, improving long-tail matching and re-ranking robustness.Huang [7] present HELM-ECS, an LLM-guided multimodal fusion with anomaly-aware decomposition and hierarchical XGBoost; this informs confidence-weighted Qwen/Gemma gating to improve long-tail robustness.

(C) Long-tail and sampling strategies. Tail performance improves with mixed-data and multi-feature designs. [8] Unified training further helps. [9] Schedules mixing hard and easy negatives aid robustness.Guo [10] proposes MHST-GB, fusing neural encoders with LightGBM using correlation-guided attention and feature-importance feedback. This motivates feature-importance guided gating and discrete metadata injection in BRIDGE to improve long-tail calibration. [11] We extend these with hierarchical negative sampling and an interleaved schedule that mitigates catastrophic forgetting across augmented sources.Sun [12] present MALLM, a LLaMA-2-70B multi-agent framework with domain-adaptive pretraining, RAG, and cross-modal fusion for low-resource concept extraction. Its agent specialization and RAG-guided standardization inform BRIDGE to improve coarse retrieval and long-tail robustness.

Dense passage retrieval and ColBERT-style systems serve as embedding and indexing baselines. Transformer fusion for multimodal tasks informs our bidirectional LLM transfer, which adds explicit gating and temperature-scaled attention to handle model-scale mismatch. [13]

3. Methodology

Large-scale advertisement retrieval systems face fundamental challenges in bridging the semantic gap between user queries and advertising content, particularly when dealing with noisy OCR-processed landing pages and heterogeneous data distributions. This paper presents BRIDGE, a neural framework that revolutionizes advertisement retrieval through strategic orchestration of multiple large language models. Our approach leverages Qwen-7B’s superior Chinese language understanding capabilities for semantic query synthesis while employing Gemma2-2b’s computational efficiency for rapid keyword extraction, creating a complementary dual-model augmentation pipeline. The framework addresses the critical issue of catastrophic forgetting in sequential training paradigms by introducing an interleaved multi-task training strategy that dynamically alternates between heterogeneous data sources within each training iteration. We develop adaptive batch interleaving with momentum-based weight scheduling to mitigate distribution shift issues that arise from dataset concatenation. The system employs Unimo-text-large as the backbone encoder, enhanced with cross-attention knowledge transfer mechanisms that enable bidirectional information flow between generative and retrieval components. A key technical innovation involves temperature-scaled attention weights that prevent gradient explosion during early training stages, addressing challenges when combining representations from models with vastly different parameter scales. Our hierarchical negative sampling strategy progressively increases training difficulty, starting with random negatives and gradually introducing semantically similar hard negatives based on cosine similarity in the embedding space. This architecture enables effective retrieval performance across diverse query types while maintaining computational efficiency through an optimized two-stage retrieval pipeline.

4. Algorithm and Model

The development of BRIDGE emerged from systematic analysis of failure modes in traditional advertisement retrieval systems. During preliminary experiments, we observed that state-of-the-art dense retrieval models consistently underperformed on queries containing colloquial expressions or abbreviated product names, a phenomenon particularly pronounced in Chinese e-commerce contexts where users frequently employ creative terminology. This observation motivated our exploration of large language models’ potential to bridge this semantic gap through sophisticated data augmentation and knowledge transfer mechanisms. Figure 1 illustrates the comprehensive BRIDGE architecture, showcasing the orchestration of multiple large language models for advertisement retrieval.

4.1. BRIDGE Architecture Overview

The BRIDGE framework represents a carefully orchestrated system of three interconnected modules, each addressing specific challenges we encountered during development. Initially, we attempted a straightforward approach using a single large model for all augmentation tasks, but quickly discovered that the computational overhead made real-time deployment infeasible. This led us to develop our dual-model architecture, where Qwen-7B handles semantically complex transformations while Gemma2-2b efficiently processes high-volume keyword extraction tasks.

4.1.1. Dual-Model Augmentation Pipeline

Our augmentation pipeline emerged from the recognition that different aspects of query understanding require distinct computational approaches. The Qwen-7B model, with its 7 billion parameters and extensive pre-training on Chinese corpora, excels at generating semantically coherent query variations. However, we discovered that directly applying Qwen-7B to all documents resulted in prohibitive latency and occasional hallucinations when processing poorly formatted OCR text. The query generation process follows:

$$Q_{syn}^{Qwen}=arg\underset{q}{max}P\left(q\right|D,{\theta }_{Qwen})·{\omega }_{semantic}\left(D\right)$$

(1)

where the semantic complexity weight ${\omega }_{semantic}\left(D\right)$ helps prioritize documents requiring sophisticated understanding:

$${\omega }_{semantic}\left(D\right)={\displaystyle \frac{1}{\left|D\right|}}\sum _{i=1}^{\left|D\right|}\mathrm{PPL}\left(d_i\right)·log(1+\mathrm{unique}\left(d_i\right))$$

(2)

A crucial trick we discovered involves temperature annealing during query generation. Starting with high temperature ($T=1.2$) encourages diversity in early training, gradually decreasing to $T=0.7$ for more focused generation. This prevents the model from generating overly generic queries that fail to capture specific product attributes.

Gemma2-2b, despite its smaller size, proved remarkably effective for keyword extraction tasks. We found that fine-tuning Gemma2-2b specifically on e-commerce terminology significantly improved its ability to identify relevant core terms:

$$K_{core}^{Gemma}=\mathrm{TopK}\left(\sum _{t\in T}P\left(t\right|C,{\theta }_{Gemma})·\mathrm{IDF}\left(t\right)\right)$$

(3)

An unexpected challenge arose when combining outputs from both models. Initial attempts at simple concatenation led to feature interference, where the model overfitted to stylistic differences between Qwen and Gemma outputs rather than learning semantic patterns. This motivated our gating mechanism:

$${\mathcal{D}}_{aug}=\alpha \left(t\right)·Q_{syn}^{Qwen}\oplus (1-\alpha \left(t\right))·K_{core}^{Gemma}$$

(4)

where $\alpha \left(t\right)=\sigma (W_g·[{\mathbf{h}}_{Qwen};{\mathbf{h}}_{Gemma}]+b_g)$ dynamically balances contributions. The key insight was initializing $W_g$ with small random values (std=0.02) to ensure stable early training.

4.1.2. Cross-Attention Knowledge Transfer

Cross-attention injects complementary LLM signals into the encoder with orthogonal projections and temperature scaling to avoid head collapse and scale mismatch:

$${\mathbf{H}}_{transfer}=\mathrm{MHA}({\mathbf{Q}}_U,[{\mathbf{K}}_Q;{\mathbf{K}}_G],[{\mathbf{V}}_Q;{\mathbf{V}}_G])$$

(5)

$${\mathrm{Attn}}_{model}=\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{Q}}_{Unimo}{\mathbf{W}}_Q^{model}{\left({\mathbf{K}}_{model}{\mathbf{W}}_K^{model}\right)}^T}{\sqrt{d_k}·{\tau }_{model}}}\right)$$

(6)

${\tau }_{Qwen}=1.5$ and ${\tau }_{Gemma}=0.8$ prevent dominance by larger Qwen representations.

4.2. Interleaved Multi-Task Training Strategy

Sequential training caused 20–30% forgetting due to late-stage gradient magnitude. We use interleaved scheduling with warm-up and performance-aware weights (Figure 2).

4.2.1. Dynamic Batch Interleaving

Batches alternate across sources with warm-up probability $p_t$ ($T_{warm}=5000$) to damp early oscillations:

$${\mathcal{B}}_t=\left\{\begin{array}{cc}{\mathcal{D}}_{orig}\hfill & \mathrm{if}tmod3=0\hfill \\ {\mathcal{D}}_{Qwen}\hfill & \mathrm{if}tmod3=1\hfill \\ {\mathcal{D}}_{Gemma}\hfill & \mathrm{if}tmod3=2\hfill \end{array}\right.·p_t$$

(7)

$$p_t=min(1,\sqrt{t/T_{warm}})$$

(8)

Adaptive loss weighting aligns with EMA-smoothed performance (decay 0.99):

$${\mathcal{L}}_{inter}=\sum _{s\in \{orig,Qwen,Gemma\}}{\lambda }_s^{\left(t\right)}{\mathcal{L}}_{task}^{\left(s\right)}+\beta {\mathcal{L}}_{reg}$$

(9)

$${\lambda }_s^{\left(t\right)}={\displaystyle \frac{exp({\gamma }_s·{\mathrm{perf}}_s^{(t-1)})}{{\sum }_{s^{\prime }}exp({\gamma }_{s^{\prime }}·{\mathrm{perf}}_{s^{\prime }}^{(t-1)})}}$$

(10)

4.3. Enhanced Unimo-text-large Encoder

Unimo-text-large offers bilingual robustness but needs fusion upgrades for heterogeneous signals. Figure 3 shows temperature-scaled cross-attention.

4.3.1. Adaptive Layer-wise Attention Fusion

We aggregate multi-layer features via query-conditioned mixing with stabilized meta-networks:

$${\mathbf{z}}_{adaptive}=\sum _{l=1}^L{\alpha }_l\left(q\right)·{\mathbf{H}}^{\left(l\right)}+{\beta }_l\left(q\right)·\mathrm{FFN}\left({\mathbf{H}}^{\left(l\right)}\right)$$

(11)

$$[{\alpha }_l\left(q\right),{\beta }_l\left(q\right)]=\mathrm{LN}(\mathrm{MLP}\left([\mathrm{len}\left(q\right),\mathrm{entropy}\left(q\right),\mathrm{rare}\left(q\right)]\right)+{\mathbf{r}}_l)$$

(12)

Residual biases ${\mathbf{r}}_l$ initialize toward mid layers (12–16), which carry the most transferable cues.

4.3.2. Hierarchical Negative Sampling

A three-tier sampler transitions from easy to hard negatives to avoid abrupt difficulty shifts:

$${\mathcal{N}}_{hier}={\mathcal{N}}_{global}\cup {\mathcal{N}}_{local}\cup {\mathcal{N}}_{hard}$$

(13)

$$P\left(\mathcal{N}\right)=\mathrm{softmax}\left([w_{global},w_{local}·e^{t/T},w_{hard}·(1-e^{-t/T})]\right)$$

(14)

$T=10000$ yields smooth progression.

4.4. Query-Aware Document Chunking

OCR noise fragments landing pages; fixed windows degrade coherence. We select chunks maximizing coherence and informativeness with smoothed embeddings:

$$C_i=arg\underset{c\in \mathcal{C}}{max}\mathrm{Coherence}\left(c\right)·\mathrm{Informativeness}\left(c\right)$$

(15)

$$\mathrm{Coherence}\left(c\right)={\displaystyle \frac{1}{\left|c\right|-1}}\sum _{j=1}^{\left|c\right|-1}cos({\mathbf{e}}_j,{\mathbf{e}}_{j+1})$$

(16)

$$\mathrm{Informativeness}\left(c\right)=H\left(c\right)-\lambda ·\mathrm{Redundancy}(c,\mathcal{C}\setminus \{c\left\}\right)$$

(17)

Exponential smoothing factor $0.8$ mitigates OCR errors; $\lambda =0.3$ trades diversity and relevance.

4.5. Optimization and Inference Acceleration

The final objective integrates interleaved training, distillation, and regularization:

$${\mathcal{L}}_{total}={\mathcal{L}}_{inter}+{\mu }_1{\mathcal{L}}_{KL}^{Qwen}+{\mu }_2{\mathcal{L}}_{KL}^{Gemma}+{\mu }_3{\left|\right|\theta \left|\right|}_2$$

(18)

We set ${\mu }_1=0.3$, ${\mu }_2=0.2$, and ${\mu }_3=1e-5$ to prevent surface imitation and overfitting. Inference uses a two-stage pipeline: Gemma2-2b with HNSW prunes 98% of candidates in $<3$ ms; full BRIDGE rescoring follows. Caching frequent Unimo representations reduces latency by 35%, and dynamic padding sustains sub-15 ms end-to-end for long queries. Figure 4 shows the pipeline.

4.6. Data Preprocessing

We address OCR noise on landing pages and extreme click imbalance.

4.6.1. Noise-Aware OCR Text Cleaning

31% of pages contain OCR artifacts. We apply confidence-weighted cleaning with domain vocabulary and Qwen-7B reconstruction for uncertain spans:

$$\mathrm{Clean}\left(t\right)=\left\{\begin{array}{cc}\mathrm{Correct}\left(t\right)\hfill & \mathrm{if}\mathrm{Conf}\left(t\right)>{\theta }_{high}\hfill \\ \mathrm{Fuzzy}(t,{\mathcal{D}}_{dict})\hfill & \mathrm{if}{\theta }_{low}<\mathrm{Conf}\left(t\right)\le {\theta }_{high}\hfill \\ ⌀\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(19)

$$\widehat{s}=arg\underset{s}{max}P\left(s\right|s_{pre},s_{post},{\theta }_{Qwen})·\mathrm{Sim}(s,s_{orig})$$

(20)

This recovers 67% of segments otherwise dropped and improves downstream generation on noisy OCR text.

4.6.2. Adaptive Click-Based Sampling Strategy

Clicks are skewed: 73% of ads have fewer than 5 clicks. We compute importance weights that downweight frequent ads while incorporating quality, then allocate augmentation:

$$w_i=\alpha ·log\left(1+{\displaystyle \frac{\mathrm{median}\left(C\right)}{c_i+ϵ}}\right)+(1-\alpha )·\mathrm{Quality}\left(a_i\right)$$

(21)

$$N_{aug}\left(a_i\right)=min\left(\lceil w_i·N_{base}\rceil ,N_{max}\right)$$

(22)

Coverage of tail ads increases from 34% to 89%, widening exposure without amplifying head dominance.

4.7. Evaluation Metrics

We report four metrics for retrieval effectiveness and long-tail robustness.

4.7.1. Recall@K

Fraction of relevant ads in the top-K:

$$\mathrm{Recall}\text{@}\mathrm{K}={\displaystyle \frac{1}{\left|Q\right|}}\sum _{q\in Q}{\displaystyle \frac{|{\mathcal{R}}_q\cap {\mathcal{T}}_q^K|}{|{\mathcal{R}}_q|}}$$

(23)

where Q is the query set, ${\mathcal{R}}_q$ the relevant ads, and ${\mathcal{T}}_q^K$ the top-K list.

4.7.2. Mean Reciprocal Rank (MRR)

Quality by first relevant position:

$$\mathrm{MRR}={\displaystyle \frac{1}{\left|Q\right|}}\sum _{q\in Q}{\displaystyle \frac{1}{{\mathrm{rank}}_q}}$$

(24)

with ${\mathrm{rank}}_q$ the rank of the first relevant ad.

4.7.3. Normalized Discounted Cumulative Gain (NDCG)

Graded relevance with position bias:

$$\mathrm{NDCG}\text{@}\mathrm{K}={\displaystyle \frac{1}{\left|Q\right|}}\sum _{q\in Q}{\displaystyle \frac{{\mathrm{DCG}}_q@K}{{\mathrm{IDCG}}_q@K}}$$

(25)

$${\mathrm{DCG}}_q@K=\sum _{i=1}^K{\displaystyle \frac{2^{re{l}_i}-1}{{log}_2(i+1)}}$$

(26)

where $re{l}_i$ is the relevance grade at position i.

4.7.4. Coverage-Weighted Precision (CWP)

Long-tail sensitivity via frequency-aware weights:

$$\mathrm{CWP}\text{@}\mathrm{K}=\sum _{g\in \mathcal{G}}{\beta }_g·{\mathrm{Precision}}_g@K$$

(27)

with $\mathcal{G}=\left\{\mathrm{rare},\mathrm{medium},\mathrm{popular}\right\}$ and ${\beta }_g$ inversely proportional to group frequency.

5. Experimental Results

Evaluated on 2.3M query-ad pairs with 580K unique ads (80/10/10 split).Table 1 reports overall effectiveness and ablations, while Table 2 details the impact of training schedules and frequency buckets.

Key Findings: BRIDGE achieves 72.8% R@10 (+8.1% over RocketQAv2) with exceptional long-tail performance (CWP@10: 64.7%). Interleaved training (-5.4% when removed) and Qwen-7B augmentation (-4.7%) prove most critical. Rare queries show 25.1% improvement, validating our long-tail optimization.

6. Conclusions

This paper presented BRIDGE, a sophisticated neural framework for advertisement retrieval that effectively leverages multiple large language models to address fundamental challenges in query-advertisement matching. Through strategic integration of Qwen-7B’s semantic understanding and Gemma2-2b’s efficient processing, combined with our innovative interleaved multi-task training strategy, BRIDGE achieves 72.8% Recall@10 on the Baidu advertising dataset, representing a 21.7% relative improvement over competitive baselines. The framework’s exceptional performance on long-tail queries (69.3% recall) while maintaining sub-15ms inference latency demonstrates its practical viability for large-scale deployment. Future work will explore cross-lingual transfer and investigate adaptive model selection based on query characteristics.