A Scalable Multi-Agent Framework for Low-Resource E-Commerce Concept Extraction and Standardization

1. Introduction

In e-commerce, understanding things like product features, brand names, and technical details helps users and supports business. Many models find this difficult. They often cannot deal with complex and varied data. This causes problems in tasks like recommendations and search.

MALLM is a new system built on LLaMA-2 70B. It uses multiple agents to handle different parts of the job. Each agent focuses on one task such as extracting entities, finding relations, or analyzing sentiment. The model learns better by using domain-specific data from e-commerce. This makes it understand product language more clearly.

MALLM also adds RAG to bring in useful outside data. It connects text with category structures and knowledge graphs. This improves what the model sees and uses. The model learns step by step and uses other models to guide its training. It works well even with little data. MALLM combines these tools to give a strong and simple way to understand shopping concepts.

This system is built to work in real cases. It can run with low delay and fixed memory. Its design helps when resources are limited. The agent layout also makes the model easier to scale or change. Each part can improve without changing the rest. This helps developers test new ideas or update tasks quickly. The goal is to make the system good at learning, easy to control, and ready to deploy.

2. Related Work

Attribute value extraction and standardization have drawn attention. Loughnane et al.[1] used weak labels and normalization in a two-step method. Ricatte and Crisostomi[2] worked with product graphs to extract and match values. These helped keep data clean and consistent.

Other work used language models made for e-commerce. Çiftlikçi et al.[3] trained a model using Turkish product text. It worked well on clear and hidden values. Gong and Eldardiry[4] made a system that could find values even for unseen labels, useful when little data is available.

Some work tried to use less labeled data. Subhalingam et al.[5] mixed rules and model guesses to learn from partial data. Roy et al.[6] used generation models to predict both attributes and values together. They needed only small amounts of training.

Efficiency was also studied. Vo et al.[7] built a smaller model using knowledge distillation. It learned from a larger model but worked faster. It kept good accuracy while using less memory.

Others looked at ideas outside of shopping. Chen[8] built a 3D vision model that breaks tasks into steps. This may help design multi-agent plans. Guan[9] used basic learning to handle health claims. It showed that even simple tools can help in low-data fields.

Guo and Yu[10] worked on keeping user data safe. Their system adds noise and clipping to protect private info in large models. This idea is useful for shopping platforms that use user data.

3. Methodology

In this section, we present MALLM (Multi-Agent Large Language Model), a sophisticated architecture designed to address the complex challenges of shopping concept understanding in e-commerce environments. Our approach leverages the powerful representation capabilities of LLaMA-2 70B as the foundation model, enhanced through domain-adaptive continued pre-training on a curated corpus of 15 million shopping-related documents. MALLM introduces a hierarchical mixture-of-agents framework where specialized neural agents collaborate through meta-reasoning controllers, enabling effective decomposition of complex shopping tasks into manageable sub-problems. The architecture incorporates retrieval-augmented generation with dynamic candidate set construction, combining dense retrieval using Sentence-BERT embeddings, sparse retrieval through BM25 scoring, and graph-based retrieval via personalized PageRank on shopping knowledge graphs. To address the challenge of domain-specific terminology and short-context understanding, we implement cross-modal fusion mechanisms that integrate textual, categorical, and relational features through graph-enhanced attention layers. Our progressive knowledge distillation pipeline transfers expertise from multiple specialized teacher models, significantly improving few-shot learning capabilities. Experimental results on the ShopBench dataset demonstrate that MALLM achieves state-of-the-art performance with an average improvement of 8.7% across five shopping concept understanding tasks, while requiring 60% fewer training examples to match baseline performance, making it particularly suitable for rapidly evolving e-commerce domains where new products and concepts emerge continuously.

4. Algorithm and Model

4.1. MALLM Architecture Overview

We propose MALLM (Multi-Agent Large Language Model), a hierarchical architecture designed for shopping concept understanding. The e-commerce domain presents unique challenges: domain-specific terminology rarely seen in general corpora, short contexts lacking disambiguation information, and rapidly evolving product categories. MALLM addresses these by combining LLaMA-2 70B’s language understanding with specialized mechanisms for shopping tasks.

Our architecture employs multiple specialized agents coordinated through a meta-reasoning controller. To handle computational overhead, we implemented dynamic batching that adaptively adjusts based on GPU memory, achieving 40% memory reduction while maintaining throughput. The meta-controller arbitrates between agents using confidence scores and task-specific priors, resolving conflicting predictions in ambiguous cases. This figure illustrates the comprehensive MALLM framework architecture designed for shopping concept understanding in e-commerce environments in Figure 1

4.2. Domain Adaptation

Building on LLaMA-2 70B, we identified critical gaps in shopping domain knowledge—the model confused technical specifications and struggled with brand nomenclature. We performed domain-adaptive continued pre-training on 15 million shopping documents using a multi-objective approach:

$${\mathcal{L}}_{\mathrm{DAPT}}={\mathcal{L}}_{\mathrm{MLM}}+{\gamma }_1{\mathcal{L}}_{\mathrm{ECL}}+{\gamma }_2{\mathcal{L}}_{\mathrm{KGC}}$$

(1)

where ${\gamma }_1=0.3$ and ${\gamma }_2=0.2$. The entity contrastive learning loss distinguishes similar shopping concepts:

$${\mathcal{L}}_{\mathrm{ECL}}=-\sum _{i=1}^{N}log{\displaystyle \frac{exp(\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_i^{+})/\tau )}{{\sum }_{j=1}^{K}exp(\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_j^{-})/\tau )}}$$

(2)

with temperature $\tau =0.07$ maintaining discriminative representations. The knowledge graph completion loss incorporates structured shopping taxonomies:

$${\mathcal{L}}_{\mathrm{KGC}}=\sum _{\mathcal{G}}{\mathcal{L}}_{\mathrm{pos}}+\sum _{{\mathcal{G}}^{-}}{\mathcal{L}}_{\mathrm{neg}}$$

(3)

To prevent catastrophic forgetting, we applied elastic weight consolidation using Fisher information matrices computed on general domain tasks.

4.3. Multi-Agent Framework

The framework consists of five specialized agents targeting distinct shopping tasks: normalization, elaboration, extraction, relational reasoning, and sentiment analysis. The multi-agent framework employs a sophisticated meta-reasoning controller to orchestrate five specialized agents, each targeting distinct shopping tasks in Figure 2

Each agent ${\mathcal{A}}_i$ maintains task-specific architectures:

$${\mathcal{A}}_i\left(\mathbf{x}\right)={\mathrm{Decoder}}_i\left(\mathrm{CrossAttn}({\mathbf{H}}_{\mathrm{enc}}^i,{\mathbf{K}}_i)\right)$$

(4)

Agent selection uses a gating mechanism considering input characteristics:

$$\mathbf{g}=\mathrm{softmax}({\mathbf{W}}_g·\mathrm{Pool}\left({\mathbf{H}}_{\mathrm{base}}\right)+{\mathbf{b}}_g)$$

(5)

$${\mathbf{z}}_{\mathrm{agent}}=\sum _{i=1}^M{g}_i·{\mathcal{A}}_i\left(\mathbf{x}\right)$$

(6)

To balance specialization with collaboration, we introduce regularization:

$${\mathcal{L}}_{\mathrm{gate}}=-{\lambda }_{\mathrm{entropy}}\sum _{i=1}^M{g}_{i}log{g}_i+{\lambda }_{\mathrm{sparse}}{\parallel \mathbf{g}\parallel }_1$$

(7)

This ensures agents work together when tasks require multiple capabilities.

4.4. Retrieval-Augmented Generation

Our hybrid retrieval system combines three strategies for comprehensive information access. Dense retrieval uses Sentence-BERT with dual encoders:

$${\mathcal{R}}_{\mathrm{dense}}\left(q\right)=\mathrm{top}-\mathrm{k}\left(\{{\mathbf{d}}_i:cos({\mathbf{E}}_q\left(q\right),{\mathbf{E}}_d\left({\mathbf{d}}_i\right))>\theta \}\right)$$

(8)

FAISS indexing with product quantization reduces latency from 200ms to 15ms. Sparse retrieval employs BM25 with optimized parameters ($k_1=0.9$, $b=0.4$) for short shopping queries:

$$\mathrm{BM}25(q,d)=\sum _{t\in q}\mathrm{IDF}\left(t\right)·{\displaystyle \frac{f(t,d)·(k_1+1)}{f(t,d)+k_1·(1-b+b·\frac{\left|d\right|}{\mathrm{avgdl}})}}$$

(9)

Graph-based retrieval explores entity relationships via personalized PageRank:

$${\mathcal{R}}_{\mathrm{graph}}\left(q\right)=\{{\mathbf{n}}_j:\mathrm{PPR}({\mathbf{n}}_q,{\mathbf{n}}_j)>ϵ\}$$

(10)

Monte Carlo approximation with early stopping enables sub-second response times on our 10M-entity knowledge graph. The retrieval-augmented generation system combines three complementary retrieval strategies to access comprehensive shopping information in Figure 3

4.5. Cross-Modal Fusion

Shopping data spans multiple modalities—text, categories, and attributes. Our fusion network integrates these through unified attention:

$${\mathbf{H}}_{\mathrm{fused}}=\mathrm{MHA}({\mathbf{Q}}_{\mathrm{text}},[{\mathbf{K}}_{\mathrm{cat}};{\mathbf{K}}_{\mathrm{attr}};{\mathbf{K}}_{\mathrm{rel}}],{\mathbf{V}}_{\mathrm{all}})$$

(11)

Category hierarchies are encoded using Graph Convolutional Networks:

$${\mathbf{H}}^{(l+1)}=\sigma \left({\tilde{\mathbf{D}}}^{-{\displaystyle \frac{1}{2}}}\tilde{\mathbf{A}}{\tilde{\mathbf{D}}}^{-{\displaystyle \frac{1}{2}}}{\mathbf{H}}^{\left(l\right)}{\mathbf{W}}^{\left(l\right)}\right)$$

(12)

Residual connections and layer normalization prevent over-smoothing in deeper layers. Attribute embeddings leverage TransE with frequency-based negative sampling:

$${\mathbf{K}}_{\mathrm{attr}}={\mathbf{W}}_{\mathrm{attr}}·\mathrm{TransE}\left({\mathbf{E}}_{\mathrm{attributes}}\right)$$

(13)

4.6. Knowledge Distillation

We distill knowledge from three specialized teachers: BERT for sentiment, T5 for summarization, and GNN for relational reasoning. The distillation objective combines prediction and feature alignment:

$${\mathcal{L}}_{\mathrm{KD}}=\sum _{t=1}^T{\alpha }_t·\mathrm{KL}(P_{\mathrm{student}}\left|\right|P_{\mathrm{teacher}}^t)+\beta ·{\mathcal{L}}_{\mathrm{feature}}$$

(14)

Temperature $T=4$ effectively transfers dark knowledge. Feature alignment uses learnable transformations:

$${\mathcal{L}}_{\mathrm{feature}}=\sum _{l=1}^L{\parallel {\mathbf{F}}_{\mathrm{student}}^{\left(l\right)}-\varphi \left({\mathbf{F}}_{\mathrm{teacher}}^{\left(l\right)}\right)\parallel }_2^2$$

(15)

4.7. Training Strategy

Training follows a staged approach: foundation pre-training (50K steps), individual agent training (20K steps each), and joint fine-tuning (30K steps). The training architecture implements progressive knowledge distillation from specialized teacher models, transferring domain expertise while preventing catastrophic forgetting in Figure 4

The complete objective balances multiple components:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{task}}+{\lambda }_1{\mathcal{L}}_{\mathrm{KD}}+{\lambda }_2{\mathcal{L}}_{\mathrm{contrast}}+{\lambda }_3{\mathcal{L}}_{\mathrm{reg}}$$

(16)

Learning rate follows cosine annealing with warm restarts:

$${\eta }_t={\eta }_{min}+{\displaystyle \frac{1}{2}}({\eta }_{max}-{\eta }_{min})(1+cos\left({\displaystyle \frac{T_{\mathrm{cur}}}{T_i}}\pi \right))$$

(17)

using ${\eta }_{max}=5e^{-5}$, ${\eta }_{min}=1e^{-6}$, with doubling restart periods. Gradient clipping (norm = 1.0) ensures stability across multi-task objectives.

4.8. Prompt Engineering and Instruction Tuning

4.8.1. Task-Specific Prompt Templates

Effective prompt design is crucial for MALLM’s performance across diverse shopping tasks. We developed a hierarchical prompt template system combining task instructions, few-shot examples, and contextual constraints:

$${\mathcal{P}}_{\mathrm{task}}=\mathrm{Concat}[{\mathcal{I}}_{\mathrm{base}},{\mathcal{E}}_{\mathrm{few}-\mathrm{shot}},{\mathcal{C}}_{\mathrm{context}},{\mathcal{Q}}_{\mathrm{query}}]$$

(18)

For concept normalization, incorporating contrastive examples proved essential:

$${\mathcal{E}}_{\mathrm{contrast}}={\{(x_i^{+},y_i^{+}),(x_j^{-},y_j^{-})\}}_{i,j=1}^k$$

(19)

where positive examples $(x_i^{+},y_i^{+})$ demonstrate correct normalizations and negative examples $(x_j^{-},y_j^{-})$ highlight common mistakes, reducing error rates by 23% on ambiguous specifications.

We optimize prompts through gradient-based search in the embedding space:

$${\mathcal{I}}^{*}=arg\underset{\mathcal{I}}{max}{\mathbb{E}}_{(x,y)\sim {\mathcal{D}}_{\mathrm{val}}}[logP\left(y\right|x,\mathcal{I})]$$

(20)

The selection policy uses a contextual bandit framework with temperature ${\tau }_{\mathrm{prompt}}=0.1$:

$$\pi \left(\mathcal{P}\right|s)={\displaystyle \frac{exp(Q(s,\mathcal{P})/{\tau }_{\mathrm{prompt}})}{{\sum }_{{\mathcal{P}}^{\prime }}exp(Q(s,{\mathcal{P}}^{\prime })/{\tau }_{\mathrm{prompt}})}}$$

(21)

4.8.2. Dynamic Few-Shot Example Selection

MALLM implements dynamic example selection using a hybrid similarity metric:

$${\mathrm{sim}}_{\mathrm{hybrid}}(q,e)=\alpha ·{\mathrm{sim}}_{\mathrm{surface}}(q,e)+(1-\alpha )·{\mathrm{sim}}_{\mathrm{semantic}}(q,e)$$

(22)

with $\alpha =0.4$ balancing lexical and semantic matching. The semantic similarity leverages learned embeddings:

$${\mathrm{sim}}_{\mathrm{semantic}}(q,e)=cos({\mathbf{h}}_q,{\mathbf{h}}_e)·exp(-\beta \parallel {\mathbf{h}}_q-{\mathbf{h}}_e{\parallel }_2)$$

(23)

Example augmentation through paraphrasing and entity substitution increases robustness by 15%:

$${\mathcal{E}}_{\mathrm{aug}}=\{T_{\mathrm{para}}\left(e\right)\cup T_{\mathrm{sub}}\left(e\right):e\in {\mathcal{E}}_{\mathrm{original}}\}$$

(24)

5. Evaluation Metrics

We employ comprehensive metrics for evaluation. For classification tasks, we use weighted F1-score:

$${\mathrm{F}1}_{\mathrm{weighted}}=\sum _{c=1}^C{\displaystyle \frac{n_c}{N}}·{\mathrm{F}1}_c$$

(25)

For generation tasks, we combine ROUGE-L and BERTScore:

$$\mathrm{ROUGE}-\mathrm{L}={\displaystyle \frac{(1+{\beta }^2)·R_{\mathrm{lcs}}·P_{\mathrm{lcs}}}{R_{\mathrm{lcs}}+{\beta }^2·P_{\mathrm{lcs}}}}$$

(26)

$$\mathrm{BERTScore}={\displaystyle \frac{1}{\left|X\right|}}\sum _{x_i\in X}\underset{y_j\in Y}{max}cos({\mathbf{h}}_{x_i},{\mathbf{h}}_{y_j})$$

(27)

We introduce Entity Consistency Score (ECS) for semantic coherence:

$$\mathrm{ECS}={\displaystyle \frac{|{\mathcal{E}}_{\mathrm{pred}}\cap {\mathcal{E}}_{\mathrm{ref}}|}{|{\mathcal{E}}_{\mathrm{ref}}|}}·exp(-\gamma ·d_{\mathrm{attr}})$$

(28)

Efficiency is measured through throughput with latency penalty:

$$\mathrm{Throughput}={\displaystyle \frac{N_{\mathrm{queries}}}{T_{\mathrm{total}}}}·{\displaystyle \frac{1}{1+{\alpha }_{\mathrm{latency}}·{\sigma }_T}}$$

(29)

6. Experiment Results

We evaluated MALLM against state-of-the-art models including Gemini-1.5 Pro, LLaMA-3 70B, and Qwen-2 72B on the ShopBench dataset. Table 1 presents comprehensive results across all evaluation dimensions.

MALLM consistently outperforms all baselines with an average improvement of 8.7%, achieving particularly strong gains in concept normalization (+5.1% over Gemini-1.5 Pro) and extraction tasks (BERTScore +0.032). The model demonstrates superior few-shot learning, matching baseline 100-shot performance with only 25 samples.

Ablation studies reveal that the multi-agent framework and domain adaptation contribute most significantly to performance (drops of 3.7% and 4.5% respectively when removed). While removing multi-agent processing improves latency from 70ms to 55ms, the performance trade-off is substantial. The cross-modal fusion and knowledge distillation components provide consistent but smaller gains across all tasks.

Notably, MALLM maintains competitive efficiency despite additional architectural complexity, achieving 70ms latency compared to 65-68ms for simpler baselines while using only 170GB memory. This efficiency, combined with superior performance, makes MALLM practical for production e-commerce deployments.

7. Conclusion

This paper presents MALLM, a multi-agent large language model architecture for shopping concept understanding. Through hierarchical agent specialization, retrieval-augmented generation, and cross-modal fusion, MALLM achieves state-of-the-art performance with 8.7% average improvement over comparable models. The model’s superior few-shot learning capabilities and practical efficiency validate our approach for e-commerce applications.