Hierarchical Expert Multi-Agent Framework for Causal Root Cause Localization in Cloud-Native Microservices

1. Introduction

Cloud-native architectures split applications into microservices that interact through complex and dynamic dependencies. This design improves scalability and resilience, but it makes root cause localization harder because anomalies can spread across services and heterogeneous telemetry data. The task is to separate true faults from propagated effects with strict limits on latency and accuracy. Traditional methods that use static graphs or single-modality inputs often fail when workloads change or when data are multimodal.

Recent work with graph neural networks and causal inference improves dependency modeling, but these approaches often misidentify hub services as root causes because of structural bias. Large language model based methods can analyze logs and metrics, but they face hallucination, context window limits, and overhead in coordination. Flat frameworks also increase communication bottlenecks as system scale grows.

HEMA-RCL is a hierarchical expert multi-agent framework that combines different large models to achieve accurate and efficient diagnosis. The layered design reduces communication cost. Dynamic agent generation adapts to new failures with small parameter overhead. Belief propagation with causal metrics helps avoid hub misidentification. Multimodal alignment combines metrics, logs, and traces in a consistent way. Context-aware prompting reduces hallucination and improves scalability for real-world use.

2. Related Work

Microservice diagnosis uses traces and metrics: Zhang et al. [1] exploit traces for fine granularity, while Wang et al.[2] integrate metrics for multimodality; both struggle with scalability and class imbalance.

Causal approaches refine localization: Xin et al.[3] apply causal reasoning, and Zhu et al.[4] extend to instance-level services; both rely on brittle, stable structures.

Graph methods improve accuracy at higher cost: Sun et al.[5] use graph autoencoders for interpretability, and Wang et al.[6] leverage knowledge graphs, increasing computation and maintenance.

Trace-based anomaly detection is sensitive yet fragile: Panahandeh et al.[7] detect deviations but degrade with noise or missing data. Broader modeling advances aid adaptability: Tian et al.[8] introduce cross-attention for heterogeneous tasks, Guan[9] applies ML to healthcare prediction, and Zhu and Liu[10] enhance NER via LoRA fine-tuning. Persistent gaps remain in scalability, robustness, and adaptation to novel failures.

3. Methodology

HEMA-RCL tackles microservice root-cause localization via a hierarchical expert multi-agent system coordinating heterogeneous LLMs: a DeepSeek-V2 (236B) Orchestrator routes to Qwen-72B and Qwen-14B specialists and dynamically spawns Qwen-7B sub-agents for compute–accuracy efficiency. For reproducibility, the appendix provides pseudocode for the Orchestrator main loop and belief-propagation message passing, detailing input formats, update rules, and termination criteria. Spawning is KL-gated on detected fault-pattern distributions, and agent disagreement is resolved by factor-graph belief propagation that fuses historical reliability with transfer entropy for causal inference. On 10,000 fault-injection scenarios, HEMA-RCL attains 87.6% top-1 root-cause accuracy and reduces mean time to detection to 38.7 s.

4. Algorithm and Model

4.1. HEMA-RCL Architecture Overview

HEMA-RCL coordinates heterogeneous LLMs in a unified diagnostic stack. DeepSeek-V2 (236B) orchestrates long-context reasoning; Qwen-72B performs high-throughput metric analysis; Qwen-14B mines log patterns. The system uses seven agent types:

$$\begin{array}{cc}\hfill {\mathcal{S}}_{HEMA}=\{& {\mathcal{A}}_{orch},{\mathcal{A}}_{mad},{\mathcal{A}}_{lpm},{\mathcal{A}}_{tr},\hfill \\ \hfill & {\mathcal{A}}_{ci},{\left\{{\mathcal{A}}_{ds}^i\right\}}_{i=1}^{N_{dynamic}},{\mathcal{A}}_{cb}\}\hfill \end{array}$$

(1)

where $N_{dynamic}$ is resource-bounded.

4.2. Hierarchical Expert System with LLM Specialization

Flat multi-agent designs incur quadratic coordination; with $>5$ agents, message passing reaches $67\%$ runtime. We adopt a three-layer hierarchy to reduce coordination while preserving information fidelity. The orchestrator uses grounded context with a compact history. History is compressed via ${\mathrm{PCA}}_{k=32}$ on recent features and centroids of similar retrieved contexts. Resource allocation uses a composite complexity metric with online weights $\alpha =0.3$, $\beta =0.2$, $\gamma =0.25$, $\delta =0.25$. The Metric Anomaly Detection Agent (MAD) fuses FFT-based spectral features, seasonal LSTM states over a sliding window, and wavelet coefficients. A learnable threshold, updated toward a conservative baseline, mitigates false positives after distribution shifts.

$${\pi }_{orch}(a_t\mid s_t)=\mathrm{DeepSeek}-\mathrm{V}2\left({\mathrm{Prompt}}_{grounded}(·)\right)$$

(2)

$$\begin{array}{cc}\hfill \mathcal{C}\left(s_t\right)& =\alpha H\left({\mathbf{M}}_t\right)+\beta {\parallel {\mathbf{L}}_t\parallel }_0\hfill \\ \hfill & +\gamma \mathrm{depth}\left({\mathcal{T}}_t\right)+\delta {\mathrm{VAR}}_{temporal}\left({\mathbf{M}}_t\right)\hfill \end{array}$$

(3)

4.3. Dynamic Agent Spawning Mechanism

Pre-configured agents degrade on novel faults (up to $40\%$ accuracy loss). We trigger spawning via a two-stage KL-gated policy (Figure 1):

$$\begin{array}{c}\hfill p(\mathrm{spawn}\mid {\mathcal{F}}_t)=\sigma \left({\mathbf{w}}^{\top }\left[{\mathcal{D}}_{KL}({\mathcal{F}}_t\parallel {\mathcal{F}}_{known}),{\mathcal{R}}_{available},{\mathcal{U}}_{current}\right]\right)\end{array}$$

(4)

Rapid adaptation uses rank-16 LoRA on Qwen-7B, cutting trainable parameters by $98.7\%$ while retaining $94\%$ performance:

$${\mathcal{A}}_{ds}^{new}=\mathrm{Qwen}-7\mathrm{B}\left({\theta }_{base}\right)+{\mathrm{LoRA}}_{r=16}\left(\Delta \theta \left({\mathcal{F}}_t\right)\right)$$

(5)

4.4. Belief Propagation-based Consensus Mechanism

Majority voting fails under correlated errors. We use factor-graph belief propagation with time-decayed reliability (Figure 2):

$$r_i\left(t\right)={\displaystyle \frac{{\sum }_{\tau =1}^t\mathbb{I}\left[{\mathrm{correct}}_i\left(\tau \right)\right]e^{-\alpha (t-\tau )}}{{\sum }_{\tau =1}^t{e}^{-\alpha (t-\tau )}}}$$

(6)

Messages encode semantic compatibility and historical agreement:

$$\begin{array}{c}\hfill {\psi }_{ij}(x_i,x_j)=exp\left(-\lambda d_{semantic}(x_i,x_j)·{\displaystyle \frac{1}{\sqrt{r_i{r}_j}}}·(1+{ϵ}_{ij})\right)\end{array}$$

(7)

Damping stabilizes loopy graphs:

$${\mu }_{i\to j}^{(t+1)}=(1-\rho ){\mu }_{i\to j}^{\left(t\right)}+\rho {\widehat{\mu }}_{i\to j}^{(t+1)}$$

(8)

This reduces consensus time from $8.3$ to $2.1$ s.

4.5. Causal Inference Enhancement

Correlation-based selection overweights hubs (43% accuracy). We incorporate transfer entropy with adaptive delay:

$$\begin{array}{c}\hfill \mathrm{TE}(u\to v)=\sum _{t}p(v_{t+\Delta t},v_t,u_t)log{\displaystyle \frac{p(v_{t+\Delta t}\mid v_t,u_t)}{p(v_{t+\Delta t}\mid v_t)}}\end{array}$$

(9)

$$\begin{array}{c}\hfill \Delta t_{optimal}=arg\underset{\Delta t\in [1,10]}{max}\left[\mathrm{TE}(u\to v;\Delta t)-{\lambda }_{penalty}\Delta t\right]\end{array}$$

(10)

We integrate with GNNs via position-agnostic attention:

$$\begin{array}{c}\hfill {\mathbf{h}}_v^{(l+1)}=\sigma \left({\mathbf{W}}_{self}^{\left(l\right)}{\mathbf{h}}_v^{\left(l\right)}+\sum _{u\in \mathcal{N}\left(v\right)}{\alpha }_{vu}^{\left(l\right)}{\mathbf{W}}_{msg}^{\left(l\right)}{\mathbf{h}}_u^{\left(l\right)}+{\mathbf{b}}_{struct}\left(\mathcal{G}\right)\right)\end{array}$$

(11)

where ${\mathbf{b}}_{struct}\left(\mathcal{G}\right)$ is structure-encoded and position-agnostic.

4.6. Data Preprocessing

Figure 3 summarizes the pipeline.

4.6.1. Multi-modal Data Alignment and Temporal Synchronization

Metrics arrive every 15 s; logs are event-driven; traces are asynchronous. Naive fusion degraded accuracy by 31%. We align to a unified grid, interpolate irregular logs, progressively encode traces, and apply robust normalization.

$${\mathcal{T}}_{unified}=\{{\tau }_k:{\tau }_k=t_0+k·\Delta t_{base},k\in [0,K]\}$$

(12)

Traces use progressive completion with confidence $\alpha =min\left(1,n_{complete}/n_{expected}\right)$:

$${\mathbf{t}}_{prog}\left({\tau }_k\right)=\alpha {\mathbf{t}}_{complete}\left({\tau }_k\right)+(1-\alpha ){\widehat{\mathbf{t}}}_{partial}\left({\tau }_k\right)$$

(13)

Robust scaling uses MAD:

$${\tilde{m}}_{ij}={\displaystyle \frac{m_{ij}-median\left({\mathbf{m}}_j\right)}{MAD\left({\mathbf{m}}_j\right)·1.4826}}$$

(14)

This alignment and scaling reduced false positives by 43%.

4.6.2. Adaptive Feature Engineering and Dimensionality Reduction

Standard PCA preserved only 67% fault-relevant variance despite 95% total variance. We compute feature importance, aggregate multi-scale statistics (windows $s\in \{1,5,15,60\}$; mean, max, gradient, FFT), and learn a supervised low-dimensional embedding with fault-aware reconstruction.

$$\begin{array}{cc}\hfill {\mathcal{I}}_{importance}\left(f_i\right)& ={\omega }_{1}I(f_i;Y_{fault})+{\omega }_2\sum _j|TE(f_i\to f_j)|\hfill \\ \hfill & +{\omega }_3{VAR}_{conditional}\left(f_i\mid Y_{fault}\right)\hfill \end{array}$$

(15)

$${\mathcal{L}}_{reduce}=\parallel \mathbf{X}-\widehat{\mathbf{X}}{\parallel }_2^2+\beta \sum _{i\in {\mathcal{F}}_{fault}}\parallel {\mathbf{x}}_i-{\widehat{\mathbf{x}}}_i{\parallel }_2^2+\gamma {\parallel {\mathbf{W}}_{encoder}\parallel }_1$$

(16)

This reduced dimensionality by 94% and improved fault-detection F1 by 12%.

4.7. Prompt Design

4.7.1. Context-Aware Dynamic Prompt Generation

Figure 4 summarizes evaluations.

Static prompts yield 52% accuracy. We construct prompts from state, history, and agent signals, and apply context filtering to control tokens. Prompt scaffolding adds intermediate reasoning and validation, raising accuracy by 31% and reducing hallucinations by 67%.

$$P_{dynamic}=T_{base}\oplus {\mathcal{F}}_{context}\left(s_t\right)\oplus {\mathcal{F}}_{history}\left({\mathcal{H}}_{relevant}\right)\oplus {\mathcal{F}}_{guide}\left({\mathcal{A}}_{detected}\right)$$

(17)

$$\mathrm{score}({\mathbf{c}}_i,\mathcal{A})=cos\left(\mathrm{embed}\left({\mathbf{c}}_i\right),\mathrm{embed}\left(\mathcal{A}\right)\right)·exp\left(-{\lambda }_t|t_i-t_{current}|\right)$$

(18)

4.7.2. Few-Shot Learning and Prompt Optimization

Random few-shot improves over zero-shot by 38%; optimal selection reaches 74%. We select examples via a relevance–diversity objective and optimize prompts with a genetic procedure; learned mutations penalize length and latency. This raises zero-shot accuracy by 23% and shortens prompts by 34%.

$$\begin{array}{cc}\hfill {\mathcal{E}}_{selected}=arg\underset{\mathcal{E}}{max}[& \sum _{e\in \mathcal{E}}\mathrm{rel}(e,s_t)\hfill \\ \hfill & +{\lambda }_{div}\sum _{e_i,e_j\in \mathcal{E}}\mathrm{dist}(e_i,e_j)]\hfill \end{array}$$

(19)

4.8. Evaluation Metrics

4.8.1. Primary Metrics

We evaluate ranking accuracy, multi-cause coverage, and timeliness with quality penalties.

$$\mathrm{Acc}\text{@}\mathrm{K}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbb{I}\left[y_i^{true}\in \mathrm{TopK}\left({\widehat{\mathbf{p}}}_i\right)\right]$$

(20)

For multi-root incidents (23%), we assign partial credit.

$$\mathrm{MR}-\mathrm{Acc}\text{@}\mathrm{K}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\displaystyle \frac{|{\mathcal{Y}}_i^{true}\cap \mathrm{TopK}\left({\widehat{\mathbf{p}}}_i\right)|}{|{\mathcal{Y}}_i^{true}|}}$$

(21)

Time efficiency with a precision-aware penalty.

$$\mathrm{QA}-\mathrm{MTTD}=\mathrm{MTTD}·\left(1+\alpha (1-\mathrm{precision})\right)$$

(22)

4.8.2. Secondary Metrics

We report NDCG with causal-distance gains and weighted F1 for class imbalance. Resource efficiency jointly scores accuracy, compute, and latency.

$$\mathrm{Efficiency}={\displaystyle \frac{\mathrm{Acc}\text{@}1}{\mathrm{GPU}-\mathrm{Hours}}}·{\displaystyle \frac{1}{1+log\left({\mathrm{Latency}}_{p99}/{\mathrm{Latency}}_{target}\right)}}$$

(23)

5. Experiment Results

5.1. Experimental Setup

We evaluate HEMA-RCL on 12,000 fault injection scenarios across three production deployments: 156-service e-commerce platform, 89-service streaming application, and 234-service financial system. Faults cover six categories: resource exhaustion, network issues, application errors, configuration drift, cascading failures, and Byzantine faults.

5.2. Main Results and Comparisons

Table 1 presents comprehensive performance comparison including baseline methods and fault type breakdown.

HEMA-RCL achieves 18.6% improvement in Acc@1 and 50.6% reduction in MTTD compared to the best baseline (Claude-Analyst). Resource exhaustion shows highest performance (F1=0.905) due to clear metric indicators, while Byzantine faults prove most challenging (F1=0.769).

5.3. Ablation Studies and Scalability

Table 2 combines ablation results and scalability analysis.

Ablation studies reveal multi-modal fusion and hierarchical structure as most critical components (9.9% and 8.9% accuracy drops respectively). The system maintains strong performance at scale, with only 6.1% accuracy degradation from 50 to 250 services, demonstrating sub-linear computational growth.

5.4. Real-Time Performance

The system meets production SLAs with P50/P90/P99 latencies of 31.4/48.6/67.2 seconds for complete diagnosis (target: 90s) and 2.1/3.8/6.4 seconds for agent consensus (target: 10s). Memory usage peaks at 52.3GB (P99), well within the 64GB budget. These results demonstrate HEMA-RCL’s production readiness for large-scale microservice environments.

6. Conclusions

This paper presented HEMA-RCL, a hierarchical expert multi-agent system that leverages diverse large language models for microservice root cause localization. Through dynamic agent spawning, belief propagation-based consensus, and sophisticated prompt engineering, our framework achieves 87.6% top-1 accuracy while reducing mean time to detection by 50.6% compared to state-of-the-art methods. The comprehensive experimental evaluation across 12,000 fault scenarios demonstrates robust performance across diverse fault types and scales effectively to hundreds of services. The ablation studies confirm the critical contribution of each architectural component, particularly the hierarchical structure and multi-modal fusion mechanisms. HEMA-RCL establishes a new paradigm for intelligent operations in cloud-native environments, paving the way for more autonomous and reliable microservice management.