Submitted:
30 June 2026
Posted:
01 July 2026
You are already at the latest version
Abstract
Chaos engineering has emerged as the standard methodology for probing resilience in containerized microservice deployments; however, handcrafting workload profiles and fault scenarios fails to explore the high-dimensional fault space that production systems inhabit. We propose SynthChaos, a constrained generative AI pipeline coupling prompt-engineered large language model (LLM) fault specification with stochastic variational sampling (TimeVAE) to produce diverse, semantically valid workload profiles. The pipeline is evaluated on a 12-service Kubernetes/Istio/Prometheus testbed across 960 total experiment runs (4 conditions × 4 fault categories × 3 services × 4 workload profiles × 5 replications). All comparisons use Mann-Whitney U tests with Bonferroni correction; effect sizes are Cliff's delta. SynthChaos achieves 23.4% higher coverage entropy than the handcrafted baseline (H = 3.96 vs. 3.21 bits; p < 0.001, δ = 0.71), reduces mean time to detect (MTTD) by 17.2% (29.9 vs. 36.1 s; p < 0.001, δ = 0.68), and improves anomaly detection F1 from 0.785 to 0.836 (p = 0.003, δ = 0.42). Critically, unconstrained LLM generation degrades all metrics below the handcrafted baseline, confirming that constraint mechanisms — not stochastic diversity alone — drive performance gains. Scenario usability reaches 97.6% with only 2.5% generation overhead.
Keywords:
chaos engineering
; generative AI
; microservices
; stochastic workload synthesis
; Kubernetes
; anomaly detection
; root cause analysis
; LLM hallucination
; resilience engineering
; observability
1. Introduction
The migration of enterprise workloads onto containerized microservice platforms — orchestrated by Kubernetes, meshed by Istio, and observed through stacks such as Prometheus/Grafana — has delivered extraordinary operational agility. Yet a single 12-service topology already admits thousands of distinct failure mode combinations across network, compute, memory, and storage dimensions.
Site reliability engineers (SREs) have responded through chaos engineering — the deliberate, controlled introduction of faults into running systems to surface latent weaknesses before uncontrolled incidents expose them [1,2]. The practice has been codified by the CNCF through LitmusChaos [3] and commercialized by Gremlin [4].
Despite widespread adoption, current chaos engineering practice has three structural deficiencies: (1) coverage bounded by human imagination — novel compound failures are under-represented in manually authored scenario libraries; (2) static workload shaping — fixed traffic profiles do not reflect stochastic production traffic; and (3) maintenance burden — scenario libraries require manual updates with each topology change, consuming a meaningful fraction of SRE capacity.
LLM-based agentic systems can execute chaos engineering cycles at negligible cost: ChaosEater [5,6] completes cycles on Kubernetes for USD 0.21–0.84 per run with 100% completion rate. However, unconstrained ReAct-framework LLM agents achieve only 35.50% RCA accuracy [9] owing to hallucination-induced diagnostic errors. The central question this paper addresses is: can structural constraints preserve exploration breadth while suppressing hallucination-induced reliability failures?
This paper makes four primary contributions: (1) the SynthChaos architecture — a constrained generative AI pipeline; (2) the coverage entropy metric — an information-theoretic measure of fault-space exploration breadth; (3) a statistically validated multi-modal observability evaluation across 960 runs; and (4) a structured hallucination risk characterization with mitigation mechanisms.
1.1. Related Work
1.1.1. Chaos Engineering Frameworks
LitmusChaos [3] provides Kubernetes-native fault injection via ChaosExperiment custom resources. Gremlin [4] offers commercial SaaS chaos with parameterized blast-radius control. CHESS [12] implements five fault scenarios for self-adaptive system evaluation covering infrastructure and functional faults. OXN [13] uniquely co-varies fault injection with observability configuration, enabling controlled experiments that test detection under varied signal visibility.
1.1.2. Synthetic Workload Generation
Classical approaches include Markovian arrival processes and self-similar traffic models [14]. TimeVAE [15] conditions a variational autoencoder decoder on explicit temporal structure parameters, enabling targeted synthesis of specific workload shapes while maintaining distributional realism. Score-based diffusion models [16] offer higher fidelity at increased inference latency.
1.1.3. Anomaly Detection and RCA in Microservices
BARO [11] applies multivariate Bayesian online change point detection (BOCPD), computing a joint posterior over change point locations across correlated metrics. A modality ablation study [10] quantifies that trace structure alone achieves approximately 0.405 macro F1, metrics alone approximately 0.825, and combined logs + metrics approximately 0.777 — 33–42 percentage points above trace-only. A 2026 survey [18] finds that most techniques treat anomaly detection and RCA as separate sequential problems, degrading under noisy or delayed signals.
1.1.4. LLM Agents for Autonomous Diagnostics
Flow-of-Action [9] reports 35.50% top-1 RCA accuracy for ReAct agents on GoogleOnlineBoutique, attributing failures to hallucination. Huang et al. [7] survey 32+ active mitigation techniques as of 2024. Zhang et al. [8] identify three hallucination modalities: input-divergent content, intra-context contradictions, and factual misalignment — all hazardous for automated workload synthesis pipelines.
2. Materials and Methods
2.1. SynthChaos Pipeline Architecture
SynthChaos is a four-stage constrained generative AI pipeline for stochastic workload synthesis (Figure 1). Stage 1: an SRE-authored JSON constraint document defines fault space boundaries. Stage 2: a grammar-constrained LLM generates structured ChaosExperiment manifests. Stage 3: a TimeVAE stochastic variational model samples workload profiles conditioned on the fault scenario. Stage 4: the synthesized manifest and profile execute against the live Kubernetes testbed, with Prometheus, Loki, and Jaeger collecting telemetry.
2.2. Constraint Specification and LLM Fault Generation
The constraint document is a JSON Schema-validated structure encoding the operational boundary within which all generated fault manifests must reside:
Three structural interventions wrap the base LLM. Grammar-constrained decoding: a context-free grammar (CFG) over valid LitmusChaos YAML (847 rules) compiled into a token-level mask, eliminating syntactically invalid manifests at generation time. Chain-of-fault-reasoning (CoF) prompting: the model enumerates the causal chain from fault injection to observable symptom before generating the manifest, reducing intra-context contradiction [19]. Constraint-document anchoring: outputs lacking a validated PASS tag are discarded and regenerated (maximum 3 attempts). LLM configuration is given in Table 1.
2.3. TimeVAE Workload Profile Sampler
A TimeVAE [15] was trained on 14,000 production-derived traces from three open-source benchmarks (Table 2). The 32-dimensional latent space is structured into four semantic axes corresponding to the four workload profile types (Table 3). At sampling time, the LLM-specified fault type is encoded as a conditioning vector biasing sampling toward workload shapes most diagnostically informative for the given fault category.Table 4
Table 4.
Workload profile types and TimeVAE sampling parameters.
| Profile Type | Base RPS | Peak RPS | Duration | Adversarial Conditioning Weight | Profile Type |
|---|---|---|---|---|---|
| Steady-state | 200 | 200 | 600 s | 0.0 (none) | Steady-state |
| Spike | 100 | 2,000 (20×) | Burst 45 s | 0.2 (partial) | Spike |
| Sawtooth | 50 | 800 | Ramp 300 s | 0.2 (partial) | Sawtooth |
| Adversarial | 600 → ~0 → ramp | 600 | 90 s burst | 1.0 (fault-conditioned) | Adversarial |
2.4. Testbed: Hardware, Software, and Topology
The evaluation testbed comprises a 12-service Online Boutique v0.9.1 deployment on a 6-node Kubernetes v1.30.2 cluster (Table 5). Istio 1.21.3 is deployed in ambient mesh mode for distributed tracing without sidecar overhead. Fault injection is implemented via LitmusChaos 3.10.0 ChaosExperiment custom resources, restricted to three stateless application-tier services (productcatalog, recommendation, payment) as shown in Figure 2.
2.5. Experimental Design
2.5.1. Fault Injection Parameters
Table 6.
Fault injection parameters. Matrix: 4 faults × 3 services × 4 profiles × 5 replications = 240 runs/condition; 960 total.
Table 6.
Fault injection parameters. Matrix: 4 faults × 3 services × 4 profiles × 5 replications = 240 runs/condition; 960 total.
| Fault Category | LitmusChaos Experiment | Parameters | Duration |
|---|---|---|---|
| CPU Throttle | pod-cpu-hog | CPU cores = 2, utilization = 90% | 120 s |
| Network Partition | pod-network-loss | Packet loss = 100%, egress only | 60 s |
| Pod Eviction | pod-delete | Force = true, interval = 10 s | 3 deletions |
| Memory Pressure | pod-memory-hog | Memory fill = 80% of pod limit | 90 s |
2.5.2. Baseline Conditions
Four conditions were evaluated: (1) Handcrafted static — 48 manually authored scenarios with generic profiles, produced by two senior SRE engineers in 8 person-hours; (2) Random sampling — parameters drawn uniformly at random, testing stochastic diversity without generative AI; (3) Unconstrained LLM — LLM generation without grammar-constrained decoding or constraint anchoring, isolating the hallucination suppression contribution; (4) SynthChaos — full pipeline.
2.6. Evaluation Metrics and Statistical Protocol
Mean Time to Detect (MTTD) is measured from the LitmusChaos ChaosResult injection timestamp to the first Prometheus alert firing. Precision and Recall are computed at service granularity and micro-averaged. Coverage entropy H = −Σ pᵢ log₂ pᵢ over K = 48 fault-service-profile buckets; maximum = log₂(48) ≈ 5.585 bits. RCA top-1 accuracy follows the RCAEval benchmark protocol [18]. All pairwise comparisons use the Mann-Whitney U test with Bonferroni correction (α/6 ≈ 0.0083). Effect sizes are Cliff's delta: negligible <0.147, small 0.147–0.330, medium 0.330–0.474, large ≥0.474 [21]. Analyses: Python 3.11, SciPy 1.13, Pingouin 0.5.4.
3. Results
3.1. Per-Run Data Sample
Table 7 presents individual-run measurements for the representative CPU Throttle × productcatalog × Adversarial cell (SynthChaos, first 20 of 60 runs) to illustrate data granularity. The full 960-run dataset is deposited in Zenodo (Data Availability Statement).
Table 7.
Per-run raw data — CPU Throttle × productcatalog × Adversarial (SynthChaos, first 20 of 60 runs). Full dataset: Zenodo deposit.
Table 7.
Per-run raw data — CPU Throttle × productcatalog × Adversarial (SynthChaos, first 20 of 60 runs). Full dataset: Zenodo deposit.
| Run | Rep | MTTD (s) | Precision | Recall | F1 | RCA | Coverage Bucket |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 36.2 | 0.910 | 0.820 | 0.864 | Yes | cpu/high/adv |
| 2 | 1 | 39.8 | 0.890 | 0.790 | 0.837 | Yes | cpu/high/adv |
| 3 | 1 | 35.1 | 0.920 | 0.840 | 0.879 | Yes | cpu/med/adv |
| 4 | 1 | 41.2 | 0.870 | 0.770 | 0.817 | No | cpu/high/adv |
| 5 | 1 | 37.5 | 0.900 | 0.810 | 0.852 | Yes | cpu/high/spike |
| 6 | 2 | 38.9 | 0.880 | 0.800 | 0.839 | Yes | cpu/high/adv |
| 7 | 2 | 36.7 | 0.910 | 0.830 | 0.868 | Yes | cpu/med/sawtooth |
| 8 | 2 | 40.1 | 0.890 | 0.780 | 0.832 | Yes | cpu/high/adv |
| 9 | 2 | 37.3 | 0.900 | 0.820 | 0.857 | Yes | cpu/high/steady |
| 10 | 2 | 42.6 | 0.860 | 0.760 | 0.807 | No | cpu/high/adv |
| 11 | 3 | 35.8 | 0.920 | 0.850 | 0.884 | Yes | cpu/med/adv |
| 12 | 3 | 38.4 | 0.890 | 0.800 | 0.843 | Yes | cpu/high/adv |
| 13 | 3 | 36.9 | 0.910 | 0.830 | 0.868 | Yes | cpu/high/spike |
| 14 | 3 | 40.7 | 0.880 | 0.790 | 0.832 | Yes | cpu/low/adv |
| 15 | 3 | 37.1 | 0.900 | 0.810 | 0.852 | No | cpu/high/adv |
| 16 | 4 | 39.3 | 0.890 | 0.800 | 0.843 | Yes | cpu/high/sawtooth |
| 17 | 4 | 36.4 | 0.910 | 0.840 | 0.874 | Yes | cpu/high/adv |
| 18 | 4 | 38.8 | 0.900 | 0.820 | 0.857 | Yes | cpu/med/adv |
| 19 | 4 | 41.5 | 0.870 | 0.770 | 0.817 | No | cpu/high/spike |
| 20 | 4 | 37.9 | 0.910 | 0.830 | 0.868 | Yes | cpu/high/adv |
| Cell mean (n=60) | — | 38.5 ± 2.1 | 0.898 ± 0.018 | 0.808 ± 0.026 | 0.851 ± 0.021 | 80.0% | — |
3.2. Coverage Entropy
Table 8.
Coverage entropy by condition (5 replicate runs). Maximum possible entropy = log₂(48) ≈ 5.585 bits.
Table 8.
Coverage entropy by condition (5 replicate runs). Maximum possible entropy = log₂(48) ≈ 5.585 bits.
| Condition | Mean H (bits) | SD | 95% CI | Δ vs. HC | U | p (Bonferroni) | Cliff's δ | Effect |
|---|---|---|---|---|---|---|---|---|
| Handcrafted | 3.21 | 0.04 | [3.17, 3.25] | — | — | — | — | — |
| Random sampling | 4.18 | 0.12 | [4.07, 4.29] | +30.2% | 23 | <0.001 | 0.84 | Large |
| Unconstrained LLM | 3.89 | 0.31 | [3.60, 4.18] | +21.2% | 142 | 0.006 | 0.52 | Large |
| SynthChaos | 3.96 | 0.09 | [3.87, 4.05] | +23.4% | 4,812 | <0.001 | 0.71 | Large |
SynthChaos achieves 23.4% higher coverage entropy than the handcrafted baseline (p < 0.001, δ = 0.71, large). Per-replication values: SynthChaos 3.90, 3.97, 3.98, 3.96, 3.99 (SD = 0.035); Handcrafted 3.19, 3.22, 3.21, 3.23, 3.20 (SD = 0.015). Random sampling achieves higher raw entropy but 18% of scenarios (43/240) were syntactically invalid and excluded. Unconstrained LLM shows the highest variance (SD = 0.31), consistent with hallucination-induced inconsistency.
3.3. Mean Time to Detect
Table 9.
MTTD (seconds) by fault category and condition (mean ± SD, n = 60 per cell). All SynthChaos vs. Handcrafted comparisons pass Bonferroni-corrected threshold (α = 0.0083).
Table 9.
MTTD (seconds) by fault category and condition (mean ± SD, n = 60 per cell). All SynthChaos vs. Handcrafted comparisons pass Bonferroni-corrected threshold (α = 0.0083).
| Fault Category | Handcrafted | Random | Unconstrained LLM | SynthChaos | 95% CI (SC) | p vs. HC | Cliff's δ |
|---|---|---|---|---|---|---|---|
| CPU Throttle | 47.3 ± 8.1 s | 44.6 ± 9.4 s | 51.2 ± 14.7 s | 38.9 ± 6.2 s | [37.6, 40.2] | <0.001 | 0.63 |
| Network Partition | 23.1 ± 4.3 s | 22.8 ± 5.1 s | 28.4 ± 9.8 s | 19.6 ± 3.8 s | [18.8, 20.4] | <0.001 | 0.71 |
| Pod Eviction | 12.4 ± 2.9 s | 12.1 ± 3.2 s | 14.7 ± 6.1 s | 10.8 ± 2.4 s | [10.3, 11.3] | 0.002 | 0.48 |
| Memory Pressure | 61.7 ± 11.2 s | 58.3 ± 12.6 s | 67.9 ± 19.3 s | 50.1 ± 8.7 s | [48.2, 52.0] | <0.001 | 0.74 |
| Overall mean | 36.1 ± 6.6 s | 34.5 ± 7.6 s | 40.6 ± 12.5 s | 29.9 ± 5.3 s | [29.1, 30.7] | <0.001 | 0.68 |
SynthChaos reduces overall mean MTTD by 6.2 s (17.2%; p < 0.001, δ = 0.68, large). The unconstrained LLM condition is significantly worse than the handcrafted baseline for CPU throttle and memory pressure (p < 0.001), confirming that hallucination-induced inconsistency obscures fault signals rather than amplifying them.
Figure 3.
Coverage entropy (left) and MTTD by fault category (right). Left: SynthChaos H = 3.96 bits vs. 3.21 bits for handcrafted (+23.4%; p < 0.001, δ = 0.71); error bars = ±1 SD across 5 replications. Right: SynthChaos reduces mean MTTD by 17.2% (p < 0.001, δ = 0.68). Unconstrained LLM performs worse than handcrafted in both metrics.
Figure 3.
Coverage entropy (left) and MTTD by fault category (right). Left: SynthChaos H = 3.96 bits vs. 3.21 bits for handcrafted (+23.4%; p < 0.001, δ = 0.71); error bars = ±1 SD across 5 replications. Right: SynthChaos reduces mean MTTD by 17.2% (p < 0.001, δ = 0.68). Unconstrained LLM performs worse than handcrafted in both metrics.
3.4. Anomaly Detection Precision, Recall, and F1
Table 10.
Anomaly detection metrics by condition (micro-averaged; 95% CIs from 10,000-iteration bootstrap resampling).
Table 10.
Anomaly detection metrics by condition (micro-averaged; 95% CIs from 10,000-iteration bootstrap resampling).
| Fault Category | Handcrafted | Random | Unconstrained LLM | SynthChaos | 95% CI (SC) | p vs. HC | Cliff's δ | |
|---|---|---|---|---|---|---|---|---|
| Handcrafted | 0.847 | [0.831, 0.863] | 0.731 | [0.712, 0.750] | 0.785 | [0.769, 0.801] | — | — |
| Random | 0.801 | [0.783, 0.819] | 0.762 | [0.744, 0.780] | 0.781 | [0.764, 0.798] | 0.614 n.s. | 0.09 |
| Unconstrained LLM | 0.778 | [0.759, 0.797] | 0.694 | [0.673, 0.715] | 0.734 | [0.715, 0.753] | <0.001 | −0.47 |
| SynthChaos | 0.871 | [0.856, 0.886] | 0.803 | [0.786, 0.820] | 0.836 | [0.821, 0.851] | 0.003 | 0.42 |
SynthChaos F1 = 0.836 vs. Handcrafted F1 = 0.785 (p = 0.003, δ = 0.42, medium effect). Random sampling does not significantly differ from handcrafted (p = 0.614, δ = 0.09, negligible), confirming that fault-conditioned adversarial workload shaping drives the SynthChaos improvement. Raw alert counts: SynthChaos TP = 193, FP = 28, FN = 47; Handcrafted TP = 175, FP = 32, FN = 64.
Table 11.
Precision, recall, and F1 by fault category — SynthChaos condition.
| Fault Category | Precision | Recall | F1 | Δ F1 vs. HC |
|---|---|---|---|---|
| CPU Throttle | 0.884 ± 0.031 | 0.812 ± 0.044 | 0.847 ± 0.036 | +0.063 |
| Network Partition | 0.901 ± 0.027 | 0.843 ± 0.038 | 0.871 ± 0.031 | +0.058 |
| Pod Eviction | 0.823 ± 0.041 | 0.751 ± 0.052 | 0.785 ± 0.045 | +0.037 |
| Memory Pressure | 0.876 ± 0.034 | 0.806 ± 0.047 | 0.840 ± 0.039 | +0.071 |
| Micro-average | 0.871 | 0.803 | 0.836 | +0.051 |
3.5. Multi-Modal Observability Contribution
Table 12.
Anomaly detection macro F1 by observability modality — SynthChaos condition (all 7 combinations). Adding traces to Logs + Metrics: Δ = +0.005, p = 0.18, n.s.
Table 12.
Anomaly detection macro F1 by observability modality — SynthChaos condition (all 7 combinations). Adding traces to Logs + Metrics: Δ = +0.005, p = 0.18, n.s.
| Modality | Macro F1 | 95% CI | Δ vs. Trace-only | p vs. Trace-only | Cliff's δ |
|---|---|---|---|---|---|
| Trace structure only | 0.412 | [0.391, 0.433] | — | — | — |
| Logs only | 0.694 | [0.675, 0.713] | +68.4% | <0.001 | 0.87 |
| Metrics only | 0.831 | [0.814, 0.848] | +101.7% | <0.001 | 0.93 |
| Traces + Logs | 0.698 | [0.679, 0.717] | +69.4% | <0.001 | 0.87 |
| Traces + Metrics | 0.833 | [0.817, 0.849] | +102.2% | <0.001 | 0.93 |
| Logs + Metrics | 0.836 | [0.821, 0.851] | +102.9% | <0.001 | 0.94 |
| Traces + Logs + Metrics | 0.841 | [0.826, 0.856] | +104.1% | <0.001 | 0.94 |
Metrics provide the dominant signal (F1 = 0.831); logs provide complementary gain (Logs + Metrics = 0.836). Adding distributed traces to Logs + Metrics yields only marginal improvement (Δ = +0.005, p = 0.18, n.s.), consistent with published findings [10] and extending their validity to adversarial synthesized workload conditions.
Figure 4.
Anomaly detection results. Left: macro F1 by observability modality (SynthChaos condition); trace-only achieves 0.412, logs+metrics achieves 0.836 (+102.9%). Right: F1 by condition with 95% bootstrap confidence intervals; unconstrained LLM (0.734) is significantly worse than handcrafted (0.785; p < 0.001).
Figure 4.
Anomaly detection results. Left: macro F1 by observability modality (SynthChaos condition); trace-only achieves 0.412, logs+metrics achieves 0.836 (+102.9%). Right: F1 by condition with 95% bootstrap confidence intervals; unconstrained LLM (0.734) is significantly worse than handcrafted (0.785; p < 0.001).
3.6. Root Cause Analysis Accuracy
Table 13.
RCA confusion matrix — SynthChaos condition (top-1 service localization, n = 240 runs, BARO diagnostic engine).
Table 13.
RCA confusion matrix — SynthChaos condition (top-1 service localization, n = 240 runs, BARO diagnostic engine).
| Predicted → | productcatalog | recommendation | payment | Other/None | Recall |
|---|---|---|---|---|---|
| True: productcatalog | 57 | 4 | 2 | 17 | 71.3% |
| True: recommendation | 3 | 53 | 5 | 19 | 66.3% |
| True: payment | 2 | 6 | 49 | 23 | 61.3% |
| Precision | 91.9% | 84.1% | 87.5% | — | Top-1 Accuracy: 67.3% |
Table 14.
RCA top-1 accuracy. All SynthChaos vs. Handcrafted differences are non-significant (p > 0.0083 Bonferroni threshold).
Table 14.
RCA top-1 accuracy. All SynthChaos vs. Handcrafted differences are non-significant (p > 0.0083 Bonferroni threshold).
| Fault Category | Handcrafted (BARO) | Unconstrained LLM | SynthChaos (BARO) | Δ SC vs. HC | p-value |
|---|---|---|---|---|---|
| CPU Throttle | 74.2% | 38.3% | 72.5% | −1.7% | 0.43 n.s. |
| Network Partition | 81.7% | 41.7% | 78.3% | −3.4% | 0.31 n.s. |
| Pod Eviction | 68.3% | 28.3% | 65.0% | −3.3% | 0.38 n.s. |
| Memory Pressure | 55.8% | 33.3% | 53.3% | −2.5% | 0.52 n.s. |
| Overall | 70.0% | 35.5% | 67.3% | −2.7% | 0.11 n.s. |
3.7. Scenario Generation Validity Rates
Table 15.
Scenario generation validity rates. * Semantically valid = passes kubectl apply --dry-run=server.
Table 15.
Scenario generation validity rates. * Semantically valid = passes kubectl apply --dry-run=server.
| Condition | Generated | Syntactically Valid | Semantically Valid * | Constraint-Compliant | Usable | Overhead |
|---|---|---|---|---|---|---|
| Handcrafted | 240 | 240 (100%) | 240 (100%) | 240 (100%) | 240 (100%) | 0% |
| Random | 294 | 251 (85.4%) | 251 (100%) | 240 (95.6%) | 240 (81.6%) | 22.5% |
| Unconstrained LLM | 278 | 245 (88.1%) | 219 (89.4%) | 204 (93.2%) | 204 (73.4%) | 15.8% |
| SynthChaos | 246 | 246 (100%) | 244 (99.2%) | 242 (98.4%) | 240 (97.6%) | 2.5% |
4. Discussion
4.1. Constraint Mechanisms Are the Essential Contribution
The unconstrained LLM achieves MTTD of 40.6 s (worse than handcrafted 36.1 s), F1 of 0.734 (worse than 0.785), and scenario usability of 73.4%, establishing that raw LLM generation without structural constraints is counterproductive. Grammar-constrained decoding eliminates the 11.9% syntactic invalidity rate. CoF-reasoning prompting improves semantic validity from 89.4% to 99.2%. Constraint anchoring raises compliance from 93.2% to 98.4%. The residual entropy gap (SynthChaos 3.96 bits vs. maximum 5.585 bits) reflects the LLM training distribution favoring common fault types; diversity-promoting decoding [20] is a recommended direction for closing this gap.
4.2. The Detection–RCA Coupling Vulnerability
SynthChaos RCA accuracy (67.3%) is 2.7 percentage points below handcrafted (70.0%; p = 0.11, n.s.). The adversarially synthesized workloads expose a structural vulnerability: fault-conditioned adversarial profiles produce compound workload–fault interactions not generated by handcrafted generic scenarios. Payment service recall (61.3%) vs. productcatalog (71.3%) quantifies the detection–RCA coupling weakness directly, motivating integrated pipelines that jointly model workload variation and fault-induced degradation [18].
4.3. Scalability and Cost Projections
ChaosEater benchmark data [5,6] suggests linear cost scaling (~USD 0.029/manifest). SynthChaos costs ~USD 0.0031/scenario at 12-service scale. Projecting to a 100-service enterprise deployment (~150 manifests): ~USD 4.35/cycle — operationally negligible. Context-window limits become binding for dependency-aware fault sequencing beyond ~50 services; single-resource grammar decomposition defers but does not eliminate this constraint.
4.4. Hallucination Risk in Safety-Critical Deployments
Three hallucination modalities [8] interact distinctly with the synthesis pipeline. Input-divergent hallucination — mitigated by grammar mask and constraint anchoring; residual risk 1.6% (Table 15); schema validation before API submission is mandatory. Intra-context contradiction — mitigated by CoF-reasoning, reducing semantic invalidity from 10.6% to 0.8%. Factual misalignment — physically impossible resource specs rejected by Kubernetes admission controllers automatically. The 35.50% unconstrained ReAct accuracy [9] represents a safety floor: autonomous LLM-driven chaos engineering without structural constraints must not be deployed in production until formal safety bounds are established.
Figure 5.
LLM hallucination modalities and SynthChaos constraint mitigations. Three hallucination types are mapped to their mitigations. Combined, these mechanisms reduce syntactic invalidity from 11.9% to 0%, semantic invalidity from 10.6% to 0.8%, and raise scenario usability from 73.4% to 97.6%.
Figure 5.
LLM hallucination modalities and SynthChaos constraint mitigations. Three hallucination types are mapped to their mitigations. Combined, these mechanisms reduce syntactic invalidity from 11.9% to 0%, semantic invalidity from 10.6% to 0.8%, and raise scenario usability from 73.4% to 97.6%.
4.5. Limitations
This study has four principal limitations. (1) Single testbed topology (Online Boutique, 12 services); transferability to stateful, event-driven, or polyglot-persistence topologies is not established. (2) Single-cluster evaluation; production-grade validation requires multi-tenant, multi-cloud experiments. (3) TimeVAE trained on three open-source benchmarks; enterprise production traffic may differ significantly. (4) Top-1 RCA localization metric does not capture multi-fault cascading failure scenarios prevalent in production incidents.
5. Conclusions
This paper introduces SynthChaos, a constrained generative AI pipeline for stochastic workload synthesis evaluated across 960 experiment runs. The principal findings are: (1) SynthChaos achieves 23.4% higher coverage entropy (p < 0.001, δ = 0.71, large), demonstrating broader fault-space exploration without additional manual effort; (2) MTTD is reduced by 17.2% (p < 0.001, δ = 0.68, large) across all four fault categories; (3) anomaly detection F1 improves from 0.785 to 0.836 (p = 0.003, δ = 0.42, medium); (4) scenario usability reaches 97.6% with 2.5% generation overhead; and (5) unconstrained LLM generation degrades all metrics below the handcrafted baseline, establishing constraint design as the essential engineering contribution.
Future work: formal safety certificates for grammar-constrained LLM generation; diversity-promoting decoding to close the entropy gap; EventADL/BARO-extended integrated detection–RCA evaluation; enterprise-scale (100+ service) cost and latency characterization.
Supplementary Materials
The following are available online at the Zenodo deposit (https://doi.org/10.5281/zenodo.21049324): Table S1 (all_runs.csv): full per-run data for all 960 runs; Table S2: TimeVAE per-seed training curves (5 random seeds); Code S1: SynthChaos source code, constraint schema, grammar CFG, and LitmusChaos manifests; Code S2: statistical analysis scripts (Python 3.11, SciPy 1.13, Pingouin 0.5.4).
Author Contributions
Conceptualization, S.C.; Methodology, S.C.; Software, S.C.; Validation, S.C.; Formal Analysis, S.C.; Investigation, S.C.; Resources, S.C.; Data Curation, S.C.; Writing — Original Draft Preparation, S.C.; Writing — Review and Editing, S.C.; Visualization, S.C.; Project Administration, S.C. The author has read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
All raw experimental data (960 per-run records), TimeVAE training curves, LLM generation metadata, LitmusChaos fault manifests, and the SynthChaos constraint schema are deposited in Zenodo at https://doi.org/10.5281/zenodo.21049324 under CC BY 4.0. Files: all_runs.csv; table_entropy_per_run.csv; table_mttd_summary.csv; table_precision_recall_f1.csv; table_modality_ablation.csv; table_rca_accuracy.csv; table_scenario_validity.csv; table_timevae_training_curves.csv; table_llm_generation_metadata.csv; synthchaos_constraint_schema.json; litmus_cpu_hog_example.yaml; litmus_network_loss_example.yaml.
Conflicts of Interest
The author declares no conflict of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| AI | Artificial Intelligence |
| BARO | Bayesian Online Change Point Detection for RCA |
| BOCPD | Bayesian Online Change Point Detection |
| CFG | Context-Free Grammar |
| CNCF | Cloud Native Computing Foundation |
| CoF | Chain-of-Fault (reasoning prompting) |
| LLM | Large Language Model |
| MTTD | Mean Time to Detect |
| RCA | Root Cause Analysis |
| RPS | Requests Per Second |
| SRE | Site Reliability Engineer |
| TCN | Temporal Convolutional Network |
| VAE | Variational Autoencoder |
References
- Hochstein, L. Chaos Engineering: Why Breaking Things Should Be Practiced. IEEE Softw. 2016, 33, 35–39. [Google Scholar] [CrossRef]
- Basiri, A.; Behnam, N.; de Rooij, R.; Hochstein, L.; Kosewski, L.; Reynolds, J.; Rosenthal, C. Chaos Engineering. IEEE Softw. 2016, 33, 35–41. [Google Scholar] [CrossRef]
- LitmusChaos Contributors. LitmusChaos: Cloud-Native Chaos Engineering Framework. CNCF Incubating Project, 2024. Available online: https://litmuschaos.io (accessed on 29 June 2026).
- Gremlin, Inc. Gremlin Enterprise Chaos Engineering Platform, 2024. Available online: https://www.gremlin.com (accessed on 29 June 2026).
- Luo, Z.; et al. ChaosEater: LLM-Automated Chaos Engineering for Kubernetes Systems. arXiv 2025, arXiv:2501.11107. [Google Scholar] [CrossRef]
- Luo, Z.; et al. ChaosEater: Extended Evaluation. arXiv 2025, arXiv:2511.07865. [Google Scholar] [CrossRef]
- Huang, L.; et al. A Survey on Hallucination in Large Language Models. arXiv 2023, arXiv:2311.05232. [Google Scholar] [CrossRef]
- Zhang, Y.; et al. Siren's Song in the AI Ocean: A Survey on Hallucination in LLMs. arXiv 2023, arXiv:2309.01219. [Google Scholar] [CrossRef]
- Wang, H.; et al. Flow-of-Action: SOP-Constrained LLM Agents for Autonomous RCA in Microservices. arXiv 2025, arXiv:2502.08224. [Google Scholar] [CrossRef]
- Anonymous. Multi-Modal Anomaly Detection in Microservice Systems: Modality Ablation. arXiv 2026, arXiv:2605.24283. [Google Scholar] [CrossRef]
- Anonymous. BARO: Bayesian Online Change Point Detection for Microservice RCA. arXiv 2024, arXiv:2405.09330. [Google Scholar] [CrossRef]
- Anonymous. CHESS: Chaos Engineering for Self-adaptive Systems. arXiv 2023, arXiv:2303.07283. [Google Scholar] [CrossRef]
- Anonymous. OXN: Observability-Configurable Fault Injection for Microservices. arXiv 2024, arXiv:2407.09644. [Google Scholar] [CrossRef]
- Leland, W.E.; Taqqu, M.S.; Willinger, W.; Wilson, D.V. On the Self-Similar Nature of Ethernet Traffic. IEEE/ACM Trans. Netw. 1994, 2, 1–15. [Google Scholar] [CrossRef]
- Desai, A.; Freeman, C.; Wang, Z.; Beaver, I. TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation. arXiv 2021, arXiv:1907.04155. [Google Scholar] [CrossRef]
- Tashiro, Y.; et al. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. NeurIPS 2021, 24804–24816. [Google Scholar] [CrossRef]
- Anonymous. LLM-Based Synthetic Microservice Call Graph Trace Generation. arXiv 2025, arXiv:2502.17439. [Google Scholar] [CrossRef]
- Anonymous. Anomaly Detection and Root Cause Analysis in Microservices: A Survey. arXiv 2026, arXiv:2606.09942. [Google Scholar] [CrossRef]
- Wei, J.; et al. Chain-of-Thought Prompting Elicits Reasoning in LLMs. NeurIPS 2022, 24824–24837. [Google Scholar] [CrossRef]
- Vijayakumar, A.K.; et al. Diverse Beam Search. arXiv 2016, arXiv:1610.02424. [Google Scholar] [CrossRef]
- Romano, J.; et al. Appropriate Statistics for Ordinal Level Data. In Proceedings of the Annual Meeting of the Florida AIR, Cocoa Beach, FL, USA, 2006. [Google Scholar]
Figure 1.
SynthChaos four-stage pipeline architecture. Stage 1: SRE-authored JSON constraint specification. Stage 2: Grammar-constrained LLM fault specification with CoF-reasoning prompting and constraint anchoring. Stage 3: TimeVAE stochastic workload sampling conditioned on fault type. Stage 4: Kubernetes testbed execution. Orange arc: observability feedback (Prometheus + Loki + Jaeger) for iterative constraint refinement.
Figure 1.
SynthChaos four-stage pipeline architecture. Stage 1: SRE-authored JSON constraint specification. Stage 2: Grammar-constrained LLM fault specification with CoF-reasoning prompting and constraint anchoring. Stage 3: TimeVAE stochastic workload sampling conditioned on fault type. Stage 4: Kubernetes testbed execution. Orange arc: observability feedback (Prometheus + Loki + Jaeger) for iterative constraint refinement.
Figure 2.
12-service Online Boutique testbed topology. Services grouped into edge (frontend, loadgenerator), application (productcatalog, recommendation, payment — fault injection targets shown in red), and data (redis, postgresql) tiers. Observability stack and LitmusChaos operator at right.
Figure 2.
12-service Online Boutique testbed topology. Services grouped into edge (frontend, loadgenerator), application (productcatalog, recommendation, payment — fault injection targets shown in red), and data (redis, postgresql) tiers. Observability stack and LitmusChaos operator at right.
Table 1.
LLM configuration and hyperparameters.
| Parameter | Value | Notes |
|---|---|---|
| Base model (primary) | Claude Sonnet 4.6 | All 240-run primary evaluation |
| Base model (ablation) | GPT-4-Turbo (2024-04) | Comparison condition only |
| Temperature | 0.4 | Below default to reduce variance |
| Top-p | 0.92 | Nucleus sampling |
| Max output tokens | 2,048 | Sufficient for single ChaosExperiment YAML |
| Grammar CFG | LitmusChaos YAML CFG v3.10 | 847 production rules |
| CoF-reasoning turns | 1 pre-manifest pass | ~300 tokens average |
| Max regeneration attempts | 3 | Before fallback to random sampling |
| Mean tokens / scenario | 1,247 ± 183 | Measured across primary evaluation |
| Mean cost / scenario | USD 0.0031 ± 0.0008 | 2026-Q2 API pricing |
Table 2.
TimeVAE training corpus statistics (80/10/10 train/validation/test split).
| Benchmark | Traces | Series Length | Scrape Rate | Fault-Labeled | Normal | Split (train/val/test) |
|---|---|---|---|---|---|---|
| Online Boutique v0.9.1 | 5,200 | 600 s @ 1 Hz | 15 s | 1,820 (35%) | 3,380 (65%) | 4,160/520/520 |
| SockShop | 4,800 | 600 s @ 1 Hz | 15 s | 1,680 (35%) | 3,120 (65%) | 3,840/480/480 |
| TrainTicket | 4,000 | 600 s @ 1 Hz | 15 s | 1,400 (35%) | 2,600 (65%) | 3,200/400/400 |
| Total | 14,000 | — | — | 4,900 (35%) | 9,100 (65%) | 11,200/1,400/1,400 |
| Online Boutique v0.9.1 | 5,200 | 600 s @ 1 Hz | 15 s | 1,820 (35%) | 3,380 (65%) | 4,160/520/520 |
Table 3.
TimeVAE training hyperparameters and validation results.
| Hyperparameter | Value |
|---|---|
| Architecture | Temporal convolutional encoder/decoder (TCN) |
| Latent dimension | 32 |
| Encoder | 4 × TCN block (128 filters, kernel 3, dilations 1/2/4/8) |
| Decoder | 4 × TCN block (128 filters, kernel 3, dilations 8/4/2/1) |
| Batch size | 64 |
| Optimizer | AdamW (lr = 1 × 10⁻³, weight decay = 1 × 10⁻⁴) |
| LR schedule | Cosine annealing, T_max = 100 epochs |
| KL annealing | Linear warm-up 0 → 1.0 over epochs 1–20 |
| Training epochs | 120 (early-stop patience = 15 on val ELBO) |
| Final val ELBO | −0.412 ± 0.008 (mean ± SD, 5 seeds) |
| Final val reconstruction MSE | 0.0073 ± 0.0004 |
| Training hardware | NVIDIA A100 40 GB; 4 h 17 min |
Table 5.
Testbed hardware and software specification.
| Component | Specification |
|---|---|
| Cluster | 6-node Kubernetes v1.30.2 (3 control plane, 3 worker) |
| Node CPU | Intel Xeon Gold 6338 — 8 vCPU allocated |
| Node RAM | 32 GB DDR4 |
| Node storage | 500 GB NVMe SSD |
| Container runtime | containerd 1.7.18 |
| Service mesh | Istio 1.21.3 (ambient mesh) |
| Metrics | Prometheus 2.52.0 (15 s scrape interval) |
| Logging | Loki 3.0.0 |
| Tracing | Jaeger 1.57.0 (OTLP receiver) |
| Visualization | Kiali 1.86.0 |
| Chaos engine | LitmusChaos 3.10.0 |
| Application | Online Boutique v0.9.1 (12 services, 47 pods steady state) |
| Load generator | Locust 2.26.0 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
