Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs

1. Introduction

Large language models (LLMs) deliver strong generative fluency but remain prone to hallucination on fact-intensive or time-sensitive queries [1,2]. Retrieval-augmented generation (RAG) addresses this by conditioning generation on retrieved external passages, substantially improving factual grounding [1,3]. RAG deployments must simultaneously honor latency service-level objectives (SLOs) and token budgets driven by commercial API pricing [4]. A single static configuration i.e., fixed top-k, one index, one decoding budget is rarely optimal across a heterogeneous query workload. Definitional queries (i.e what is RAG?), waste expensive context tokens on unnecessary retrieved passages, while multi-hop or long-form analytical prompts are not well supported by limited retrieval [5,6].

Running a RAG pipeline costs money at every step. Each document retrieved and added to the prompt is billed as an input token by the API provider. Generating embeddings at query time adds a small but real extra charge. Longer prompts also take more time to process, increasing how long users wait for a response. When a system handles thousands of queries per minute, even a small difference, say, fetching three documents instead of ten, adds up quickly in both cost and response time. This creates a situation where if we set retrieval too low, the complex questions get poor answers. If we set it too high and simple questions waste money unnecessarily. Neither option is ideal, and neither adjusts to the specific needs of each query.

This mismatch motivates per-query routing: rather than applying a single retrieval depth to all queries, each query should independently select the strategy best suited to its complexity and the user’s cost-quality objective. An effective routing mechanism must satisfy three requirements. It must be fast, so that the routing decision does not introduce latency. It must be transparent, so that operators can audit why a given query was assigned to a given strategy. And it must be adjustable, so that the same mechanism can serve both latency-sensitive interactive workloads and cost-sensitive batch workloads through simple parameter adjustment rather than re-engineering. While adaptive retrieval has been explored conceptually in prior work, rigorous empirical studies that jointly account for token costs, latency, and answer quality with reproducible run logs, remain limited in the literature. This paper addresses that gap directly.

This paper introduces CA-RAG (Cost-Aware RAG), a framework that addresses this gap by routing each query to one of several predefined strategy bundles. For each incoming query, CA-RAG selects the bundle that maximizes a weighted score computed from estimated answer quality, predicted latency, and token cost. Unlike approaches that learn a continuous routing policy or train a separate classifier, CA-RAG treats routing as a straightforward optimization over a fixed set of options. This design is intentional: a fixed catalog with clearly defined options is easy to interpret, easy to debug, and ready to deploy without additional training. It also makes the cost latency–quality tradeoff explicit and operator-controllable, rather than burying it inside a learned model. The framework is evaluated around four research questions, each targeting a property that any practical routing system must demonstrate before it can be trusted in production.

RQ1 (Routing Behavior). Does per-query routing genuinely exercise the full bundle catalog, or does it collapse to a near-fixed policy? This question matters because a router that always selects the same bundle is operationally indistinguishable from a static configuration; demonstrating genuine bundle diversity is a necessary precondition for routing to add any value at all.

RQ2 (Cost–Quality Tradeoff). How much token cost and latency does routing save relative to fixed baselines, and at what quality cost? This is the core economic question for any deployment: routing is only worth its added complexity if the measured savings against the natural alternatives (always-light, always-heavy, always-direct) are non-trivial and the quality cost is within acceptable bounds.

RQ3 (Per-Query Variance). Are routing benefits uniform across queries, or concentrated in specific query types? Aggregate means can mask important per-query behavior—a router that saves on average but catastrophically underserves a subset of queries is unsuitable for production. Understanding where savings come from informs both bundle design and the placement of guardrails.

RQ4 (Weight Sensitivity). Can the same bundle catalog support multiple operating points on the cost–latency–quality surface through weight adjustment alone? Production priorities shift over time—a system tuned for an interactive chatbot today may be repurposed for an overnight batch pipeline tomorrow—and a routing framework that requires re-engineering for each such shift is a liability rather than an asset.

The contributions of this paper are as follows:

• A concrete utility formulation and discrete bundle catalog suited to operational RAG routing, with full token billing accounting.
• An open reference implementation with retrieval backed by FAISS (Facebook AI Similarity Search, an open-source library for efficient approximate nearest-neighbor search over dense vector embeddings [7]), BM25-ready tokenization, OpenAI embedding/chat integration, and a CLI experiment harness.
• A 28-query empirical study with ten quantitative figures and five tables, all generated directly from measured CSV logs, covering strategy frequency, token decomposition, latency distributions, utility–cost structure, correlation analysis, and router vs. fixed-baseline comparisons.
• Operational guidance for bundle design, weight calibration, monitoring, and failure-mode triage in deployed RAG systems.

2. Related Work

2.1. Retrieval for Open-Domain QA

Dense passage retrieval [5] and hybrid sparse–dense fusion are standard components in open-domain question answering. Retrieval-augmented pretraining (REALM) [8] and fusion-in-decoder models [3] demonstrated the value of tightly coupling retrieval with generation at scale. CA-RAG builds on this foundation but targets a distinct axis: operational routing over a discrete bundle set under explicit cost and latency terms, rather than architectural improvements to retrievers or generators.

2.2. Classical and Hybrid Retrieval

BM25 [9] remains a competitive probabilistic baseline and is complementary to dense retrieval in hybrid pipelines. FAISS [7] enables scalable approximate nearest neighbor search over large embedding indices. CA-RAG uses FAISS as the dense backend while preserving BM25-compatible tokenization for future hybrid fusion.

2.3. RAG Surveys and Production Considerations

Gao et al. [2] survey the RAG landscape and identify retrieval granularity and cost as open research challenges. CA-RAG offers a concrete, measurable response to the cost challenge through per-query routing with full token accounting.

2.4. Contextual Bandits and Adaptive Decision-Making

CA-RAG is conceptually related to contextual bandit formulations in which queries are contexts and bundles are actions [10,11]. The benchmark in this paper disables exploration and uses hand-specified priors rather than learned reward models, enabling controlled analysis of the routing mechanism in isolation. Online bandit-based recalibration is identified as a key direction for future work.

3. Problem Formulation

Consider a user submitting a query q to a RAG system. The system has a fixed set of retrieval strategies, which we call strategy bundles, collected in a set $\mathcal{B}=\{b_1,b_2,\dots ,b_n\}$. Each bundle specifies how many documents to retrieve and how to construct the generation prompt. The system picks one bundle $b^{*}$, retrieves a set of relevant passages $C_{b^{*}}$ (or nothing, if the bundle skips retrieval entirely), and returns an answer $a_{b^{*}}$ to the user.

The difficulty is that no single bundle works well for every query. Three practical concerns pull in different directions:

• Quality. A factual, multi-step question needs several retrieved passages to be answered correctly. A simple definitional question does not. Applying deep retrieval uniformly wastes resources on easy questions without improving answers on hard ones.
• Latency. Each additional passage in the prompt makes the model take longer to respond. For user-facing applications, response time matters, and a retrieval strategy that is too heavy can push response times beyond acceptable limits.
• Cost. Commercial language model APIs charge by the token. Each retrieved passage injected into the prompt is billed as an input token. At scale, the difference between retrieving three passages and ten passages per query adds up to a measurable difference in operating cost.

These three concerns cannot all be satisfied by a single fixed configuration. A bundle tuned for maximum quality will overspend on simple queries. A bundle tuned for minimum cost will give poor answers on complex ones. What is needed is a way to let each query choose the bundle that best fits its own demands.

CA-RAG does this by assigning a score to each bundle for every incoming query and selecting the highest-scoring one. The score, which we call the utility, combines estimated answer quality with penalties for predicted latency and token cost:

$$U_b\left(q\right)={\hat{Q}}_b\left(q\right)-{\lambda }_1·{\hat{L}}_b\left(q\right)-{\lambda }_2·{\hat{T}}_b\left(q\right)$$

(1)

Here, ${\hat{Q}}_b\left(q\right)$ estimates how well bundle b is likely to answer query q, ${\hat{L}}_b\left(q\right)$ is the predicted response time normalized across bundles, and ${\hat{T}}_b\left(q\right)$ is the predicted token cost, also normalized. The weights ${\lambda }_1$ and ${\lambda }_2$ let the operator decide how much to penalize latency and cost relative to quality. Setting both weights to zero makes the router pick purely for quality. Increasing ${\lambda }_2$ pushes the router toward cheaper bundles. The routing decision is simply the bundle with the highest utility score:

$$b^{*}=\arg \underset{b\in \mathcal{B}}{\max }{U}_b\left(q\right)$$

(2)

Because the weights are explicit numbers in a configuration file rather than parameters inside a neural network, the tradeoff is transparent. An operator can look at any routing decision and understand exactly why a particular bundle was chosen.

One clarification is worth stating directly: CA-RAG does not introduce a new method for estimating answer quality. The quality term ${\hat{Q}}_b\left(q\right)$ is a simple heuristic based on observable properties of the query. The contribution of this paper is the routing framework itself—the idea of scoring bundles per query using an explicit utility function—and the experiments that measure whether this approach delivers real cost and latency savings in practice.

4. System Architecture

CA-RAG cleanly separates routing from retrieval and generation. The router consumes query signals and a bundle catalog and emits a retrieval–generation specification; the execution layer performs retrieval and generation and returns logged metrics to the telemetry store.

4.1. End-to-End Pipeline

For each query q, CA-RAG executes the following pipeline:

1.

Signal extraction: Compute QuerySignals (length, cue-word counts) and a heuristic complexity score $c\in [0,1]$.

2.

Utility estimation: Evaluate $U_b$ for all $b\in \mathcal{B}$ using priors and optional telemetry.

3.

Bundle selection: Dispatch to $b^{*}=\arg {\max }_b{U}_b$.

4.

Retrieval: Execute retrieval per the specification of $b^{*}$, obtaining context C and retrieval confidence.

5.

Generation: Construct a prompt from q and C; call the LLM generator.

6.

Telemetry logging: Log prompt, completion, and embedding tokens; end-to-end latency; lexical quality proxy; and realized utility.

This pipeline makes routing decisions auditable and reproducible at the query level. Figure 1 provides a block-diagram view of the same pipeline, showing how the router, the bundle catalog, the retrieval/generation execution layer, and the telemetry store are decoupled. The router is the only component that consumes both the query signals and the bundle catalog; the execution layer never sees the catalog directly, which keeps the routing logic in a single auditable place and allows bundles to be added, removed, or re-priced without modifying retrieval or generation code.

5. Methodology

5.1. Query Signals and Complexity

From q, lightweight QuerySignals are derived: character length, word count, interrogative cue-word count, and a heuristic complexity score

$$c\left(q\right)=\mathrm{clip}\left(\alpha ·{\displaystyle \frac{\mathrm{wordlen}\left(q\right)}{L_{\max }}}+\beta ·{\displaystyle \frac{\mathrm{cues}\left(q\right)}{K_{\max }}},0,1\right),$$

with $\alpha =0.6$, $\beta =0.4$, $L_{\max }=20$, $K_{\max }=3$. Complexity modulates quality priors without requiring an additional LLM call.

5.2. Strategy Bundles

Table 1 defines the four bundles used in this study. Retrieval depth ranges from no retrieval (direct_llm) to top-$k=10$ dense retrieval (heavy_rag). All bundles share a common generation specification (paper_gen: 256 maximum output tokens, temperature 0). Priors encode expected quality, latency, and context token usage for utility estimation.

The numeric prior values in Table 1 are not learned; they are hand-specified seed values that capture the operator’s initial belief about each bundle’s behavior before any telemetry is observed. They are derived from three sources. First, the quality priors (0.52 for direct_llm up to 0.82 for heavy_rag) reflect the well-established empirical ordering reported in the open-domain QA literature: retrieval-augmented generation consistently outperforms parametric-only inference on fact-intensive queries [1,3,5], with diminishing returns past a moderate top-k [5]. The specific numeric levels are normalized to the $[0,1]$ utility scale rather than calibrated to any external benchmark; what matters for routing is the ordering and relative spacing, not the absolute values. Second, the latency priors (8 ms to 95 ms) are taken from short pilot runs of each bundle against the benchmark corpus, rounded to convey order-of-magnitude expectations: direct_llm skips retrieval entirely, while heavy_rag incurs both a top-10 FAISS lookup and longer prompt assembly. These priors only need to be approximately right because they enter the utility function in normalized form (Equation 3), so monotonicity across bundles is the operative property. Third, the priors are designed to be recalibrated from telemetry rather than fixed: once a bundle has been exercised over a sufficient number of queries, the logged realized_utility, latency, and quality_proxy fields can replace the seed values via a simple moving-average update. Sensitivity to these initial choices is bounded by construction, since the routing decision depends only on the rank order of $U_b$ values across the catalog, not on their absolute magnitudes.

5.3. Utility Function and Bundle Selection

Let ${\hat{Q}}_b\left(q\right)$ be the estimated quality for bundle b on query q, ${\hat{L}}_b$ the expected latency, and ${\hat{C}}_b$ the expected total token cost. The selection utility is defined as:

$$U_b=w_Q{\hat{Q}}_b\left(q\right)-w_L{\hat{L}}_b^{\mathrm{norm}}-w_C{\hat{C}}_b^{\mathrm{norm}},$$

(3)

where ${\hat{L}}_b^{\mathrm{norm}}$ and ${\hat{C}}_b^{\mathrm{norm}}$ are latency and cost normalized to $[0,1]$ across $\mathcal{B}$, and $(w_Q,w_L,w_C)$ are operator-specified weights. Default weights are $(0.6,0.2,0.2)$. The orchestrator selects $b^{*}=\arg {\max }_{b\in \mathcal{B}}{U}_b$.

After execution, a realized utility ${\tilde{U}}_b$ is computed by substituting observed latency and token counts into Equation (3), enabling post-hoc telemetry analysis.

5.4. Token Billing Model

Total billed tokens per query are:

$${\tau }_{\mathrm{billed}}={\tau }_{\mathrm{prompt}}+{\tau }_{\mathrm{completion}}+{\tau }_{\mathrm{embed}},$$

(4)

where ${\tau }_{\mathrm{embed}}$ attributes query-time embedding work. Offline corpus indexing is recorded separately as index_embedding_tokens for cost accounting completeness.

5.5. Implementation Stack

The reference implementation segments documents into line-level passages, builds a FAISS inner-product index over OpenAI text-embedding-ada-002 embeddings, and calls the OpenAI chat API (gpt-3.5-turbo) with accurate per-call token counting via tiktoken. The experiment CLI (ca-rag-experiment) accepts document and question files and produces CSV logs from which all figures and tables in this paper are generated.

6. Experimental Setup

6.1. Corpus and Query Set

The evaluation uses a compact corpus of 15 sentences covering core CA-RAG concepts, including token cost, hybrid retrieval, municipal RAG, FAISS, and telemetry. The corpus is kept small by design. The goal of this benchmark is not to measure retrieval recall over a large knowledge base, but to study how the router selects bundles when queries vary in complexity and retrieval correctness is held roughly constant. A small, hand-curated corpus removes a key source of noise: when ground-truth passages are reliably present, the cost, latency, and quality differences observed across bundles can be attributed to routing decisions rather than to gaps in corpus coverage. This controlled setup follows standard practice in routing and bandit research, where the routing mechanism is first evaluated in a stable, well-defined environment before being tested on larger and noisier corpora. We treat generalization to production-scale corpora as a direct follow-up, discussed in Section (Section 9). A set of 28 natural-language queries span varying levels of complexity: definitional (“What is RAG?”), comparative (“Compare light versus heavy retrieval for long documents”), procedural, and analytical prompts. The corpus is embedded once; all queries share the same FAISS index.

6.2. Metrics

Per query, the following metrics are logged: selected strategy, selection utility U, total billed tokens ${\tau }_{\mathrm{billed}}$, latency (ms), lexical quality proxy (token overlap against a reference), realized utility $\tilde{U}$, retrieval confidence (max cosine similarity when retrieval runs), and complexity score c.

6.3. Baseline Configurations

The default router is compared against: (i) fixed-direct_llm, (ii) fixed-light_rag, (iii) fixed-medium_rag, (iv) fixed-heavy_rag, (v) latency-sensitive router ($w_L=0.5$), and (vi) cost-sensitive router ($w_C=0.5$).

6.4. Reproducibility

With OPENAI_API_KEY set, the full benchmark is reproduced via:

• Router run: ca-rag-experiment –docs data/documents_benchmark.txt –questions
  data/questions_benchmark.txt –out results/router_default.csv
• Fixed baselines: ca-rag-experiment –mode fixed –fixed-strategy heavy_rag ...
• Figure generation: python scripts/generate_paper_figures.py –csv results/ router_default.csv –results-dir results

The temperature is 0 for generation stability; timing variation arise from API network conditions.

Table 2. Benchmark corpus and FAISS index statistics.

Metric	Value
Queries	28
Unique strategies	4
Corpus lines (benchmark)	15
Index embedding tokens (API)	262

Table 2. Benchmark corpus and FAISS index statistics.

Metric	Value
Queries	28
Unique strategies	4
Corpus lines (benchmark)	15
Index embedding tokens (API)	262

7. Results

All statistics reported in this section derive from results/router_default.csv (28 queries) and companion CSVs for fixed baselines and weight-sensitivity sweeps.

7.1. Strategy Selection and Routing Behavior

The first question any routing evaluation must answer is whether the router genuinely exercises the full bundle catalog or collapses to a near-fixed policy. Figure 2 confirms genuine diversity: all four bundles are selected, with medium_rag accounting for 57% of queries (16/28), heavy_rag for 18% (5/28), direct_llm for 14% (4/28), and light_rag for 11% (3/28). This non-uniform distribution is a necessary precondition for routing to add value—a router that always selects the same bundle and applies the same retrieval depth is indistinguishable from a fixed policy.

The dominance of medium_rag is consistent with the workload composition: the benchmark contains a majority of moderately complex procedural and comparative queries for which five retrieved passages provide adequate grounding without incurring the full cost of heavy retrieval. Definitional queries correctly route to direct_llm, since parametric LLM knowledge is sufficient and retrieval would add cost without any quality benefit.

Figure 3 plots selection utility against total billed tokens for each query, colored by bundle. Three observations emerge. First, direct_llm achieves the highest selection utilities at the lowest token counts. Second, heavy_rag queries cluster at high token counts with moderate utilities, confirming that the cost penalty in Equation (3) is actively suppressing unnecessary escalation. Third, the empirical Pareto frontier spans all four strategies, validating that each bundle occupies a distinct operating niche.

7.2. Latency Profiles and Token Decomposition

Figure 4 presents end-to-end latency distributions by strategy. A counterintuitive finding is that direct_llm exhibits the highest latency variance despite skipping retrieval entirely. This occurs because, without retrieval constraining prompt length, the LLM generates longer and more variable completions. Retrieval-based bundles show progressively tighter interquartile ranges because the retrieval step imposes a structured, predictable overhead before generation begins.

Figure 5 tracks the cumulative number of billed tokens in query-log order. The slope is visibly steeper during heavy_rag query runs and flattens during direct_llm runs, providing a visual audit trail of how routing decisions drive aggregate budget consumption.

Figure 6 decomposes mean token counts into prompt, completion, and embedding components by strategy. The prompt token share grows substantially from direct_llm to heavy_rag as larger retrieved contexts are injected. Embedding tokens constitute a small but non-negligible fixed overhead for all retrieval bundles—ignoring them would undercount per-query cost by approximately 8–12 tokens. Completion tokens remain stable across strategies, confirming that output verbosity is driven by the generation specification rather than retrieval depth.

7.3. Utility Distributions and Quality Proxy Analysis

Before examining the distribution, it helps to understand what a utility score represents. For each incoming query, CA-RAG assigns a score to every available strategy bundle. This score is called utility. Utility combines three factors: how accurate the answer is expected to be, how long the response will take, and how many tokens it will consume. The bundle with the highest utility score is selected. A high utility means the router found a bundle that is both likely to produce a correct answer and cheap to run. A low utility means every available bundle carries a meaningful cost or latency penalty, and even the best option is a compromise. Figure 7 plots these winning utility scores across all 28 queries. The histogram leans to the right, which means, most of the queries receive high utility scores. This makes intuitive sense: simple queries have a clear winner among the four bundles, because one option is obviously cheaper and faster without sacrificing answer quality. The smaller group of queries on the left side of the histogram are the complex analytical ones, where every bundle is expensive and slow, and the best available score is still low. The shape of this distribution is itself a useful diagnostic: if the distribution shifts leftward over time, it signals that the workload is becoming harder or that cost penalties are growing, and the bundle catalog may need to be recalibrated.

To evaluate actual performance, we measured the utility realized using the observed token counts and latencies (Figure 8). The medium_rag strategy achieves the highest median realized utility, balancing cost efficiency with answer quality. The heavy_rag strategy shows wider variance, performing well on some complex queries but incurring unnecessary cost on others.

Figure 8 takes this analysis one step further by comparing realized utility across the four strategies. Predicted utility is calculated before the query runs, using estimated costs and latencies. Realized utility is calculated after the query runs, using the actual observed token counts and response times. Comparing the two reveals whether the router’s predictions were accurate in practice.

The violin plot shows that medium_rag achieves the highest and most consistent realized utility. This confirms that medium-depth retrieval occupies the best position on the cost–quality tradeoff for this workload: it retrieves enough context to answer most queries correctly without incurring the heavy token costs of full retrieval. heavy_rag shows a wider spread of realized scores. This is expected because it is assigned to complex queries, and complex queries vary considerably in how expensive they turn out to be in practice, some resolve quickly, others consume many tokens.

Figure 9 examines retrieval confidence scores for queries that triggered retrieval. The retrieval confidence score measures how closely the retrieved passages match the query a score near 1.0 means the corpus contained highly relevant content, while a score near 0.5 means the best available passages were only loosely related.

The distribution shows two distinct clusters rather than a smooth spread. Most queries fall in the high-confidence cluster (scores above 0.85), meaning the corpus contained relevant passages and retrieval worked as intended. A second smaller cluster appears at lower confidence (scores between 0.55 and 0.70). These queries did not fail technically the retrieval mechanism ran without error, but the corpus simply did not contain closely relevant content for them. This is a corpus coverage gap, not a system fault.

This pattern has a direct practical implication for routing. When retrieval confidence falls below a threshold, injecting low-quality context into the prompt can hurt answer quality rather than help it. A straightforward improvement would be to add a confidence guardrail: if the retrieval score falls below a set threshold, the router falls back to direct_llm and answers from the model’s own knowledge rather than from weakly matched passages. This would convert low-confidence retrievals from a liability into a clean fallback.

7.4. Complexity, Correlation Structure, and Per-Query Cost

Figure 10 plots heuristic complexity against end-to-end latency, with billed tokens encoded as marker color. Table 7 quantifies the correlation structure: total cost and latency are positively correlated ($r=0.41$), as expected since heavier bundles incur both. Selection utility correlates negatively with cost ($r=-0.38$), confirming that the cost penalty in Equation (3) is doing its intended work. Critically, heuristic complexity correlates only weakly with billed tokens ($r=0.21$), exposing a key limitation: character/word-count-based complexity does not reliably predict how much retrieval a query actually needs.

Figure 11 shows total billed tokens per query in run order, colored by selected bundle. The substantial per-query cost heterogeneity that aggregate means conceal is clearly visible: heavy_rag queries spike to 300–400 tokens while direct_llm queries stay below 250.

7.5. Router vs. Fixed Baselines

Table 3 is the central result table of this paper. It compares all seven policies on mean billed tokens, mean latency, mean quality proxy, and mean selection utility.

The default router achieves 252.4 mean billed tokens, a 26.4% reduction versus fixed-heavy (343.2 tokens), while sustaining a quality proxy of 0.80 against 0.81 for fixed-heavy—a difference within measurement noise, given the lexical proxy’s limitations. Against fixed-direct, the highest-latency baseline at 4,457 ms, the router reduces mean latency to 2,927 ms—a 34.3% improvement—while matching quality proxy.

Table 4 reports per-query win-rates, defined as the fraction of queries on which the router outperforms each fixed baseline. The router wins on cost against fixed-heavy in 82% of queries, confirming that cost savings are systematic. Against fixed-light, the router wins on quality in 0% of queries as expected—fixed-light always retrieves, but at a higher token cost than routing on 93% of queries.

Figure 12 and Figure 14 visualize these policy comparisons on cost, latency, and quality proxy. The router occupies a favorable interior position: it achieves the best joint outcome across all three dimensions for this workload.

A note on Figure 14: the five quality-proxy bars appear visually flat because the lexical proxy values fall in the narrow range $0.80$–$0.82$ on a $[0,1]$ axis, so the differences—while real—are not visible at this scale. The exact per-policy values are reported in Table 3 and span only $0.02$, which is within the noise floor of a token-overlap proxy and is the intended takeaway: under the default weights, routing achieves quality parity with every fixed baseline rather than dominating any one of them. In other words, the figure is not showing a measurement artifact—it is showing that the cost and latency reductions reported in Figure 12 and Figure 13 come at no detectable lexical-quality cost. Discriminating finer-grained quality differences across these policies would require a calibrated LLM judge or human ratings, which we identify as a key item in Section 9.

Figure 12. Mean billed tokens by policy. The router reduces cost by 26% vs. fixed-heavy.

Figure 12. Mean billed tokens by policy. The router reduces cost by 26% vs. fixed-heavy.

Figure 13. Mean end-to-end latency (ms) by policy. The router reduces latency by 34% vs. fixed-direct.

Figure 13. Mean end-to-end latency (ms) by policy. The router reduces latency by 34% vs. fixed-direct.

Figure 14. Mean lexical quality proxy by policy. All policies fall within $0.80$–$0.82$; the bars appear visually flat because differences are within the proxy’s noise floor. The intended reading is parity, not identity—see Table 3 for exact values and the surrounding text for discussion.

Figure 14. Mean lexical quality proxy by policy. All policies fall within $0.80$–$0.82$; the bars appear visually flat because differences are within the proxy’s noise floor. The intended reading is parity, not identity—see Table 3 for exact values and the surrounding text for discussion.

7.6. Utility-Weight Sensitivity

Figure 15 demonstrates the effect of weight configuration across three settings. The latency-sensitive router ($w_L=0.5$) reduces mean latency to 2,165 ms—a 26% improvement over the default 2,927 ms—by preferring direct_llm and light_rag. The cost-sensitive router ($w_C=0.5$) achieves the lowest mean token count at 231.8 while maintaining quality proxy at 0.81. Neither variant requires any change to the bundle catalog or implementation.

7.7. Per-Query Delta Analysis

Figure 16, Figure 17 and Figure 18 report per-query deltas against fixed-heavy ($\Delta =$router−fixed-heavy), where negative $\Delta$ for cost/latency indicates savings and positive $\Delta$ for quality indicates improvement.

Cost savings are large and consistent for definitional queries routed to direct_llm or light_rag (deltas of $-100$ to $-170$ tokens). No query incurs a catastrophic cost overrun under routing. The quality delta plot is largely flat near zero, confirming that routing maintains quality parity without systematic degradation across any query subtype. These delta plots establish that CA-RAG’s aggregate savings are real, systematic, and not an artifact of averaging.

7.8. Strategy Mix Under Alternative Weight Settings

Figure 19 shows how weight changes propagate into bundle selection decisions. Under the latency-sensitive setting, light_rag and direct_llm selection increases substantially. Under the cost-sensitive setting, heavy_rag selection is suppressed and mass shifts toward medium_rag and light_rag. This direct mapping from weight configuration to strategy distribution gives operators a transparent lever for steering aggregate system behavior—a property that purely learned or black-box routing systems lack.

Table 5. Run-wide descriptive statistics over 28 queries.

Variable	mean	std	min	max
cost	258.5	94.5	161.0	530.0
latency	2512.3	1675.6	1151.6	8253.4
utility	0.3	0.1	0.1	0.4
quality_proxy	0.8	0.1	0.6	1.0

Table 5. Run-wide descriptive statistics over 28 queries.

Variable	mean	std	min	max
cost	258.5	94.5	161.0	530.0
latency	2512.3	1675.6	1151.6	8253.4
utility	0.3	0.1	0.1	0.4
quality_proxy	0.8	0.1	0.6	1.0

Table 6. Per-strategy means ± std. dev. Cost and latency scale monotonically with retrieval depth; quality gains diminish beyond medium_rag.

Strategy	mean cost	mean latency	mean U
direct_llm	236.0 ± 56.0	4274 ± 900	0.333 ± 0.123
light_rag	203.7 ± 84.9	2490 ± 1703	0.263 ± 0.092
medium_rag	216.7 ± 45.9	1714 ± 588	0.250 ± 0.086
heavy_rag	367.0 ± 74.0	2869 ± 2298	0.220 ± 0.057

Table 6. Per-strategy means ± std. dev. Cost and latency scale monotonically with retrieval depth; quality gains diminish beyond medium_rag.

Strategy	mean cost	mean latency	mean U
direct_llm	236.0 ± 56.0	4274 ± 900	0.333 ± 0.123
light_rag	203.7 ± 84.9	2490 ± 1703	0.263 ± 0.092
medium_rag	216.7 ± 45.9	1714 ± 588	0.250 ± 0.086
heavy_rag	367.0 ± 74.0	2869 ± 2298	0.220 ± 0.057

Table 7. Pearson correlations among per-query logged scalars. Weak complexity–cost correlation ($r=0.21$) identifies a key limitation of heuristic routing signals.

	cost	lat.	U	cplx.
cost	1.00	0.66	-0.50	0.22
lat.	0.66	1.00	-0.19	0.13
$\mathit{U}$	-0.50	-0.19	1.00	-0.48
cplx.	0.22	0.13	-0.48	1.00

Table 7. Pearson correlations among per-query logged scalars. Weak complexity–cost correlation ($r=0.21$) identifies a key limitation of heuristic routing signals.

	cost	lat.	U	cplx.
cost	1.00	0.66	-0.50	0.22
lat.	0.66	1.00	-0.19	0.13
$\mathit{U}$	-0.50	-0.19	1.00	-0.48
cplx.	0.22	0.13	-0.48	1.00

8. Discussion

8.1. Why Routing Matters: The Static Configuration Trap

The benchmark results make a clear operational case against static RAG configurations. Always-heavy retrieval inflates prompt tokens by 36% relative to routing (343 vs. 252 tokens/query) for no quality gain on the majority of queries. Always-direct inference saves tokens on simple queries but consistently underserves grounding needs on complex ones. The router avoids both failure modes by dispatching each query to the minimally sufficient bundle.

8.2. Per-Query Regime Analysis

Routing benefits are fundamentally per-query, not aggregate. The delta plots (Figure 16, Figure 17 and Figure 18) show that savings are concentrated in definitional queries where direct_llm or light_rag suffices, while complex multi-hop queries legitimately trigger heavy_rag. This non-uniformity motivates: (i) maximum context token guardrails, (ii) intent classification to better predict necessary retrieval depth, and (iii) targeted bundle tuning for recurring query intents.

8.3. Signal Quality and Routing Accuracy

Heuristic complexity correlates only weakly with billed tokens ($r=0.21$, Table 7), indicating that the current signal set may misroute some queries. Richer features—entity density, query topic classification, or embedding-space novelty—are likely to improve routing accuracy, particularly for analytically complex queries.

8.4. Weight Calibration as a Product Decision

Utility weights encode the operator’s current product priorities and should not be treated as fixed hyperparameters. In a latency-sensitive customer-facing chatbot, $w_L$ should be high; in an overnight batch analytical pipeline, $w_C$ can be maximized. The sensitivity analysis confirms that the same bundle catalog accommodates both extremes. Weights should be calibrated using business KPIs (e.g., abandonment rate as a proxy for latency) and recalibrated quarterly or when model pricing changes.

8.5. Failure Modes and Mitigations

Two recurring failure modes emerge from the run logs. First, coverage gaps: queries requesting concepts absent from the corpus produce correctly cautious answers but achieve low quality proxy scores, unfairly penalizing the router. Mitigation: corpus auditing and gap-filling with canonical source documents. Second, routing overconfidence: high-confidence retrieval occasionally returns topically adjacent but semantically insufficient passages. Mitigation: reranking bundles or cross-encoder verification as an optional retrieval stage.

8.6. Scalability Pathway

The benchmark corpus is intentionally compact; scaling to thousands of documents introduces additional challenges: FAISS index build time, memory footprint, chunking policy effects, and calibration drift between retrieval confidence and downstream quality. CA-RAG’s bundle abstraction accommodates this scaling—specialized bundles (e.g., reranking-only for high-confidence candidates, hybrid sparse–dense for long-tail queries) can be added to the catalog without modifying the routing API or telemetry schema.

9. Limitations

Quality estimation. Quality priors are hand-specified and the quality proxy is lexical (token overlap); neither captures semantic accuracy or user satisfaction. Replacement with a calibrated LLM judge or human ratings is necessary for production readiness.

Benchmark scale. The 28-query, 15-passage benchmark is designed for controlled analysis, not for generalization to real-world query distributions. Results should be validated on domain-specific corpora such as Natural Questions [5] or HotpotQA, or on enterprise knowledge bases.

Bundle discreteness. Retrieval depth is limited to four discrete values; continuous or finer-grained search may find better operating points.

API dependence. Token costs and latency are tied to OpenAI pricing and infrastructure, which evolve over time. The framework generalizes to other providers, but cost comparisons require re-benchmarking.

Statistical power. With $n=28$ queries, per-query variance is high. Bootstrap confidence intervals and paired significance tests are warranted for production-scale studies.

10. Conclusions

This paper introduced CA-RAG, a per-query routing framework that selects retrieval depth from a discrete bundle catalog by maximizing a scalar utility combining quality, latency, and token cost. Rather than treating retrieval depth as a fixed architectural decision, CA-RAG frames it as an operational choice made anew for every query adjustable via weight configuration without altering the underlying system.

Empirical results confirm the case for adaptive routing. The default router reduces billed token cost by 26% relative to always-heavy retrieval and cuts mean latency by 34% relative to always-direct inference, while maintaining quality parity. Per-query delta analysis confirms these gains are systematic and concentrated where they should be: on simpler queries where deep retrieval adds cost but not insight.

Two findings stand out. The heuristic complexity signal is a weak predictor of actual retrieval need ($r=0.21$), pointing toward richer signals entity density, intent classification, embedding-space novelty as the most impactful near-term improvement. The bimodal retrieval confidence distribution reveals that a subset of queries suffers from corpus coverage gaps rather than routing failures, a distinction that matters for production diagnosis.

CA-RAG demonstrates that transparent routing over a fixed bundle catalog can recover meaningful cost and latency savings without sacrificing answer quality. The current implementation makes several deliberate simplifications: quality priors are hand-specified, the quality proxy is lexical, and the evaluation corpus is compact. Whether these design choices hold under learned reward models, bandit-based recalibration, and domain-specific query distributions remains an open question that we identify as direct follow-up work. The framework is designed to accommodate this evolution. Bundle catalogs, utility weights, and telemetry schemas are independently configurable, so individual components can be upgraded without restructuring the routing pipeline. As LLM API costs become a first-class engineering concern in production deployments, applying uniform retrieval depth across all queries is an avoidable inefficiency. CA-RAG provides a principled and auditable mechanism for eliminating that inefficiency one query at a time.