A Time-Resolved, SLO-Aware and Bi-Objective Framework to Measure and Minimize LLM Serving’s Carbon and Water Footprints

1. Introduction

The deployment of large language models on scale has reshaped the sustainability discussion for artificial intelligence [1,2]. Early work focused almost exclusively on the training phase because one-off runs for frontier models were shown to consume on the order of thousands of megawatt-hours, emit hundreds of tons of CO₂ equivalent, and use large volumes of cooling water for a single training job [3,4,5]. Those numbers rightly drew attention and motivated research on efficient architectures and carbon-aware training [6,7]. The landscape has since changed. Public adoption has turned inference into a continuous and interactive service that runs every minute of the day. The cumulative footprint of serving billions of queries now dominates the life-cycle of many deployments and can exceed training by a wide margin [5,8]. Several analyzes estimate that inference can account for up to 90% of the lifetime energy and that the annual operating energy of the large-scale service can be tens of times higher than the energy used to train the model [5,9,10]. A single short prompt appears small when measured in watt-hours, yet at hundreds of millions of prompts per day, the aggregate demand reaches utility-scale electricity and meaningful volumes of water [5]. It follows that sustainability efforts that target training alone are no longer sufficient.

Accurate accounting for inference is challenging, and the literature has only recently converged on methods that match production practice [11]. Early estimates combined accelerator nameplate power, theoretical FLOPs, and assumed token lengths, producing per-prompt energies that differed by an order of magnitude because each study chose different utilization factors and boundaries [5,12,13]. Empirical instrumentation is now emerging through industry studies [1]. A key result from this line of work is that narrow measurement—accelerator only—misses material contributors such as host CPU and DRAM, provisioned idle, and facility overhead captured by power usage effectiveness. Google’s recent production study reports medians at a comprehensive serving boundary and shows that accelerator-only accounting can underestimate per-prompt energy by more than a factor of two for the same workload [1]. This boundary choice explains much of the spread in earlier per-prompt claims and makes clear that scope-transparent reporting is a prerequisite for credible comparison and for effective optimization.

Water has been even less visible in model reporting despite its central role in data center operations [3,5,14]. Cooling systems withdraw and consume water on site, and electricity generation for computing carries a significant upstream water intensity [15]. Model cards and environmental summaries often report scope 2 carbon while omitting water entirely [16,17]. The omission matters because the water intensity of electricity and the carbon intensity of the grid vary by region and time, and they are only weakly correlated. A policy that pursues the lowest carbon intensity without regard to water can inadvertently increase total water consumption, for example by shifting the load to nuclear-heavy or hydro-dominated regions where the liters per kilowatt-hour are higher, while the converse can also occur [3,14]. Recent studies document that carbon and water can move in opposite directions, which means sustainability must be treated as a bi-objective problem rather than a single number to be minimized [14].

Any practical solution must also respect the realities of an interactive service [18]. LLM serving is governed by strict service level objectives for responsiveness. Time to first token and time per output token directly shape perceived latency and throughput [17]. Production systems cannot delay user requests to wait for a greener grid, nor can they indiscriminately route traffic to distant regions if that violates latency budgets. The optimization space is therefore bounded by performance constraints and by the capacity of the serving fleet. A sustainability framework that ignores these constraints does not translate into production.

This paper proposes a deployment-aware framework that unifies measurement and control. The approach is provider-agnostic, scope transparent by construction, and explicitly bi-objective in carbon and water. Measurement follows production practice and reports daily medians at a comprehensive serving boundary that includes active accelerators, host CPU and DRAM, provisioned idle, and PUE uplift [1]. Consumptive water is computed as the site plus the source using $W=\mathrm{PUE}·{\mathrm{WUE}}_{\mathrm{site}}+\mathrm{EWIF}$ [3]. Carbon is location based by default with a market-based sensitivity when portfolio factors are disclosed. Optimization casts serving orchestration as a mixed-integer linear program solved over 288 five-minute windows per day. For each prompt profile the solver chooses region, batch size, and a phase-aware hardware assignment that separates prefill and decode. The formulation enforces $p95$ constraints on time to first token and time per output token, augmented by a per-region network term. Capacity constraints derive from measured tokens-per-second tables. Token-length directives are modeled as controllable reductions in decode work.

Our contributions are fourfold. First, we provide a full-stack, scope-transparent accounting for inference that reproduces the production gap between accelerator-only and comprehensive boundaries and supplies a defensible translation between the two, with a narrow-to-comprehensive median ratio of about 0.417 for the workloads we study [1]. Second, we formulate a time-resolved, bi-objective optimizer that co-minimizes location-based carbon and consumptive water under explicit $p95$ SLOs and fleet capacity. Third, we demonstrate the framework on four representative models using public per-prompt energy tables and provider PUE, WUE, and carbon factors, and we show that a single SLO-aware policy reduces comprehensive medians by roughly 57–59% for energy, 59–60% for water, and 78–80% for location-based CO₂ while meeting latency and throughput targets in every window. Fourth, we analyze routing-only and joint routing+batch+token frontiers to make the carbon–water geometry explicit, which tells operators when improvements move both metrics together and when trade-offs must be managed.

The remainder of the paper is organized as follows. Section 2 surveys the literature and motivates comprehensive boundaries, water-aware metrics, and SLO-respecting orchestration. Section 3 details the methodology, including the functional unit, scope definitions, impact accounting, time resolution, decision variables, constraints, and parameterization. Section 4 presents the results, reporting comprehensive-boundary medians and daily totals, scope reconciliation between accelerator-only and comprehensive views, and carbon–water movement at fixed quality of service with routing, batch, token, and phase-split analyses. Section 5 discusses implications, deployment guidance, and limitations, and outlines directions for future work. Section 6 concludes the paper.

2. Optimizing the Environmental Footprint of LLM Inference: A Literature Review

2.1. From Training to Inference: Why the Burden Has Shifted

Early work on AI sustainability emphasized training, where single jobs for frontier models consumed thousands of MWh and emitted hundreds of tons of CO₂e while using significant cooling water [3,4,5,6,7]. As generative systems moved into daily use, the continuous nature of serving billions of prompts has become the dominant lifecycle driver for many deployments [5,8,9,10,19,20,21]. Per-request impacts that seem small in isolation compound at scale: a single ChatGPT-style prompt has been estimated at several grams CO₂e, far above a web search [22], and daily volumes in the hundreds of millions translate to utility-scale electricity and meaningful water withdrawals [5]. These observations motivate methods that measure inference accurately and optimize it under interactive quality constraints.

2.2. Full Stack per Prompt Accounting: Energy, Carbon, Water, and Embodied Impacts

Production studies recommend a comprehensive serving boundary that includes the active accelerator, host CPU and DRAM, provisioned idle, and facility overhead via PUE [1,23,24]. Under this boundary, “accelerator only” views can undercount by more than a factor of two. The Google study reports a narrow-to-comprehensive median ratio near $0.417$ for similar workloads [1]. Carbon per prompt should be reported both location based (grid average) and, when portfolios are disclosed, market based [25], and it should include amortized embodied impacts from hardware manufacturing where possible [26,27,28]. Water deserves first-class treatment: consumptive site cooling scaled by PUE plus off-site electricity–water intensity (EWIF) yields a scope-consistent measure of liters per kWh that varies widely by region and generation mix [3,29,30,31,32,33]. Because carbon intensity (CIF) and EWIF are only weakly correlated, any realistic framework should measure both and co-optimize them [3,14].

2.3. Measurement Boundaries: Why Scope Transparency Matters

Inconsistent boundaries explain much of the order-of-magnitude spread in earlier per-prompt estimates. Studies that measured only chip power (sometimes at unrealistic utilization) omitted host, idle, and cooling overheads and thus overstated efficiency [5,34,35]. Converging practice now recommends a comprehensive inference boundary under operator control, with explicit exclusions outside that boundary (e.g., end user devices or wide-area network transit) and with scope-transparent reporting so results are comparable across systems [1,24]. The ≈0.417 narrow-to-comprehensive ratio provides a defensible translation when only accelerator-level numbers are available and site factors and mixes are held constant [1]. Our methods section adopts this convention and reports daily medians to handle skewed mixes.

2.4. Real-Time Orchestration: Carbon- and Water-Aware Routing Under SLOs

With consistent measurement in place, the next step is optimization. Carbon-aware request routing has been shown to cut serving emissions substantially without violating latency when traffic is steered to cleaner grids in space and time [10,36,37,38,39]. Projections suggest even larger reductions as grids decarbonize and capacity headroom grows [10]. Emerging frameworks treat scheduling as a multi-objective problem, co-optimizing carbon, water, and cost while enforcing latency SLOs and site capacity constraints; practical solutions range from MILPs to learning-based controllers [9,40,41,42,43]. Our

2.5. Phase-Aware Hardware Scheduling (Prefill vs. Decode)

Transformer serving has two phases with different bottlenecks. Prefill is parallel and compute-heavy. Decode is sequential and often memory-bound [44,45]. Phase-aware scheduling assigns prefill and decode to different hardware or configurations to improve both responsiveness and efficiency [1,46,47,48]. Speculative decoding and KV-cache reuse further reduce decode work [49,50]. Our formulation exposes prefill and decode decisions and allows second-life hardware to serve decode when SLOs permit [22,51].

2.6. Semantic-Level Interventions

Beyond systems levers, generation can be made more efficient with semantic directives that reduce unnecessary output length or avoid high-compute behaviors while preserving usefulness. SPROUT exemplifies this approach, reporting large carbon savings with minimal impact on quality by selectively applying concise directives [52]. Related work on eco-adaptive services explores small, user-acceptable adjustments during high-carbon periods to achieve large operational reductions [21]. These methods complement, rather than replace, hardware and orchestration optimizations.

2.7. Lifecycle and Circular Economy Strategies

Holistic sustainability includes upstream manufacturing and end of life. Hardware production for AI accelerators is energy- and material-intensive; embodied emissions can dominate in low-carbon operational settings [53,54]. Scenario analyses warn of rapid growth in AI-related e-waste, and point to device life extension, refurbishment, parts harvesting, and improved recycling as the most effective mitigations [55]. Second-life deployment and modular upgrades align with extended producer responsibility and reduce both embodied and disposal impacts [1,22,55].

2.8. Toward a Unified, Deployment-Aware Framework

Across the literature a consensus is emerging: inference now dominates impacts [5,8,9,10,19,20,21]; scope-transparent, full-stack per-prompt accounting is essential [1,23,24,25]; optimization must be bi-objective in carbon and water and must observe latency SLOs [3,9,10,14,36,37,38,39]; and lifecycle thinking is required to avoid shifting costs upstream or downstream [22,53,54,55]. Persistent gaps remain—especially inconsistent boundaries and limited treatment of water. The present work addresses these gaps by (i) adopting comprehensive, production-style measurement with explicit narrow↔comprehensive translation; (ii) integrating EWIF alongside CIF so water is optimized jointly with carbon; (iii) enforcing $p95$ TTFT/TPOT and capacity constraints so results are deployable; and (iv) exposing phase-aware and semantic levers that reduce watt-hours per prompt while maintaining interactive quality. Together, these elements provide a practical path for operators to measure, compare, and materially lower the environmental footprint of LLM serving.

3. Methodology

The aim of the methodology is to turn non-comparable measurements of LLM inference into one portable, auditable workflow that: (i) reports apples-to-apples per-prompt medians under a comprehensive serving boundary consistent with production practice and, when needed for comparison, under a narrower accelerator-only boundary; (ii) enforces real SLOs through explicit capacity and latency constraints derived from public tokens-per-second and latency quantiles; and (iii) co-optimizes energy, location-based (LB) greenhouse-gas emissions, consumptive water, and amortized embodied impacts through a mixed-integer linear program (MILP). The comprehensive boundary follows the decomposition implemented in a recent first-party production study for Gemini Apps, which measures the median text prompt at $0.24$ Wh (comprehensive) with an accelerator-only median of $\approx 0.10$ Wh when active accelerator, host CPU/DRAM, provisioned idle, and facility overhead (PUE) are aggregated [1]. These medians imply an uplift of $\approx 2.4\times$ due purely to boundary choice (comprehensive vs. accelerator-only). We adopt that full-stack paradigm for comparability and to avoid systematic undercounting identified by the production study [1].

3.1. Functional Unit and System Boundaries

The functional unit is one served prompt. We compute energy per prompt in watt-hours at two nested system boundaries so that our results are comparable to both “narrow” and “comprehensive” studies [1]. Accelerator-only boundary counts only the active accelerator energy directly attributable to the prompt. Comprehensive serving boundary adds host CPU/DRAM, provisioned idle capacity, and facility overhead captured by PUE. We adopt this boundary as our default because it aligns with production telemetry and produces apples-to-apples medians across models and regions [1].

Let $E^{\mathrm{active}}$ denote accelerator-only energy per prompt (i.e., the GPU/TPU compute energy for the prefill and decode phases). It excludes host CPU/DRAM energy, provisioned-but-idle energy, and any facility overhead captured by $\mathrm{PUE}$; $E^{\mathrm{IT}}$ the IT energy per prompt (accelerators + host CPU/DRAM + provisioned idle); $E^{\mathrm{host}}$ and $E^{\mathrm{idle}}$ the per-prompt energy used on non-accelerator hardware that is part of serving the request: host CPUs, DRAM, NICs, disks on the inference nodes (and sometimes service sidecars); ${\kappa }_{\mathrm{host}+\mathrm{idle}}$ the ratio of total IT-side energy (accelerator active + host + idle) to the accelerator’s active energy:

$${\kappa }_{\mathrm{host}+\mathrm{idle}}={\displaystyle \frac{E^{\mathrm{IT}}}{E^{\mathrm{active}}}}={\displaystyle \frac{E^{\mathrm{active}}+E^{\mathrm{host}}+E^{\mathrm{idle}}}{E^{\mathrm{active}}}}.$$

(1)

To be numerically consistent with the reported medians ($E^{\mathrm{fac}}\approx 0.24$ Wh and $E^{\mathrm{active}}\approx 0.10$ Wh [1]), we set $E^{\mathrm{IT}}=E^{\mathrm{fac}}/\mathrm{PUE}\approx 0.24/1.09\approx 0.220\mathrm{Wh}$ and thus obtain ${\kappa }_{\mathrm{host}+\mathrm{idle}}\approx E^{\mathrm{IT}}/E^{\mathrm{active}}\approx 0.220/0.10\approx 2.20.$ When host/idle shares are undisclosed, we use this aggregated ${\kappa }_{\mathrm{host}+\mathrm{idle}}$ rather than point estimates for $E^{\mathrm{host}}$ and $E^{\mathrm{idle}}$.

We define: $E^{\mathrm{fac}}=\mathrm{PUE}\times E^{\mathrm{IT}}$ the facility energy per prompt (what the whole data center expends to deliver the prompt), with $\mathrm{PUE}\approx 1.09$ we have $E^{\mathrm{fac}}\approx 0.24$ Wh per text prompt. This matches the comprehensive median found in [1] and implies a $\sim 2.4\times$ uplift over the accelerator-only median ($\sim 0.10$ Wh) for the same workload.The overhead energy is exactly the extra above IT:

$$E^{\mathrm{overhead}}=(\mathrm{PUE}-1)\times E^{\mathrm{IT}}\approx 0.02\mathrm{Wh}.$$

3.2. Impact Accounting

We compute operational emissions using the location-based (LB) approach recommended for data-center Scope-2 reporting [1]. The facility energy associated with serving one prompt, $E^{\mathrm{fac}}$ (kWh prompt⁻¹), is multiplied with location-based grid carbon intensity for region r, ${\mathrm{CIF}}_r^{\mathrm{LB}}$ (kgCO₂kWh⁻¹). This yields the LB, per-prompt operational emissions:

$$g{\mathrm{CO}}_2^{\mathrm{LB}}=E^{\mathrm{fac}}\times {\mathrm{CIF}}_r^{\mathrm{LB}}\times {10}^3.$$

(2)

LB attributes emissions to the electricity actually drawn from the local grid at the point of consumption and is the basis for statistically comparable across operators and regions, aligning with routing-aware decisions, and avoids year-to-year swings caused by portfolio accounting. For the sensitivity we also compute the provider’s prior-year portfolio emission factor $E{F}_r^{\mathrm{MB}}$ (kgCO₂e kWh⁻¹) [25]. MB, when disclosed, can be reported as a sensitivity and makes results operationally interpretable while retaining comparability. We use LB as the objective default and MB as a sensitivity. Operational emissions are reported in g CO₂ (LB); g CO₂e is reserved for MB factors and embodied/lifecycle terms.

For water, we follow the scope-consistent “site + source” rule advocated in the AI water-footprint literature [3]. Consumptive site water (cooling, at the facility) is proportional to facility energy via the site Water Usage Effectiveness ${\mathrm{WUE}}_r^{\mathrm{site}}$ (LkWh⁻¹) while source water accounts for the electricity-generation water intensity of the regional grid via the Electricity-Water Intensity Factor ${\mathrm{EWIF}}_r^{\mathrm{source}}$ (LkWh⁻¹), separating site water (cooling; WUE Category 2) from source water associated with electricity generation (EWIF), as recommended in the water-footprint literature [3]. Using $E^{\mathrm{IT}}[kWh/prompt]$ for the electrical work of the IT load, the per-prompt consumptive water is:

$$m{L}_r=[E^{\mathrm{IT}}\times \left({\mathrm{PUE}}_r\times {\mathrm{WUE}}_r^{\mathrm{site}}\right)+E^{\mathrm{IT}}\times {\mathrm{EWIF}}_r^{\mathrm{source}}]\times {10}^3.$$

(3)

This form makes explicit that site water scales with the facility energy (hence the $\mathrm{PUE}$ multiplier), while source water scales with the electricity used to power the IT equipment and is governed by the grid mix. Analysis emphasizes both the need to include scope-2 water and the empirical weak correlation between carbon and water intensities [3], motivating dual-metric reporting and optimization. The macro background on facility-level $\mathrm{PUE}$ and $\mathrm{WUE}$ levels across the U.S. fleet is taken from the 2024 U.S. Data Center Energy Usage Report, which shows hyperscale-weighted averages below ∼1.4 $\mathrm{PUE}$ and site $\mathrm{WUE}$ around $0.32$–$0.40$ LkWh⁻¹ in 2023 (with a likely rise as liquid cooling penetrates), giving realistic bounds for sensitivity tests [56].

We summarize per-prompt footprints (Wh, mL, gCO₂e) as daily medians by profile and season, mirroring the production study’s choice of medians as the right statistic for skewed mixes; that study reports comprehensive-boundary medians near $0.24$ Wh, $0.03$ gCO₂e, and $0.26$ mL for the median text prompt and documents the $\sim 2.4\times$ uplift from accelerator-only to comprehensive accounting. Our framework reproduces this uplift when we apply ${\kappa }_{\mathrm{host}+\mathrm{idle}}$ and $\mathrm{PUE}$ to chip-only numbers, providing scope-transparent comparability [1].

We include embodied impacts and e-waste on a serving-time basis. For accelerator class h with embodied carbon $C_h^{\mathrm{emb}}$ (kgCO₂e board⁻¹), board mass $M_h^{\mathrm{board}}$ (g), and assumed service lifetime $L_h$ (days), we compute daily rates $C_h^{\mathrm{emb}}/L_h$ (kgCO₂e day⁻¹) and $M_h^{\mathrm{board}}/L_h$ (g day⁻¹). When a class is activated (binary variable in the solver), we allocate its daily embodied rates to the prompts it serves that day, adding an amortized per-prompt embodied term that brings hardware lifecycle into scope. This aligns with the lifecycle scope of the production study [1,22].

3.3. Time Resolution, Traffic Mix and SLOs

We partition a day into $\left|T\right|=288$ decision windows of five minutes each with budget $\Delta t=300$ s. Demand is described by three prompt profiles $p\in P=\{\mathit{short},\mathit{medium},\mathit{long}\}$ with token pairs $(T_p^{\mathrm{in}},T_p^{\mathrm{out}})$ equal to $(100, 300)$, $(1000, 1000)$, and $(10,000, 15,000)$. Unless noted, arrivals total $N_{\mathrm{day}}=500$ M prompts per day with a $70/25/5\%$ mix. Windowed arrivals $Q_{t,p}$ are either provided or constructed from $N_{\mathrm{day}}$, the mix ${\pi }_p$, and diurnal weights $f\left(t\right)$ as $Q_{t,p}=\mathrm{round}\left(N_{\mathrm{day}}{\pi }_{p}f\left(t\right)/{\sum }_{t^{\prime }}f\left(t^{\prime }\right)\right)$.

The router chooses non-negative integer assignments $x_{t,r,h,b,\phi ,p,d}$ by region r, hardware class h, batch size b, phase $\phi \in \{\mathit{prefill},\mathit{decode}\}$, and directive $d\in \{\mathit{default},\mathit{brief}\}$. Prefill and decode are two legs of the same request, so conservation holds per window and profile as in Equation (6). Capacity in a five-minute window is enforced with quantile throughputs ${\mathrm{TPS}}_{h,b}\left(q\right)$, which bound the total tokens processed per provisioned node at batch b; Equation (7) applies this limit with the window budget $\Delta t$.

Latency SLOs use profile-specific $p95$ targets. We work with the published per-hardware, per-batch quantiles for TTFT and TPOT and the associated $\mathrm{TPS}$ rows. The formulation imposes the tail-latency guard of Equation (8) at $p=95$ and uses the same $p95$ throughputs in the capacity constraint so feasibility reflects the stochastic nature of serving. Unless otherwise stated, reported summaries are daily medians by profile and a $70/25/5$ mix of those medians, which is consistent with production practice for skewed mixes and utilization patterns [1].

3.4. Decision Variables, objEctive and Constraints

The decision variables are the assignment of prompts $x_{t,r,h,b,\phi ,p,d}$ in each window (non-negative integers), binary activation variables $u_{t,r,h,b}\in \{0, 1\}$ for hardware classes at a site, the chosen batch size b from a small discrete set per hardware, binary indicators $z_{r,h}\in \{0, 1\}$ that switch on embodied amortization when a hardware class is used and $n_{t,r,h,b}\in {\mathbb{Z}}_{\ge 0}$: the number of replicas provisioned at $(r,h,b)$ in window t. Parameters include the per-site replica cap $U_{r,h,b}\in {\mathbb{Z}}_{\ge 0}$ and the big-M constants $M_{h,b,\phi ,p,d}$ and $M_{h,b,p,d}^{\prime }$ used in activation/feasibility linkages.

Each prompt of profile p consists of $T_p^{\mathrm{in}}$ input tokens in the prefill phase and $T_p^{\mathrm{out}}$ output tokens in the decode phase. Decode tokens are scaled by the directive factor ${\alpha }_d$, which can take values such as $0.70$, $0.75$, or $0.80$ depending on whether the output is shortened. The number of tokens served by a routed block is therefore given by:

$${\tau }_{r,h,b,\phi ,p,d}=\left\{\begin{array}{cc}{T}_p^{\mathrm{in}},\hfill & \phi =\mathit{prefill},\hfill \\ {\alpha }_d{T}_p^{\mathrm{out}},\hfill & \phi =\mathit{decode}.\hfill \end{array}\right.$$

(4)

Throughput is modeled as ${\mathrm{TPS}}_{h,b}\left(q\right)$, the number of tokens per second at quantile q for hardware class h under batch size b. Latency SLOs are profile-specific constants $L_p^{★}$, for example $0.9$ s for “short” prompts.

We aggregate to daily medians per profile and mix by 70/25/5. The objective is a weighted-sum scalarization with non-negative policy weights:

$$\underset{x,u,z,n}{min}\sum _{t,r,h,b,\phi ,p,d}\left(\alpha E_{t,r,h,b,\phi ,p,d}^{\mathrm{IT}}+\beta m{L}_{t,r,h,b,\phi ,p,d}+\lambda {\mathrm{gCO}}_2^{\mathrm{LB}}t,r,h,b,\phi ,p,d\right)+\delta {\mathrm{CO}}_2^{\mathrm{emb}}\mathrm{day}+\epsilon W_{\mathrm{day}}^{\mathrm{ewaste}},$$

(5)

where minimization is over the decision variables $x_{t,r,h,b,\phi ,p,d},u_{t,r,h,b},z_{r,h}$. The weights $\alpha ,\beta ,\lambda ,\delta ,\epsilon \ge 0$ are chosen by the operator/policy. Equation (5) mirrors multi-impact scoring used in recent infrastructure-aware benchmarking (energy, water, carbon) while adding embodied and e-waste terms for life-cycle completeness [5].

Prefill and decode are two legs of the same request; counts must match and meet demand. The first constraint to satisfy this, a linear one, is:

$$\sum _{r,h,b,d}{x}_{t,r,h,b,\mathit{prefill},p,d}=\sum _{r,h,b,d}{x}_{t,r,h,b,\mathit{decode},p,d}=Q_{t,p}.$$

(6)

As we cannot push more tokens through a node in five minutes than the node can handle at the selected batch and the specified throughput quantile, we impose the second constraint, capacity using quantile throughput from the benchmark (per hardware/batch):

$$\sum _{p,d}\left({\displaystyle \frac{x_{t,r,h,b,\mathit{prefill},p,d}{T}_p^{\mathrm{in}}}{{\mathrm{TPS}}_{h,b}\left(q\right)}}+{\displaystyle \frac{x_{t,r,h,b,\mathit{decode},p,d}{\alpha }_d{T}_p^{\mathrm{out}}}{{\mathrm{TPS}}_{h,b}\left(q\right)}}\right)\le \Delta t{n}_{t,r,h,b}\forall t,r,h,b,$$

(7)

where $\Delta t=300$ s is the five-minute budget and $q=95$ unless noted. We enforce $0\le n_{t,r,h,b}\le U_{r,h,b}{u}_{t,r,h,b}$ and $z_{r,h}\ge u_{t,r,h,b}$ for all $(t,r,h,b)$, and gate assignments by $x_{t,r,h,b,\varphi ,p,d}\le M_{h,b,\varphi ,p,d}{u}_{t,r,h,b}$ for all $(t,r,h,b,\varphi ,p,d)$.

The third constraint is tail latency controlled by bounding the $p95$ response time per profile to its SLO $L_p^{★}$:

$${\mathrm{TTFT}}_{h,b}\left(95\right)+{\alpha }_d{T}_p^{\mathrm{out}}\times {\mathrm{TPOT}}_{h,b}\left(95\right)\le L_p^{★},{\mathrm{TPOT}}_{h,b}\left(95\right)={\displaystyle \frac{1}{{\mathrm{TPS}}_{h,b}\left(95\right)}}.$$

(8)

To evaluate the objective, we expand each impact term at the level of a single routing assignment $(t,r,h,b,\phi ,p,d)$. We first compute accelerator-only energy (Wh), after which the solver chooses an assignment count. In each 5-minute window t:

$$E_{t,r,h,b,\phi ,p,d}^{\mathrm{acc}}=x_{t,r,h,b,\phi ,p,d}\times e^{\mathrm{Wh}/\mathrm{token}}(h,\phi ,b;q)\times {\tau }_{r,h,b,\phi ,p,d},$$

(9)

with energy per output token in decode:

$$e^{\mathrm{Wh}/\mathrm{token}}(h,\mathit{decode},b;q)={\overline{P}}_{h,b}^{\mathrm{acc}}\left(\mathit{decode}\right){\displaystyle \frac{1}{{\mathrm{TPS}}_{h,b}\left(q\right)}}\times {\displaystyle \frac{1}{3600}},$$

(10)

where ${\overline{P}}_{h,b}^{\mathrm{acc}}$ is the accelerator-only average power (W).

Before the first token appears, the system spends ${\mathrm{TTFT}}_{h,b}\left(q\right)$ seconds on prefill; spreading that energy over the $T_p^{\mathrm{in}}$ input tokens for profile p makes the prefill term linear and comparable:

$$e^{\mathrm{Wh}/\mathrm{token}}(h,\mathit{prefill},b;q)={\overline{P}}_{h,b}^{\mathrm{acc}}\left(\mathit{prefill}\right){\displaystyle \frac{{\mathrm{TTFT}}_{h,b}\left(q\right)}{T_p^{\mathrm{in}}}}\times {\displaystyle \frac{1}{3600}}.$$

(11)

We then lift to IT energy with the production-observed host+idle scaler ${\kappa }_{\mathrm{host}+\mathrm{idle}}$ (cf. Equation (1)), and to facility (comprehensive) energy via the site’s $\mathrm{PUE}$:

$$E_{t,r,h,b,\phi ,p,d}^{\mathrm{IT}}={\kappa }_{\mathrm{host}+\mathrm{idle}}\times E_{t,r,h,b,\phi ,p,d}^{\mathrm{acc}},$$

(12)

$$E_{t,r,h,b,\phi ,p,d}^{\mathrm{fac}}={\mathrm{PUE}}_r\times E_{t,r,h,b,\phi ,p,d}^{\mathrm{IT}}.$$

(13)

Consumptive water is obtained by applying the site-level cooling term and source-site electricity water to IT energy:

$${\mathrm{mL}}_{t,r,h,b,\phi ,p,d}=E_{t,r,h,b,\phi ,p,d}^{\mathrm{IT}}\times \left({\mathrm{PUE}}_r\times {\mathrm{WUE}}_r^{\mathrm{site}}\right)+E_{t,r,h,b,\phi ,p,d}^{\mathrm{IT}}\times {\mathrm{EWIF}}_r^{\mathrm{source}}.$$

(14)

Operational emissions use facility energy times either the provided market-based factor or the location-based grid factor; with LB as default:

$$g{\mathrm{CO}}_{2,t,r,h,b,\phi ,p,d}^{\mathrm{LB}}=E_{t,r,h,b,\phi ,p,d}^{\mathrm{fac}}\times {\mathrm{CIF}}_r\times {10}^3.$$

(15)

Finally, the embodied CO₂ and e-waste are amortized as daily charges that activate once per hardware class used:

$${\mathrm{CO}}_{2,\mathrm{day}}^{\mathrm{emb}}=\sum _{r,h}{z}_{r,h}{\displaystyle \frac{C_h^{\mathrm{emb}}}{L_h}},W_{\mathrm{day}}^{\mathrm{ewaste}}=\sum _{r,h}{z}_{r,h}{\displaystyle \frac{M_h^{\mathrm{board}}}{L_h}}.$$

(16)

Summing Equations (9)–(16) over all $(r,h,b,p,\phi ,d)$ assignments in a window yields the window-level $E_t,m{L}_t,gC{O}_{2,t}$ that enter the objective in (5); summing over t and taking daily medians produces the reported statistics, with LB as default and MB as sensitivity [1,3,25,56].

Algorithm 1 $\Sigma$-Scale (daily-coupled MILP; p95 SLOs, replicas, daily binaries)

Inputs:  Sets: regions R, hardware classes H, batches B, phases $\Phi =\{\mathrm{prefill},\mathrm{decode}\}$, profiles P, directives $D=\{\mathrm{default},\mathrm{brief}\}$, windows $T=\{1,\dots ,288\}$; window budget $\Delta t=300$ s. Inputs:  Arrivals $Q_{t,p}$ per window and profile, either provided or constructed from $N_{\mathrm{day}}$, mix ${\pi }_p$, and diurnal weights $f\left(t\right)$: $Q_{t,p}=\mathrm{round}\left(N_{\mathrm{day}}{\pi }_{p}f\left(t\right)/{\sum }_{t^{\prime }}f\left(t^{\prime }\right)\right)$. Inputs:  p95 quantile tables: ${\mathrm{TPS}}_{h,b}\left(95\right)$, ${\mathrm{TTFT}}_{h,b}\left(95\right)$, ${\mathrm{TPOT}}_{h,b}\left(95\right)=1/{\mathrm{TPS}}_{h,b}\left(95\right)$. Inputs:  Site rows per r: ${\mathrm{PUE}}_r$, ${\mathrm{WUE}}_{\mathrm{site},r}$, ${\mathrm{EWIF}}_r^{\mathrm{source}}$, ${\mathrm{CIF}}_r^{\mathrm{LB}}$ (optionally $E{F}_r^{\mathrm{MB}}$). Inputs:  Lift and SLOs: ${\kappa }_{\mathrm{host}+\mathrm{idle}}\ge 1$, $L_p^{★}$; weights $\alpha ,\beta ,\lambda ,\delta ,\epsilon \ge 0$. Inputs:  Embodiment: $C_h^{\mathrm{emb}}$ (kgCO2e/board), $M_h^{\mathrm{board}}$ (kg), lifetime $L_h$ (days). Inputs:  Accelerator-only per-token models $e_{\mathrm{Wh}/\mathrm{token}}(h,\phi ,b)$; tokens $T_p^{\mathrm{in}}$, $T_{p,\mathrm{default}}^{\mathrm{out}}$; directive multipliers ${\alpha }_d\in (0,1]$; replica caps $U_{r,h,b}\in {\mathbb{Z}}_{\ge 0}$. Outputs:  Decision variables:                $x_{t,r,h,b,\phi ,p,d}\in {\mathbb{Z}}_{\ge 0}$ (assignments),    $n_{t,r,h,b}\in {\mathbb{Z}}_{\ge 0}$ (replicas),    $u_{t,r,h,b}\in \{0,1\}$ (activation),    $z_{r,h}\in \{0,1\}$ (daily activation).                Optional:$y_{t,r,h,b,p,d}\in \{0,1\}$ (usage, for gated SLO).   1: Per-prompt coefficients for each $(r,h,b,\phi ,p,d)$ in Wh/prompt:       $T_{p,d}^{\phi }\leftarrow \left(T_p^{\mathrm{in}}\mathrm{if}\phi =\mathrm{prefill};{\alpha }_d{T}_{p,\mathrm{default}}^{\mathrm{out}}\mathrm{if}\phi =\mathrm{decode}\right)$       $E_{h,b,\phi ,p,d}^{\mathrm{acc}}\leftarrow e_{\mathrm{Wh}/\mathrm{token}}(h,\phi ,b)T_{p,d}^{\phi }$       $E_{h,b,\phi ,p,d}^{\mathrm{IT}}\leftarrow {\kappa }_{\mathrm{host}+\mathrm{idle}}{E}_{h,b,\phi ,p,d}^{\mathrm{acc}}$       $E_{r,h,b,\phi ,p,d}^{\mathrm{fac}}\leftarrow {\mathrm{PUE}}_r{E}_{h,b,\phi ,p,d}^{\mathrm{IT}}$       ${\mathrm{mL}}_{r,h,b,\phi ,p,d}\leftarrow E_{h,b,\phi ,p,d}^{\mathrm{IT}}\left({\mathrm{PUE}}_r{\mathrm{WUE}}_{\mathrm{site},r}+{\mathrm{EWIF}}_r^{\mathrm{source}}\right)$       $g{\mathrm{CO}}_{2,r,h,b,\phi ,p,d}^{\mathrm{LB}}\leftarrow E_{r,h,b,\phi ,p,d}^{\mathrm{fac}}{\mathrm{CIF}}_r^{\mathrm{LB}}$   2: Minimize (global daily sum):       ${\displaystyle \alpha \sum _{t,r,h,b,\phi ,p,d}{x}_{t,r,h,b,\phi ,p,d}{E}_{h,b,\phi ,p,d}^{\mathrm{IT}}+\beta \sum _{t,r,h,b,\phi ,p,d}{x}_{t,r,h,b,\phi ,p,d}{\mathrm{mL}}_{r,h,b,\phi ,p,d}+\lambda \sum _{t,r,h,b,\phi ,p,d}{x}_{t,r,h,b,\phi ,p,d}g{\mathrm{CO}}_{2,r,h,b,\phi ,p,d}^{\mathrm{LB}}}$       $+{\sum }_{r,h}{z}_{r,h}\left(\delta {\displaystyle \frac{C_h^{\mathrm{emb}}}{L_h}}+\epsilon {\displaystyle \frac{M_h^{\mathrm{board}}}{L_h}}\right)$   3: Subject to   4: (i) Conservation per $(t,p)$ and phase link:       ${\sum }_{r,h,b,d}{x}_{t,r,h,b,\mathrm{prefill},p,d}={\sum }_{r,h,b,d}{x}_{t,r,h,b,\mathrm{decode},p,d}=Q_{t,p}\forall t,p.$   5: (ii) Capacity with replicas (RHS scaled by n):       ${\displaystyle \sum _{p,d}\left(\frac{x_{t,r,h,b,\mathrm{prefill},p,d}{T}_p^{\mathrm{in}}}{{\mathrm{TPS}}_{h,b}\left(95\right)}+\frac{x_{t,r,h,b,\mathrm{decode},p,d}{\alpha }_d{T}_{p,\mathrm{default}}^{\mathrm{out}}}{{\mathrm{TPS}}_{h,b}\left(95\right)}\right)\le \Delta t{n}_{t,r,h,b}\forall t,r,h,b.}$   6: (iii) Replica bounds and activation:       $0\le n_{t,r,h,b}\le U_{r,h,b}{u}_{t,r,h,b}\forall t,r,h,b.$   7: (iv) Daily coupling (embodiment once/day):       $z_{r,h}\ge u_{t,r,h,b}\forall t,r,h,b.$   8: (v) Link assignments to activation (tight big-M):       $x_{t,r,h,b,\phi ,p,d}\le M_{h,b,p,d,\phi }{u}_{t,r,h,b}\forall t,r,h,b,\phi ,p,d.$   9: (vi) Latency SLO at p95 (choose one form):       Ungated: ${\mathrm{TTFT}}_{h,b}\left(95\right)+{\alpha }_d{T}_{p,\mathrm{default}}^{\mathrm{out}}{\mathrm{TPOT}}_{h,b}\left(95\right)\le L_p^{★}\forall h,b,p,d.$       Optional gated: ${\mathrm{TTFT}}_{h,b}\left(95\right)+{\alpha }_d{T}_{p,\mathrm{default}}^{\mathrm{out}}{\mathrm{TPOT}}_{h,b}\left(95\right)\le L_p^{★}+M_{h,b,p,d}^{\prime }\left(1-y_{t,r,h,b,p,d}\right)$, with       $x_{t,r,h,b,\phi ,p,d}\le {\widehat{M}}_{h,b,p,d,\phi }{y}_{t,r,h,b,p,d}$ and $y_{t,r,h,b,p,d}\le u_{t,r,h,b}$. 10: Solve the MILP once over all $t\in T$; return $x^{★},n^{★},u^{★},z^{★}$.

Algorithm 2 Aggregation and scope-transparent reporting (post-solve)

Inputs:  Optimal decisions $x^{★},n^{★},u^{★},z^{★}$; coefficients $E^{\mathrm{IT}},E^{\mathrm{fac}},\mathrm{mL},g{\mathrm{CO}}_2^{\mathrm{LB}}$; arrivals $Q_{t,p}$; profile mix $(0.70,0.25,0.05)$. Outputs:  Daily per-prompt medians by profile, then mixed medians; comprehensive default, accelerator-only whiskers; LB default with MB sensitivity.   1: for all $(t,p)$ do   2:       $E_{t,p}^{\mathrm{IT}}\leftarrow {\sum }_{r,h,b,\phi ,d}{x}_{t,r,h,b,\phi ,p,d}^{★}{E}_{h,b,\phi ,p,d}^{\mathrm{IT}}$   3:       ${\mathrm{mL}}_{t,p}\leftarrow {\sum }_{r,h,b,\phi ,d}{x}_{t,r,h,b,\phi ,p,d}^{★}{\mathrm{mL}}_{r,h,b,\phi ,p,d}$   4:       $g{\mathrm{CO}}_{2,t,p}^{\mathrm{LB}}\leftarrow {\sum }_{r,h,b,\phi ,d}{x}_{t,r,h,b,\phi ,p,d}^{★}g{\mathrm{CO}}_{2,r,h,b,\phi ,p,d}^{\mathrm{LB}}$   5:       Normalize to per-prompt: $E_{t,p}^{\mathrm{IT}}=E_{t,p}^{\mathrm{IT}}/Q_{t,p}$, $w_{t,p}={\mathrm{Water}}_{t,p}/Q_{t,p}$, $c_{t,p}^{\mathrm{LB}}=g{\mathrm{CO}}_{2,t,p}^{\mathrm{LB}}/Q_{t,p}$.   6: end for   7: For each profile p, take the median over $t=1:288$ of $(E_{t,p}^{\mathrm{IT}},w_{t,p},c_{t,p}^{\mathrm{LB}})$; then mix by $0.70/0.25/0.05$ to get per-model medians.   8: Embodied and e-waste (once/day): $\mathrm{CO}{2}_{\mathrm{emb},\mathrm{day}}={\sum }_{r,h}{z}_{r,h}^{★}{C}_h^{\mathrm{emb}}/L_h$, ${\mathrm{Waste}}_{\mathrm{day}}={\sum }_{r,h}{z}_{r,h}^{★}{M}_h^{\mathrm{board}}/L_h$.   9: Scopes. Report comprehensive (default) via $E^{\mathrm{fac}}={\mathrm{PUE}}_r{E}^{\mathrm{IT}}$; add accelerator-only whiskers using $E^{\mathrm{acc}}$ (or the observed narrow→comprehensive ratio when shares are undisclosed). 10: Emissions. Default LB via ${\mathrm{CIF}}_r^{\mathrm{LB}}$; add MB sensitivity by substituting $E{F}_r^{\mathrm{MB}}$.

3.5. Parameterization from Public Sources

We adopt the comprehensive serving boundary by default: active accelerators together with host CPU/DRAM and provisioned idle, lifted to facility energy by the site PUE in Equations (12)–(13). When only accelerator-only measurements are disclosed, we translate to the comprehensive boundary with a host+idle lift and PUE, or when shares are undisclosed, by using the production-observed narrow→comprehensive median ratio $\approx 0.417$ for comparable workloads [1]. Reporting uses daily medians to mirror production practice and to keep results comparable across models and regions [1].

Baseline per-prompt energy and throughput/latency quantiles by model and profile are taken from the cross-model API benchmark [5]. We ingest watt-hours per prompt together with the p-quantile tokens-per-second and latency rows, for example p95 TTFT, TPS, and TPOT. These quantiles parameterize the capacity and tail-latency constraints in Equations (7)–(8). As an illustration, the table for GPT-4o lists a short-profile energy near $0.421$ Wh and provides site multipliers; we keep the published provider/region mapping [5].

Per-token coefficients for prefill and decode follow the device-level accelerator-only power models used in Equations (10)–(11). The average accelerator power ${\overline{P}}_{\mathrm{acc},h,b}\left(\phi \right)$ is taken per hardware class and batch. We lift accelerator-only energy to IT with the host+idle scaler ${\kappa }_{\mathrm{host}+\mathrm{idle}}$ in Equation (12) and then to facility with the site PUE in Equation (13), consistent with the comprehensive boundary in [1].

Consumptive water is computed with the scope-consistent site+source formulation in Equation (14). The site component scales with facility energy via ${\mathrm{WUE}}_{\mathrm{site},r}$. The source component scales with IT energy via the regional electricity–water intensity factor ${\mathrm{EWIF}}_{\mathrm{source},r}$. Priors and bounds for site WUE come from the LBNL synthesis and AI water-footprint guidance [3,56]. Electricity–water intensity values follow the scope-2 definition in [3] and are applied as annual, location-based factors when hourly mixes are unavailable.

Operational emissions are location-based by default using regional CIF with facility energy in Equation (15). We also provide a market-based sensitivity by substituting the provider’s prior-year portfolio factor when disclosed [1,25]. Units are reported in the native form for each metric.

For the phase-aware variant, we use accelerator-only per-token energy curves that separate compute-bound prefill from memory-bound decode across relevant batches. This enables assignments that reduce Wh per prompt while respecting p95 SLOs [22]. Embodied carbon per board, board mass, and lifetimes are taken from serving and lifecycle assessments [16,53] and are amortized once per day when a hardware class is activated, as in Equation (16).

Reference ranges for site factors follow [56]. Hyperscale PUE trends to roughly $1.15$–$1.35$. Site WUE is around $0.32$–$0.40$ L kWh⁻¹ in 2023 with a likely rise as liquid cooling penetrates. Scenario sweeps therefore use $\pm 0.10$ around a regional PUE baseline and $\pm 25\%$ around site WUE. When we construct hydro-like or nuclear-like sites for Pareto illustrations, CIF and electricity–water intensity are kept within documented ranges and flagged as replaceable medians when provider rows are disclosed [3,56].

All numerical inputs—per-model Wh per prompt, p95 TTFT/TPS/TPOT by batch, site rows with PUE, site WUE, electricity–water intensity, and location-based carbon intensity, and embodiment parameters—are taken from peer-reviewed or provider-authored sources [1,3,5,56]. We package them into machine-readable tables so figures and results regenerate one-for-one when providers update disclosures. When deployment caps are known, we set $U_{r,h,b}$ to the maximum provisionable replicas for class h at region r and batch b. Otherwise $U_{r,h,b}$ is a large operator-chosen bound.

Because grid carbon intensity and electricity–water intensity can be weakly correlated at fine time scales, improving one can worsen the other [3]. This motivates the bi-objective treatment and the routing-only and joint routing+batch+token analyses that make the trade-off geometry explicit under p95 SLOs.

4. Results

We simulate a single 24 hour production day split into 288 five-minute decision windows and a fixed demand of 500 million prompts with a $70/25/5$ short/medium/long mix. Short, medium, and long profiles use token pairs $(100, 300)$, $(1000, 1000)$, and $(10,000, 15,000)$, respectively. In every window the $\Sigma$ Scale optimizer chooses, for each profile, the region, the batch size, and the phase-specific hardware (prefill vs. decode) to minimize an equal-weighted sum of energy, consumptive water, and location-based CO₂ subject to explicit $p95$ SLOs and the capacity and conservation constraints. Capacity is enforced with per-hardware, per-batch tokens-per-second quantiles, and latency feasibility uses $p95$ time-to-first-token and time-per-output-token; the short prompt target is $\le 0.9$ s. Window outputs are aggregated to daily medians per profile and then mixed by $70/25/5$ to obtain per-model medians. By default we report the comprehensive serving boundary—active accelerator + host CPU/DRAM + provisioned idle—lifted to facility energy by the site’s $\mathrm{PUE}$. Consumptive water is computed with the scope-consistent site+source rule, $E^{\mathrm{IT}}(\mathrm{PUE}\times {\mathrm{WUE}}_{\mathrm{site}}+\mathrm{EWIF})$; CO₂ is location-based using $\mathrm{CIF}$, with a market-based sensitivity when prior-year portfolio factors are available. For reconciliation with chip-only studies we also compute an accelerator-only subset and translate between scopes using the empirically observed narrow/comprehensive median ratio of $0.417$ (approximately $2.4\times$ uplift; e.g., $0.10$ Wh accelerator-only vs. $0.24$ Wh comprehensive at the median) [1]. Regional multipliers ($\mathrm{PUE}$, ${\mathrm{WUE}}_{\mathrm{site}}$, $\mathrm{EWIF}$, $\mathrm{CIF}$) and the throughput/latency quantiles that bind the SLOs come from the cross-model public benchmark; a phase-aware variant uses device-level per-token energy curves for prefill/decode and includes embodied-carbon amortization when a hardware class is activated during the day [5]. Two policies are evaluated under identical demand and site multipliers. The baseline fixes batch $= 8$, uses the home region only, applies no token directive, and does not split phases. The $\Sigma$ Scale optimized policy right-sizes batch over the day, applies a concise token directive when SLO-safe, routes across sites in response to CIF and $\mathrm{EWIF}$, assigns prefill and decode to different hardware when that lowers Wh/prompt, and amortizes embodied impacts; the contrast between these policies isolates the value of orchestration at a fixed model and boundary.

4.1. Comprehensive Boundary Medians and Daily Totals

Table 1 reports per-prompt median-scaled total and associated daily totals for energy, water (consumptive, site+source), and location-based CO₂ for four representative models. The Optimized rows correspond to the $\Sigma$ Scale loop with all levers active (batch right-sizing, semantic token control, geo-routing, phase-aware assignment, and second-life amortization). Baseline uses the same comprehensive boundary and medians but with batch $=8$, no token control, home region only, and no phase split. Across the four evaluated models, the $\Sigma$ Scale policy delivers large and consistent savings at the comprehensive serving boundary. After aggregating window outputs to daily medians per profile and mixing by $70/25/5$, the median energy per prompt falls by 57–$59\%$, the median consumptive water (site+source) by 59–$60\%$, and the median location-based CO₂ by 78–$80\%$, all while meeting $p95$ capacity and $p95$ latency in every five-minute window. For GPT-4o, the comprehensive baseline is $0.6876$ Wh/prompt, $2.3915$ mL/prompt, and $0.2426$ g CO₂/prompt; after optimization the medians become $0.2898$ Wh, $0.9803$ mL, and $0.0507$ g CO₂, respectively. At a volume of 500 M queries per day these medians translate to energy dropping $0.344\to 0.145$ GWh, water $1.196\to 0.490$ ML, and LB CO₂$121\to 25$ t under the same SLA. The other models follow the same pattern, with absolute magnitudes scaling with baseline Wh/prompt (e.g., Claude 3.7 Sonnet shows the largest absolute reductions because its baselines are highest). These gains reflect full-stack serving and scope-consistent water accounting rather than chip-only power, and they arise from the combined effect of batch right-sizing (e.g., $8\to 16$ off-peak), token-length directives, carbon- and water-aware geo-routing, phase-aware hardware assignment, and second-life amortization. See Figure 1 for a side-by-side visualization of the comprehensive-boundary medians (baseline vs. optimized) for energy, consumptive water, and LB CO₂, with lower ‘scope whiskers’ showing accelerator-only values via the observed narrow/comprehensive $\approx 0.417$ ratio.

4.2. Scope Reconciliation: Accelerator-Only vs. Comprehensive

A persistent source of disagreement in the literature is the accounting boundary used for inference. To make our results directly comparable to chip-only studies, we compute both scopes for the same traffic, mix, and site multipliers and place them side by side. The empirical narrow/comprehensive median energy ratio in our runs is $\approx 0.417$, reproducing the $\approx 2.4\times$ uplift observed in production telemetry when host CPU/DRAM, provisioned idle, and facility overheads ($\mathrm{PUE}$) are included [1]. This factor is not universal—it varies with architecture and fleet management—but when underlying shares are unavailable it provides a defensible translation between scopes. Concretely, if a study reports only accelerator-level energy $E_{\mathrm{acc-only}}$ (Wh prompt⁻¹), the comprehensive median may be approximated by $E_{\mathrm{comprehensive}}\approx E_{\mathrm{acc-only}}/0.417$, holding $\mathrm{PUE}$, ${\mathrm{WUE}}_{\mathrm{site}}$, $\mathrm{EWIF}$, and the prompt mix constant.

Table 2 provides scope translation to reconcile chip-only studies with comprehensive accounting. The numerical proximity between optimized comprehensive and baseline narrow in our run ($\sim 0.42\times$ of baseline comprehensive in both cases) is coincidental: the former reflects optimization levers and SLOs; the latter reflects boundary accounting shares measured in production. Together with Table 1 (policy effects at the comprehensive boundary), Table 2 furnishes a clear, citable translation layer so that chip-only results can be reconciled with full-stack accounting without ambiguity.

4.3. Carbon–Water Movement at Fixed QoS

Because grid carbon intensity ($\mathrm{CIF}$) and electricity-water intensity ($\mathrm{EWIF}$) are weakly correlated, carbon-optimal routing can be water-suboptimal. Our $\Sigma$ Scale policy jointly optimizes both metrics. Figure 2 shows each model’s movement from baseline to optimized in the $(\mathrm{g}{\mathrm{CO}}_2,\mathrm{mL})$ plane. For each model we plot the comprehensive-boundary median at the baseline configuration and the corresponding median under the $\Sigma$ Scale policy; the two points are connected by an arrow. In every case the arrow points down and to the left, showing that the optimized policy reduces both location-based CO₂ and consumptive water while honoring the $p95$ SLOs. This simultaneous reduction is non-trivial because $\mathrm{CIF}$ and $\mathrm{EWIF}$ are only weakly correlated; a carbon-only router can increase water if it shifts load to a clean but water-intensive region. The $\Sigma$ Scale objective avoids that pitfall by jointly reducing Wh per prompt—through batch right-sizing, concise token directives, and phase-aware placement—and by routing remaining Wh to regions and hours with favorable $\mathrm{CIF}$ and $\mathrm{PUE}\times {\mathrm{WUE}}_{\mathrm{site}}+\mathrm{EWIF}$.

The geometry is intuitive. Each site i defines a ray through the origin in the $(\mathrm{g}{\mathrm{CO}}_2,\mathrm{mL})$ plane whose slope is $W_i/{\mathrm{CIF}}_i$, with $W_i={\mathrm{PUE}}_i\times {\mathrm{WUE}}_{\mathrm{site},i}+{\mathrm{EWIF}}_i$. Moving along a ray changes only energy (Wh) via batching and token control; switching rays changes the water-to-carbon ratio by altering site factors while holding Wh fixed. The optimizer exploits both degrees of freedom: it lowers Wh and, subject to the latency budget, selects rays with more favorable slopes. In a two-region sensitivity where the alternative site is hydro-like with both lower $\mathrm{CIF}$ and a lower combined water factor, routing alone becomes a Pareto improvement; the arrow’s down-left direction then reflects a pure siting effect. In other configurations, the arrow still points down-left because energy reductions combine with selective routing to offset sites where low carbon coincides with higher water.

To keep scope visible without obscuring directionality, Figure 2 overlays asymmetric boundary whiskers on both endpoints. The barbed end of each whisker marks the accelerator-only value obtained by multiplying the comprehensive coordinate by $0.417$ on both axes; there is no upper whisker because comprehensive is our defined upper boundary. The tip-to-tail arrows and their whiskered counterparts are nearly parallel and scaled, illustrating that the direction and magnitude of the optimization gain are robust to scope choice: changing scope shifts absolute values by the familiar $\approx 2.4\times$ but does not alter the qualitative improvement.

This figure is diagnostic rather than a full carbon–water Pareto frontier. Constructing that frontier would require multiple MILP runs with different objective weights. At fixed QoS, the comprehensive, SLO-aware policy achieves concurrent reductions in CO₂ and water for the evaluated workloads. In the next section we quantify the additional reductions available if small latency relaxations or stronger token-length directives are permitted.

4.4. Routing-Only Carbon–Water Pareto Under SLOs

The previous subsection established that the $\Sigma$ Scale policy moves every model down and to the left in the $(\mathrm{g}{\mathrm{CO}}_2,\mathrm{mL})$ plane: batching and token directives reduce watt-hours per prompt, while geo-routing chooses sites whose environmental multipliers further shrink carbon and water, all without violating the p95 latency SLO. We now isolate the routing lever to understand the geometry of those gains. To avoid conflating routing with operational changes, we hold fixed each model’s optimized comprehensive-boundary energy per prompt $E_m^{★}$ (Wh prompt⁻¹) and vary only the regional mix subject to the same SLO. This yields an interpretable map of what geo placement alone can deliver once the runtime system has already right-sized batch and applied concise generation.

Let $s=(s_b,s_h,s_n)$ denote the shares routed to a baseline U.S. thermal site b, a hydro-dominated site h, and a nuclear-like thermal site n, with $s_i\ge 0$ and $s_b+s_h+s_n=1$. Each site i is characterized by a location-based grid carbon intensity ${\mathrm{CIF}}_i$ (kg CO₂ kWh⁻¹) and a consumptive water factor $W_i$, following a scope-consistent convention (on-site cooling scaled by PUE, plus source-side electricity–water intensity added directly). With $E_m^{★}$ fixed, the per-prompt impacts for any routing mix s are linear:

$$\mathrm{g}{\mathrm{CO}}_2/\mathrm{prompt}\left(s\right)=E_m^{★}\sum _{i\in \{b,h,n\}}{s}_i{\mathrm{CIF}}_i,\mathrm{mL}/\mathrm{prompt}\left(s\right)=E_m^{★}\sum _{i\in \{b,h,n\}}{s}_i{W}_i.$$

(17)

Equation (17) implies that the feasible set in the $(\mathrm{g}{\mathrm{CO}}_2,\mathrm{mL})$ plane is the convex hull of the three single-site vertices $\{E_m^{★}{\mathrm{CIF}}_i,E_m^{★}{W}_i\}$; the Pareto frontier is the lower-left convex envelope of those vertices. Each site also defines a ray through the origin with slope $W_i/{\mathrm{CIF}}_i$: moving along a ray changes only the energy scale, whereas switching rays changes the water-to-carbon ratio at fixed energy.

Because the hydro site in our parameterization couples very low ${\mathrm{CIF}}_h$ with a high electricity–water intensity (we use $5.50$ L kWh⁻¹ to stress the conflict), while the nuclear-like site has a lower combined water factor $W_n$ but a higher carbon intensity ${\mathrm{CIF}}_n$ than hydro, the non-dominated set collapses to the straight edge joining hydro and nuclear. The baseline site lies to the right of both in carbon and does not beat nuclear on water; hence it is dominated. Along the efficient edge, a single parameter $\lambda \in [0,1]$ (the nuclear share) describes the trade-off curve:

$$\begin{array}{cc}\hfill \mathrm{mL}/\mathrm{prompt}\left(\lambda \right)& =E_m^{★}\left[(1-\lambda )W_h+\lambda W_n\right],\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill \mathrm{g}{\mathrm{CO}}_2/\mathrm{prompt}\left(\lambda \right)& =E_m^{★}\left[(1-\lambda ){\mathrm{CIF}}_h+\lambda {\mathrm{CIF}}_n\right].\hfill \end{array}$$

(19)

The slope of this edge,

$${\displaystyle \frac{d(\mathrm{mL}/\mathrm{prompt})}{d(\mathrm{g}{\mathrm{CO}}_2/\mathrm{prompt})}}={\displaystyle \frac{W_n-W_h}{{\mathrm{CIF}}_n-{\mathrm{CIF}}_h}},$$

(20)

depends only on site properties and is therefore independent of $E_m^{★}$. This scaling invariance explains why the four model panels share the same shape but span different numerical ranges: increasing $E_m^{★}$ stretches both axes uniformly without changing which combinations are efficient.

Latency feasibility is enforced by a conservative p95 proxy:

$${\mathrm{TTFT}}_{\mathrm{p}95}\left(s\right)=\sum _{i\in \{b,h,n\}}{s}_i{\mathrm{TTFT}}_{i,\mathrm{p}95}\le \mathrm{SLO},$$

(21)

so the routing cloud shows exactly those mixes that respect the interactive QoS; infeasible mixes are shaded separately for transparency. With this setup in place, the four panels in Figure 3, for GPT-4o, GPT-4o-mini, Claude-3.7 Sonnet, and LLaMA-3-70B respectively, the feasible routing simplex in $(\mathrm{g}{\mathrm{CO}}_2,\mathrm{mL})$ space, the hydro and nuclear vertices that determine the efficient edge via (18)–(20), and the dominated baseline vertex. Selecting any point on the red edge is equivalent to choosing $\lambda$ to meet a carbon or water budget at unchanged quality of service.

4.5. Joint Frontiers from Site + Batch + Token Sweeps Under the SLO

The routing-only picture is intentionally conservative because it holds $E_m^{★}$ fixed. In practice, the serving stack can reduce energy substantially by right-sizing batch and encouraging concise generations when quality allows. To quantify how these operational levers compound with routing, we extend the enumeration to include a batch multiplier b and a token-length multiplier t (both in $[0,1]$; in our grids $b\in [0.561,1]$ for batch $8\to 16$ medians, $t\in [0.7175,1]$ for default→brief). For a given routing mix s, the effective energy becomes

$$E(b,t)=E_m^{★}bt,$$

and the per-prompt impacts generalize to:

$$\begin{array}{cc}\hfill \mathrm{g}{\mathrm{CO}}_2/\mathrm{prompt}(s,b,t)& =E_m^{★}bt\sum _{i\in \{b,h,n\}}{s}_i{\mathrm{CIF}}_i,\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill \mathrm{mL}/\mathrm{prompt}(s,b,t)& =E_m^{★}bt\sum _{i\in \{b,h,n\}}{s}_i{W}_i.\hfill \end{array}$$

(22b)

Because both axes are linear in energy, every point in the routing triangle is translated down and left by the factor $bt$. As b and t vary over their feasible ranges, the single routing triangle thickens into a wedge of scaled triangles, and the joint Pareto frontier is the lower-left envelope of that wedge. In the feasibility test we retain the conservative p95 proxy of (21), i.e., we mix per-site p95 TTFT linearly by s. When providers supply measured p95 curves as functions of batch and region, those can be dropped into the same mechanism to further tighten feasibility; the frontier construction itself is unchanged.

The four panels in Figure 4 sweep s, b, and t for GPT-4o, GPT-4o-mini, Claude-3.7 Sonnet, and LLaMA-3-70B. Gray points satisfy the p95 SLO; salmon points would violate it. The red curve in each panel is the feasible lower-left envelope. Black stars mark the three single-site anchors at $b=t=1$; by construction these stars sit inside the feasible cloud and are typically dominated, because there exist mixes with the same s but smaller $bt$ that strictly reduce both axes while remaining inside the SLO. Relative to the routing-only panels, the frontiers shift markedly down and left, quantifying the additional reductions unlocked by batching and concise tokens. For GPT-4o and GPT-4o-mini the feasible wedge is dense and the frontier hugs the extreme lower left. For Claude-3.7 Sonnet, which has the largest $E_m^{★}$, the SLO trims the feasible set more aggressively near the high-batch regime, revealing the practical limit imposed by interactivity. Across all models the direction of the frontier continues to align with the hydro–nuclear edge because hydro remains carbon-optimal while nuclear remains water-optimal. Operationally, routing determines where energy is consumed (through $\mathrm{CIF}$ and W); batching and tokens determine how much energy per prompt is consumed (through $E_m^{★}bt$). The levers compound.

4.6. Ablation: Where the Gains Come from (Fixed p95 SLOs)

We quantify the marginal contribution of each operational lever at fixed quality of service, using the same data and boundary choices as in Methods and in Section 4.1. We evaluate feasible point clouds in the $({\mathrm{gCO}}_2,\mathrm{mL})$ plane under the p95 latency constraint in (8), capacity in (7), and conservation in (6). We vary exactly one degree of freedom at a time: routing; routing+batch; routing+batch+token; routing+phase split. As in the main results, carbon is location based by default and water is consumptive site+source via (14). When an ablation holds energy fixed we use the per-model optimized energy per prompt $E_m^{★}$ from Section 4.1. Enumeration is deterministic with no RNG. Daily medians are computed by profile and then mixed $70/25/5$ for short, medium, and long prompts.

• A1 Routing-only. Holding $E_m^{★}$ fixed, let $s=(s_b,s_h,s_n)$ denote the shares across the three representative sites with $s_i\ge 0$ and ${\sum }_i{s}_i=1$. Impacts are affine in s by (17), so the feasible set is the routing simplex and the efficient set is its lower-left envelope. For very low ${\mathrm{CIF}}_h$ but higher ${\mathrm{EWIF}}_h$ for hydro, the efficient edge collapses to hydro↔nuclear and the baseline vertex is dominated (see Figure 5). Latency feasibility overlays follow the conservative p95 mix proxy in (21).
• A2 Routing + Batch. We add a batch multiplier $b\in [b_{min},1]$ that scales energy nearly multiplicatively (while respecting (8)):

  $$E_m^{★}\mapsto b{E}_m^{★},(\mathrm{g},\mathrm{mL})\mapsto b(\mathrm{g},\mathrm{mL}).$$

  Each routing triangle is scaled down and left by b, which shifts the Pareto envelope toward the origin (see Figure 6).
• A3 Routing + Batch + Token (wider wedge). We also sweep a token-length multiplier $t\in [t_{min},1]$ (default→brief). With $E_m^{★}\mapsto bt{E}_m^{★}$, both axes contract by the same $bt$ factor and the impacts follow the joint form already defined in (21a)–(21b). The feasible cloud thickens and the efficient frontier extends down and left relative to A2 (see Figure 7).
• A4 Routing + Phase split (prefill vs. decode). Using the phase-aware per-token energy models $e_{\mathrm{Wh}/\mathrm{token}}(h,\phi ,b)$ and the prefill and decode constructions in (10)–(11), we expose the energy-shaping effect of placing prefill and decode on different cohorts. For visualization we sweep phase multipliers $({\eta }_{\mathrm{prefill}},{\eta }_{\mathrm{decode}})\in {[0.90,1]}^2$ with decode share $\rho \in [0,1]$ and map $E_m^{★}\mapsto E_m^{★,\mathrm{phase}}=E_m^{★}\left((1-\rho ){\eta }_{\mathrm{prefill}}+\rho {\eta }_{\mathrm{decode}}\right)$ subject to the same SLO guard. The frontier shifts further toward the origin. The hydro↔nuclear edge remains slope-setting with slope given by (20) (see Figure 8).

In all four panels, ∘ points satisfy p95 SLOs; × points violate. To keep uncertainty visible without clutter, we show vertex whiskers: for water, $\mathrm{PUE}\in [{\mathrm{PUE}}_0\pm 0.10]$ and ${\mathrm{WUE}}_{\mathrm{site}}\in [0.75,1.25]\times {\mathrm{WUE}}_{\mathrm{site},0}$ (holding $\mathrm{EWIF}$ at the published median); for carbon, a $\pm 15\%$ CIF envelope when only LB is available, otherwise an LB↔MB swap whisker when provider MB factors are disclosed. These match the uncertainty bands and scope practice used elsewhere in the paper.

Across A1 to A4 the feasible set widens and the frontier moves toward the origin. Routing chooses the slope through the site mix. Batch, token, and phase split reduce watt-hours per prompt at fixed QoS, which shrinks both axes. This explains why the optimized medians in Table 1 land down and left of the baseline points without relaxing SLOs and why single-site anchors are dominated once the operational levers are enabled.

5. Discussion

This work introduces a provider-agnostic, time-resolved framework that couples scope-transparent measurement with a mixed-integer orchestration loop ($\Sigma$-Scale) to co-minimize the carbon and water footprints of LLM serving under production-grade SLOs. Reporting daily medians at a comprehensive boundary—active accelerators + host CPU/DRAM + provisioned idle with $\mathrm{PUE}$—resolves a major source of disagreement in prior studies and aligns with operational telemetry. In this boundary, the empirical translation between accelerator-only and comprehensive scopes (narrow/comprehensive $\approx 0.417$) enables direct, auditable comparisons across systems and papers; interventions that appear large at the chip boundary can attenuate—or reverse—once host, idle, and facility overheads are included [1].

Operationally, a single SLO-aware policy achieves large, consistent reductions without relaxing interactivity. Across four models, median per-prompt impacts fall by roughly 57–$59\%$ (energy), 59–$60\%$ (consumptive water; site + source), and 78–$80\%$ (location-based CO₂), with $p95$$\mathrm{TTFT}/\mathrm{TPOT}$ and capacity constraints met in every five-minute window. For a representative 500 M-query day on GPT-4o, totals drop from $0.344\to 0.145$ GWh, $1.196\to 0.490$ ML, and $121\to 25$ t CO₂ (LB). These gains arise from complementary levers—batch right-sizing, concise token directives, carbon- and water-aware geo-routing (via $\mathrm{CIF}$ and $\mathrm{EWIF}$), and phase-aware prefill/decode placement—rather than from any single mechanism.

A central conceptual contribution is to treat water as a first-class objective, computed as site cooling scaled by $\mathrm{PUE}$ plus source-side electricity–water intensity, and to optimize it jointly with carbon [3]. Because $\mathrm{CIF}$ and $\mathrm{EWIF}$ are only weakly correlated, optimizing one can worsen the other. The framework’s dual-objective design and the “ray” geometry in the (g CO₂, mL) plane make these trade-offs explicit: routing alone traces a Pareto edge determined by site factors, while adding batch and token controls translates the entire feasible cloud toward the origin, compounding siting with operational efficiency (Figure 1, Figure 2, Figure 3 and Figure 4).

Choosing medians over means for skewed mixes matches production practice and provides a stable basis for policy comparison. Table 2 makes scope reconciliation explicit by showing the $\sim 2.4\times$ uplift from accelerator-only to comprehensive across models; pairing comprehensive medians with scope “whiskers” avoids ad hoc conversion factors and helps readers reconcile results across boundary choices [1].

From a deployment standpoint, three near-term steps follow. (i) Integrate carbon- and water-aware geo-routing into global load balancers, enforcing $p95$ latency and using realistic tokens-per-second quantiles. (ii) Apply concise generation directives to curb unnecessary tokens, especially when combined with higher off-peak batching. (iii) Use phase-aware placement—fast, compute-efficient cohorts for prefill and memory-efficient or second-life cohorts for decode—to extend hardware life and bring embodied impacts into scope without sacrificing responsiveness [16,22].

Limitations point directly to future work. Hourly variability in $\mathrm{PUE}$, site $\mathrm{WUE}$, and electricity-mix water intensities suggests moving from annual to time-resolved site and grid factors to sharpen routing signals. Market-based carbon is treated here as a sensitivity; incorporating temporal matching and procurement constraints (e.g., REC/PPA portfolios) would enable joint LB/MB evaluations [25]. Water could be weighted by basin-level scarcity to reflect environmental equity [3]. Finally, richer user-experience models (percentile bands by profile/region) and live-traffic experiments would strengthen external validity; deeper lifecycle modeling (refurbishment yields, end-of-life pathways) would close the loop from serving policy to circularity outcomes [16].

6. Conclusions

This work delivers an operational template for reducing both the carbon and water footprints of LLM serving without compromising interactive quality. By adopting a comprehensive serving boundary, summarizing impacts with daily medians, and co-optimizing carbon and water under explicit $p95$ latency and capacity constraints, the framework turns a fragmented literature into a deployable control policy with repeatable gains.

Across four models, the SLO-aware policy cuts comprehensive-boundary medians by about 58% in energy, about 59% in consumptive water, and about 79% in location-based CO₂. For a representative day with 500 M GPT-4o queries, totals fall from 0.344 to 0.145 GWh, from 1.196 to 0.490 ML, and from 121 to 25 t CO₂ (LB), with $p95$ SLOs satisfied in every five-minute window. These reductions stem from coordinated levers—carbon- and water-aware routing, batch right-sizing, concise token directives, and phase-aware assignment of prefill and decode—rather than from a single intervention.

The scope reconciliation module reproduces the production-observed narrow/comprehensive $\approx 0.417$ ratio, which enables apples-to-apples comparison between chip-only and full-stack accounting. Pareto views make the carbon–water geometry explicit and give operators a practical way to navigate trade-offs at fixed service levels. Taken together, these elements support industry adoption, transparent reporting, and continuous improvement as grids, cooling systems, and inference stacks evolve.