Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving

The gist Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions, and this paper addresses how to reduce these emissions by designing and evaluating latency-centric, geo-distributed inference simulators and schedulers that account for queueing and continuous batching dynamics.

Problem Context

Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions, although carbonaware scheduling can reduce these emissions in geo-distributed datacenters by exploiting the spatiotemporal variations in grid carbon intensity, the latency-sensitive nature of LLM serving constrains this design space I. INTRODUCTION

The growth of LLM-powered applications has spurred massive investments in accelerator-rich datacenters and raised concerns about their environmental impacts The International Energy Agency projects the energy use of accelerator-equipped servers to grow by 30% annually, over three times faster than conventional ones, and to reach nearly half of net datacenter energy use through 2030 Recent studies show that inference already emits over 25× more carbon than training throughout a model’s lifetime, underscoring the need for carbon-efficient inference serving Temporal scheduling is appropriate for delaytolerant workloads but not for latency-critical ones, such as LLM inference with stringent time-to-first-token (TTFT) and time-per-output-token (TPOT) objectives Spatial scheduling can meet latency constraints, subject to data-placement regulations such as GDPR, but suffers from load imbalance

Simulator and Modeling

To study carbon-aware LLM scheduling under realworld serving conditions, the authors developed a latency-centric, geo-distributed serving simulator This simulator instantiates datacenters across multiple regions, each with GPU-equipped nodes hosting configurable data-parallel inference engines and forwards requests between regions over measured inter-region network latencies The engine captures iterative continuous batching behavior, models queueing delays by admitting requests only when sufficient memory is available, and estimates per-inference phase latency and power using analytical models calibrated across different models, degrees of parallelism, and datasets The simulator allows schedulers to optimize latency and carbon objectives together through distinct policies

The simulator incorporates several key dynamics:

** It emulates continuous-batching behavior in wall-clock time, with a configurable maximum batch size similar to vLLM’s max num seqs [69] The queueing delay depends on the engine’s occupancy rather than being a fixed constant Admitted requests can be processed in several prefill steps, depending on the input sequence length (ISL) The updated batch then advances by one decode step lasting tdec, recomputed for the current batch and KV cache size, so that admissions and completions change the inference step time at every iteration Engines sleep for each predicted step duration, so latencies accumulate in wall-clock time Alongside latency, engines accumulate energy throughout inference, charging Epre per prefill step, Edec per decode step, and idle energy when there are no requests in the inference pipeline The simulator reproduces the latency and energy dynamics of modern inference engines, such as vLLM, which the authors validate within a node deployment in Section V-B The simulator captures interregion transfers and key serving features, such as continuous batching and engine queueing, using calibrated per-phase latency and power models over configurable topologies The authors’ novelty lies in identifying the enginelevel datacenter state, namely the active batch load and queue depth, from which a carbon-aware scheduler can estimate TTFT and TPOT at scheduling time, which to their knowledge no prior carbon-aware geo-distributed scheduler has explored The simulator is composed of four major components communicating over gRPC: (i) a load generator, (ii) regional schedulers, (iii) DCs, and (iv) a provisioner In every polling period ∆p, the provisioner pulls these counters averaged over each DC’s engines, along with the grid CI of the DC 4, and broadcasts them to all schedulers with the inter-region latency matrix 5 Schedulers consume these to optimize latency and carbon objectives together through distinct policies The authors’ main contributions include a high-fidelity simulator, SLO- and carbon-aware schedulers, and a characterization of the carbon–SLO frontier The facility MILP achieves the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3%, while the most carbon-aggressive greedy setting, at comparable savings, violates the SLO for up to 55% of requests The authors further analyze the TTFT tail behavior, scheduling overhead, sensitivity to load-update frequency, topology, robustness to facility overheads, and predicted output lengths Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average SLO violation of 3% Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon–SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy with an average

1. Bold scheduling for carbon-SLO trade-offs: The facility MILP scheduler models batched energy as a shared per-datacenter cost and a marginal per-request cost, discouraging the scattering of a batch across many datacenters, achieving the best carbon–SLO balance by pricing energy collectively.

2. High-fidelity simulation for dynamic engine modeling: The simulator emulates inference features such as iterative continuous batching and captures dynamics like queueing delay by admitting requests only when sufficient memory is available, enabling the study of carbon savings under strict service-level objectives (SLOs).

3. Latency-centric scheduling estimation: Schedulers use a common estimator to predict metrics, such as TTFT being estimated as TTFTr,d = RTT(s, s′) + Wd + tpre(ISLr + Qd), allowing for the identification of SLO-feasible DCs based on predicted latency.

4. Load-coupled iterative assignment: The iterative ILP scheduler addresses the limitations of static batch assignments by re-projects datacenter batch states after each ILP solve to capture load coupling and re-solves until the assignment converges, which is crucial for capturing how continuous batching affects latency.

5. Tail latency analysis: The system can analyze the p99 TTFT across policies, as well as the per-request scheduling overhead, revealing that at high load, the facility MILP or standard ILP maintains better tail latency than greedy policies by enforcing capacity constraints.

6. Load-aware engine activation: A future system can implement a load-aware engine activator that adjusts the number of active engines to the demand, based on observations that idle engines account for significant carbon and host energy, improving efficiency beyond simple scheduling decisions.