KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving".
Jane: Here's the gist of the paper. Power has become a central bottleneck for AI inference This paper presents KAIROS, a context-aware power optimization system for agentic AI serving that reduces power consumption while maintaining a target performance SLO The key contribution is a detailed characterization of agentic inference serving that reveals why power optimization is fundamentally different and harder compared to single-turn LLM serving KAIROS achieves an average of 27% (up to 39.8%) power reduction while meeting the performance targets.
Tom: Characterization of Agentic Inference Serving The analysis reveals that agentic workloads maintain long-lived state for each request through a context cache that grows across tool-interleaved turns and varies widely across agents This context growth is highly dynamic, varying substantially over time within and across agents, and the distributions of agent lifetimes, turn counts, and growth rates all exhibit long tails.
Jane: The study uncovers a new power-management phenomenon: lowering GPU frequency beyond a certain value can drive the system into a thrashing regime where accumulated context exceeds GPU memory capacity.
Tom: This strong coupling between power and memory stability shows that agentic serving is not merely a scaled-up version of single-turn LLM serving, but a fundamentally different workload whose stateful and dynamic behavior reshapes the power-performance trade-off.
Title and authors: Tom: So we're talking about this paper called KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving. It looks like they’re tackling a problem where power is becoming a huge bottleneck for AI inference, especially with agentic AI workloads.
Jane: That sounds technical but super relevant because we all know running these complex agents is getting way more demanding on hardware than just doing a single prompt. So, what's the main idea behind KAIROS?
Lu: The core idea is that agentic serving is fundamentally different from just serving one prompt at a time because each request keeps its context for many turns, and this context grows in a really dynamic way. They found that lowering the GPU frequency too much can actually push the system into a thrashing state where memory pressure gets worse both in terms of performance and power consumption.
Meng: Thrashing sounds like when your computer is constantly swapping things around because it runs out of RAM, so if you lower the power, you might just make that memory issue much worse. What's their solution to stop that cycle?
Lalam: KAIROS introduces a context-aware power management system that uses the agent context as a signal to control GPU frequency, manage how many agents run at once, and route requests across different machines in a power-aware way.
Tom: That sounds like three main levers they're pulling: adjusting the GPU speed based on how much context is being used, controlling concurrency to keep the total memory demand safe, and smart routing to consolidate traffic when things are quiet or spread it out when things are busy.
Jane: Let's unpack that a bit more. They suggest dynamically adjusting GPU frequency based on context pressure to safely trade performance for power savings whenever there's enough memory headroom available.
Lu: Exactly, they’re trying to find that sweet spot where you can save power without letting the agent progress slow down too much, and they do this by looking at the context usage.
Meng: And on top of that, they are regulating concurrency based on an upper bound on total context usage to make sure the aggregate context demand stays within a safe memory envelope so you don't hit that thrashing regime.
Lalam: And for routing, they use a global router with both an assignment policy for where requests start and a reassignment policy to dynamically balance the load across instances.
Tom: So, what's the actual result? How much power did they save while keeping performance stable?
Jane: The evaluation shows KAIROS achieves an average of twenty-seven percent power reduction, but it can go up to thirty-nine point eight percent on a single serving instance, while still meeting the per-agent SLO of twenty tokens per second.
Lu: That’s a solid number for a system that handles this kind of stateful workflow; they manage to balance those competing demands effectively across their design principles.
Meng: I wonder how much more power they could save if the context growth distributions weren't so long-tailed, but for what it is, twenty-seven percent is a decent saving when you're dealing with these kinds of complex agents.
Lalam: The paper also argues that for agentic serving, you should use a throughput-based SLO instead of latency metrics like time to first token because that better captures the agent's progress over its whole lifecycle.
Tom: That makes sense for real-world deployment; if you're measuring how much actual work an agent gets done, that’s more meaningful for controlling it online.
Jane: So, what does this mean for someone who is just running these agents on their own setup without a massive cluster? It suggests a way to optimize power right where the computation is happening.
Lu: It means you can use context-aware control to safely trade performance for power when you have memory headroom, which is a big relief because before this, it felt like any reduction in frequency was risky.
Meng: For us shipping models, it means we need to build this kind of context monitoring into our runtime decisions before we scale up to massive infrastructure.
Lalam: This KAIROS paper points toward a future where power optimization isn't just about making things faster or smaller, but specifically about managing the stateful behavior of these multi-turn workflows.
Tom: So, to wrap this up, they’ve presented KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving as a way to reduce power consumption by using agent context to manage GPU frequency and concurrency smartly.
Jane: It’s a design focused on ensuring that when you optimize for power on these complex agents, you don't accidentally destabilize the entire system by letting the context grow too large or pushing the hardware into thrashing.
Lu: The main implication is that agentic inference needs a totally new kind of power management because it’s stateful and dynamic in a way single-turn serving isn't.
Meng: From an engineering standpoint, this gives us clear rules on how to structure our runtime control policies based on the memory envelope, not just arbitrary frequency targets.
Lalam: This work shows that you can achieve significant power savings by making context awareness a first-class signal in your serving system and managing it with these three principles.
The paper's summary: Tom: So, we’re talking about KAIROS now—this paper is about making AI agents use less power without losing performance, and the main takeaway is that agentic work is really different from just running a single prompt.
Jane: Exactly. The authors found that because these agents keep their memory going through many turns while using tools, the context they hold isn't static; it grows in this very dynamic way across different steps and even between different agents.
Lu: That growth is the key here—the paper shows that when you try to save power by lowering the GPU clock too much, you hit a wall where that accumulated context just overflows what the GPU memory can handle. It drives the system into a thrashing state where everything slows down way more than it should because of memory instability.
Meng: So, they’re saying that trying to optimize power in this specific workload isn't as simple as just turning the chip down and hoping for the best; you have to manage the memory envelope carefully.
Lalam: KAIROS proposes three main ways to handle this: first, they dynamically adjust how fast the GPU runs based on how much context is currently being used, saving power when there’s room, but boosting it back up if an agent's progress stalls below a certain speed target.
Tom: That sounds like smart control—balancing that trade-off between energy and throughput based on real-time memory pressure.
Jane: They also regulate concurrency by setting a hard limit on the total context usage across all agents, basically saying, "don't let the aggregate demand exceed this safe memory envelope," which prevents that thrashing we talked about.
Lu: And they use a global router to manage where requests land; it consolidates work onto fewer GPUs when demand is low to exploit lower idle power states, but spreads things out under high context pressure so no single GPU gets overloaded with unstable state.
Meng: From my side, that distributed load management makes sense for production. It’s about making sure we aren't just optimizing one machine in a vacuum; we’re managing the whole serving fleet intelligently.
Tom: The evaluation shows KAIROS manages to get an average power saving of twenty-seven percent on a single instance, up to nearly forty percent with multiple instances, while still hitting that twenty tokens per second speed target for every agent.
Jane: That means for someone just watching this show, the big picture is that you can make these complex agents much more efficient in terms of energy use without sacrificing the progress they actually make over time.
Lu: It’s a fundamental shift because it shows that stateful, context-heavy AI serving needs a totally different power management approach than what we used for single-turn tasks.
Meng: For us shipping models, this gives us concrete rules on how to structure our runtime control policies based on the memory envelope rather than just picking an arbitrary frequency target.
Lalam: This paper suggests that context awareness needs to be a first-class signal in your serving system, and managing it through these three levers is how you achieve significant power savings.
Tom: So, KAIROS gives us a specific blueprint for how to make agentic AI serving more sustainable on hardware without compromising the work itself.
The paper's improvements: Tom: So, KAIROS isn't just about managing frequency; they suggest three specific ways to improve the system even further for better power efficiency.
Jane: They’re talking about improving how GPU frequency control works—they say you can linearly partition safe regions based on aggregate context usage, so you save power when there’s headroom but boost the speed back up instantly if an agent's performance drops below its target.
Lu: That makes sense because it avoids the binary on or off switch for the clock speed; it’s a smoother transition based on actual need.
Meng: And then there's concurrency regulation—they propose regulating how many agents are admitted based on an upper bound of total context usage, making sure the aggregate context demand stays inside that safe memory envelope so you actively prevent thrashing.
Tom: That moves it from a reactive control system to a proactive one, which is what I love when we're training models.
Jane: They also have this power-aware multi-instance scaling and routing idea—using a global router to consolidate agents onto fewer GPUs when demand is low to exploit those low idle power states.
Lu: And under high context pressure, it spreads the load across more instances so no single machine gets pushed into instability locally.
Meng: That’s smart infrastructure thinking; consolidating load at light demand saves energy on unused hardware, which is a huge win for deployment costs.
Tom: It really shows they are looking at the entire serving stack, not just the chip itself.
Jane: This whole set of improvements means that instead of just one solution, you have a layered approach to managing power based on context pressure and system load.
Lu: The limitation they state is that this method is heavily tied to their specific characterization of agentic inference—it works best when the context growth patterns follow those long-tailed distributions they measured.
Tom: So, if we tried this on a completely different kind of workload, like something with very short memory lifetimes, would it still work as well?
Jane: That’s exactly where the caveat comes in; their findings are specific to how state builds up over many turns using tools and agents.
Lu: The implication is that for agentic AI, we need to focus our power optimization efforts on capturing and controlling that long-lived context growth, because that's the source of all the instability they identified.
Conclusion: Tom: So to wrap up, KAIROS is this paper’s full title: "Stateful, Context-Aware Power-Efficient Agentic Inference Serving," and they show that you can use context awareness to save significant power in agentic AI serving.
Jane: It means we can design systems for complex AI agents that are more energy efficient because they understand the stateful nature of their work.
Lu: The core idea is that agentic inference isn't just a scaled-up version of single-turn serving; the dynamic context growth requires a fundamentally different power management strategy.
Meng: For production, it gives us concrete rules on how to structure our runtime control policies based on memory envelopes instead of just guessing frequency targets.
Lalam: I’m very confident in this result because the system they designed handles the actual stateful behavior of these agents quite well under heavy load.
Tom: It really boils down to treating context usage as a first-class signal for hardware control, which is something we need to push harder on across all our models.
Jane: The implication for people watching this is that the AI they use won't just be faster; it’ll also be more sustainable in terms of energy consumption when running those long, multi-step tasks.
Lu: This paper sets a new baseline for how we think about the power-performance trade-off in stateful systems.
Meng: We still need to see how this scales on truly massive clusters, but it’s a solid starting point for designing better inference infrastructure that actually respects memory limits.
Yichao Yuan, Mosharaf Chowdhury, Nishil Talati
University of Illinois Urbana Champaign · University of Michigan
cs.DC, cs.AI
Submitted: 2026-04-17
Updated: 2026-11-10
Code: https://github.com/harbor-framework/harbor
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 93/100
The gist: Power has become a central bottleneck for AI inference This paper presents KAIROS, a context-aware power optimization system for agentic AI serving that reduces power consumption while maintaining a…
Key concepts
- Agentic Inference Serving
- This refers to serving AI tasks where an agent interacts with tools over multiple turns. Unlike simple single-turn LLM serving, these agents maintain a long-lived state by accumulating context across tool uses and different agents, making power management significantly more complex.
- Context Growth and Thrashing
- Agentic workloads cause context to grow dynamically across turns and agents, exhibiting long tails in distribution. Lowering GPU frequency too much can push the system into a 'thrashing' regime where accumulated context exceeds GPU memory capacity, showing a strong link between power settings and memory stability.
- Throughput-Based SLO
- The paper advocates using throughput (tokens per second) as the primary Service Level Objective (SLO) for agentic serving instead of latency metrics. This metric better captures the progress of an entire agent workflow over its full lifecycle, providing a more actionable target for online power control.
- Context-Aware Frequency Control
- This mechanism dynamically adjusts GPU frequency based on how much context pressure exists. It saves power by lowering the frequency when memory headroom is available but boosts it to maximum levels if individual agent performance falls below its required throughput SLO.
Terminology
Summary
Power has become a central bottleneck for AI inference This paper presents KAIROS, a context-aware power optimization system for agentic AI serving that reduces power consumption while maintaining a target performance SLO The key contribution is a detailed characterization of agentic inference serving that reveals why power optimization is fundamentally different and harder compared to single-turn LLM serving KAIROS achieves an average of 27% (up to 39. p. 1 Our key contributions are:
• A detailed characterization of agentic inference serving
that reveals why power optimization is fundamentally
different and harder compared to single-turn LLM serving. • A context-aware control mechanism that jointly manages
GPU frequency and concurrency using context growth,
memory pressure, and agent progress to reduce power
while avoiding thrashing and preserving SLO attainment. Evaluated across diverse software and data engineering agentic
tasks, KAIROS achieves an average of 27% (up to 39. 8%)
power reduction while meeting the performance targets.Where the paper says it
Where the paper says it
Characterization of Agentic Inference Serving
The analysis reveals that agentic workloads maintain long-lived state for each request through a context cache that grows across tool-interleaved turns and varies widely across agents This context growth is highly dynamic, varying substantially over time within and across agents, and the distributions of agent lifetimes, turn counts, and growth rates all exhibit long tails. The study uncovers a new power-management phenomenon: lowering GPU frequency beyond a certain value can drive the system into a thrashing regime where accumulated context exceeds GPU memory capacity. p. 0 This makes
memory pressure difficult to predict and control, and creates a fundamental tension for power management: lowering
GPU frequency saves power, but also slows execution, and in
agentic workloads that slowdown accumulates across turns
and can directly hurt per-agent progress. Our analysis also uncovers a new power-management
phenomenon: lowering GPU frequency beyond a certain
value can drive the system into a thrashing regime, where
accumulated context exceeds GPU memory capacity. As a
result, the context needs to be recomputed [24, 29] or loaded
from slower memory tiers [8] that degrade throughput, latency, and power efficiency. This strong coupling between
power and memory stability shows that agentic serving is
not merely a scaled-up version of single-turn LLM serving,
but a fundamentally different workload whose stateful and
dynamic behavior reshapes the power-performance trade-off.Where the paper says it
Where the paper says it
Design Goals and Overview
KAIROS is designed around three key principles to achieve its optimization objective 1) dynamically adjusting GPU frequency based on context pressure to safely trade performance for power savings when memory headroom exists, 2) regulating concurrency to prevent excessive context growth that can destabilize the system, and 3) scaling and routing requests across multiple serving instances in a power-aware manner to consolidate load at low demand and spread load under high context pressure. The design overview shows that KAIROS is designed around the three goals in §5. p. 4 Figure 6 shows the high-level design of KAIROS. KAIROS
is designed around the three goals in §5. KAIROS
is designed around the three goals in §5. 1 to address the
challenges (§3.Where the paper says it
Where the paper says it
KAIROS Control Mechanisms
The unified runtime combines lightweight request tracking, per-instance control, and global routing using agent context as a first-class signal The Context-Aware Per-Instance Controller monitors context growth and system load to adapt GPU frequency and admission decisions. This controller includes Control Policy Formulation, Context-Aware Frequency Control, SLO-based Frequency Boosting, and Thrashing Avoidance. p. 7 At each
control epoch, the controller first updates the current context
usage, per-agent throughput, and average power, and then
selects a baseline frequency using the context-aware policy. It next applies two corrective mechanisms: SLO-based boosting to recover performance when throughput falls below
target, and thrashing avoidance to regulate admissions and
keep context usage within a safe memory region. Under peak
demand, the router tends to utilize all available instances,
while under lighter load it consolidates agents onto a smaller
subset of GPUs and leaves the remaining instances idle to
avoid unnecessary power overhead1
. The router achieves
this through two components: an assignment policy for initial placement and a reassignment policy for dynamic load
balancing.Where the paper says it
Where the paper says it
Evaluation Results
KAIROS reduces average GPU power by 27% (up to 39.8%) for a single serving instance and 46. p. 1 KAIROS
reduces average GPU power by 27% (up to 39. 8%) for a single
serving instance and 46. 8%) for a single
serving instance and 46. 3% for multi-instance serving while
maintaining per-agent SLO of P5 throughput 20 tokens/s. 0
terminus-2
Uncontrolled Fixed Freq (810Mhz) STEER
Figure 7. Comparison of throughput, SLO attainment (20 tokens/s per-agent target), energy, and power across request rates
for no frequency control, fixed 810 MHz, and KAIROS. While both fixed
frequency baseline and KAIROS reduce power relative to
no control, the fixed-frequency baseline achieves slightly
larger power savings at the cost of significantly degraded
SLO attainment. In contrast, KAIROS strikes a balance by
achieving an average power saving of 27% (up to 39. In contrast, KAIROS strikes a balance by
achieving an average power saving of 27% (up to 39. 8%) while
meeting the per-agent SLO target of 20 tokens/second.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Discussion on Metrics and Tradeoffs
The paper argues for a throughput-based SLO for agentic serving instead of latency-based metrics like TTFT or TBT because it better captures end-to-end agent progress over its full lifecycle. Throughput directly reflects runtime agent progress under shared execution, making it a more actionable SLO for online control. p. 4 TTLT is less
actionable for online control because it depends on an isolated request-specific latency that is not available in online
serving at runtime. In contrast, throughput directly reflects
runtime agent progress under shared execution, making it a
more actionable SLO for agentic serving. Context-Aware Frequency Control
addresses DG1 and tackles C1/C3 by trading power for performance based on context pressure. SLO-based Frequency
Boosting preserves throughput targets under slowdown, directly addressing C1. 08 agents/s. At 1185 MHz, execution slows only slightly
relative to 1680 MHz, so tasks still complete fast enough to
keep memory pressure stable.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Context-aware frequency control: The system can reduce GPU frequency by linearly partitioning safe regions based on aggregate context usage to save power when headroom exists, while dynamically boosting frequency to maximum levels if per-agent throughput falls below a target SLO.
-
Concurrency regulation: The system can regulate the number of admitted agents based on an upper bound on total context usage, ensuring that
aggregate context demand remains within a safe memory envelope
and preventing the system from entering a thrashing regime whereaccumulated context exceeds GPU memory capacity.
-
Power-aware multi-instance scaling and routing: The system can employ a global router to dynamically adapt instance utilization, consolidating agents onto fewer GPUs under light demand to exploit low idle-power states, while spreading agents under high context pressure to avoid local memory instability.
Abstract
Power has become a central bottleneck for AI inference. This problem is becoming more urgent as agentic AI emerges as a major workload class, yet prior power-management techniques focus almost entirely on single-turn LLM serving. Our analysis shows that agentic serving behaves fundamentally differently: each request carries long-lived context that evolves across tool-interleaved turns, and lowering GPU frequency can push the system into a thrashing regime where memory pressure sharply worsens both performance and power efficiency. These observations show that power optimization for agentic serving requires rethinking. We present KAIROS, a context-aware power optimization system for agentic AI serving. KAIROS uses agent context as a first-class control signal to jointly manage GPU frequency, per-instance concurrency, and multi-instance request placement. This enables KAIROS to save power when memory headroom exists while avoiding thrashing and preserving performance targets. At a high level, KAIROS tracks requests at agent granularity, adapts local control to context growth and agent progress, and routes agents across instances to jointly improve power efficiency and memory stability. Evaluated across diverse software and data engineering agentic tasks, KAIROS achieves an average of 27% (up to 39.8%) power reduction while meeting the performance targets.
Sources
- No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
- Efficient and Scalable Agentic AI with Heterogeneous Systems
- DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS
- Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization
- Where Do the Joules Go? Diagnosing Inference Energy Consumption
- DABstep: Data Agent Benchmark for Multi-step Reasoning
- Efficient Multi-round LLM Inference over Disaggregated Serving
- The Price of Prompting: Profiling Energy Use in Large Language Models Inference
- SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
- SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
- ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
- Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading
- TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
- Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
- EcoServe: Designing Carbon-Aware AI Inference Systems
- Autellix: An Efficient Serving Engine for LLM Agents as General Programs
- Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Related papers
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends
- AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition