Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI
cs.OS
Submitted: 2026-05-30
Updated: 2026-11-11
Code: https://github.com/openai/codex
License: http://creativecommons.org/licenses/by/4.0/
The gist: Agentic programs alternate between LLM inference and tool calls, each request extending the previous one's context.
Terminology
Abstract
Agentic programs alternate between LLM inference and tool calls, each request extending the previous one's context. Program-aware schedulers pin a program's KV cache in GPU HBM across tool calls to avoid re-prefilling it at every step, but as contexts grow, pinned KV cache exceeds HBM, raising program eviction: deciding when a program releases GPU memory and when it returns. Offloading to CPU DRAM lets an evicted program wait out a tool call instead of recomputing its context, but existing schedulers leave DRAM to LRU, so offloaded programs thrash. Making these decisions is hard: tool-call durations vary by orders of magnitude and are unknown in advance. We observe that programs nevertheless stay busy (short tool calls) or idle (long ones) for extended periods, and use a program's idleness, its recent fraction of time in tool calls, as a continuous, relative metric. MORI ranks programs by idleness, keeps the least idle in HBM and the next in DRAM, and drops the rest, enforcing admission control at both tiers under any GPU-to-CPU capacity ratio; it also proactively demotes idle programs to make room for ready ones. On real coding-agent traces, MORI achieves on average 64% higher throughput and 72% lower p90 TTFT than the best offloading baseline under high load.
Sources
- InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- The Llama 3 Herd of Models
- M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
- SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
- ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
- Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
- Combee: Scaling Prompt Learning for Self-Improving Language Model Agents
- Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- Autellix: An Efficient Serving Engine for LLM Agents as General Programs
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- Fast Transformer Decoding: One Write-Head is All You Need
- Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
- Teola: Towards End-to-End Optimization of LLM-based Applications
- Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective