Scaling LLM Agent Tool Access: The MCP Gateway
cs.DC, cs.AI, cs.NI
Submitted: 2026-07-17
Updated: 2026-11-02
Comments: Accepted at the 2026 ACM SIGOPS Annual Technical Conference (SIGOPS ATC 2026). Compared with the ATC version, this version adds supplementary appendix material and corrects an error in Figure 9. See the statement for details
Project page: https://microsoft.github.io
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: LLM agents increasingly rely on tool calling to complete tasks.
Terminology
Abstract
LLM agents increasingly rely on tool calling to complete tasks. Model Context Protocol (MCP) standardizes tool calling between LLM agents and tool providers. It reduces integration cost and drives rapid growth in tool scale. In production deployments, tool access becomes difficult to scale. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development creates ongoing compatibility cost. On the agent side, the number of mounted tools is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale MCP Gateway system. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing at the gateway. The conversion module supports 315 OpenAPI services in the cloud and integrates MCP transport variants. Session-aware routing decouples the scalability of the gateway layer and the MCP service layer. The hybrid retrieval engine sustains about 98% Top-15 recall; it scales agent tool access to 3, 000+ tools with improved tool selection accuracy. We also share the lessons learned from deploying the gateway system in production.
Sources
- TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
- BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models
- Infrastructure for AI Agents
- Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud
- AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework
- RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation
- AgentScope: A Flexible yet Robust Multi-Agent Platform
- AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications
- A Measurement Study of Model Context Protocol Ecosystem
- MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Security
- Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
- Nalar: An agent serving framework
- Autellix: An Efficient Serving Engine for LLM Agents as General Programs
- AIOS: LLM Agent Operating System
- MemGPT: Towards LLMs as Operating Systems
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- Improving API Documentation Comprehensibility via Continuous Optimization and Multilingual SDK
- A-MEM: Agentic Memory for LLM Agents
- A Survey of AI Agent Protocols
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends