EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities
cs.OS, cs.AI, cs.CL, cs.SE
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/pytorch/executorch
License: http://creativecommons.org/licenses/by/4.0/
The gist: Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge.
Abstract
Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.
Sources
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- OpenAI GPT-5 System Card
- Gemma 3 Technical Report
- Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks
- Learning Globally Reusable Skills for Coding Agents
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
- debug-gym: A Text-Based Environment for Interactive Debugging
- Where LLM Agents Fail and How They can Learn From Failures
- MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?