te-comp: A Complete MLIR Compiler Generating Self-Contained C Inference Engines for MCUs
cs.PL, cs.RO
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/nickoala/cardputer-ai
License: http://creativecommons.org/licenses/by/4.0/
The gist: Deploying deep learning on microcontrollers (MCUs) requires fitting both the model weights and runtime activations within tens to hundreds of kilobytes of SRAM and Flash.
Terminology
Abstract
Deploying deep learning on microcontrollers (MCUs) requires fitting both the model weights and runtime activations within tens to hundreds of kilobytes of SRAM and Flash. Existing approaches either use an interpreter with a pre-written kernel library or combine a Python code generator with hand-written C kernels. This incurs dispatch overhead and requires framework dependencies. None produces self-contained, dependency-free C code specialized to a given model. In this paper, we present te-comp, the first complete MLIR-based compiler that takes a standard model in.tflite or.safetensors format and emits all C source files needed for deployment: the inference engine (kernels, weight arrays, memory layout) and the RTOS integration layer with zero runtime dependencies. On 10 MCUNet CNN benchmarks, te-comp outperforms MCUNet TinyEngine by using 38% less SRAM (up to 5.9 times) and 15% less Flash, while generating complete C code rather than linking a pre-written kernel library. To demonstrate, we use te-comp to compile an 8.3M-parameter GPT-Neo transformer (TinyStories-v2) and deploy it on a single-core Cortex-M7 at 160MHz. The full firmware uses 249KB of 512KB SRAM and 3.62MB of 4MB Flash and achieves approximately 1 token/s interactive text generation via Q4 0 weight quantization, a stateful int4 KV cache, and vocabulary pruning -- a capability no existing MCU inference framework supports.
Sources
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs
- ExecuTorch -- A Unified PyTorch Solution to Run AI Models On-Device
- Glow: Graph Lowering Compiler Techniques for Neural Networks
Related papers
- The Best of Times, the Worst of Times: Moment-Based Analysis of Probabilistic Cost Structures
- Formal Verification of an Out-of-Order Multiprocessor against an In-Order Weak-Memory ISA
- Fuss-free cumulative universes: theory and practice
- High-Level Synthesis of Efficient Pipelines with Visibility Control
- Model Checking Probabilistic Multiparty Session Types
- A Type System for Communication-Safe Hardware Design