ConvLUT: Rethinking Convolutional Architectures for Ultra-Low-Latency LUT-Based Inference
cs.LG
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/mlcommons/tiny_results_v1.1
License: http://creativecommons.org/licenses/by/4.0/
The gist: Lookup-table (LUT)-based neural networks (NNs) offer a compelling path to ultra-low-latency inference on FPGAs.
Abstract
Lookup-table (LUT)-based neural networks (NNs) offer a compelling path to ultra-low-latency inference on FPGAs. However, existing LUT-based neural networks have largely been restricted to MLP-style topologies, leaving structured vision and audio tasks underexplored. We present ConvLUT, a family of fully unrolled LUT-mappable convolutional architectures targeting the MLPerf Tiny benchmarks. It is comprised of a standard convolutional model built around a hierarchical tree of LUTs and a depthwise separable model following the MobileNet design philosophy. These architectures are enabled by four key contributions: a unified LUT-based convolutional framework supporting standard, depthwise, and pointwise convolutions under explicit fan-in constraints; a LUT-tree that addresses the k 2c receptive field dimensionality bottleneck; a channel-partitioned LUT pooling layer that replaces conventional spatial pooling; and a hardware-aware iterative pruning strategy for stable convergence at scale. Evaluated on CIFAR-10, Visual Wake Words, and Keyword Spotting, ConvLUT achieves competitive accuracy with fully unrolled inference latencies below about 50 ns. On CIFAR-10 it reaches at least a 8844 times latency reduction over non-logic-based prior implementations, and outperforms existing MLP-style LUT-based models on the same task by at least 20 pp in test accuracy.
Sources
- MLPerf Tiny Benchmark
- Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark
- Visual Wake Words Dataset
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- TreeLUT: An Efficient Alternative to Deep Neural Networks for Inference Acceleration Using Gradient Boosted Decision Trees
- LUTNet: Learning FPGA Configurations for Highly Efficient Neural Network Inference
- Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis