AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V".
Jane: Here's the gist of the paper. Deploying Vision Transformers (ViTs) on low-power edge devices is challenging due to high computational demands.
Tom: Conventional pruning frameworks require a separate compiled binary for each sparsity level, increasing storage overhead and limiting runtime adaptability.
Jane: This paper presents an end-to-end deployment pipeline that transforms pretrained ViTs into a single runtime-configurable binary, enabling dynamic compute-budget switching on embedded CPUs.
Tom: Pruning redundant reasoning tokens The proposed method achieves significant reductions in storage requirements and increases inference speedup through hardware-aware structured pruning and ISA extensions.
Title and authors: Tom: So we're looking at this paper now called AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V. It sounds like they are tackling the problem of getting these big vision models onto low-power devices without needing a separate compiled binary for every single level of sparsity.
Jane: That’s right, Tom, and it seems the core idea is making one binary that can switch its compute budget dynamically on an embedded CPU just by loading a small configuration file. It's about flexibility for those edge devices we use all the time.
Lu: What’s really interesting here is how they handle the transformation from a high-level model to bare-metal C code, because usually, when you do dynamic selection of active dimensions in Python, it’s easy, but translating that into safe skipping at coarsegrained block boundaries in C without adding too much overhead is tough.
Meng: From my side on deployment, I worry about that control-flow overhead you mentioned. If the transformation adds too much complexity to the execution path inside the compute-intensive regions, it just slows down inference instead of helping.
Lalam: I'm seeing that they are using TVM to map ViT operations into C kernels across encoder blocks, and profiling on the target RISC-V processor shows these specific kernels are doing most of the work—about seventy-nine percent for ViT Small and eighty-three percent for ViT Base >
Tom: Exactly, and that leads us to how they tackle the actual computation. They introduce a hardware-aligned block pruning strategy specifically for Multi-Layer Perceptron or MLP layers to make sure that pruning is as efficient as possible when you're targeting this specific processor architecture.
Jane: And on top of that, they propose a custom instruction called MAC2 to exploit how input activations are reused in linear projection kernels, which they say helps with speedup and energy efficiency.
Lu: That ISA extension is significant because it's tailored to the dominant operations; it’s not just some generic vector extension, but something that directly addresses the specific pattern of input reuse in those projection layers.
Title and authors: Meng: I’m curious about what this MAC2 instruction actually does in terms of instructions saved per iteration, because if it reduces the instruction count significantly, that translates directly into lower power consumption when we're running these models at scale on our hardware.
Tom: The paper claims that this custom MAC2 instruction provides a one point five six times speedup and a thirty-three percent lower inference energy specifically for the unpruned model, which is pretty compelling.
Jane: But let’s look at the trade-offs, because they aren't just talking about speed; they are also talking about storage. They claim this runtime-adaptive approach reduces on device storage by up to four point eight six times compared to using separate binaries for each pruning level.
Lu: That massive reduction is achieved because they store only one full model and use lightweight metadata for switching, which is much better than having nearly eight hundred megabytes on device >
Meng: four point eight six times sounds huge, but I need to know what that means in practice when we have to manage those configuration files for switching between levels. Is the overhead of switching actually small?
Tom: They say the switching cost is minimal, just about fifty-seven thousand cycles for ViT Small and one hundred fourteen thousand cycles for ViT Base, which they call classifier handling >
Jane: That’s a very low number when you think about how often we might need to switch during a real inference run. So, if we can get that kind of speedup and storage saving while keeping switching overhead this low, what does that mean for deploying these kinds of vision models on things like mobile phones or even small embedded systems?
Lu: It means the architecture is designed to be highly flexible; the C code generation is structured so you can skip computation at coarsegrained block boundaries with minimal overhead, which solves that control-flow problem we talked about earlier >
Meng: So it’s less about having a fixed, perfect model and more about having a single deployable artifact that adapts to the immediate constraints of the device running it >
Title and authors: Tom: That’s the core idea—a single runtime-configurable binary—and they show that this approach allows dynamic compute budget switching on embedded CPUs, which was previously very hard to achieve >
Jane: So we've seen how they generate these multiple sparsity levels offline using post-training pruning, and then how the unpruned backbone is quantized to INT8 using TORCHAO while keeping nonlinear operations in floating point for accuracy >
Lu: That quantization step is interesting because it shows they are balancing storage reduction with maintaining reasonable accuracy, as the accuracy drops by five point zero seven percent when they apply the INT8 quantization on top of the pruning >
Meng: So you get a trade-off: huge storage savings versus a small drop in accuracy from that INT8 move, and then you've got this runtime switching mechanism to toggle between those states >
Tom: And that dynamic switching is what makes it useful for real-world edge deployment where the computational budget changes based on the workload at any given moment >
Jane: So, to wrap up, AdaptViT is an end-to-end framework for runtime adaptive Vision Transformer inference on RISC-V processors, showing up to two point eight times speedup and a thirty-three percent energy reduction at zero pruning with the MAC2 extension >
Lu: The authors also flag that they are exploring extending this mask-based skipping to nonlinear layers in future work, which suggests they aren't trying to solve everything perfectly right now >
Meng: And for me, from a production standpoint, the focus is on getting this single binary and its runtime adaptability working reliably across different model sizes like ViT Small and ViT Base >
Lalam: I think what really stands out is how they align structured sparsity with hardware execution granularity to make it efficient and flexible for edge devices >
Tom: So, if you're deploying a Vision Transformer on an embedded RISC-V processor and you need dynamic compute budget switching, the AdaptViT framework shows a path toward achieving significant speedup and storage savings through hardware-aware pruning and custom ISA extensions >
The paper's summary: Tom: So, to recap, this paper is about taking those big Vision Transformers and making them fit on low-power edge devices by turning them into a single binary that can change its speed and memory usage on the fly, instead of needing a whole new file for every level of compression >
Jane: Exactly. The big thing here is moving away from having to compile a separate program for each sparsity level, which used to make things take up huge amounts of storage on the device >
Tom: Right. And they achieved this by doing some smart pruning—specifically block pruning for the MLP layers and a custom instruction called MAC2 that helps with how the processor handles input reuse >
Jane: That custom instruction is pretty interesting because it’s specifically designed for those linear projection kernels, which are where most of the work happens in these models >
Tom: And the results are pretty solid, they show up to two point eight times speedup on a RISC-V processor like the TRV32P3FX when they use this setup and they see that massive storage reduction too >
Jane: That storage saving is huge—they're talking about reducing the memory needed for ViT-Base by almost five times compared to having those separate binaries >
Tom: It’s not just about speed, though. They also saw a thirty-three percent energy reduction at zero pruning just because of that MAC2 extension itself, which is pretty significant for battery life >
Jane: But the paper also shows they have to be careful with accuracy. they use INT8 quantization on parts of the model and see a drop in accuracy, about five point zero seven percent relative to the full floating-point version >
Tom: That’s the trade-off you always run into when you try to squeeze performance and size together, but they say it’s a reasonable balance for edge deployment >
Jane: So, what this means for us is that we can finally take these powerful Vision Transformers and deploy them on devices with tight constraints, like mobile phones or small embedded systems, without having to compromise too much on speed or storage >
Tom: It really changes how you think about deploying these models; it moves from a static deployment to something truly dynamic where the model adapts to the exact resources available at that moment >
Jane: And this flexibility is what makes it possible for vision AI to move beyond just big data centers and get into real-world edge applications, where things are constantly changing >
Tom: It’s about making these models practical, not just theoretical demonstrations on a massive GPU >
Jane: So next up, we’re going to look at how they actually built this system. We need to see how they take the trained model and turn it into this single runtime configuration file that can switch between states.
The paper's improvements: Tom: So, moving on to what they actually suggest doing next—the improvements—they’re talking about extending this idea beyond just MLP layers >
Jane: Right, they want to see if we can apply this same runtime-adaptive skipping logic to the nonlinear parts of the ViT, not just the linear projections >
Tom: That makes sense. Right now, they've focused on structured pruning for those MLP layers—grouping them into blocks and ranking them before pruning—but that’s just a start >
Jane: And then there’s this whole idea of using vector or matrix compute engines in the future to handle things differently, which could help with the energy efficiency gains we saw at zero pruning >
Tom: Yeah, they want to see if you can get better hardware support for those patterns so you can skip computations more aggressively across different layers >
Jane: It’s about making sure that this runtime-adaptive approach isn't just a trick for the MLP, but something that works consistently across the whole model structure >
Tom: And they are also looking at how to use this mechanism to manage the compute budget even more dynamically based on what kind of input we are processing >
Jane: That means in theory, we could have a system that decides which parts of the model to run with full precision and which parts can be aggressively pruned based on battery level or processing load >
Tom: It’s about making the deployment truly adaptive to the environment, not just static selection based on a pre-set configuration file >
Jane: And if they succeed in extending this mask-based skipping to nonlinear layers, it would mean these complex Vision Transformers could run much more efficiently on devices that have very limited power budgets >
Tom: It’s ambitious. They’re essentially trying to bring the flexibility of software control flow into the actual hardware execution pipeline so it can adapt on its own >
Jane: So, while they haven't fully done it yet—they flag that extending mask-based skipping to nonlinear layers is future work—the direction they’re pushing is toward a model that’s truly self-optimizing for the hardware it’s running on >
Tom: It’s about taking this from just a clever pruning technique to a full, flexible inference system for edge AI >
Jane: And that leads us perfectly into how they built the pipeline in the first place. We need to see how they took a massive pre-trained model and turned it into this single binary that can switch its behavior on the fly.
Conclusion: Tom: So we’re wrapping up AdaptViT, which is this framework for runtime-adaptive Vision Transformer deployment on custom RISC-V processors that lets you switch compute budgets dynamically >
Jane: It was a lot of technical stuff, but the main point is that they managed to turn a massive model into something flexible enough to run on small hardware without needing eight hundred megabytes of storage >
Tom: Right. The speedup numbers are impressive, up to two point eight times faster than the unpruned version on that specific processor when they use this structured pruning and the custom instruction >
Jane: That’s a big win because it shows that even though we are moving towards smaller devices, we can still get really high performance if we design the software to be adaptable rather than just optimizing for one fixed scenario >
Tom: It’s about making these models practical for real-world edge deployment, not just theoretical demos on a huge GPU cluster >
Jane: And they did manage to show that there's a trade-off with accuracy when you use that INT8 quantization layer, which is something we need to keep in mind >
Tom: True. It’s a balancing act between how much space you save and how good the Vision Transformer actually is at seeing things >
Jane: So what does this mean for us? It means vision AI can get onto devices that aren't designed for massive computation, provided we use smart techniques to make the model fit the hardware constraints dynamically >
Tom: Exactly. This AdaptViT paper shows a clear path forward for making Vision Transformers more accessible on constrained edge devices >
Jane: Before we move on to another paper, I want a quick word from Lu about what this means for representation learning in general >
Lu: Well, this proves that the idea of runtime adaptability is viable even when you’re dealing with very complex architectures like ViTs because the structural transformations are consistent across different model sizes >
Meng: For me, it’s about deployment cost. If we can reduce the storage footprint by nearly five times, that lowers our entire operational cost for running inference at scale, even if the switching overhead is small >
Lalam: From my side, I’m sure of this. The mechanism they use to store just one full model and swap configurations is robust; I’m confident in the results of AdaptViT >
Tom: Okay, so that covers AdaptViT for now. We'll be looking at some papers on session types next to see how other models handle dynamic constraints >
University College Dublin
cs.AR
Submitted: 2026-10-01
Updated: 2026-11-13
Comments: Accepted for publication at the 2026 IEEE 33rd International Conference on Electronics, Circuits and Systems (ICECS)
Code: https://github.com/psvishnu1/AdaptViT
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 89/100
The gist: Deploying Vision Transformers (ViTs) on low-power edge devices is challenging due to high computational demands.
Key concepts
- Runtime-Adaptive Inference Pipeline
- This is the core innovation where a single compiled binary supports multiple sparsity levels. Instead of needing separate files for each level, the system uses modified loop bounds and mask logic within one executable. This allows the device to switch between different computational budgets dynamically using simple external configuration files during runtime.
- Hardware-Aware Structured Pruning
- This strategy focuses pruning efforts on Multi-Layer Perceptron (MLP) layers by grouping neurons into blocks of eight. By ranking these blocks based on importance before pruning, the method maximizes efficiency and achieves a 65% reduction in MAC operations, which is crucial for hardware compatibility.
- Custom ISA Extension (MAC2 Instruction)
- The authors propose a custom instruction for the RISC-V processor that exploits input reuse. This instruction allows the processor to perform two parallel multiply-accumulate operations using a single shared activation and two weights simultaneously. This reduces the number of instructions needed in critical loops, leading to up to 1.56x speedup.
- Storage Reduction Analysis
- The runtime-adaptive approach drastically cuts storage needs by only storing one full model and using small metadata for switching configurations. This method reduces on-device storage requirements by up to 4.86 times compared to deploying separate binaries for every possible sparsity level.
Terminology
Summary
Deploying Vision Transformers (ViTs) on low-power edge devices is challenging due to high computational demands. p. 0 Conventional pruning frameworks require a separate compiled binary for each sparsity level, increasing storage overhead and limiting runtime adaptability. p. 0 p. 0 This paper presents an end-to-end deployment pipeline that transforms pretrained ViTs into a single runtime-configurable binary, enabling dynamic compute-budget switching on embedded CPUs. p. 0 p. 0
Pruning redundant reasoning tokens
The proposed method achieves significant reductions in storage requirements and increases inference speedup through hardware-aware structured pruning and ISA extensions. The authors introduce a hardware-aligned block pruning strategy for Multi-Layer Perceptron (MLP) layers to maximize pruning efficiency. p. 0 Furthermore, a custom ISA extension is proposed to exploit input-reuse patterns in linear projection kernels. p. 0 On the Synopsys TRV32P3FX RISC-V processor, the full system achieves up to 2. p. 08× speedup at 65% MLP and 50% attention-head pruning for ViT-Base. p. 0
How the training signal is built
The offline processing stages involve post-training pruning to generate multiple sparsity levels, extending SnapViT [10] with hardware-aligned block pruning for MLP layers while retaining per-head pruning for attention blocks. The unpruned backbone is then partially quantized to INT8 using TORCHAO [11], where linear projection layers are quantized while nonlinear operations remain in floating point to preserve accuracy
Where the gains come from
The speedup of up to 2.8× at the highest pruning level for ViT-Base is achieved through several optimizations. p. 3 The ISA extension alone provides a 1. p. 056× speedup and 33% lower inference energy. p. 0 The proposed runtime-adaptive approach reduces on-device storage by up to 4. p. 086×, requiring only 163 MB for ViT-Base instead of nearly 800 MB. p. 0
Inference and Hardware Efficiency
The framework enables runtime adaptability by restructuring generated C kernels with modified loop bounds and binary-mask control logic. This allows execution to switch across discrete sparsity levels via compact external configuration files. p. 0 The storage analysis shows that the proposed approach reduces on-device storage by up to 4. p. 086× over separate per-pruning-level binaries. p. 2 Switching between pruning configurations incurs minimal overhead, reducing it to redirecting pointers to the corresponding mask buffers. p. 2
Conclusion
This work presents an end-to-end framework for runtimeadaptive Vision Transformer (ViT) inference on embedded RISC-V processors. The proposed system achieves up to 2. p. 08× inference speedup, 33% energy reduction, and up to 4. p. 386× storage savings. p. 3 Overall, aligning structured sparsity with hardware execution granularity enables efficient and flexible ViT deployment on edge devices. p. 3 The future work will extend mask-based skipping to nonlinear layers and explore heterogeneous acceleration using vector or matrix compute engines. p. 3
--- Page 1 ---
AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V
Abstract—Deploying Vision Transformers (ViTs) on low-power edge devices is challenging due to high computational demands. Conventional pruning frameworks require a separate compiled binary for each sparsity level, increasing storage overhead and limiting runtime adaptability. This paper presents an end-to-end deployment pipeline that transforms pretrained ViTs into a single runtime-configurable binary, enabling dynamic compute-budget switching on embedded CPUs.
I. INTRODUCTION
Deploying deep learning models on edge devices requires high computational efficiency without significant loss in accuracy [1]. Vision Transformers (ViTs) achieve strong performance but incur high computational and memory costs due to Multi-Head Self-Attention (MHSA) and large Feed-Forward Networks (FFNs), limiting their suitability for embedded deployment [2]–[4]. p. 0 However, edge environments are dynamic, with changing constraints due to battery levels, workload changes, or user demands. p. 0
--- Page 2 ---
Fig. 1: MLP Pruning Method. p. 1
A. C Code Transformation
While dynamic selection of active dimensions is straightforward in high-level environments such as Python, reproducing this behavior in bare-metal C is challenging. Introducing conditional mask checks inside compute-intensive regions adds control-flow overhead, reducing efficiency. p. 1 Therefore, exploiting sparsity on embedded targets requires a post-compilation transformation that restructures loops to enable safe skipping of computation at coarsegrained block boundaries with minimal overhead. p. 1
--- Page 3 ---
Fig. 2: Accuracy vs. p. 0 pruning level. p. 0
A. C Code Transformation
The TVM compiler maps ViT operations into C kernels across encoder blocks. Profiling on the target RISC-V processor shows these kernels dominate execution, accounting for approximately 79% and 83% of total inference cycles for ViT-Small and ViT-Base, respectively. p. 1
--- Page 4 ---
B. Framework Generalization
Although ViT variants differ in dimensions, the TVMgenerated kernel structure and execution order remain consistent. This enables the same post-compilation transformations to be applied across models ViT-Small and ViT-Base. p. 1
--- Page 5 ---
C. ISA Extension
The innermost loop of all four dense pack kernels performs eight MAC operations per iteration, reusing a single input activation across eight weights. While the baseline TRV32P3FX core supports fused floating-point MACs via the F extension, the quantized workload is dominated by integer operations, requiring 16 instructions (8 mul + 8 add) per iteration. p. 1 The custom hardware instantiates two parallel MAC pipelines sharing operand a, broadcasting it to both multipliers while independently processing b0 and b1. p. 1
--- Page 6 ---
III. RESULTS
Pruning mask generation across multiple sparsity levels, quantization, and C code generation using TVM (v0.14.dev273) are performed offline on an NVIDIA A100 GPU in Google Colab. p. 2 Evaluation is conducted on the Oxford-IIIT Pet dataset (37 classes) Fig. p. 2 4 presents energy and accuracy trends across pruning levels on the extended RISC-V, relative to the unpruned baseline. p. 3
--- Page 7 ---
A. Accuracy Impact
Fig. 2 shows accuracy across six pruning configurations for two ViT variants. p. 2 The x-axis denotes runtime settings as X/Y,where X and Y represent MLP group and attention head pruning ratios, respectively. p. 2 The third applies INT8 quantization, leading to an additional 5. p. 207% drop relative to FP32. p. 2
--- Page 1 ---
B. Storage Requirement Analysis
Let d, dmlp, L, and b denote the embedding dimension, MLP dimension, number of transformer blocks, and bytes per weight. For pruning configuration i with MLP-pruning ratio r(i) mlp and head-pruning ratio r(i) head the retained weight fraction fi = 1/3 (1 − r(i)head) + 2/3 (1 − r(i)mlp), Wi = 12Ld squared bfi The proposed method reduces on-device storage by up to 4. p. 286× over separate per pruning level binaries. p. 2
--- Page 1 ---
C. Runtime Switching Overhead
Switching between pruning configurations incurs minimal overhead, where the only remaining overhead is classifier handling. The switching costs ∼57K / 114K cycles for ViT-Small / Base. p. 3
--- Page 3 ---
D. Inference Speedup Evaluation
Fig. 3 shows speedup across pruning levels for ViT-Small and ViT-Base (DINO), across two settings. p. 3 The MAC2 instruction provides up to 1. p. 356× speedup for the unpruned model. p. 3
--- Page 4 ---
E. Hardware Overhead
The hardware synthesis results in Table II show that FPGA implementation adds 6 DSPs with ∼11.8% higher dynamic power over the base core. p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
- Runtime-Adaptive Inference Pipeline
The system can support dynamic compute-budget switching on embedded CPUs by transforming pretrained ViTs into a single binary with modified loop bounds and binary-mask control logic, which allows execution to switch across discrete sparsity levels via compact external configuration files.
- Hardware-Aware Structured Pruning for MLP Layers
The introduction of a hardware-aligned block pruning strategy for Multi-Layer Perceptron (MLP) layers maximizes pruning efficiency by grouping neurons into blocks of eight and ranking them using mean importance before pruning at the block level, achieving 65.0% MAC reduction (4.6x improvement).
- ISA-Level Acceleration via Custom Instruction
The proposed custom MAC2 instruction for the RISC-V ISA exploits input-reuse patterns in linear projection kernels by operating on a shared activation and two weights to perform two parallel multiply-accumulate operations, reducing the innermost loop from 16 instructions to fewer.
- Significant Storage Reduction
The runtime-adaptive approach reduces on-device storage by up to 4.86 times compared to multibinary deployment, requiring only 163 MB for ViT-Base instead of nearly 800 MB, by storing a single full model and using lightweight metadata for switching between configurations.
- Energy Efficiency Gains
The system can achieve up to a 33% energy reduction at zero pruning solely due to the MAC2 extension, and progressively lower energy consumption is achieved as pruning reduces MAC operations, allowing for selection of operating points based on application constraints.
- Flexible Operating Points
The framework enables low-overhead switching between multiple sparsity configurations during inference because the same executable and shared weight tensor are reused across all operating points, reducing switching costs to less than 10-5 times single inference cycles.
Sources
- Comprehensive Survey of Model Compression and Speed up for Vision Transformers
- Once-for-All: Train One Network and Specialize it for Efficient Deployment
- Elastic ViTs from Pretrained Models without Retraining
- TorchAO: PyTorch-Native Training-to-Serving Model Optimization
Related papers
- Embedded Arena: Iterative Optimization via Hardware Feedback
- WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
- SparsePixels: Efficient Convolution for Sparse Data on FPGAs
- Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats
- AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
- RFChipAgent: Multi-Agentic AI Flow for Analog/RF Chip Design