AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V

summary

The gist

Deploying Vision Transformers (ViTs) on low-power edge devices is challenging due to high computational demands.

In short

The paper introduces AdaptViT, an end-to-end pipeline for deploying Vision Transformers (ViTs) on low-power edge CPUs. It transforms a single model into a runtime-configurable binary that dynamically switches between different sparsity levels. This enables significant speedups (up to 2.8x), energy savings, and massive storage reductions (up to 4.86x) by using hardware-aware pruning and custom CPU instructions.

Key concepts

Runtime-Adaptive Inference Pipeline
This is the core innovation where a single compiled binary supports multiple sparsity levels. Instead of needing separate files for each level, the system uses modified loop bounds and mask logic within one executable. This allows the device to switch between different computational budgets dynamically using simple external configuration files during runtime.
Hardware-Aware Structured Pruning
This strategy focuses pruning efforts on Multi-Layer Perceptron (MLP) layers by grouping neurons into blocks of eight. By ranking these blocks based on importance before pruning, the method maximizes efficiency and achieves a 65% reduction in MAC operations, which is crucial for hardware compatibility.
Custom ISA Extension (MAC2 Instruction)
The authors propose a custom instruction for the RISC-V processor that exploits input reuse. This instruction allows the processor to perform two parallel multiply-accumulate operations using a single shared activation and two weights simultaneously. This reduces the number of instructions needed in critical loops, leading to up to 1.56x speedup.
Storage Reduction Analysis
The runtime-adaptive approach drastically cuts storage needs by only storing one full model and using small metadata for switching configurations. This method reduces on-device storage requirements by up to 4.86 times compared to deploying separate binaries for every possible sparsity level.

Terminology used across episodes

This episode discusses

The paper

AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V · Read on arXiv

University College Dublin

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V".

Jane: Here's the gist of the paper. Deploying Vision Transformers (ViTs) on low-power edge devices is challenging due to high computational demands.

Tom: Conventional pruning frameworks require a separate compiled binary for each sparsity level, increasing storage overhead and limiting runtime adaptability.

Jane: This paper presents an end-to-end deployment pipeline that transforms pretrained ViTs into a single runtime-configurable binary, enabling dynamic compute-budget switching on embedded CPUs.

Tom: Pruning redundant reasoning tokens The proposed method achieves significant reductions in storage requirements and increases inference speedup through hardware-aware structured pruning and ISA extensions.

Title and authors: Tom: So we're looking at this paper now called AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V. It sounds like they are tackling the problem of getting these big vision models onto low-power devices without needing a separate compiled binary for every single level of sparsity.

Jane: That’s right, Tom, and it seems the core idea is making one binary that can switch its compute budget dynamically on an embedded CPU just by loading a small configuration file. It's about flexibility for those edge devices we use all the time.

Lu: What’s really interesting here is how they handle the transformation from a high-level model to bare-metal C code, because usually, when you do dynamic selection of active dimensions in Python, it’s easy, but translating that into safe skipping at coarsegrained block boundaries in C without adding too much overhead is tough.

Meng: From my side on deployment, I worry about that control-flow overhead you mentioned. If the transformation adds too much complexity to the execution path inside the compute-intensive regions, it just slows down inference instead of helping.

Lalam: I'm seeing that they are using TVM to map ViT operations into C kernels across encoder blocks, and profiling on the target RISC-V processor shows these specific kernels are doing most of the work—about seventy-nine percent for ViT Small and eighty-three percent for ViT Base >

Tom: Exactly, and that leads us to how they tackle the actual computation. They introduce a hardware-aligned block pruning strategy specifically for Multi-Layer Perceptron or MLP layers to make sure that pruning is as efficient as possible when you're targeting this specific processor architecture.

Jane: And on top of that, they propose a custom instruction called MAC2 to exploit how input activations are reused in linear projection kernels, which they say helps with speedup and energy efficiency.

Lu: That ISA extension is significant because it's tailored to the dominant operations; it’s not just some generic vector extension, but something that directly addresses the specific pattern of input reuse in those projection layers.

Title and authors: Meng: I’m curious about what this MAC2 instruction actually does in terms of instructions saved per iteration, because if it reduces the instruction count significantly, that translates directly into lower power consumption when we're running these models at scale on our hardware.

Tom: The paper claims that this custom MAC2 instruction provides a one point five six times speedup and a thirty-three percent lower inference energy specifically for the unpruned model, which is pretty compelling.

Jane: But let’s look at the trade-offs, because they aren't just talking about speed; they are also talking about storage. They claim this runtime-adaptive approach reduces on device storage by up to four point eight six times compared to using separate binaries for each pruning level.

Lu: That massive reduction is achieved because they store only one full model and use lightweight metadata for switching, which is much better than having nearly eight hundred megabytes on device >

Meng: four point eight six times sounds huge, but I need to know what that means in practice when we have to manage those configuration files for switching between levels. Is the overhead of switching actually small?

Tom: They say the switching cost is minimal, just about fifty-seven thousand cycles for ViT Small and one hundred fourteen thousand cycles for ViT Base, which they call classifier handling >

Jane: That’s a very low number when you think about how often we might need to switch during a real inference run. So, if we can get that kind of speedup and storage saving while keeping switching overhead this low, what does that mean for deploying these kinds of vision models on things like mobile phones or even small embedded systems?

Lu: It means the architecture is designed to be highly flexible; the C code generation is structured so you can skip computation at coarsegrained block boundaries with minimal overhead, which solves that control-flow problem we talked about earlier >

Meng: So it’s less about having a fixed, perfect model and more about having a single deployable artifact that adapts to the immediate constraints of the device running it >

Title and authors: Tom: That’s the core idea—a single runtime-configurable binary—and they show that this approach allows dynamic compute budget switching on embedded CPUs, which was previously very hard to achieve >

Jane: So we've seen how they generate these multiple sparsity levels offline using post-training pruning, and then how the unpruned backbone is quantized to INT8 using TORCHAO while keeping nonlinear operations in floating point for accuracy >

Lu: That quantization step is interesting because it shows they are balancing storage reduction with maintaining reasonable accuracy, as the accuracy drops by five point zero seven percent when they apply the INT8 quantization on top of the pruning >

Meng: So you get a trade-off: huge storage savings versus a small drop in accuracy from that INT8 move, and then you've got this runtime switching mechanism to toggle between those states >

Tom: And that dynamic switching is what makes it useful for real-world edge deployment where the computational budget changes based on the workload at any given moment >

Jane: So, to wrap up, AdaptViT is an end-to-end framework for runtime adaptive Vision Transformer inference on RISC-V processors, showing up to two point eight times speedup and a thirty-three percent energy reduction at zero pruning with the MAC2 extension >

Lu: The authors also flag that they are exploring extending this mask-based skipping to nonlinear layers in future work, which suggests they aren't trying to solve everything perfectly right now >

Meng: And for me, from a production standpoint, the focus is on getting this single binary and its runtime adaptability working reliably across different model sizes like ViT Small and ViT Base >

Lalam: I think what really stands out is how they align structured sparsity with hardware execution granularity to make it efficient and flexible for edge devices >

Tom: So, if you're deploying a Vision Transformer on an embedded RISC-V processor and you need dynamic compute budget switching, the AdaptViT framework shows a path toward achieving significant speedup and storage savings through hardware-aware pruning and custom ISA extensions >

The paper's summary: Tom: So, to recap, this paper is about taking those big Vision Transformers and making them fit on low-power edge devices by turning them into a single binary that can change its speed and memory usage on the fly, instead of needing a whole new file for every level of compression >

Jane: Exactly. The big thing here is moving away from having to compile a separate program for each sparsity level, which used to make things take up huge amounts of storage on the device >

Tom: Right. And they achieved this by doing some smart pruning—specifically block pruning for the MLP layers and a custom instruction called MAC2 that helps with how the processor handles input reuse >

Jane: That custom instruction is pretty interesting because it’s specifically designed for those linear projection kernels, which are where most of the work happens in these models >

Tom: And the results are pretty solid, they show up to two point eight times speedup on a RISC-V processor like the TRV32P3FX when they use this setup and they see that massive storage reduction too >

Jane: That storage saving is huge—they're talking about reducing the memory needed for ViT-Base by almost five times compared to having those separate binaries >

Tom: It’s not just about speed, though. They also saw a thirty-three percent energy reduction at zero pruning just because of that MAC2 extension itself, which is pretty significant for battery life >

Jane: But the paper also shows they have to be careful with accuracy. they use INT8 quantization on parts of the model and see a drop in accuracy, about five point zero seven percent relative to the full floating-point version >

Tom: That’s the trade-off you always run into when you try to squeeze performance and size together, but they say it’s a reasonable balance for edge deployment >

Jane: So, what this means for us is that we can finally take these powerful Vision Transformers and deploy them on devices with tight constraints, like mobile phones or small embedded systems, without having to compromise too much on speed or storage >

Tom: It really changes how you think about deploying these models; it moves from a static deployment to something truly dynamic where the model adapts to the exact resources available at that moment >

Jane: And this flexibility is what makes it possible for vision AI to move beyond just big data centers and get into real-world edge applications, where things are constantly changing >

Tom: It’s about making these models practical, not just theoretical demonstrations on a massive GPU >

Jane: So next up, we’re going to look at how they actually built this system. We need to see how they take the trained model and turn it into this single runtime configuration file that can switch between states.

The paper's improvements: Tom: So, moving on to what they actually suggest doing next—the improvements—they’re talking about extending this idea beyond just MLP layers >

Jane: Right, they want to see if we can apply this same runtime-adaptive skipping logic to the nonlinear parts of the ViT, not just the linear projections >

Tom: That makes sense. Right now, they've focused on structured pruning for those MLP layers—grouping them into blocks and ranking them before pruning—but that’s just a start >

Jane: And then there’s this whole idea of using vector or matrix compute engines in the future to handle things differently, which could help with the energy efficiency gains we saw at zero pruning >

Tom: Yeah, they want to see if you can get better hardware support for those patterns so you can skip computations more aggressively across different layers >

Jane: It’s about making sure that this runtime-adaptive approach isn't just a trick for the MLP, but something that works consistently across the whole model structure >

Tom: And they are also looking at how to use this mechanism to manage the compute budget even more dynamically based on what kind of input we are processing >

Jane: That means in theory, we could have a system that decides which parts of the model to run with full precision and which parts can be aggressively pruned based on battery level or processing load >

Tom: It’s about making the deployment truly adaptive to the environment, not just static selection based on a pre-set configuration file >

Jane: And if they succeed in extending this mask-based skipping to nonlinear layers, it would mean these complex Vision Transformers could run much more efficiently on devices that have very limited power budgets >

Tom: It’s ambitious. They’re essentially trying to bring the flexibility of software control flow into the actual hardware execution pipeline so it can adapt on its own >

Jane: So, while they haven't fully done it yet—they flag that extending mask-based skipping to nonlinear layers is future work—the direction they’re pushing is toward a model that’s truly self-optimizing for the hardware it’s running on >

Tom: It’s about taking this from just a clever pruning technique to a full, flexible inference system for edge AI >

Jane: And that leads us perfectly into how they built the pipeline in the first place. We need to see how they took a massive pre-trained model and turned it into this single binary that can switch its behavior on the fly.

Conclusion: Tom: So we’re wrapping up AdaptViT, which is this framework for runtime-adaptive Vision Transformer deployment on custom RISC-V processors that lets you switch compute budgets dynamically >

Jane: It was a lot of technical stuff, but the main point is that they managed to turn a massive model into something flexible enough to run on small hardware without needing eight hundred megabytes of storage >

Tom: Right. The speedup numbers are impressive, up to two point eight times faster than the unpruned version on that specific processor when they use this structured pruning and the custom instruction >

Jane: That’s a big win because it shows that even though we are moving towards smaller devices, we can still get really high performance if we design the software to be adaptable rather than just optimizing for one fixed scenario >

Tom: It’s about making these models practical for real-world edge deployment, not just theoretical demos on a huge GPU cluster >

Jane: And they did manage to show that there's a trade-off with accuracy when you use that INT8 quantization layer, which is something we need to keep in mind >

Tom: True. It’s a balancing act between how much space you save and how good the Vision Transformer actually is at seeing things >

Jane: So what does this mean for us? It means vision AI can get onto devices that aren't designed for massive computation, provided we use smart techniques to make the model fit the hardware constraints dynamically >

Tom: Exactly. This AdaptViT paper shows a clear path forward for making Vision Transformers more accessible on constrained edge devices >

Jane: Before we move on to another paper, I want a quick word from Lu about what this means for representation learning in general >

Lu: Well, this proves that the idea of runtime adaptability is viable even when you’re dealing with very complex architectures like ViTs because the structural transformations are consistent across different model sizes >

Meng: For me, it’s about deployment cost. If we can reduce the storage footprint by nearly five times, that lowers our entire operational cost for running inference at scale, even if the switching overhead is small >

Lalam: From my side, I’m sure of this. The mechanism they use to store just one full model and swap configurations is robust; I’m confident in the results of AdaptViT >

Tom: Okay, so that covers AdaptViT for now. We'll be looking at some papers on session types next to see how other models handle dynamic constraints >

More episodes

← Home