Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators

arXiv:2606.06818 · cs.DC, cs.AR, cs.LG · Submitted 2026-06-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators".

Jane: The gist: Terastal introduces a soft real-time framework that jointly designs layer variants offline and schedules them online to reduce deadline misses for multi-DNN workloads on heterogeneous accelerators.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up "Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators", it's a framework that combines offline design, variant construction, and online scheduling to handle multi-DNN workloads on heterogeneous accelerators.

Jane: The authors introduce layer variants as customized layer implementations that reduce latency on non-preferred accelerators and then build a budget-guided framework for constructing those variants.

Lu: And finally, they design an online scheduler that uses layer-wise slack to jointly choose the accelerator mappings and variant usage, making sure to respect accuracy constraints during runtime.

Meng: For practical implementation, this suggests we can design specialized layers for specific hardware targets before deployment and then let the scheduler handle the dynamic runtime decisions based on real-time slack measurements.

Tom: It addresses how large layer-latency differences across accelerators limit scheduling flexibility and increase deadline misses under skewed workloads by optimizing both mapping and variant selection simultaneously.

Jane: The main implication is a more robust approach to achieving soft real-time performance for complex AI inference tasks running on diverse, heterogeneous hardware setups.

Lu: It shows how fine-grained, layer-granularity scheduling can be powerfully augmented by customized computation rather than just relying on static mapping choices.

Tom: We'll keep an eye on how this approach scales when we move from a few models to a much larger collection of concurrent DNN tasks across many devices.

Conclusion: Tom: So, Terastal is about taking multiple deep neural networks that need to run at specific deadlines on different kinds of hardware, like GPUs or NPUs, and making sure they don't miss those deadlines.

Jane: Right. The authors are looking at how to manage that complexity by combining three things: designing specialized layer versions, figuring out how much time each layer gets, and then scheduling everything online.

Lu: What’s really interesting is the idea of these layer variants—basically making a layer run faster on one type of chip by changing how the math flows, like using space-todepth transformations.

Meng: From an engineering side, that means you can tailor the execution path for a specific accelerator without having to rewrite the whole model from scratch every time.

Lalam: And thinking about that customization, it could eventually mean we can deploy AI on devices with much lower power constraints by making the hardware utilization much more precise.

Tom: So why does this matter right now? Because as models get bigger and we have more diverse chips, the old scheduling methods just don't keep up with those huge latency differences between accelerators.

Jane: Exactly. The paper shows that they use virtual budgets assigned offline, which tells the system how much time each layer *should* get to finish its job.

Lu: And then online, the scheduler uses those budgets and checks available hardware to pick the best combination of variant and chip for every layer running right now.

Meng: The results show a significant drop in deadline misses—like forty percent or thirty percent better than just using standard scheduling techniques like FCFS or EDF.

Tom: And they didn't sacrifice too much accuracy; the average loss was quite small, around two to three percent, which is pretty good for real-time systems.

Jane: It’s a solid framework because it tackles the problem at three levels simultaneously: design, budgeting, and scheduling.

Lalam: The vision here is that this isn't just about faster inference; it’s about creating a more resilient AI system capable of meeting strict timing guarantees in unpredictable environments.

Tom: So the big picture is that we can move toward truly reliable, real-time AI deployment on whatever hardware you have available, rather than just hoping the scheduling works out okay.

Jane: It’s a lot of heavy lifting done offline to get a much smoother ride online for those demanding workloads.

Lu: The future work would probably look at how this framework handles even more complex dependencies or maybe extends these layer variants to include different types of neural network operations.

Department of Computer Science, University of California, Irvine

cs.DC, cs.AR, cs.LG

Submitted: 2026-06-05

Updated: 2026-06-05

Comments: 8 pages, 6 figures. Accepted by RTCSA 2026. Author accepted manuscript

Journal ref: 2026 IEEE 32nd International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), pp. 176-182

DOI: 10.1109/RTCSA71153.2026.00029

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: The gist: Terastal introduces a soft real-time framework that jointly designs layer variants offline and schedules them online to reduce deadline misses for multi-DNN workloads on heterogeneous

Key concepts

Layer Variant Architecture: This involves modifying a specific layer within a neural network so it runs faster on a particular accelerator. It achieves this by reshaping the computation using transformations like space-todepth or depth-to-space, effectively changing how data flows across spatial and channel dimensions to better suit the target hardware's architecture.
Virtual Budget Distribution: The system breaks down a model's total deadline into smaller 'virtual budgets' for each individual layer. This allows the framework to assign these budgets during offline planning, ensuring that even if some accelerators are slower, the layer has a realistic time constraint based on its assigned budget.
Online Scheduler Design: The scheduler makes real-time decisions by prioritizing layers with the most urgent remaining slack (best-case slack). It jointly decides which accelerator to use and which layer variant to apply, ensuring that the most critical tasks are scheduled first while respecting their virtual deadlines and accuracy limits.
Latency Reduction on Non-Preferred Accelerators: This is achieved by designing specific layer variants that are trained independently. These variants are optimized so that the execution time of a layer on a non-preferred accelerator becomes comparable to or even lower than its execution time on the preferred accelerator, thus boosting performance where it is needed most.

Terminology

Accuracy — 2× in this paper · explained in 1 episode

Summary

The gist: Terastal introduces a soft real-time framework that jointly designs layer variants offline and schedules them online to reduce deadline misses for multi-DNN workloads on heterogeneous accelerators.

How it works

Terastal combines offline heterogeneity-aware virtual budget assignment and layer-variant design, and online scheduling to jointly optimize accelerator mapping and variant selection under timing and accuracy constraints The framework addresses the challenge that large layer-latency differences across accelerators limit scheduling flexibility and increase deadline misses under skewed workloads.

Layer Variant Architecture

A layer variant reduces the latency of executing a layer on a non-preferred accelerator by reshaping the computation to better match the target dataflow DNN accelerators primarily use output-stationary (OS) or weight-stationary (WS) dataflows, and different DNN layers vary in spatial and channel size, which can cause significant latency differences on non-preferred accelerators. To mitigate this, layer variants are designed using the space-todepth (S2D) and depth-to-space (D2S) transformations. These operations redistribute information across spatial and channel dimensions while preserving the layer’s input-output functionality at the model level.

Virtual Budget Distribution

Offline, Terastal decomposes each model deadline into layer-wise virtual budgets For a model Mm with relative deadline Dm, a virtual budget bm,l is assigned to each layer l such that X Lm l=1 bm,l = Dm (1) The virtual deadline of layer l of request Jj,m during runtime is computed as d v j,m,l = t a j,m + X l l′=1 bm,l′ (2) A key insight is that a layer budget need not accommodate all accelerators; if bm,l lies in the interval c↓(r) m,… c↓(r+1) m,… then accelerators with latency larger than c↓(r) m,l are excluded from the budget assignment.

Layer Variant Design

The layer variant design module identifies latency-critical layers and selectively constructs variants that reduce latency on non-preferred accelerators while controlling accuracy loss. For each candidate layer, the minimum γ is chosen that reduces latency on the target non-preferred accelerator to the next constraint level or below the preferred accelerator's latency. Variants are trained independently by replacing the original layer and freezing all other layers, and their OS latency becomes comparable to or lower than that on WS, showing the latency reduction on non-preferred accelerators. Accuracy loss is controlled by assigning each model an accuracy threshold θm, and only valid variant combinations are passed to the scheduler if their offline-measured accuracy exceeds θm.

Online Scheduler Design

The online scheduler uses the offline-generated budgets, layer latency profiles, and valid variant combinations to jointly decide accelerator mapping and variant usage. The scheduler sorts all ready layers by non-decreasing best-case slack s∗ j,m,l(t) and prioritizes the most urgent layer that can still meet its virtual deadline on some idle accelerator. If a layer has a variant, it checks if applying it keeps the current request within th threshold. The second stage backfills remaining idle accelerators to improve utilization by assigning the layer with the highest ∆sj,m,l→k(t) to each remaining accelerator.

Evaluation

Terastal reduces average per-model deadline miss rate by 40.58%, 30.53%, and 36.27% over FCFS, EDF, and DREAM, respectively, while incurring only 2.24% average normalized accuracy loss on models with layer variants. The ablation study shows that virtual budgets and layer variants play complementary roles. Full Terastal further improves over Terastal–no variants, showing that layer variants provide additional benefit on top of virtual budgeting. The resulting storage overhead is modest, increasing per-model storage by only 0.5% to 5.9% relative to the original model sizes. The scheduler's total per-invocation complexity is O(nJ · n2a + nJ log nJ), which remains lightweight relative to layer execution time. The early-drop policy is applied if the current time plus the minimum possible latency of all remaining layers already exceeds a request’s absolute deadline.

The paper's main contributions are summarized as follows: 1) We introduce layer variants, customized layer implementations that reduce latency on non-preferred accelerators. 2) We propose a budget-guided framework for layer variant construction and virtual-budget/virtual-deadline assignment. 3) We design an online scheduler that uses layer-wise slack to jointly choose accelerator mappings and layer variants, reducing deadline misses while respecting accuracy constraints.

The paper's main contributions are summarized as follows: 1) We introduce layer variants, customized layer implementations that reduce latency on non-preferred accelerators. 2) We propose a budget-guided framework for layer variant construction and virtual-budget/virtual-deadline assignment. 3) We design an online scheduler that uses layer-wise slack to jointly choose accelerator mappings and variant usage, reducing deadline misses while respecting accuracy constraints.

The paper'

Improvements for AI systems

textbf Rewriting layer implementations for non-preferred accelerators: This involves designing layer variants using space-todepth (S2D) and depth-to-space (D2S) transformations to reduce latency on a non-preferred accelerator by reshaping the computation to better match the target dataflow.
textbf Budgeted layer variant construction: The system should employ a budget-guided framework for layer variant construction and virtual-budget/virtual-deadline assignment, where offline, Terastal decomposes each model deadline into layer-wise virtual budgets to identify layers where variants are beneficial.
textbf Joint online scheduling: The scheduler must utilize the offline outputs to jointly decide accelerator mapping and variant usage, using budgetderived virtual deadlines and layer-wise slack to make decisions at runtime, thereby reducing deadline misses while controlling accuracy degradation.