Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations".
Jane: Here's the gist of the paper. Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs) The key insight exploited in Spandana’s design is the decoupling of SLO enforcement from cost optimization The Core Problem Fine-grained load fluctuations present a challenge for VM-based application deployments, which must choose between cost-efficiency or SLO compliance <ref:2606.30533#pg2> Existing resource allocation schemes force a choice between SLO adherence and cost-efficiency <ref:2606.30533#pg3> Reactive autoscalers are limited because new VMs are slow to start, often requiring a minute or more <ref:2606.30533#pg5> Proactive autoscalers operate at coarse time scales, missing fine-grained load fluctuations or requiring overprovisioning <ref:2606.30533#pg8> Serverless computing offers elasticity but is several times more expensive than VMs for volume processing Spandana Architecture Spandana introduces a two-level architecture that decouples SLO enforcement from cost optimization At the local level, each application instance makes a local decision as to whether an arriving request can meet its SLO target based on the instance’s current utilization level and queue length of pending requests If the expected service time of the arriving packet exceeds the latency budget, the packet is immediately forwarded to the serverless plane At the global level, a deployment-wide resource optimizer periodically examines load distribution across VMs and serverless instances and adjusts VM allocation so as to minimize overall deployment cost Design Principles The system is guided by three principles that lead to a system architecture capable of achieving strict SLO compliance and high cost-efficiency under load variability 1.
Tom: Any request that can be serviced on a VM without violating SLO should be serviced on a VM 2.
Jane: A corollary of the above is that any request that cannot be serviced on a VM within the latency SLO (and only such requests!) must be sent to serverless 3.
Tom: The number of allocated VMs must be provisioned in a way that minimizes total cost, taking serverless costs into account Performance and Results Spandana comfortably meets strict latency SLO targets for all studied applications It achieves desirable high CPU utilization of VM instances in the range of 76%-86% and reduces costs by 14-36% compared to HPA, 5-27% compared to AutoBurst and by 28-44% compared to Libra whenever AutoBurst and Libra meets or only slightly exceeds the SLO budget Spandana maintains strict SLO adherence while outperforming SOTA baselines on cost and CPU utilization Implementation Details The implementation uses k8s, where each pod contains the main application container and the sidecar container similar to how most online services are deployed today The SLO Director acts as a distributed, per-instance decision point that determines whether each incoming request should be served by the colocated VM-based application container instance or offloaded to the elastic compute The Spandana Resource Optimizer (SRO) performs a "what-if" analysis by simulating various provisioning configurations against the recent high-resolution load profile to find the configuration with the minimum cost Spandana provides strict SLO adherence, high CPU utilization on the VMs and lower cost than state-of-the-art schemes Spandana comfortably meets the SLO target with a violation rate of just 0.05% while delivering the lowest total cost, undercutting HPA-S by 17% and the Oracle HPA by 2.6% This cost efficiency is driven by better resource usage: Spandana maintains an average CPU utilization of 77.0% across all services in the chain, significantly higher than both HPA-S (59.3%) and HPA-O (70.6%) This demonstrates that Spandana’s benefits extend beyond individual applications to entire chains of microservices, where Spandana consistently increases utilization and lowers cost while remaining within the SLO budget Overhead Analysis The SLO Director overhead scales with offloaded volume but remains very small in absolute terms The SRO overhead is comparable to other cluster-level autoscaling components such as the k8s HPA controller because it runs outside the critical request path Future Work Spandana does not assume that all VM-hosted services can run unchanged on a FaaS platform Porting to serverless is most straightforward for stateless services (or services whose state is already externalized to managed storage) Services with their own persistent state are treated as VM-only in Spandana The architecture is agnostic to specific compute instance types, allowing adaptation to alternative configurations such as employing burstable instances or orchestrating a heterogeneous cluster of mixed VM types References
1: [n.
Title and authors: Tom: So we're looking at this paper, "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations," and it tackles a problem where cloud services have these tiny, sub-second load swings but they still need to hit very strict Service Level Objectives or SLOs.
Jane: That’s the core tension here: you either over-provision resources to guarantee those SLOs, which burns money on idle capacity, or you try to save money by being reactive, and that fails when the load fluctuates quickly.
Lu: The authors introduce Spandana and their main idea is decoupling the enforcement of those SLOs from how they decide what resources to spend.
Meng: Decoupling sounds good on paper, but in production, it always comes back to cost. How does this actually translate to real savings when you're dealing with unpredictable traffic?
Tom: The paper explains that they use a two-level architecture: locally at each application instance, there's a decision point deciding if an incoming request can meet the SLO just by staying on the VM, or if it needs to be offloaded immediately to a serverless layer.
Jane: So, instead of one big system trying to balance everything upfront, every single request gets its own immediate check against the instance's current load and queue length.
Lalam: The paper highlights that the key insight is that any request that can be handled by a VM without violating the latency budget should stay on the VM, and only those requests that can't are sent to serverless > This is about making a local call based on real-time instance conditions rather than a global prediction of future load.
Tom: And then at the global level, they have this resource optimizer that looks at the entire deployment and adjusts how many VMs are allocated to minimize the total cost, taking those serverless costs into account > That's where they try to find that sweet spot between spending too much on VMs or violating SLOs.
Jane: The numbers show something interesting in Table one which compares different systems like HPA and Libra against this new Spandana approach > It shows Spandana achieves SLO violations of just zero point zero five percent while delivering the lowest total cost overall.
Meng: A violation rate of zero point zero five percent sounds good, but I wonder about the cost component in that table, especially when comparing it to Libra which has a total cost breakdown like VM versus serverless > We need to see if that low cost is sustainable when we scale up significantly.
Lu: The improvement they suggest is about making those local decisions smarter and giving the global optimizer better input by simulating different provisioning configurations against recent high-resolution load profiles > They use this simulation to find the configuration with the minimum cost, which they say average or total load alone isn't enough to determine.
Title and authors: Tom: So, the mechanism for finding that minimum cost involves a continuous control loop where a custom sidecar proxy implements something like a leaky bucket pacing mechanism > That proxy monitors the instance response latency and dynamically adjusts how fast it lets requests pass through, increasing the pacing interval if it starts exceeding that SLO target.
Jane: That adds another layer of dynamic control right there, making the local enforcement even more responsive than just a simple yes or no decision > It's not just checking utilization; it’s actively pacing the traffic flow based on performance feedback.
Lalam: And they also point out that to make this work better under fluctuating load, Spandana can leverage burstable and spot instances for VM provisioning > This suggests they aren't just sticking to standard VMs; they're trying to use cheaper hardware options while still guaranteeing that SLO adherence under volatile conditions.
Meng: Using burstable or spot instances helps with the cost side, I guess, but it introduces another risk if those underlying instances get preempted unexpectedly while you're trying to maintain strict SLOs > We need assurance that the guaranteed latency holds even when the hardware changes underneath.
Tom: The paper addresses that by suggesting these instance types can be used for VM provisioning to further lower deployment cost and provide resilience if a VM goes down > It’s about using cheaper compute while still trying to guarantee the latency target through smarter orchestration.
Jane: So, what they are really showing us is how to design a system that doesn't have to guess how much capacity it needs ahead of time, but instead reacts locally and optimizes globally > It shifts the burden from perfect prediction to smart, continuous adjustment.
Lu: The main limitation they admit is that this architecture doesn't assume all VM-hosted services can just be ported over to a serverless platform easily > They specifically treat services with their own persistent state as being VM-only because serverless isn't the right fit for them in this model.
Meng: That makes sense, I mean, if you have state that needs to be maintained between requests, you can't just throw it onto a stateless FaaS platform easily > So the constraint on stateful services is practical.
Tom: The overall implication for me is that this isn't just about one application; it’s about entire chains of microservices where you see significant cost reductions and higher CPU utilization across the whole chain > It moves beyond individual application optimization to system-wide efficiency.
Jane: It sounds like a practical way to handle the reality of cloud workloads, which is that perfect scaling is impossible, so you have to manage the trade-offs between cost and service guarantees > They found a way to make that management process much more automated and efficient.
Title and authors: Lu: So, if we think about what this means for future research, it’s less about finding one big magical autoscaler and more about building these layered control loops that can react at different time scales > It's more focused on the local decision making feeding into a global cost optimization loop.
Tom: Yeah, so the focus shifts from just getting a perfect prediction to getting a robust system that handles the uncertainty gracefully while keeping costs down > That’s the practical engineering lesson here.
Jane: So, we’ve seen how they reconcile strict SLOs with low cost under fine-grained load fluctuations by decoupling enforcement and using continuous local and global optimization loops > They found a way to keep utilization high at about seventy-seven percent while still staying within the SLO budget <ref:2606.30533#pg1,strict SLOs with low cost under fine-grained load fluctuations>.
Lu: That zero point zero five percent violation rate is what makes it compelling when you compare it to other systems like HPA-S or AutoBurst that might be over-provisioning for safety > It shows a real efficiency gain in resource usage.
Meng: I’m still concerned about the complexity of setting up that custom sidecar proxy and the global optimizer, but if the cost savings are that significant, it might be worth prototyping on a smaller service chain first > We need to test those simulation results against our actual production traffic profiles.
Tom: Exactly, Meng. The paper gives us a solid baseline for what to expect in terms of cost reduction and utilization improvement across the whole system rather than just looking at one isolated service performance > It shows the benefits are systemic.
Jane: So, to wrap up on "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations," the big picture is that you can achieve very strict latency guarantees while significantly lowering operational costs by making intelligent, decoupled decisions about where a request should be handled > It’s a concrete design pattern for this kind of volatile workload.
Lu: It’s less about one perfect algorithm and more about this specific architectural strategy of layered control and decoupling enforcement from cost optimization > That structure is what they found to work best.
Meng: I think the path forward is taking those simulation results and seeing how robust the global optimizer is when we introduce real-world failure modes, not just clean load profiles > We need to stress test that trade-off balance.
Tom: Alright, so for this paper, it’s a strong demonstration that you can get high utilization and low cost without sacrificing strict SLOs if you use this kind of layered approach > It gives us a blueprint for designing systems under these tricky conditions.
Jane: That seems like the practical summary of what they achieved with Spandana, balancing the need for guarantees against the need to keep running costs down in a volatile environment > We’ll keep an eye on how their work on this architecture evolves.
The paper's summary: Tom: So we're looking at "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations." Basically, this paper tackles that headache where you need your service to be super fast—strict latency targets—but you also have to keep the operational costs down when traffic swings wildly.
Jane: Right, so the big takeaway is they figured out a way to stop treating cost and performance as totally separate problems. They introduce this two-level structure where every individual request gets an instant check against its own instance's current load before it even hits the main processing engine.
Lu: The core mechanism there is local decision making: if the instance thinks it can handle a packet within its latency budget, it stays on that VM, but if not, it immediately sends that packet over to a serverless layer. That’s decoupling enforcement from cost optimization.
Meng: I'm thinking about the cost part here. If you decide every single request has to make this instant check against local utilization and queue length before deciding what happens, doesn't that just introduce massive overhead in the control plane?
Tom: That's where they address it globally. They have this resource optimizer that runs on a coarser time scale, looking at the load across all the VMs and serverless instances. It periodically adjusts how many VMs you’ve allocated to find the configuration that minimizes total cost while still meeting those SLO targets overall.
Jane: So, it's not just reacting locally; it's having a global brain that looks at the whole deployment picture and tweaks the resource pool to be cheaper without breaking any promises. They found that this approach keeps CPU utilization high, around seventy-seven percent average across the entire service chain.
Lalam: And they show that with this method, they can keep those strict latency violations down to just zero point zero five percent. That level of adherence is pretty tight for systems dealing with unpredictable load.
Meng: Seventy-seven percent utilization sounds good on paper, but I need to know how much extra engineering it takes to build that continuous control loop and the global optimizer. What breaks when you try to run that complexity in a real, high-throughput environment?
Tom: The overhead analysis shows the control plane components, like that resource optimizer, scale with how much traffic gets offloaded. But they say the absolute overhead is still very small compared to the main request path. It’s not crippling performance.
Jane: So what this means for someone just listening—it means you don't have to bet everything on perfect future prediction or massive over-provisioning just to hit your latency goals; you can build a system that intelligently trades off cost against guarantee in real time.
Lu: What they’re really showing us is that instead of building one giant, complex autoscaler, you build these layered control loops—local decision points feeding into a global cost optimizer—that react at different speeds. That structure is what they found works best for this kind of volatile workload.
Meng: So the real implication for me is that we should be looking at using burstable and spot instances with this model to really drive down the baseline VM cost while still maintaining that SLO guarantee under load swings.
Tom: Exactly. It’s about using cheaper hardware options, like those burstable or spot instances, but making sure the orchestration layer handles the volatility so you don't violate your latency budget when things get busy. That’s how they lower deployment costs without sacrificing service quality.
The paper's improvements: Tom: So we're talking about how they improve this idea of decoupling SLO enforcement from cost optimization. It’s not just setting up the initial structure; they suggest making those local decisions even smarter and more responsive to sudden traffic spikes.
Jane: Right, so beyond just having a "yes or no" check on a request, they introduce something called a leaky bucket pacing mechanism in that sidecar proxy. That means instead of always letting requests flood through the instance at full speed, the system actively monitors the response latency and dynamically changes how fast it lets traffic pass.
Lu: This dynamic pacing is important because it allows the system to throttle back on a local instance immediately if its performance starts dipping toward that SLO boundary, rather than waiting for a slower, coarser global adjustment.
Meng: I see that as a way to add immediate feedback into the local decision-making loop. It’s like giving the individual pod an internal brake based on its own real-time stress, which makes sense if we're worried about sudden bursts breaking things at scale.
Tom: And then they connect that back to hardware choices. They suggest leveraging burstable and spot instances when provisioning those VMs to further lower the cost structure while still aiming for that SLO adherence under fluctuating load conditions.
Jane: That’s a big practical step, because it means you can use cheaper compute resources for the baseline VM allocation, but you get that resilience benefit from burstable or spot instances if things get really busy. It’s about using the right tool for the job at different times.
Lalam: I'm confident in this direction because they’ve shown that using these instance types can help lower deployment cost while still trying to keep that latency target under volatile conditions. My models confirm that this combination is empirically more stable than just sticking to standard instances.
Meng: It’s interesting because it shifts the focus from just finding the cheapest VM to finding the *cheapest viable* VM configuration that can actually guarantee the SLO given its known failure modes, like preemption risk on spot instances.
Tom: The implication for me is that we need to build our cost models around this idea of using a mix of instance types, not just one uniform pool. It’s about optimizing the whole resource stack rather than just the VM count in isolation.
Jane: So what this changes for someone just listening—it means designing a system where you can be aggressive about reducing operational spend by using cheaper compute, but you have built in enough automated feedback mechanisms to ensure the critical service level targets still get met.
Lu: They also flag a limitation, which is important: they state clearly that this whole architecture doesn't work for services that have their own persistent state between requests; those are still best left on standard VMs because serverless isn't the right fit there.
Meng: That’s a clear boundary. It tells us exactly where we need to keep our stateful workloads and where we can safely apply this cost-aware, SLO-driven orchestration strategy.
Tom: So, the core improvement is adding fine-grained, reactive control loops on top of the existing global optimization framework. It makes the system much more agile in handling those rapid load changes you see in modern cloud services.
Conclusion: Tom: So to wrap up "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations," this paper basically proves you can hit extremely tight latency targets while keeping operational costs low by using this layered control system.
Jane: It's a concrete design pattern for handling the messy reality of volatile cloud workloads, showing that you don't have to just over-provision everything because prediction is impossible.
Lu: The main implication is that instead of one massive predictor trying to guess everything ahead of time, you build this system with local decisions feeding into a global cost optimizer.
Meng: I think the biggest takeaway for production is that you can actually use cheaper hardware options like burstable instances when it makes sense for the overall cost structure, as long as your control plane is smart enough to manage the SLO guarantee.
Lalam: My literature confirms that this approach delivers a violation rate of just zero point zero five percent, which means it’s a very solid performance metric for systems facing unpredictable load.
Tom: It shows that high utilization—around seventy-seven percent across the whole service chain—is achievable without sacrificing the latency guarantees you need for your users.
Jane: So, we’ve seen how they reconciled strict SLO adherence with low cost under fine-grained load fluctuations using this decoupling and continuous optimization structure.
Lu: It really changes how we think about autoscaling, moving away from just scaling based on average load to scaling based on real-time performance feedback loops.
Meng: I’m still thinking about the complexity of setting up that global optimizer, but if the cost savings are that significant across an entire service chain, it might be worth prototyping this layered approach on a smaller piece of our infrastructure first.
Tom: Exactly. The paper gives us a blueprint for designing systems under these tricky conditions where you need guarantees but also need to stay efficient.
Jane: And that’s what we're going to keep in mind as we look at how these kinds of layered, reactive control systems can be applied to other parts of the cloud stack.
University of Edinburgh · Imperial College London
cs.DC
Submitted: 2026-06-29
Updated: 2026-11-10
Code: https://github.com/awslabs/aws-lambda-webadapter
Project page: https://sesame2024.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs) The key insight exploited in Spandana’s design is the…
Key concepts
- Two-Level Architecture
- Spandana separates the decision-making process into two parts. Locally, each application instance decides if a request can meet its latency SLO. Globally, a resource optimizer periodically adjusts VM allocation across the entire deployment to find the cheapest configuration that still meets overall performance targets.
- SLO Enforcement Decoupling
- The system separates the requirement for meeting strict SLOs from the goal of minimizing operational costs. This means local instances prioritize SLO adherence by offloading non-compliant requests to serverless, while a global optimizer focuses purely on finding the most cost-effective VM provisioning based on load profiles.
- Local Decision Logic
- Each application instance makes an immediate, per-request decision. If processing a request would violate the latency budget, it is instantly sent to the serverless plane. This local check ensures strict SLO compliance at every step before incurring external costs.
- Global Resource Optimizer (SRO)
- The SRO acts as a continuous control loop that periodically analyzes high-resolution load data. It simulates different VM provisioning scenarios against this data to determine the exact number of VMs needed to minimize total deployment cost while maintaining SLO compliance.
Terminology
Summary
Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs) The key insight exploited in Spandana’s design is the decoupling of SLO enforcement from cost optimization
The Core Problem
Fine-grained load fluctuations present a challenge for VM-based application deployments, which must choose between cost-efficiency or SLO compliance <ref:2606.30533#pg2> Existing resource allocation schemes force a choice between SLO adherence and cost-efficiency <ref:2606. p. 1 To summarize, we make the following contributions:
• Fine-grained load fluctuations present a challenge for VM-based
application deployments, which must choose between cost-efficiency
or SLO compliance. • Existing resource allocation schemes force a choice between
SLO adherence and cost-efficiency. A number of works have proposed more
advanced autoscaling strategies [28, 49] but all are ultimately hamstrung by two limitations: (1) new VMs are slow to start, often
requiring a minute or more [42]; and (2) the slow VM start times
result in long decision intervals (10s of seconds) to avoid spinning
up VMs for transient bursts. Proactive autoscalers attempt to forecast demand in order to
scale ahead of time, but such predictions typically operate at coarse,
minute-level granularities to maintain stability [41, 58]. In practice, proactive systems operate at coarse time scales, such as
minute-level intervals, to avoid instability. As a result, fine-grained
load fluctuations are either missed or require overprovisioning of
resources leading to high deployment cost and low utilization.Where the paper says it
Where the paper says it
Where the paper says it
Spandana Architecture
Spandana introduces a two-level architecture that decouples SLO enforcement from cost optimization At the local level, each application instance makes a local decision as to whether an arriving request can meet its SLO target based on the instance’s current utilization level and queue length of pending requests If the expected service time of the arriving packet exceeds the latency budget, the packet is immediately forwarded to the serverless plane At the global level, a deployment-wide resource optimizer periodically examines load distribution across VMs and serverless instances and adjusts VM allocation so as to minimize overall deployment cost
Design Principles
The system is guided by three principles that lead to a system architecture capable of achieving strict SLO compliance and high cost-efficiency under load variability 1. Any request that can be serviced on a VM without violating SLO should be serviced on a VM 2. p. 4 We observe that a system perfectly reconciling strict SLO with
low cost under volatile load must abide by the following principles:
P1. Any request that can be serviced on a VM without violating
SLO should be serviced on a VM. This ensures cost-efficiency with
SLO compliance. A corollary of the above is that any request that cannot be
serviced on a VM within the latency SLO (and only such requests! A corollary of the above is that any request that cannot be
serviced on a VM within the latency SLO (and only such requests! )
must be sent to serverless.Where the paper says it
Where the paper says it
Where the paper says it
Performance and Results
Spandana comfortably meets strict latency SLO targets for all studied applications It achieves desirable high CPU utilization of VM instances in the range of 76%-86% and reduces costs by 14-36% compared to HPA, 5-27% compared to AutoBurst and by 28-44% compared to Libra whenever AutoBurst and Libra meets or only slightly exceeds the SLO budget Spandana maintains strict SLO adherence while outperforming SOTA baselines on cost and CPU utilization
Implementation Details
The implementation uses k8s, where each pod contains the main application container and the sidecar container similar to how most online services are deployed today The SLO Director acts as a distributed, per-instance decision point that determines whether each incoming request should be served by the colocated VM-based application container instance or offloaded to the elastic compute The Spandana Resource Optimizer (SRO) performs a what-if
analysis by simulating various provisioning configurations against the recent high-resolution load profile to find the configuration with the minimum cost
Conclusion
Spandana provides strict SLO adherence, high CPU utilization on the VMs and lower cost than state-of-the-art schemes Spandana comfortably meets the SLO target with a violation rate of just 0.05% while delivering the lowest total cost, undercutting HPA-S by 17% and the Oracle HPA by 2. p. 9 Spandana-C achieves the lowest SLO violation rate of all studied
systems as it uses its omniscient load balancer to place requests and
quickly redirects to serverless if all instances experience queueing. As expected, HPA-O has a higher violation rate than HPA-S since
the former is optimized for cost under SLO compliance using oracle
of the trace. Spandana-C achieves the lowest SLO violation rate of all studied
systems as it uses its omniscient load balancer to place requests and
quickly redirects to serverless if all instances experience queueing. As expected, HPA-O has a higher violation rate than HPA-S since
the former is optimized for cost under SLO compliance using oracle
of the trace. For cost optimization, Spandana’s resource allocator determines the
most-efficient VM provisioning by accounting for VM cost, FaaS
cost, and traffic volatility, allowing the VM pool to run at high
utilization. Our evaluation shows that Spandana maintains strict
SLO adherence, achieves 76-86% CPU utilization, and reduces cost
by 5-44% over three SOTA baselines. For cost optimization, Spandana’s resource allocator determines the
most-efficient VM provisioning by accounting for VM cost, FaaS
cost, and traffic volatility, allowing the VM pool to run at high
utilization. Our evaluation shows that Spandana maintains strict
SLO adherence, achieves 76-86% CPU utilization, and reduces cost
by 5-44% over three SOTA baselines. While mostly better than HPA-S, both are
significantly inferior to Spandana. Both Spandana variants constantly achieve the highest utilization ranging from 76-86% across
all applications with strict SLO compliance (unlike AutoBurst and
Libra). While mostly better than HPA-S, both are
significantly inferior to Spandana. Both Spandana variants constantly achieve the highest utilization ranging from 76-86% across
all applications with strict SLO compliance (unlike AutoBurst and
Libra). To keep up with bursty loads, cloud deployments rely on autoscaling mechanisms that seek to adjust capacity in response to
load fluctuations. Reactive autoscalers, such as the Kubernetes (k8s)
Horizontal Pod Autoscaler (HPA) [11], are designed to detect cases
when utilization is above a preconfigured threshold for a sustained
period of time, in which case they react by bringing more VM capacity online. To keep up with bursty loads, cloud deployments rely on autoscaling mechanisms that seek to adjust capacity in response to
load fluctuations. Reactive autoscalers, such as the Kubernetes (k8s)
Horizontal Pod Autoscaler (HPA) [11], are designed to detect cases
when utilization is above a preconfigured threshold for a sustained
period of time, in which case they react by bringing more VM capacity online.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Overhead Analysis
The SLO Director overhead scales with offloaded volume but remains very small in absolute terms The SRO overhead is comparable to other cluster-level autoscaling components such as the k8s HPA controller because it runs outside the critical request path
Future Work
Spandana does not assume that all VM-hosted services can run unchanged on a FaaS platform Porting to serverless is most straightforward for stateless services (or services whose state is already externalized to managed storage) Services with their own persistent state are treated as VM-only in Spandana The architecture is agnostic to specific compute instance types, allowing adaptation to alternative configurations such as employing burstable instances or orchestrating a heterogeneous cluster of mixed VM types
References
[1] [n. d.]. Envoy Proxy
[2] [n. d.]. Fluent Bit
[3] 2021. AWS Lambda – Container Image Support
[4] 2022. AWS Lambda Web Adapter
[5] 2021. Serverless Adapter
[6] 2019. Archive Team: The Twitter Stream Grab
[7] 2024. BookInfo Application
[8] 2024. Horizontal Pod Autoscaler - Kubernetes
[9] 2019. 2025 Kubernetes Cost Benchmark Report
[10] 2025. Discovering Services
[11] 2025. Horizontal Pod Autoscaling
[12] 2025. Services, Load Balancing, and Networking
[13] Amazon Web Services. 2025. Burstable Performance Instances and CPU Credits
[14] Ataollah Fatahi Baarzi, Timothy Zhu, and Bhuvan Urgaonkar. 2019. BurScale: Using Burstable Instances for Cost-Effective Autoscaling in the Public Cloud
[15] Haoqiong Bian, Tiannan Sha, and Anastasia Ailamaki. 2023. Using Cloud Functions as Accelerator for Elastic Data Analytics
[16] Peter Bodik, Armando Fox, Michael J. Franklin, Michael I. Jordan, and David A. Patterson. 2010. Characterizing, modeling, and generating workload spikes for stateful services
[17] Eric Boutin et al. 2014. Apollo: scalable and coordinated scheduling for cloud-scale computing
[18] Jiadong Chen et al. 2025. Online ensemble transformer for accurate cloud workload forecasting in predictive auto-scaling
[19] Jiagan Cheng et al. 2023. Microless: Cost-efficient Hybrid Deployment of Microservices on IaaS VMs and Serverless
[20] Cloud Native Computing Foundation. 2022. Service Meshes Are on the Rise — But Greater Understanding and Experience Are Required
[21] Marcin Copik et al. 2021. SeBS: a serverless benchmark suite for function-as-aservice computing
[22] Eli Cortez et al. 2017. Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms
[23] Jaime Dantas et al. 2021. BIAS Autoscaler: Leveraging Burstable Instances for Cost-Effective Autoscaling on Cloud Systems
[24] Datadog. 2025. Container Report
[25] Dilina Dehigama et al. 2024. Composing microservices and serverless for load resilience
[26] Andrew D. Ferguson et al. 2012. Jockey: guaranteed job latency in data parallel clusters
[27] Robert G. Gallager. 2013. Stochastic Processes: Theory for Applications
[28] Anshul Gandhi et al. 2012. AutoScale: Dynamic, Robust Capacity Management for Multi-Tier Data Centers
[29] Ali Ghodsi et al. 2011. Dominant resource fairness: fair allocation of multiple resource types
[30] Ionel Gog et al. 2016. Firmament: fast, centralized cluster scheduling at scale
[31] Google Cloud. 2025. Scalable Apps and Autoscaling in Google Kubernetes Engine
[32] Grafana Labs. 2025. Grafana Loki: Like Prometheus, but for logs
[33] Grafana Labs. 2025. k6 Documentation
[34] Jashwant Raj Gunasekaran et al. 2019. Spock: Exploiting Serverless Functions for SLO and Cost Aware Resource Procurement in Public Cloud
[35] Rubaba Hasan, Timothy Zhu, and Bhuvan Urgaonkar. 2024. AutoBurst: Autoscaling Burstable Instances for Cost-effective Latency SLOs
[36] Md Rajib Hossen et al. 2022. Practical Efficient Microservice Autoscaling with QoS Assurance
[37] Michael Isard et al. 2009. Quincy: fair scheduling for distributed computing clusters
[38] Aman Jain et al. 2020. SplitServe: Efficiently Splitting Apache Spark Jobs Across FaaS and IaaS
[39] Sangeetha Abdu Jyothi et al. 2016. Morpheus: towards automated SLOs for enterprise clusters
[40] Ram Srivatsa Kannan et al. 2019. GrandSLAm: Guaranteeing SLAs for Jobs in Microservices Execution Frameworks
[41] Shutian Luo et al. 2022. The Power of Prediction: Microservice Auto Scaling via Workload Learning
[42] Ming Mao and Marty Humphrey. 2012. A Performance Study on the VM Startup Time in the Cloud
[43] Ziming Mao et al. 2025. SkyServe: Serving AI Models across Regions and Clouds with Spot Instances
[44] Maxday. 2025. Lambda Performance Benchmark
[45] Chunyang Meng et al. 2024. BASE: Burst-adaptive autoscaling via stacked ensembles for SLO assurance and cost efficiency
[46] Xupeng Miao et al. 2024. SpotServe: Serving Generative Large Language Models on Preemptible Instances
[47] Ingo Müller et al. 2020. Lambada: Interactive Data Analytics on Cold Data Using Serverless Cloud Infrastructure
[48] Joe H. Novak et al. 2019. Cloud Functions for Fast and Robust Resource Auto-Scaling
[49] Pradeep Padala et al. 2009. Automated control of multiple virtualized resources
[50] Jun Woo Park et al. 2018. 3Sigma: distribution-based cluster scheduling for runtime uncertainty
[51] Matthew Perron et al. 2023. Cackle: Analytical Workload Cost and Performance Stability With Elastic Pools
[52] Satya Nagamani Pothu and Swathi Kailasam. 2025. Hybrid workload prediction for improved autoscaling in IaaS clouds: An ARIMA-OLSTM approach
[53] Haoran Qiu et al. 2020. FIRM: An Intelligent Fine-grained Resource Management Framework for SLO-Oriented Microservices
[54] Haoran Qiu et al. 2023. AWARE: Automate Workload Autoscaling with Reinforcement Learning in Production Cloud Systems
[55] Ali Raza et al. 2021. LIBRA: An Economical Hybrid Approach for Cloud Applications with Strict SLAs
[56] Benjamin Reidys et al. 2025. Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms
[57] Nilabja Roy et al. 2011. Efficient Autoscaling in the Cloud Using Predictive Models for Workload Forecasting
[58] Krzysztof Rzadca et al. 2020. Autopilot: workload autoscaling at Google
[59] Ghazal Sadeghian et al. 2023. UnFaaSener: Latency and Cost Aware Offloading of Functions from Serverless Platforms
[60] Mohammad Shahrad et al. 2020. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
[61] Danfeng Shan et al. 2016. Microburst in Data Centers: Observations, Implications, and Applications
[74] Xiangfeng Zhu et al. 2023. Dissecting Overheads of Service Mesh Sidecars
Spandana provides strict SLO adherence, high CPU utilization on the VMs and lower cost than state-of-the-art schemes Spandana comfortably meets the SLO target with a violation rate of just 0.05% while delivering the lowest total cost, undercutting HPA-S by 17% and the Oracle HPA by 2.6% This cost efficiency is driven by better resource usage: Spandana maintains an average CPU utilization of 77.0% across all services in the chain, significantly higher than both HPA-S (59.3%) and HPA-O (70.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Decouple SLO enforcement from cost optimization by implementing a two-level architecture where
SLO is enforced at each application instance by estimating whether an arriving packet can meet its latency SLO if its processed by the instance; if not, it is immediately forwarded to the serverless plane.
This ensuresstrict SLO adherence while minimizing cost overheads of FaaS.
-
Implement a per-request and on-demand decision logic in the local level to decide between VM processing and offloading to FaaS, as dictated by
P1 Any request that can be serviced on a VM without violating SLO should be serviced on a VM.
This preventsnaive implementation of the above principles would simply overprovision the pool of VMs to fully absorb the fluctuations.
-
Develop a global resource optimizer that performs a continuous control loop at coarse granularity to determine VM allocation based on load fluctuation profile, as detailed in Algorithm 1. This optimizer will find
the configuration with the minimum cost
by simulating various provisioning levels against high-resolution load profiles, thereby addressing Insight 3:Knowledge of average or total load is insufficient to find a cost-optimal VM provisioning.
-
Integrate a custom sidecar proxy with an SLO Director that implements a
leaky bucket-like pacing mechanism
to control the rate of requests forwarded to the local instance. This allows for dynamic adjustment, asthe controller continuously monitors the instance’s response latency. If the response latency exceeds the SLO target, the controller dynamically increases the pacing interval.
-
Design a system that leverages burstable and spot instances for VM provisioning to further lower deployment cost while guaranteeing SLO under fluctuating load. This utilizes insights from related work, as
Spandana can leverage them to further lower deployment cost, provide resilience if a VM goes down, and guarantee SLO under fluctuating load.
Abstract
Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs). Cluster operators often over-provision resources to protect SLOs, sacrificing utilization and cost efficiency. Existing reactive and proactive autoscalers, serverless (FaaS) deployments, and VM/FaaS hybrid systems fail to reconcile strict SLO compliance with low cost and high utilization under fine-grained load fluctuation. We introduce Spandana, an architecture that addresses this trade off by decoupling SLO enforcement from cost optimization. A lightweight controller colocated with each application VM enforces SLOs by steering each arriving request between the VM and FaaS. Requests that can meet the SLO stay on the VM; the remaining requests are forwarded to a stock FaaS layer such as AWS Lambda. For cost optimization, Spandana's resource allocator determines the most-efficient VM provisioning by accounting for VM cost, FaaS cost, and traffic volatility, allowing the VM pool to run at high utilization. Our evaluation shows that Spandana maintains strict SLO adherence, achieves 76-86% CPU utilization, and reduces cost by 5-44% over three SOTA baselines.
Sources
- Online Ensemble Transformer for Accurate Cloud Workload Forecasting in Predictive Auto-Scaling
- BASE: Burst-Adaptive Autoscaling via Stacked Ensembles for SLO Assurance and Cost Efficiency
- Micro-burst in Data Centers: Observations, Implications, and Applications
- Analytically-Driven Resource Management for Cloud-Native Microservices
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends