Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations

summary

The gist

Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs) The key insight exploited in Spandana’s design is the…

In short

Spandana addresses cloud service challenges where fine-grained load fluctuations threaten strict Service Level Objectives (SLOs) while demanding cost efficiency. It introduces a two-level architecture: local instance decisions to enforce SLOs and a global optimizer to minimize VM costs. This decouples performance guarantees from cost optimization, achieving strict SLO adherence with significant cost reductions compared to existing autoscaling methods.

Key concepts

Two-Level Architecture
Spandana separates the decision-making process into two parts. Locally, each application instance decides if a request can meet its latency SLO. Globally, a resource optimizer periodically adjusts VM allocation across the entire deployment to find the cheapest configuration that still meets overall performance targets.
SLO Enforcement Decoupling
The system separates the requirement for meeting strict SLOs from the goal of minimizing operational costs. This means local instances prioritize SLO adherence by offloading non-compliant requests to serverless, while a global optimizer focuses purely on finding the most cost-effective VM provisioning based on load profiles.
Local Decision Logic
Each application instance makes an immediate, per-request decision. If processing a request would violate the latency budget, it is instantly sent to the serverless plane. This local check ensures strict SLO compliance at every step before incurring external costs.
Global Resource Optimizer (SRO)
The SRO acts as a continuous control loop that periodically analyzes high-resolution load data. It simulates different VM provisioning scenarios against this data to determine the exact number of VMs needed to minimize total deployment cost while maintaining SLO compliance.

Terminology used across episodes

This episode discusses

The paper

Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations · Read on arXiv

University of Edinburgh · Imperial College London

Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs). Cluster operators often over-provision resources to protect SLOs, sacrificing utilization and cost efficiency. Existing reactive and proactive autoscalers, serverless (FaaS) deployments, and VM/FaaS hybrid systems fail to reconcile strict SLO compliance with low cost and high utilization under fine-grained load fluctuation. We introduce Spandana, an architecture that addresses this trade off by decoupling SLO enforcement from cost optimization. A lightweight controller colocated with each application VM enforces SLOs by steering each arriving request between the VM and FaaS. Requests that can meet the SLO stay on the VM; the remaining requests are forwarded to a stock FaaS layer such as AWS Lambda. For cost optimization, Spandana's resource allocator determines the most-efficient VM provisioning by accounting for VM cost, FaaS cost, and traffic volatility, allowing the VM pool to run at high utilization. Our evaluation shows that Spandana maintains strict SLO adherence, achieves 76-86% CPU utilization, and reduces cost by 5-44% over three SOTA baselines.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations".

Jane: Here's the gist of the paper. Cloud-based online services face significant sub-second load fluctuations while needing to meet strict Service Level Objectives (SLOs) The key insight exploited in Spandana’s design is the decoupling of SLO enforcement from cost optimization The Core Problem Fine-grained load fluctuations present a challenge for VM-based application deployments, which must choose between cost-efficiency or SLO compliance <ref:2606.30533#pg2> Existing resource allocation schemes force a choice between SLO adherence and cost-efficiency <ref:2606.30533#pg3> Reactive autoscalers are limited because new VMs are slow to start, often requiring a minute or more <ref:2606.30533#pg5> Proactive autoscalers operate at coarse time scales, missing fine-grained load fluctuations or requiring overprovisioning <ref:2606.30533#pg8> Serverless computing offers elasticity but is several times more expensive than VMs for volume processing Spandana Architecture Spandana introduces a two-level architecture that decouples SLO enforcement from cost optimization At the local level, each application instance makes a local decision as to whether an arriving request can meet its SLO target based on the instance’s current utilization level and queue length of pending requests If the expected service time of the arriving packet exceeds the latency budget, the packet is immediately forwarded to the serverless plane At the global level, a deployment-wide resource optimizer periodically examines load distribution across VMs and serverless instances and adjusts VM allocation so as to minimize overall deployment cost Design Principles The system is guided by three principles that lead to a system architecture capable of achieving strict SLO compliance and high cost-efficiency under load variability 1.

Tom: Any request that can be serviced on a VM without violating SLO should be serviced on a VM 2.

Jane: A corollary of the above is that any request that cannot be serviced on a VM within the latency SLO (and only such requests!) must be sent to serverless 3.

Tom: The number of allocated VMs must be provisioned in a way that minimizes total cost, taking serverless costs into account Performance and Results Spandana comfortably meets strict latency SLO targets for all studied applications It achieves desirable high CPU utilization of VM instances in the range of 76%-86% and reduces costs by 14-36% compared to HPA, 5-27% compared to AutoBurst and by 28-44% compared to Libra whenever AutoBurst and Libra meets or only slightly exceeds the SLO budget Spandana maintains strict SLO adherence while outperforming SOTA baselines on cost and CPU utilization Implementation Details The implementation uses k8s, where each pod contains the main application container and the sidecar container similar to how most online services are deployed today The SLO Director acts as a distributed, per-instance decision point that determines whether each incoming request should be served by the colocated VM-based application container instance or offloaded to the elastic compute The Spandana Resource Optimizer (SRO) performs a "what-if" analysis by simulating various provisioning configurations against the recent high-resolution load profile to find the configuration with the minimum cost Spandana provides strict SLO adherence, high CPU utilization on the VMs and lower cost than state-of-the-art schemes Spandana comfortably meets the SLO target with a violation rate of just 0.05% while delivering the lowest total cost, undercutting HPA-S by 17% and the Oracle HPA by 2.6% This cost efficiency is driven by better resource usage: Spandana maintains an average CPU utilization of 77.0% across all services in the chain, significantly higher than both HPA-S (59.3%) and HPA-O (70.6%) This demonstrates that Spandana’s benefits extend beyond individual applications to entire chains of microservices, where Spandana consistently increases utilization and lowers cost while remaining within the SLO budget Overhead Analysis The SLO Director overhead scales with offloaded volume but remains very small in absolute terms The SRO overhead is comparable to other cluster-level autoscaling components such as the k8s HPA controller because it runs outside the critical request path Future Work Spandana does not assume that all VM-hosted services can run unchanged on a FaaS platform Porting to serverless is most straightforward for stateless services (or services whose state is already externalized to managed storage) Services with their own persistent state are treated as VM-only in Spandana The architecture is agnostic to specific compute instance types, allowing adaptation to alternative configurations such as employing burstable instances or orchestrating a heterogeneous cluster of mixed VM types References

1: [n.

Title and authors: Tom: So we're looking at this paper, "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations," and it tackles a problem where cloud services have these tiny, sub-second load swings but they still need to hit very strict Service Level Objectives or SLOs.

Jane: That’s the core tension here: you either over-provision resources to guarantee those SLOs, which burns money on idle capacity, or you try to save money by being reactive, and that fails when the load fluctuates quickly.

Lu: The authors introduce Spandana and their main idea is decoupling the enforcement of those SLOs from how they decide what resources to spend.

Meng: Decoupling sounds good on paper, but in production, it always comes back to cost. How does this actually translate to real savings when you're dealing with unpredictable traffic?

Tom: The paper explains that they use a two-level architecture: locally at each application instance, there's a decision point deciding if an incoming request can meet the SLO just by staying on the VM, or if it needs to be offloaded immediately to a serverless layer.

Jane: So, instead of one big system trying to balance everything upfront, every single request gets its own immediate check against the instance's current load and queue length.

Lalam: The paper highlights that the key insight is that any request that can be handled by a VM without violating the latency budget should stay on the VM, and only those requests that can't are sent to serverless > This is about making a local call based on real-time instance conditions rather than a global prediction of future load.

Tom: And then at the global level, they have this resource optimizer that looks at the entire deployment and adjusts how many VMs are allocated to minimize the total cost, taking those serverless costs into account > That's where they try to find that sweet spot between spending too much on VMs or violating SLOs.

Jane: The numbers show something interesting in Table one which compares different systems like HPA and Libra against this new Spandana approach > It shows Spandana achieves SLO violations of just zero point zero five percent while delivering the lowest total cost overall.

Meng: A violation rate of zero point zero five percent sounds good, but I wonder about the cost component in that table, especially when comparing it to Libra which has a total cost breakdown like VM versus serverless > We need to see if that low cost is sustainable when we scale up significantly.

Lu: The improvement they suggest is about making those local decisions smarter and giving the global optimizer better input by simulating different provisioning configurations against recent high-resolution load profiles > They use this simulation to find the configuration with the minimum cost, which they say average or total load alone isn't enough to determine.

Title and authors: Tom: So, the mechanism for finding that minimum cost involves a continuous control loop where a custom sidecar proxy implements something like a leaky bucket pacing mechanism > That proxy monitors the instance response latency and dynamically adjusts how fast it lets requests pass through, increasing the pacing interval if it starts exceeding that SLO target.

Jane: That adds another layer of dynamic control right there, making the local enforcement even more responsive than just a simple yes or no decision > It's not just checking utilization; it’s actively pacing the traffic flow based on performance feedback.

Lalam: And they also point out that to make this work better under fluctuating load, Spandana can leverage burstable and spot instances for VM provisioning > This suggests they aren't just sticking to standard VMs; they're trying to use cheaper hardware options while still guaranteeing that SLO adherence under volatile conditions.

Meng: Using burstable or spot instances helps with the cost side, I guess, but it introduces another risk if those underlying instances get preempted unexpectedly while you're trying to maintain strict SLOs > We need assurance that the guaranteed latency holds even when the hardware changes underneath.

Tom: The paper addresses that by suggesting these instance types can be used for VM provisioning to further lower deployment cost and provide resilience if a VM goes down > It’s about using cheaper compute while still trying to guarantee the latency target through smarter orchestration.

Jane: So, what they are really showing us is how to design a system that doesn't have to guess how much capacity it needs ahead of time, but instead reacts locally and optimizes globally > It shifts the burden from perfect prediction to smart, continuous adjustment.

Lu: The main limitation they admit is that this architecture doesn't assume all VM-hosted services can just be ported over to a serverless platform easily > They specifically treat services with their own persistent state as being VM-only because serverless isn't the right fit for them in this model.

Meng: That makes sense, I mean, if you have state that needs to be maintained between requests, you can't just throw it onto a stateless FaaS platform easily > So the constraint on stateful services is practical.

Tom: The overall implication for me is that this isn't just about one application; it’s about entire chains of microservices where you see significant cost reductions and higher CPU utilization across the whole chain > It moves beyond individual application optimization to system-wide efficiency.

Jane: It sounds like a practical way to handle the reality of cloud workloads, which is that perfect scaling is impossible, so you have to manage the trade-offs between cost and service guarantees > They found a way to make that management process much more automated and efficient.

Title and authors: Lu: So, if we think about what this means for future research, it’s less about finding one big magical autoscaler and more about building these layered control loops that can react at different time scales > It's more focused on the local decision making feeding into a global cost optimization loop.

Tom: Yeah, so the focus shifts from just getting a perfect prediction to getting a robust system that handles the uncertainty gracefully while keeping costs down > That’s the practical engineering lesson here.

Jane: So, we’ve seen how they reconcile strict SLOs with low cost under fine-grained load fluctuations by decoupling enforcement and using continuous local and global optimization loops > They found a way to keep utilization high at about seventy-seven percent while still staying within the SLO budget <ref:2606.30533#pg1,strict SLOs with low cost under fine-grained load fluctuations>.

Lu: That zero point zero five percent violation rate is what makes it compelling when you compare it to other systems like HPA-S or AutoBurst that might be over-provisioning for safety > It shows a real efficiency gain in resource usage.

Meng: I’m still concerned about the complexity of setting up that custom sidecar proxy and the global optimizer, but if the cost savings are that significant, it might be worth prototyping on a smaller service chain first > We need to test those simulation results against our actual production traffic profiles.

Tom: Exactly, Meng. The paper gives us a solid baseline for what to expect in terms of cost reduction and utilization improvement across the whole system rather than just looking at one isolated service performance > It shows the benefits are systemic.

Jane: So, to wrap up on "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations," the big picture is that you can achieve very strict latency guarantees while significantly lowering operational costs by making intelligent, decoupled decisions about where a request should be handled > It’s a concrete design pattern for this kind of volatile workload.

Lu: It’s less about one perfect algorithm and more about this specific architectural strategy of layered control and decoupling enforcement from cost optimization > That structure is what they found to work best.

Meng: I think the path forward is taking those simulation results and seeing how robust the global optimizer is when we introduce real-world failure modes, not just clean load profiles > We need to stress test that trade-off balance.

Tom: Alright, so for this paper, it’s a strong demonstration that you can get high utilization and low cost without sacrificing strict SLOs if you use this kind of layered approach > It gives us a blueprint for designing systems under these tricky conditions.

Jane: That seems like the practical summary of what they achieved with Spandana, balancing the need for guarantees against the need to keep running costs down in a volatile environment > We’ll keep an eye on how their work on this architecture evolves.

The paper's summary: Tom: So we're looking at "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations." Basically, this paper tackles that headache where you need your service to be super fast—strict latency targets—but you also have to keep the operational costs down when traffic swings wildly.

Jane: Right, so the big takeaway is they figured out a way to stop treating cost and performance as totally separate problems. They introduce this two-level structure where every individual request gets an instant check against its own instance's current load before it even hits the main processing engine.

Lu: The core mechanism there is local decision making: if the instance thinks it can handle a packet within its latency budget, it stays on that VM, but if not, it immediately sends that packet over to a serverless layer. That’s decoupling enforcement from cost optimization.

Meng: I'm thinking about the cost part here. If you decide every single request has to make this instant check against local utilization and queue length before deciding what happens, doesn't that just introduce massive overhead in the control plane?

Tom: That's where they address it globally. They have this resource optimizer that runs on a coarser time scale, looking at the load across all the VMs and serverless instances. It periodically adjusts how many VMs you’ve allocated to find the configuration that minimizes total cost while still meeting those SLO targets overall.

Jane: So, it's not just reacting locally; it's having a global brain that looks at the whole deployment picture and tweaks the resource pool to be cheaper without breaking any promises. They found that this approach keeps CPU utilization high, around seventy-seven percent average across the entire service chain.

Lalam: And they show that with this method, they can keep those strict latency violations down to just zero point zero five percent. That level of adherence is pretty tight for systems dealing with unpredictable load.

Meng: Seventy-seven percent utilization sounds good on paper, but I need to know how much extra engineering it takes to build that continuous control loop and the global optimizer. What breaks when you try to run that complexity in a real, high-throughput environment?

Tom: The overhead analysis shows the control plane components, like that resource optimizer, scale with how much traffic gets offloaded. But they say the absolute overhead is still very small compared to the main request path. It’s not crippling performance.

Jane: So what this means for someone just listening—it means you don't have to bet everything on perfect future prediction or massive over-provisioning just to hit your latency goals; you can build a system that intelligently trades off cost against guarantee in real time.

Lu: What they’re really showing us is that instead of building one giant, complex autoscaler, you build these layered control loops—local decision points feeding into a global cost optimizer—that react at different speeds. That structure is what they found works best for this kind of volatile workload.

Meng: So the real implication for me is that we should be looking at using burstable and spot instances with this model to really drive down the baseline VM cost while still maintaining that SLO guarantee under load swings.

Tom: Exactly. It’s about using cheaper hardware options, like those burstable or spot instances, but making sure the orchestration layer handles the volatility so you don't violate your latency budget when things get busy. That’s how they lower deployment costs without sacrificing service quality.

The paper's improvements: Tom: So we're talking about how they improve this idea of decoupling SLO enforcement from cost optimization. It’s not just setting up the initial structure; they suggest making those local decisions even smarter and more responsive to sudden traffic spikes.

Jane: Right, so beyond just having a "yes or no" check on a request, they introduce something called a leaky bucket pacing mechanism in that sidecar proxy. That means instead of always letting requests flood through the instance at full speed, the system actively monitors the response latency and dynamically changes how fast it lets traffic pass.

Lu: This dynamic pacing is important because it allows the system to throttle back on a local instance immediately if its performance starts dipping toward that SLO boundary, rather than waiting for a slower, coarser global adjustment.

Meng: I see that as a way to add immediate feedback into the local decision-making loop. It’s like giving the individual pod an internal brake based on its own real-time stress, which makes sense if we're worried about sudden bursts breaking things at scale.

Tom: And then they connect that back to hardware choices. They suggest leveraging burstable and spot instances when provisioning those VMs to further lower the cost structure while still aiming for that SLO adherence under fluctuating load conditions.

Jane: That’s a big practical step, because it means you can use cheaper compute resources for the baseline VM allocation, but you get that resilience benefit from burstable or spot instances if things get really busy. It’s about using the right tool for the job at different times.

Lalam: I'm confident in this direction because they’ve shown that using these instance types can help lower deployment cost while still trying to keep that latency target under volatile conditions. My models confirm that this combination is empirically more stable than just sticking to standard instances.

Meng: It’s interesting because it shifts the focus from just finding the cheapest VM to finding the *cheapest viable* VM configuration that can actually guarantee the SLO given its known failure modes, like preemption risk on spot instances.

Tom: The implication for me is that we need to build our cost models around this idea of using a mix of instance types, not just one uniform pool. It’s about optimizing the whole resource stack rather than just the VM count in isolation.

Jane: So what this changes for someone just listening—it means designing a system where you can be aggressive about reducing operational spend by using cheaper compute, but you have built in enough automated feedback mechanisms to ensure the critical service level targets still get met.

Lu: They also flag a limitation, which is important: they state clearly that this whole architecture doesn't work for services that have their own persistent state between requests; those are still best left on standard VMs because serverless isn't the right fit there.

Meng: That’s a clear boundary. It tells us exactly where we need to keep our stateful workloads and where we can safely apply this cost-aware, SLO-driven orchestration strategy.

Tom: So, the core improvement is adding fine-grained, reactive control loops on top of the existing global optimization framework. It makes the system much more agile in handling those rapid load changes you see in modern cloud services.

Conclusion: Tom: So to wrap up "Reconciling Strict SLOs with Low Cost under Fine-Grained Load Fluctuations," this paper basically proves you can hit extremely tight latency targets while keeping operational costs low by using this layered control system.

Jane: It's a concrete design pattern for handling the messy reality of volatile cloud workloads, showing that you don't have to just over-provision everything because prediction is impossible.

Lu: The main implication is that instead of one massive predictor trying to guess everything ahead of time, you build this system with local decisions feeding into a global cost optimizer.

Meng: I think the biggest takeaway for production is that you can actually use cheaper hardware options like burstable instances when it makes sense for the overall cost structure, as long as your control plane is smart enough to manage the SLO guarantee.

Lalam: My literature confirms that this approach delivers a violation rate of just zero point zero five percent, which means it’s a very solid performance metric for systems facing unpredictable load.

Tom: It shows that high utilization—around seventy-seven percent across the whole service chain—is achievable without sacrificing the latency guarantees you need for your users.

Jane: So, we’ve seen how they reconciled strict SLO adherence with low cost under fine-grained load fluctuations using this decoupling and continuous optimization structure.

Lu: It really changes how we think about autoscaling, moving away from just scaling based on average load to scaling based on real-time performance feedback loops.

Meng: I’m still thinking about the complexity of setting up that global optimizer, but if the cost savings are that significant across an entire service chain, it might be worth prototyping this layered approach on a smaller piece of our infrastructure first.

Tom: Exactly. The paper gives us a blueprint for designing systems under these tricky conditions where you need guarantees but also need to stay efficient.

Jane: And that’s what we're going to keep in mind as we look at how these kinds of layered, reactive control systems can be applied to other parts of the cloud stack.

More episodes

← Home