Adapting Kubernetes controllers to the edge: on-demand control planes using Wasm and WASI

summary

The gist

Kubernetes’ high resource requirements hamper its adoption in constrained environments such as the edge and fog This paper presents a WebAssembly-based framework for running lightweight controllers…

In short

Kubernetes controllers are resource-heavy processes that consume memory even when idle, which is problematic for low-resource edge environments. This work introduces a WebAssembly (WASM) framework to run lightweight operators on demand, achieving a 64% memory reduction compared to traditional container methods. It enables scalable, on-demand control planes suitable for constrained systems.

Key concepts

Kubernetes Controllers
These are long-lived processes within Kubernetes that constantly monitor the cluster state and react to changes. They require continuous resources even when there is no immediate activity, making them costly in low-resource settings.
WebAssembly (WASM)
WASM allows running code securely and efficiently outside of a traditional container. It enables the framework to run small, isolated operator logic directly within a process with minimal overhead, facilitating on-demand execution.
Scale-to-zero Architecture
This design allows inactive child operators to be dynamically unloaded when they have been idle for a long time. This feature ensures that resources are only consumed when needed, optimizing memory usage in environments where resources are scarce.

Terminology used across episodes

This episode discusses

The paper

Adapting Kubernetes controllers to the edge: on-demand control planes using Wasm and WASI · Read on arXiv

Merlijn Sebrechts, Tim Ramlot, Sander Borny, Tom Goethals, Bruno Volckaert, Filip De Turck

IDLab Department of Information Technology Ghent University imec Belgium

Kubernetes' high resource requirements hamper its adoption in constrained environments such as the edge and fog. Its extensible control plane is a significant contributor to this, consisting of long-lived processes called "controllers" that constantly listen for state changes and use resources even when they are not needed. This paper presents a WebAssembly-based framework for running lightweight controllers on-demand, only when they are needed. This framework extends the WebAssembly System Interface (WASI), in order to run Kubernetes controllers as lightweight Wasm modules. The framework runs these Wasm controllers in a modified version of Wasmtime, the reference WebAssembly (Wasm) runtime, that swaps idle controllers to disk and activates them when needed. A thorough evaluation shows this framework achieves a 64% memory reduction compared to traditional container-based controller frameworks.

DOI: 10.1109/CloudNet55617.2022.9978884

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Adapting Kubernetes controllers to the edge".

Title and authors: Tom: So we're looking at this paper called "Adapting Kubernetes controllers to the edge: on-demand control planes using Wasm and WASI". It sounds like they’re tackling a huge problem with Kubernetes running on low-resource hardware, specifically the edge stuff.

Jane: Right, and what I hear from the title is that they're trying to make those long-running controllers run only when they actually need to, instead of always being on. It seems like they’re focusing on how WebAssembly and WASI can help them achieve this lightweight execution.

Lu: The core idea is moving away from those always-on processes—the controllers—and making them run on demand using WebAssembly modules that are only loaded when necessary <ref:2209.01077#pg1>. It’s about making the control plane much more efficient for places with very limited power and memory.

Meng: From my perspective, I'm thinking about how this translates to deployment costs on real hardware, because right now, we’re paying for that overhead constantly. If we can scale down those controllers dynamically, it means less persistent compute resources needed for the control plane itself <ref:2209.01077#pg1>.

Lalam: I see the main thrust is using a framework that lets them swap idle controllers to disk and only activate them when an event happens, which sounds like a big win for scalability <ref:2209.01077#pg1>.

The paper's summary: Tom: So the paper explains this idea of having these lightweight controllers that live on demand, and they’re using this WebAssembly-based framework built on WASI to run them. They are extending the WebAssembly System Interface, which is key for letting these controllers interact with the system in a standardized way <ref:2209.01077#pg1>.

Jane: It seems like they’re using a specific runtime called Wasmtime, and they modify it so that instead of just running the modules constantly, it swaps them out to disk when they are idle and brings them back up when something triggers a change <ref:2209.01077#pg1>.

Lu: The architecture involves a parent operator that manages an event loop, and this loop is what handles calling host functions from inside the WebAssembly module, which is crucial for making these child operators asynchronous <ref:2209.01077#pg3>.

Meng: As an engineer shipping things in production, I’m curious about that parent operator loop—how does it manage the work queue so we know exactly when to unload or reload a controller? That sounds like a complex scheduling problem to get right at scale <ref:2209.01077#pg3>.

Lalam: They mention that inactive operators are unloaded after a long period of inactivity, which is essentially the scale-to-zero capability they’re aiming for <ref:2209.01077#pg3>.

The paper's improvements: Tom: One of the major things they highlight is how this framework achieves a sixty-four percent memory reduction compared to traditional container-based controller frameworks, which is a huge number for resource savings <ref:2209.01077#pg1>.

Jane: That memory saving is directly tied to how they structure the architecture, focusing on isolation overhead and modularity, so you can add or remove child operators without messing up the others <ref:2209.01077#pg1>.

Lu: They also move the low-level operator logic into the parent operator, which simplifies things for those client operators and reduces their complexity <ref:2209.01077#pg1>. And they chose Rust for the child operators because of its support for WASI host functions and better interoperability than Golang IMPROVEMENT three.

Meng: Rust sounds promising from a deployment standpoint if it means less memory overhead per operator, especially when dealing with garbage collection compared to other languages IMPROVEMENT three. But what about the performance hit? Assembly runtimes usually have some performance cost compared to native code, which is something I need to check on <ref:2209.01077#pg2>.

Lalam: They also mention that the parent operator manages the asynchronous runtime for a child operator and unloads it if it hasn't received any asynchronous results over a long time, which optimizes resource utilization IMPROVEMENT two <ref:2209.01077#pg3,any asynchronous results over a long>.

Conclusion: Tom: So to wrap up, the main point of "Adapting Kubernetes controllers to the edge: on-demand control planes using Wasm and WASI" is that this framework lets you replace those heavy, always-on controllers with lightweight WebAssembly modules that only run when needed <ref:2209.01077#pg1>.

Jane: It really boils down to achieving a sixty-four percent memory reduction compared to the old container methods, and they’ve built an asynchronous runtime in the parent operator to handle those calls efficiently so things don't block IMPROVEMENT two <ref:2209.01077#pg3>.

Lu: The modularity is important too, allowing you to swap out components easily, and using Rust for the child operators helps keep the per-operator memory footprint low IMPROVEMENT three. It’s about making Kubernetes control planes viable in those constrained edge environments where resources are tight <ref:2209.01077#pg1>.

Meng: I still have to be concerned about that performance hit they mentioned regarding assembly runtimes versus native code, and how reliable that swap-to-disk and reload cycle is under real traffic load <ref:2209.01077#pg2>.

Lalam: The authors admit a limitation, which is that even with the optimizations, there's still some overhead associated with the runtime itself when you’re constantly cycling modules in and out IMPROVEMENT four <ref:2209.01077#pg2>. But overall, it shows a path toward running these kinds of management tasks on very small devices <ref:2209.01077#pg1>.

Tom: So that's the big picture here, using this paper to look at how we can actually make Kubernetes control planes fit where they need to be deployed, which is a really tangible goal for edge computing <ref:2209.01077#pg1>.

More episodes

← Home