Waiting at the front door: Continuous monitoring of latency in the host network stack
summary
The gist
With networking moving into the sub-millisecond latency domain, latency in the end host itself can become a significant barrier to achieving consistently low application latency The authors present…
In short
netstacklat is a monitoring tool that tracks latency at multiple points within the Linux kernel network stack, from packet arrival to application read. It uses eBPF and kernel timestamps to provide continuous, low-overhead monitoring of host network latency under various workloads. Results show it has minimal overhead and effectively detects long latency tails and anomalous events.
Key concepts
- Host Network Latency
- This measures the total delay a data packet experiences on a server, starting from when it hits the network interface card (NIC) until an application finally reads that data. It is crucial for modern, low-latency applications where even small delays can impact performance.
- eBPF Tracing
- eBPF (extended Berkeley Packet Filter) allows the tool to safely run custom monitoring programs directly inside the Linux kernel. This enables netstacklat to observe network events at precise points in the kernel stack without needing complex external state management or high-overhead system calls.
- SKB Timestamping
- Socket Buffer (SKB) timestamping is a mechanism used by the Linux kernel to record exactly when a packet arrives at different stages of the network stack. netstacklat leverages this built-in kernel feature to calculate latency directly within its eBPF programs, making measurements accurate and efficient.
- Low Monitoring Overhead
- The tool is designed to have a very small performance footprint, averaging only 0.81% CPU utilization across tests. This low overhead is vital because it allows the monitoring system to run continuously in production environments without significantly degrading the actual application's performance.
Terminology used across episodes
This episode discusses
- Waiting at the front door: Continuous monitoring of latency in the host network stack · Paper Radio
- Computing Extremely Accurate Quantiles Using t-Digests
- The eBPF Runtime in the Linux Kernel
- Circllhist -- A Log-Linear Histogram Data Structure for IT Infrastructure Monitoring
The paper
Waiting at the front door: Continuous monitoring of latency in the host network stack · Read on arXiv
Simon Sundberg, Anna Brunstrom, Simone Ferlin-Reiter, Jesper Dangaard Brouer, Toke Høiland-Jࡡlandsen
Karlstad University · Red Hat Research and the Swedish Knowledge Foundation
With networking moving into the sub-millisecond latency domain, latency in the end host itself can become a significant barrier to achieving consistently low application latency. Both the physical interconnect between the network card and the CPU, the kernel network stack, and the scheduling of applications themselves can be considerable sources of latency. Previous work has studied host latency at various levels, yet there remains a lack of methods and tools to continuously monitor host latency in production. To remedy this, we present netstacklat, a monitoring tool that captures latency at several points in the host network, from the early parts of the Linux kernel network stack all the way until the application reads the data. We evaluate netstacklat in a testbed, demonstrating its ability to capture host latency across 144 variations of HTTP workloads for Nginx and Apache, while also showing how the low monitoring overhead does not inflate tail latency by more than 6%, where previous monitoring solutions increase it by over 100%. Furthermore, we share our initial findings from deploying netstacklat in Cloudflare's global CDN network.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Waiting at the front door".
Jane: Here's the gist of the paper. With networking moving into the sub-millisecond latency domain, latency in the end host itself can become a significant barrier to achieving consistently low application latency The authors present netstacklat, a monitoring tool that captures latency at several points in the host network, from the early parts of the Linux kernel network stack all the way until the application reads the data They demonstrate its ability to capture host latency across 144 variations of HTTP workloads for Nginx and Apache, while also showing how the low monitoring overhead does not inflate tail latency by more than 6%, where previous monitoring solutions increase it by over 100% Host network latency, encompassing the delay from a packet arriving at the NIC to its final read by the application, is becoming increasingly important as modern networks push for sub-millisecond performance Previous methods like actively measuring end-to-end application latency or analyzing packet captures are unsuitable for continuously monitoring host network latency on production servers The work addresses this gap by designing netstacklat, a tool that combines the Linux kernel’s timestamping mechanism with eBPF to track host latency for every packet system-wide with a low enough footprint for continuous monitoring Tool Design and Methodology Netstacklat's core idea is to observe SKBs at several strategically selected points across the kernel network stack to break down how network stack latency builds up They use eBPF tracing programs with fentry or fexit attach points, leveraging a trampoline mechanism for very small overhead jumps into kernel functions The tool utilizes the SKB receive timestamp generated by SOF TIMESTAMPING RX SOFTWARE to calculate latency directly in the eBPF programs, avoiding external state management and lookup overhead They probe four different layers: ip-start, udp-start and tcp-start, socket-enqueued, and socket-read Evaluation Results The evaluation across 144 Nginx and Apache workloads showed that netstacklat has reasonably low overhead, averaging 0.81% CPU utilization across all workloads and only inflating end-to-end tail latency by 6% in a heavily loaded scenario where other state-of-the-art solutions increase tail latency by over 100% The results show that the latency typically grows at all four probe points as the load increases from a single connection up to 100 connections, while application latency with tcp-socket-read is typically inflated by three orders of magnitude or more Furthermore, netstacklat can detect anomalous events where host latency continuously increased over a 3-hour period before returning to normal Comparative Analysis and Future Work When comparing monitoring overhead, netstacklat is shown to be the only tool suitable for continuous monitoring as it maintains a small footprint both during lightly and heavily loaded scenarios While its ingress monitoring can be expanded to cover egress packets or user space components, future work could extend its capability by calculating delay relative to hardware timestamps from NICs The authors recommend using netstacklat's high-level overview to guide the collection of more detailed traces for root cause identification using complementary tools like pwru or retis In this work, we have presented netstacklat, a novel monitoring tool that collects latency distributions within several layers of the host network stack for ingress traffic Netstacklat achieves compatibility with a wide variety of systems and a low enough overhead to support continuous monitoring in production environments The tool successfully detects long latency tails, reaching hundreds of microseconds in the early parts of the network stack, tens of milliseconds before the packet is enqueued to the socket, and seconds before the application reads the data The research suggests that tracking latency within the host network stack will become a crucial aspect of performance monitoring as end hosts increasingly become bottlenecks Acknowledgments This research was funded by Red Hat Research and the Swedish Knowledge Foundation (grant 20220072) Additional thanks to Felix Maurer for assisting with lattrace, and to the anonymous reviewers for their helpful feedback References
Title and authors: Tom: So, we're talking about this paper today called "Waiting at the front door: Continuous monitoring of latency in the host network stack". This research looks at how latency inside your computer itself becomes a big problem when we push networks to sub-millisecond speeds.
Jane: That sounds like it hits a nerve because if the network gets super fast, that delay in the host can totally bottleneck everything we're trying to do. So, what's the core idea here?
Tom: Well, they present a monitoring tool called netstacklat. It captures latency at several points across the whole Linux kernel network stack, all the way until an application actually reads the data. It’s designed for continuous monitoring in production environments where you can’t just do a one-off measurement.
Lu: What's interesting about its design is that they use eBPF tracing programs with fentry or fexit attach points to probe these specific layers of the stack, like ip start, tcp start, socket enqueued, and socket read. It tries to avoid slow external state management by using the SKB receive timestamp generated by SOF TIMESTAMPING RX SOFTWARE directly inside the eBPF programs.
Meng: That sounds complicated to set up on a production server with low overhead. What’s the practical trade-off they found between having this detailed view and actually slowing down your system?
Jane: They did some testing across one hundred forty-four different HTTP workloads, including Nginx and Apache, and they found that the monitoring overhead was really low—averaging only about zero point eight one percent CPU utilization across all those tests.
Tom: And the key result there is that this low overhead doesn't actually inflate the tail latency by more than six percent in a heavily loaded scenario, which is significant because previous tools we've seen have increased tail latency by over one hundred percent.
Lu: The paper shows that latency typically grows as you increase load, from just a single connection up to one hundred connections, at all four probe points they measure. But the application latency measured at the socket read point can be inflated by three orders of magnitude or more under those conditions.
Meng: So if I'm running a system that handles thousands of requests per second, this tool seems like it could give me visibility into where the delay is building up, even when things are busy. But what about detecting weird spikes?
Jane: They also showed that netstacklat can detect anomalous events where host latency keeps increasing for a whole three hours before it goes back to normal, which points toward catching those intermittent issues that are really hard to find with standard monitoring.
Tom: It seems like the main improvement they're pushing is moving beyond just end-to-end latency measurements and providing a continuous, deep look *inside* the host network stack for production use. The paper calls this "Waiting at the front door: Continuous monitoring of latency in the host network stack".
Title and authors: Lu: What I find compelling is how they map out that entire ingress path, showing exactly where each probe point sits relative to functions like netif receive skb and tcp v4 rcv, which gives a really concrete view of the delay accumulation.
Meng: From a deployment standpoint, if this is so low overhead across those one hundred forty-four workloads, it suggests it could be feasible to run this kind of monitoring continuously without needing massive dedicated resources just for the monitoring itself.
Jane: The conclusion is that tracking latency within the host network stack will become a crucial part of performance monitoring as end hosts start acting like bottlenecks rather than just being the endpoint. It's about seeing where time is actually spent in those early kernel layers, not just waiting at the very end.
Tom: So to wrap up this paper on "Waiting at the front door: Continuous monitoring of latency in the host network stack", it gives us a way to monitor latency distributions across multiple layers with very low overhead, and it successfully detects long latency tails that can happen deep in the stack.
Lu: What I think is that for representation learning or performance analysis, seeing this kind of detailed breakdown of where time accumulates—from the driver level up to the socket read—is really useful context for understanding system behavior.
Meng: For me, it’s about deployability. If you can continuously monitor without spiking your CPU utilization by more than six percent during peak hours, that makes it something I could actually consider putting into a production pipeline at scale.
Jane: So we've seen how netstacklat works and what the numbers say about its performance and detection capabilities in this paper on "Waiting at the front door: Continuous monitoring of latency in the host network stack". What comes next for us?
Tom: Well, I think the next step is seeing how this specific tool fits into larger systems, maybe integrating its insights with other tools like pwru or retis to get more detailed root cause analysis on those latency tails.
Lu: And from a theoretical side, they suggest that while ingress monitoring is covered now, future work could extend it to calculate delay relative to actual hardware timestamps coming directly from the NICs, which would give an even finer measurement of the physical layer impact.
Meng: If they can get closer to hardware timestamps without adding massive overhead, that makes this tool even more valuable for diagnosing those very early delays we see in production.
Jane: It really shows how important it is to have these kinds of continuous, deep insights into the host when we're dealing with sub-millisecond network demands. That's what this paper on "Waiting at the front door: Continuous monitoring of latency in the host network stack" is all about.
The paper's summary: Tom: So, to recap where we are: these authors wrote this paper about a tool called netstacklat that watches latency all the way from when a packet hits your network card to when your application finally reads the data.
Jane: Right, and what they found is that this tool lets you see exactly how much time is being lost at each step inside the operating system’s network stack.
Tom: The main point is that in today's fast networks, those tiny delays inside your computer are becoming a huge problem for keeping applications running smoothly.
Lu: They built it using eBPF tracing programs, which is this powerful way to hook into the Linux kernel and measure things without adding a lot of overhead.
Meng: But there’s always that trade-off, right? How much does this monitoring tool actually slow down my production server when I have real traffic going?
Tom: And that’s where they got some really strong numbers. They tested it across one hundred forty-four different web workloads, like Nginx and Apache.
Jane: So the key result is that the monitoring overhead was very low, averaging about zero point eight one percent CPU utilization across all those tests.
Tom: And even under heavy load, this low overhead didn't inflate the tail latency by more than six percent, which is a big deal because other tools we’ve seen have increased tail latency by way over one hundred percent.
Lu: The paper shows that as you increase the traffic, from just one connection up to a hundred connections, the latency typically grows at all four places they are measuring.
Meng: So it means this tool can act as a load signal; you can use it to see exactly when your server is getting overloaded because that specific latency metric starts spiking significantly.
Jane: And what’s really interesting for continuous monitoring is that netstacklat can actually detect weird spikes, like when the host latency keeps climbing for three hours before it settles back down to normal.
Tom: That ability to catch those long latency tails, which can reach hundreds of microseconds deep in the network stack, is something we haven't seen as consistently measured before.
Lu: It changes how we think about performance monitoring because instead of just looking at the end-to-end time, you get a detailed map of exactly where time is accumulating in that complex system.
Jane: So for someone just listening, it means that when you're trying to make your network sub-millisecond, you can start looking inside the computer itself for the hidden delays.
Tom: It’s about seeing what happens before the packet even gets put into the socket queue, which gives you a much clearer picture of where to focus your tuning efforts.
Lu: The implication is that tracking latency within those kernel layers will become a really crucial part of performance monitoring as we deal with these increasingly bottlenecked end hosts.
Jane: And since they can detect these long tails, it suggests that understanding the inner workings of the host network stack is going to be a major focus for making systems faster in the coming years.
Tom: It’s a big step toward making sure that when we push networks to sub-millisecond speeds, we aren't just chasing end-to-end time but actually understanding what’s happening at every single layer along the way.
The paper's improvements: Tom: So we’ve seen how netstacklat works and what the numbers say about its performance, and now we need to talk about what they suggest next for improving this tool.
Jane: What are the authors actually proposing to do with this research going forward? Are they just happy with where they landed, or is there more work ahead?
Lu: They are looking at a few things. One thing is expanding the monitoring beyond just ingress traffic, which means looking at egress packets too.
Tom: So if we can monitor traffic leaving the host as well, it gives us a much more complete picture of where all that latency is actually coming from in the network path.
Jane: That makes sense. And another thing they suggest is moving the measurement point even closer to the physical hardware itself, specifically using actual hardware timestamps from the Network Interface Cards.
Meng: Using hardware timestamps sounds ambitious, though I have to ask, how much overhead does that introduce? Does it make this monitoring tool unusable in a real production environment?
Tom: That’s a valid concern. But they are pushing the idea because those direct hardware measurements would give us an even finer measurement of the physical layer impact on latency.
Lu: The authors suggest using netstacklat's high-level overview as a guide to collect those more detailed traces, and then use complementary tools like pwru or retis to figure out what’s causing the specific delays they find.
Jane: So it’s about using this tool not just as a measurement device, but as a way to pinpoint where you need other specialized tools for deep root cause analysis.
Tom: It implies that the next step isn't just having more data, but having a better system for using that data to actually fix the performance issues we find.
Meng: From my side, if they can get closer to those hardware timestamps without adding massive overhead, this tool becomes way more valuable for diagnosing those really early delays we see in production traffic.
Lu: They are essentially saying that while ingress monitoring is covered now, the next big step is calculating delay relative to actual hardware timestamps to get a finer measurement of the physical layer impact.
Jane: It’s about moving from just knowing *where* the delay is building up in the software stack to knowing *exactly* what’s happening at the wire itself.
Tom: So, we’re moving from just capturing latency distributions to building a framework that connects those kernel measurements with real hardware timing data for precise diagnosis.
Conclusion: Tom: So we're wrapping up this look at netstacklat, which is titled "Waiting at the front door: Continuous monitoring of latency in the host network stack".
Jane: To recap, they built a tool that tracks latency all the way from when a packet hits your network card to when your application reads the data.
Tom: Right, and they proved it can do this with very low overhead across one hundred forty-four different web workloads.
Lu: The paper shows that this method is suitable for continuous monitoring in production because it maintains a small footprint both during light and heavy loads.
Meng: So you’re saying an engineer could actually deploy this kind of deep monitoring without needing massive dedicated resources just for the monitoring itself?
Tom: Exactly, and they showed it doesn't inflate end-to-end tail latency by more than six percent in a heavily loaded scenario where other tools increase it by way over one hundred percent.
Jane: That’s a huge win because it means you can monitor performance without introducing your own significant bottleneck into the system.
Lu: The implication is that tracking latency within those kernel layers will become a crucial part of performance monitoring as we deal with these increasingly bottlenecked end hosts.
Tom: It's about seeing what happens before the packet even gets put into the socket queue, which gives you a much clearer picture of where to focus your tuning efforts.
Jane: So for someone just listening, it means that when you're trying to make your network sub-millisecond, you can start looking inside the computer itself for those hidden delays.
Lu: They are also suggesting that while ingress monitoring is covered now, future work could extend it to calculate delay relative to actual hardware timestamps from the Network Interface Cards.
Meng: If they can get closer to those hardware timestamps without adding massive overhead, this tool becomes way more valuable for diagnosing those very early delays we see in production traffic.
Tom: So, the next step is using this tool not just as a measurement device, but building a framework that connects those kernel measurements with real hardware timing data for precise diagnosis.
Jane: It’s about moving from just knowing where the delay is building up in the software stack to knowing exactly what’s happening at the wire itself.
Lu: The implication is that this detailed breakdown of where time accumulates, from driver level up to socket read, provides really useful context for understanding system behavior.
Meng: For me, it’s about deployability. If you can continuously monitor without spiking your CPU utilization by more than six percent during peak hours, that makes it something I could actually consider putting into a production pipeline at scale.
Tom: So to wrap up this work on "Waiting at the front door: Continuous monitoring of latency in the host network stack", they’ve given us a way to monitor latency distributions across multiple layers with very low overhead.
Jane: It successfully detects long latency tails that can happen deep in the stack, reaching hundreds of microseconds before a packet is enqueued to the socket.
Lu: What I think is that for representation learning or performance analysis, seeing this kind of detailed breakdown of where time accumulates—from the driver level up to the socket read—is really useful context for understanding system behavior.
Meng: It’s about making sure that when we push networks to sub-millisecond speeds, we aren't just chasing end-to-end time but actually understanding what’s happening at every single layer along the way.
Tom: The research suggests that tracking latency within the host network stack will become a crucial aspect of performance monitoring as end hosts increasingly become bottlenecks.
Jane: So next time you hear about network performance, think about where that delay is hiding inside the computer itself, not just at the very edge.
More episodes
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines