FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence
cs.NI, cs.AI
Submitted: 2024-02-06
Updated: 2026-10-11
Comments: Accepted by SoCC 2026
Code: https://github.com/noise-lab/FlowWise
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryption and service consolidation make rule-based
Terminology
Abstract
Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryption and service consolidation make rule-based classification unreliable. At high flow-arrival rates, however, new flows can arrive faster than end-to-end analysis pipelines can process. Unlike typical cloud inference workloads, packets from the same flow arrive over time and require per-flow state. This streaming, stateful setting limits naive scale-out with stateless workers. This paper presents FlowWise, a serving system for streaming, stateful flow intelligence on commodity CPU/GPU servers that jointly selects (i) how many packets to collect and (ii) which model(s) to apply for each flow to balance latency, service rate, and accuracy. Across three representative tasks, inference time across candidate models differs by 1.8x-141.3x, while packet waiting time can exceed feature computation and inference by 6-8 orders of magnitude at the tail. FlowWise exploits these gaps with an uncertainty-driven fast-slow cascade: it issues early predictions from a one-packet model and escalates only uncertain flows to slower models that wait for additional packets and use richer features. A queue-centric runtime maintains per-flow state, performs asynchronous feature joining, and uses brokered, flow-ID-keyed streams to distribute workload across heterogeneous workers. FlowWise serves 76.3% of flows within 16 ms, improving median end-to-end serving latency by 40.5x while increasing service rate and maintaining similar accuracy to always running the slow model. With thousands of packet-header features per flow, our scale-up benchmark sustains over 48.5k new flows per second on a single 16-core commodity CPU server, comparable to reported flow-arrival rates in city-backbone traces. FlowWise is open sourced at https://github.com/noise-lab/FlowWise.
Sources
- pForest: In-Network Inference with Random Forests
- Training with Quantization Noise for Extreme Model Compression
- netFound: Principled Design for Network Foundation Models
- LoRA: Low-Rank Adaptation of Large Language Models
- NetDiffusion: Network Data Augmentation Through Protocol-Constrained Traffic Generation
- AC-DC: Adaptive Ensemble Classification for Network Traffic Identification
- AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
- Rethinking the Value of Network Pruning
- Automated Website Fingerprinting through Deep Learning
- EdgeServe: A Streaming System for Decentralized Model Serving
Related papers
- HiFiNet: Hierarchical Fault Identification in Wireless Sensor Networks via Edge-Based Classification and Graph Aggregation
- PACC: Protocol-Aware Cross-Layer Compression for Compact Network Traffic Representation
- TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces
- Embodied AI in 6G Networks: From Intelligent Connectivity to Physical Intelligence
- Lightweight GenAI for Network Traffic Generation: Fidelity, Augmentation, and Classification
- NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning