denet, A lightweight command-line tool for process monitoring in benchmarking and beyond
summary
The gist
denet is a lightweight process monitoring tool providing real-time resource profiling of running processes It reports CPU, memory, disk I/O, network activity, and thread usage…
In short
denet is a lightweight Rust process monitoring tool offering real-time resource profiling of running processes. It provides deep insights like CPU, memory, and I/O metrics with adaptive sampling rates for better performance analysis in benchmarks. It features both a command-line interface and a Python API for structured data output.
Key concepts
- Adaptive Sampling
- denet adjusts its monitoring frequency based on the process's runtime. It samples at a high rate initially to capture startup spikes and transient activity, then slows down for steady state monitoring. This ensures detailed observation of both initial behavior and long-term resource usage.
- Recursive Child Monitoring
- The tool tracks not just the main process but also all its child processes. This allows users to see how a parent process spawns new tasks and monitor their resource consumption in relation to the parent's activity, which is crucial for understanding complex workflows.
- eBPF Support
- denet optionally uses eBPF technology for off-CPU analysis. This enables monitoring kernel-level events without relying solely on standard process metrics. It provides deeper visibility into system interactions that are otherwise invisible to traditional tools.
- Structured Output (JSONL)
- Instead of plain text, denet outputs performance metrics and process metadata in a structured JSON Lines format. This makes the data machine-readable and ideal for integration into automated systems, such as Snakemake workflows or custom analysis scripts.
Terminology used across episodes
This episode discusses
- denet, A lightweight command-line tool for process monitoring in benchmarking and beyond · Paper Radio
- The eBPF Runtime in the Linux Kernel
The paper
denet, A lightweight command-line tool for process monitoring in benchmarking and beyond · Read on arXiv
Ben Carrillo, Izaskun Mallona
University of Zurich · SIB Swiss Institute of Bioinformatics
Summary: denet is a lightweight process monitoring tool providing real-time resource profiling of running processes. It reports CPU, memory, disk I/O, network activity, and thread usage, including recursive child monitoring, with adaptive sampling rates. denet exposes two interfaces: a CLI with colorized output and a Python API. Output format is JSONL. Each record carries performance metrics and process metadata (PID and the executed command). Structured output makes denet suitable for benchmarking and tuning data-intensive pipelines in bioinformatics and beyond. We provide CLI and API examples, including a bioinformatics workflow in Snakemake, showcasing denet's diagnostic properties. Availability and implementation: denet is open-source software released under the GPLv3 terms, and maintained at https://github.com/btraven00/denet. It is implemented in Rust, with Python bindings provided via maturin, and installs from Cargo (cargo install denet), PyPI (pip install denet) or conda-forge. Most functionality does not require administrative privileges, enabling use on cloud platforms, HPC clusters, and standard Linux workstations.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "denet, A lightweight command-line tool for process monitoring in benchmarking and beyond".
Jane: Here's the gist of the paper. denet is a lightweight process monitoring tool providing real-time resource profiling of running processes It reports CPU, memory, disk I/O, network activity, and thread usage, including recursive child monitoring, with adaptive sampling rates denet exposes two interfaces: a CLI with colorized output and a Python API Output format is JSONL Each record carries performance metrics and process metadata (PID and the executed command) Structured output makes denet suitable for benchmarking and tuning data-intensive pipelines in bioinformatics and beyond Tool Capabilities denet offers several advanced monitoring features to provide deep process insight These capabilities include adaptive profiling with user-provided resolution, where the sampling rate varies with the elapsed runtime of the monitored process Metrics collected by denet encompass CPU usage, memory separation into RSS and VMS to expose peak usage and swapping events, GPU memory and percent usage (requiring NVML), disk reads and writes per interval, network bytes received and transmitted per interval, thread counts, child processes spawned from a parent, and the parent process exit code It also supports experimental eBPF support for off-CPU analysis Implementation Details The architecture of denet is modular, consisting of components such as core (sampling from /proc on Linux), config (user inputs), error, cpu-sampler (CPU time akin to top/htop), monitor (sampling loop and JSONL output), ebpf (optional kernel-level backend via BCC), and python (PyO3 bindings exposing the Rust API to Python) Adaptive sampling is implemented such that during the first second, denet samples at the highest frequency, resolving process startup and transient activity spikes Benchmarking Application The paper demonstrates denet's utility by integrating it with a Snakemake workflow in two ways First, in a wrap mode, each rule’s shell calls denet’s attach subcommand to record timeseries performance data in JSONL Second, in native mode, denet on PATH replaces Snakemake’s default psutil profiler to produce performance summaries This allows the timeseries recorded in wrap mode to resolve spikes and bottlenecks that would be hidden otherwise Workflow Diagnostics The fine-grained denet readout is showcased for workflow diagnostics, such as in the mark duplicates step where it indicates two distinct phases For instance, the trace shows a first phase where CPU is highly used across multiple cores with no disk writes, identifying in-memory sorting The second phase shows RSS dropping abruptly and then stabilizing while disk writes rise linearly, pointing to data stream output This joint reading of the timeseries benchmarking data applies elsewhere, attributing resource peaks to specific pipeline stages Comparison and Context Compared with other tools, denet provides adaptive sampling and recursive process-tree monitoring which top/htop and psutil only partially support In terms of structured output, denet streams structured text, unlike bpftrace and top/htop which are designed for interactive terminal sessions The comparison table indicates that denet supports adaptive sampling, programmatic API access, recursive child tracking, and both eBPF backend and GPU observability denet is a lightweight and customizable process monitoring tool built for workflows and benchmarks It adds adaptive sampling, recursive process-tree monitoring, and eBPF-based off-CPU analysis Its dual interface (CLI for interactive use and Python API for scripting) fits both settings On a Snakemake workflow, denet does not distort the default profiling readouts Its joint CPU, memory, and I/O timeseries attribute resource peaks to specific pipeline stages denet is Linux-first macOS support via host APIs is experimental, and eBPF needs a Linux 4.x or newer kernel with CONFIG BPF SYSCALL Availability and Implementation denet is open-source software released under the GPLv3 terms It is implemented in Rust, with Python bindings provided via maturin, and installs from Cargo (cargo install denet) or PyPI (pip install denet) Most functionality does not require administrative privileges, enabling use on cloud platforms, HPC clusters, and standard Linux workstations The eBPF backend requires elevated permissions in general Data and Code Availability denet is available at https://github.com/btraven00/denet and the bioinformatics workflow at https://github.com/imallona/denet example, both under the GPLv3 terms Supplementary material (CLI and Python API usage, extended comparison table, installation notes) describes them in detail Funding This project received no specific funding IM acknowledges the University of Zurich GRC Career Grant 2025 Q1 CG 001 Acknowledgements We thank Mark D.
Paper summary: Tom: We’re looking at denet today. The central thesis here is that there’s a need for a lightweight process monitoring tool specifically built for benchmarking and beyond, where you need real-time resource profiling of running processes without bringing in heavy overhead.
Jane: So the paper claims denet fills that gap by reporting CPU usage, memory separation into RSS and VMS to expose peak usage and swapping events, disk I/O, network activity, and thread usage. Plus it includes recursive child monitoring with adaptive sampling rates.
Lu: The core claim is this adaptive profiling which varies the sampling rate with the elapsed runtime of a monitored process. During the first second, denet samples at a high frequency to resolve startup spikes, then it slows down linearly over nine seconds to keep overhead low for long-running tasks.
Meng: That sounds like a smart way to balance detail and performance on systems that might have both very fast initial setup phases and then steady operation. But does the author mention what this adaptive resolution is tied to? Is it configurable by the user?
Tom: Yeah, they say adaptive profiling is implemented with user-provided resolution, meaning you can actually set a fixed interval in milliseconds for sampling. If you want to be more specific about how often it samples, you can tell denet that.
Jane: And the output format is structured: JSONL. This means each record carries performance metrics and process metadata like the PID and the executed command, which makes it easy to parse later.
Lu: The authors also highlight that they provide both a CLI interface with colorized output for interactive use and a Python API for scripting, which broadens its utility beyond just running simple commands on your terminal.
Lalam: They emphasize that this structured output makes denet suitable for benchmarking data-intensive pipelines in bioinformatics and beyond, because you get concrete performance metrics linked to the specific process they are monitoring.
Meng: So it’s not just a pretty graph; it’s raw, structured data about what happened during the run, which is what I need when I'm trying to debug a complex AI pipeline that runs for hours.
Tom: It means you can query those specific metrics programmatically, which is great for automated analysis of performance regressions across many different runs.
Conclusion: Jane: So looking at this paper, denet is a lightweight command-line tool for process monitoring in benchmarking and beyond by Carrillo, Mallona, and SIB Swiss Institute of Bioinformatics.
Tom: The authors are focused on creating a tool that doesn't impose heavy overhead while providing deep visibility into what’s happening inside running processes.
Lu: The implication is that you can get highly detailed performance data—CPU, memory, I/O—that is structured and time-series based, which lets you connect resource usage peaks directly to the specific stages of your pipeline.
Meng: So for a production engineer like me, it means I get better diagnostic information when something goes wrong because I can pinpoint exactly which part of the workflow is causing the slowdown.
Tom: It’s about moving beyond just knowing that a job is slow and finding out precisely where in the workflow that slowness originated.
Jane: The tool offers both interactive CLI use and a Python API, so you can either run it quickly on your machine or integrate it into a larger automation script.
Lu: And they’ve shown that this combination of adaptive sampling, recursive monitoring, and structured data makes denet a strong contender for workflow diagnostics in data science environments.
Tom: It’s not about replacing existing tools; it's about adding new capabilities—adaptive sampling and detailed process trees—to what you already have.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game