PhotonZip: Lifecycle-Aware Compression for HDF5-Based Synchrotron Workflows
cs.DC
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: 12 pages, 5 figures, 3 tables, accepted by IFIP NPC 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Synchrotron detectors generate data at rates that challenge storage and I/O systems.
Abstract
Synchrotron detectors generate data at rates that challenge storage and I/O systems. Compression can reduce data movement, but codec-only measurements do not capture the end-to-end behavior of HDF5 workflows. We present PhotonZip, a lifecycle-aware compression framework that tests candidate modes through a common HDF5 path and selects among eligible measured configurations according to either compression ratio or round-trip throughput. Its modular filters preserve standard HDF5 access when the required plugin is available. Across four beamline-derived datasets, PhotonZip reaches lossless compression ratios up to 1.629 times and HDF5 write throughput up to 3.40GB/s write and 7.75GB/s read. The throughput-oriented configuration achieves 1.529 times lossless data reduction while improving HDF5 write and read throughput by 21.2% and 15.0%, respectively. Lossy preview modes provide up to 16 times data reduction and reduce preview latency by up to 8.35 s in the evaluated file-per-rank HDF5 configuration.
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends