Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring
cs.SE, cs.LG
Submitted: 2026-09-22
Updated: 2026-09-22
Comments: 17 pages; preprint published at Zenodo, DOI 10.5281/zenodo.22832168
Code: https://github.com/phamquiluan/RCAEval
License: http://creativecommons.org/licenses/by/4.0/
The gist: Graph neural networks dominate recent work on microservice root-cause analysis, yet recent results question whether the graph contributes.
Terminology
Abstract
Graph neural networks dominate recent work on microservice root-cause analysis, yet recent results question whether the graph contributes. Those results compare whole pipelines, so when a flat model wins one cannot tell whether structure is useless or redundant. We run the comparison they imply on RCAEval: three learned arms with identical features, optimiser, validation split, early-stopping rule and scoring head, in which a single term separates the graph arms. Across two RCAEval benchmarks, two topology sources and four regimes we find no reliable graph-specific effect: in-distribution the graph model leads the conventional flat model by 0.003 Avg@5 (p = 0.844, n = 6 disjoint folds). Auditing the pipeline surfaced two benchmark properties that condition any result on it. RCAEval injects faults into only five services per system while exposing 12 to 70 in telemetry, and the headline metric is Avg@5: a ranker reading no telemetry at all places the true culprit in the top five on 99.7 percent of held-out incidents, scoring Avg@5 0.488. That prior, not the uniform-random 0.137, is the honest in-distribution floor, and it collapses to 0.192 across systems. The second property is a non-uniform column schema that silently zeroes telemetry for most RE1 cases. We reproduce a published baseline, BARO, RCAEval's own reference implementation; on the one system with a clean schema it reaches similar aggregate accuracy to our heuristic, within 0.004, under a different scoring rule. The audit motivated a new model. PSC-GRCA separates a candidate score into a system prior, telemetry evidence and a centred graph residual, and reaches mean Avg@5 0.915 against 0.864 for the flat baseline, while its ablations locate most of the gain in the prior term rather than the graph. We close with a twelve-item checklist for graph-versus-flat ablation studies, distilled from sixty-two recorded defects.
Sources
- Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?
- Graph-Free Root Cause Analysis
- Rethinking the Evaluation of Microservice RCA with a Fault Propagation-Aware Benchmark
- RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data
- CHASE: A Causal Hypergraph based Framework for Root Cause Analysis in Multimodal Microservice Systems
- LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis
- Accounting for Variance in Machine Learning Benchmarks
- Deep Reinforcement Learning that Matters
- Deep Reinforcement Learning at the Edge of the Statistical Precipice
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties