Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing
summary
The gist
Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles…
This episode discusses
- Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing · Paper Radio
- Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT
The paper
Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing · Read on arXiv
Bin Duan, Ruican Dong, Naipeng Dong, Dan Dongseong Kim, Guowei Yang
School of Electrical Engineering and Computer Science, The University of Queensland
Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU--GPU differential oracles and may miss defects that behave consistently across hardware backends. We present Xamt, a cross-framework differential fuzzing approach for deep learning library APIs. Xamt constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries. It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs. Crash and inconsistency oracles flag executions exhibiting abnormal termination or inconsistent outputs for subsequent reproduction and analysis. Across the seven libraries, Xamt constructs 676 execution-validated groups containing 2,563 matched APIs. Among these, Xamt identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies. Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing".
Jane: Here's the gist of the paper. Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles and may miss defects that behave consistently across hardware backends.
Tom: XAMT constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries.
Jane: XAMT Methodology XAMT constructs candidate correspondences using explicit API aliases and parameter-role normalization to align library-specific interfaces through shared logical inputs.
Tom: This process involves several steps: 1.
Paper summary: Tom: So to wrap up, this paper "Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing" is proposing XAMT as a way to test deep learning libraries by comparing operations across seven different frameworks >
Jane: It claims XAMT constructs candidate correspondences using explicit API aliases and parameter role normalization, and then validates these through pairwise execution and group level behavioral checks on canonical ordinary inputs >
Lu: They use variance guided differential fuzzing over ordinary, boundary, and non-finite inputs to explore the differences between APIs intended to implement the same operation across all seven libraries >
Meng: The results show that this approach identifies seventy-two distinct discrepancy cases, including four abnormal process termination cases and sixty-eight output inconsistencies >
Lalam: They confirmed twenty-five of those reports, including twenty-three that have been fixed, and crucially, these cross library comparisons exposed API level discrepancies that provide no CPU to GPU disagreement signal under the evaluated fixed inputs and configurations >
Tom: It means that testing deep learning libraries doesn't always need hardware specific checks; it can rely on finding bugs in how the APIs themselves are implemented across different software environments >
Jane: That’s a big shift for reliability engineering. It moves the focus from just checking if things run fast on one chip to ensuring the logic is consistent everywhere, which is what this work points toward >
Conclusion: Tom: So we're wrapping up this look at XAMT, which is this new way to test deep learning library APIs by comparing operations across seven different frameworks, and it really focuses on finding bugs where the libraries disagree even if they look similar on the surface.
Jane: It sounds like you’re saying this whole method is about checking if the same math actually produces the same results when you run it in different software environments, not just whether a single piece of code crashes?
Tom: Exactly. The paper shows they build these candidate groups by aligning the APIs using explicit aliases and parameter role normalization, then they run execution validation on those pairs to see if they produce consistent outputs.
Lu: What I find interesting is how XAMT uses variance-guided differential fuzzing over ordinary, boundary, and non-finite inputs to specifically hunt for those subtle behavioral differences that a simple comparison might miss.
Meng: From my side, the real win here isn't just finding a bug in one library; it’s seeing how these cross-library comparisons expose API level discrepancies that don't show up as CPU versus GPU disagreement signal under fixed inputs.
Lalam: I can confirm that they constructed six hundred seventy-six execution-validated groups involving two thousand five hundred sixty-three matched APIs across all seven libraries, and this process retained seventy two distinct discrepancy cases including four crash cases and sixty eight output inconsistencies.
Jane: Seventy two distinct cases, including crashes and inconsistencies—that’s a solid number to have when you’re trying to ensure reliability in these complex systems. So what does that mean for the people who actually use these libraries every day?
Tom: It means that when we test an operation like matrix multiplication or a specific tensor function, we don't need to just trust the hardware-specific testing; we can trust the cross-framework comparison to flag inconsistencies in the underlying API implementation itself.
Jane: If this holds up, it shifts our focus from just hardware acceleration performance to making sure the logic is sound across different software stacks before it ever hits production. Next time, we need to look at how researchers are actually using these findings in their actual training pipelines.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game