Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing".
Jane: Here's the gist of the paper. Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles and may miss defects that behave consistently across hardware backends.
Tom: XAMT constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries.
Jane: XAMT Methodology XAMT constructs candidate correspondences using explicit API aliases and parameter-role normalization to align library-specific interfaces through shared logical inputs.
Tom: This process involves several steps: 1.
Paper summary: Tom: So to wrap up, this paper "Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing" is proposing XAMT as a way to test deep learning libraries by comparing operations across seven different frameworks >
Jane: It claims XAMT constructs candidate correspondences using explicit API aliases and parameter role normalization, and then validates these through pairwise execution and group level behavioral checks on canonical ordinary inputs >
Lu: They use variance guided differential fuzzing over ordinary, boundary, and non-finite inputs to explore the differences between APIs intended to implement the same operation across all seven libraries >
Meng: The results show that this approach identifies seventy-two distinct discrepancy cases, including four abnormal process termination cases and sixty-eight output inconsistencies >
Lalam: They confirmed twenty-five of those reports, including twenty-three that have been fixed, and crucially, these cross library comparisons exposed API level discrepancies that provide no CPU to GPU disagreement signal under the evaluated fixed inputs and configurations >
Tom: It means that testing deep learning libraries doesn't always need hardware specific checks; it can rely on finding bugs in how the APIs themselves are implemented across different software environments >
Jane: That’s a big shift for reliability engineering. It moves the focus from just checking if things run fast on one chip to ensuring the logic is consistent everywhere, which is what this work points toward >
Conclusion: Tom: So we're wrapping up this look at XAMT, which is this new way to test deep learning library APIs by comparing operations across seven different frameworks, and it really focuses on finding bugs where the libraries disagree even if they look similar on the surface.
Jane: It sounds like you’re saying this whole method is about checking if the same math actually produces the same results when you run it in different software environments, not just whether a single piece of code crashes?
Tom: Exactly. The paper shows they build these candidate groups by aligning the APIs using explicit aliases and parameter role normalization, then they run execution validation on those pairs to see if they produce consistent outputs.
Lu: What I find interesting is how XAMT uses variance-guided differential fuzzing over ordinary, boundary, and non-finite inputs to specifically hunt for those subtle behavioral differences that a simple comparison might miss.
Meng: From my side, the real win here isn't just finding a bug in one library; it’s seeing how these cross-library comparisons expose API level discrepancies that don't show up as CPU versus GPU disagreement signal under fixed inputs.
Lalam: I can confirm that they constructed six hundred seventy-six execution-validated groups involving two thousand five hundred sixty-three matched APIs across all seven libraries, and this process retained seventy two distinct discrepancy cases including four crash cases and sixty eight output inconsistencies.
Jane: Seventy two distinct cases, including crashes and inconsistencies—that’s a solid number to have when you’re trying to ensure reliability in these complex systems. So what does that mean for the people who actually use these libraries every day?
Tom: It means that when we test an operation like matrix multiplication or a specific tensor function, we don't need to just trust the hardware-specific testing; we can trust the cross-framework comparison to flag inconsistencies in the underlying API implementation itself.
Jane: If this holds up, it shifts our focus from just hardware acceleration performance to making sure the logic is sound across different software stacks before it ever hits production. Next time, we need to look at how researchers are actually using these findings in their actual training pipelines.
Bin Duan, Ruican Dong, Naipeng Dong, Dan Dongseong Kim, Guowei Yang
School of Electrical Engineering and Computer Science, The University of Queensland
cs.SE
Submitted: 2026-08-12
Updated: 2026-10-29
Code: https://github.com/beanduan22/SupXamt
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles…
Terminology
Summary
Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles and may miss defects that behave consistently across hardware backends. p. 0 au, guowei. au
Abstract—Deep learning libraries underpin many safety- and
reliability-critical applications, yet existing API-level testing
techniques often rely on intra-library properties or CPU–GPU
differential oracles and may miss defects that behave consistently
across hardware backends. We present XAMT, a cross-framework
differential fuzzing approach for deep learning library APIs. XAMTconstructs and tests execution-validated groups of APIs
intended to implement equivalent operations across seven libraries.Where the paper says it
Where the paper says it
XAMT Methodology
XAMT constructs candidate correspondences using explicit API aliases and parameter-role normalization to align library-specific interfaces through shared logical inputs. This process involves several steps:
-
Collecting callable APIs from the configured namespaces of the seven target libraries and normalizing their operation names and parameter roles. p. 1
Where the paper says it
1 presents an overview of XAMT. First, XAMT collects callable APIs from the configured namespaces of the seven target libraries and normalizes their operation names and parameter roles.
-
Aligning input parameters by mapping framework-specific parameter names into common semantic roles. p. 1
Where the paper says it
XAMT therefore normalizes framework-specific parameter names into common semantic roles. The mapping contains 111 parameter-name aliases covering 24 roles.
-
Validating each supported cross-library API pair through actual execution on canonical-operation-specific ordinary inputs. p. 1
Where the paper says it
APIs with the same normalized operation name are placed into candidate groups, while coarse namespacederived categories are used only to exclude obvious semantic mismatches. Second, XAMT aligns their input parameters and validates each supported cross-library API pair through actual execution on canonical-operation-specific ordinary inputs.
-
Applying variance-guided differential fuzzing over ordinary, boundary, and non-finite inputs. p. 0
Where the paper says it
It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs.
API Group Construction and Validation
XAMT constructs candidate groups based on normalized operation names and coarse operation categories to reject obvious semantic mismatches. After pairwise validation, APIs are represented as nodes and PASS pairs as undirected edges, which are then processed using union–find to retain connected components that contain APIs from at least two target libraries. p. 2 4) Constructing Executable API Groups: After pairwise
validation, XAMT represents each API as a node and each
PASS pair as an undirected edge. It applies union–find to the
PASS edges and retains connected components that contain
APIs from at least two target libraries. XAMT therefore reruns all APIs in each component
on the same canonical validation input used during pairwise
validation. A component is retained as an execution-validated
API group only when all members execute successfully and
produce mutually consistent outputs; otherwise, the component
is excluded from subsequent fuzzing.Where the paper says it
Where the paper says it
Fuzzing Strategy and Oracles
XAMT applies variance-guided differential fuzzing to explore behavioral differences among execution-validated APIs intended to implement the same operation. Input seeds are generated using a unified representation of Tensor, Value Scalar, Index Scalar, Shape, and Boolean Flag inputs. p. 3 1) Input Seed Generation: XAMT generates input seeds
using a unified parameter representation:
• Tensor: Synthetic tensors are generated as multidimensional
arrays with valid ranks and shapes and are converted into
each library’s native tensor representation. • Value Scalar: Floating-point and integer values are sampled
from ranges supported by the corresponding operation. Oracles
During differential execution, XAMT applies two oracles to
flag individual executions that exhibit abnormal termination or
inconsistent outputs. 1) Crash: The crash oracle detects unexpected crashes
during API execution, including aborts, segmentation faults,
assertion failures, and memory violations.Where the paper says it
Where the paper says it
Evaluation Results and Effectiveness
XAMT constructs 676 execution-validated groups containing 2,563 matched APIs across seven libraries. The testing campaign identified 72 independently reproduced discrepancy cases, including four Crash cases and 68 output inconsistencies. p. 0 Across these seven
libraries, XAMT constructs 676 execution-validated crosslibrary groups spanning 74 distinct library combinations and
containing 2,563 matched APIs. The testing campaign identifies 72 independently reproduced discrepancy cases, including
four Crash cases and 68 output inconsistencies. Among these, XAMT identifies 72
independently reproduced discrepancy cases, including 4 crash
cases and 68 output inconsistencies. Among the 72 developer
reports, 25 have been confirmed, including 23 that have been
fixed. Moreover, among the 50 CPU–GPUapplicable cases in the backend-replay subset, none produces
an observable backend disagreement. These results show that
the same triggering inputs provide no CPU–GPU disagreement
signal under the evaluated configurations.Where the paper says it
Where the paper says it
Where the paper says it
Complementarity and Sensitivity Analysis
The analysis shows complementarity at three levels: fixed-input backend replay yields no observable backend disagreement, method-level rediscovery finds 2 of 15 applicable defects using existing baselines, and TensorScope covers only 9.7% of the 72 cases fully represented in its scope. p. 7 As shown in Table X, only 7 of the 72 cases (9. 7%) are
fully represented in TensorScope’s converter-derived correspondence scope. Answer to RQ5. Execution validation reduces the density of
raw discrepancy candidates from 25. As shown in Table XI, the full configuration
produces 109 raw discrepancy candidates, corresponding to
16. 1 candidates per 100 tested groups. Among the 22 cases not reached, 20 require boundary
or non-finite triggering values, while the remaining two are
not reached within the testing budget. These results show that
extended inputs primarily improve the exploration of boundary
and non-finite behaviors.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
XAMT constructs candidate API correspondences through explicit API aliases and parameter-role normalization, retains supported correspondences after pairwise and group-level execution validation, and applies variance-guided fuzzing over ordinary, boundary, and non-finite inputs. Across seven libraries, XAMT constructs 676 execution-validated groups containing 2,563 API members. p. 0 Crash and inconsistency oracles
flag executions exhibiting abnormal termination or inconsistent
outputs for subsequent reproduction and analysis. Across the
seven libraries, XAMT constructs 676 execution-validated groups
containing 2,563 matched APIs. 1) Discrepancy Categories: We classify the independently
reproduced discrepancy cases by their observed behavior: abnormal process termination or output inconsistency. As shown
in Table VII, four cases trigger abnormal process termination
and 68 involve output inconsistencies, including differences in
output structures, shapes, non-finite masks, infinity signs, or
finite values. XAMT identifies 72 validated discrepancy
cases, including four Crash cases and 68 output inconsistencies. Among the 72 submitted reports, 25 have been confirmed,
including 23 that have been fixed; 41 remain pending and six
have other dispositions. In RQ4,
we replay the fixed XAMT triggering inputs under a CPU–
GPU differential oracle. None of the 50 applicable output
inconsistencies produces an observable backend disagreement,
showing that these cross-library discrepancies are not exposed
by CPU–GPU comparison under the evaluated configurations.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Cross-framework API comparison allows for detection of defects that
behave consistently across hardware backends,
providing a signal when an implementation producesthe same divergent result on both CPU and GPU.
-
The system can identify 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies, which are then used to generate developer reports for fixes.
-
XAMT can construct execution-validated API groups across seven libraries, such as PyTorch, TensorFlow, Keras, JAX, MindSpore, PaddlePaddle, and Chainer to compare equivalent operations.
-
The system can perform method-level rediscovery by running existing baselines like FreeFuzz and DeepREL on affected APIs to see if they expose the same underlying defects.
-
The system can identify which API pairs are fully represented in TensorScope's converter-derived correspondence scope, helping researchers understand the limitations of automated correspondence mapping.
Abstract
Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU--GPU differential oracles and may miss defects that behave consistently across hardware backends. We present Xamt, a cross-framework differential fuzzing approach for deep learning library APIs. Xamt constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries. It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs. Crash and inconsistency oracles flag executions exhibiting abnormal termination or inconsistent outputs for subsequent reproduction and analysis. Across the seven libraries, Xamt constructs 676 execution-validated groups containing 2,563 matched APIs. Among these, Xamt identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies. Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed.
Sources
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters