Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing

arXiv:2608.11886 · cs.SE · Submitted 2026-08-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing".

Jane: Here's the gist of the paper. Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles and may miss defects that behave consistently across hardware backends.

Tom: XAMT constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries.

Jane: XAMT Methodology XAMT constructs candidate correspondences using explicit API aliases and parameter-role normalization to align library-specific interfaces through shared logical inputs.

Tom: This process involves several steps: 1.

Paper summary: Tom: So to wrap up, this paper "Testing Deep Learning Library APIs via Cross-Framework Differential Fuzzing" is proposing XAMT as a way to test deep learning libraries by comparing operations across seven different frameworks >

Jane: It claims XAMT constructs candidate correspondences using explicit API aliases and parameter role normalization, and then validates these through pairwise execution and group level behavioral checks on canonical ordinary inputs >

Lu: They use variance guided differential fuzzing over ordinary, boundary, and non-finite inputs to explore the differences between APIs intended to implement the same operation across all seven libraries >

Meng: The results show that this approach identifies seventy-two distinct discrepancy cases, including four abnormal process termination cases and sixty-eight output inconsistencies >

Lalam: They confirmed twenty-five of those reports, including twenty-three that have been fixed, and crucially, these cross library comparisons exposed API level discrepancies that provide no CPU to GPU disagreement signal under the evaluated fixed inputs and configurations >

Tom: It means that testing deep learning libraries doesn't always need hardware specific checks; it can rely on finding bugs in how the APIs themselves are implemented across different software environments >

Jane: That’s a big shift for reliability engineering. It moves the focus from just checking if things run fast on one chip to ensuring the logic is consistent everywhere, which is what this work points toward >

Conclusion: Tom: So we're wrapping up this look at XAMT, which is this new way to test deep learning library APIs by comparing operations across seven different frameworks, and it really focuses on finding bugs where the libraries disagree even if they look similar on the surface.

Jane: It sounds like you’re saying this whole method is about checking if the same math actually produces the same results when you run it in different software environments, not just whether a single piece of code crashes?

Tom: Exactly. The paper shows they build these candidate groups by aligning the APIs using explicit aliases and parameter role normalization, then they run execution validation on those pairs to see if they produce consistent outputs.

Lu: What I find interesting is how XAMT uses variance-guided differential fuzzing over ordinary, boundary, and non-finite inputs to specifically hunt for those subtle behavioral differences that a simple comparison might miss.

Meng: From my side, the real win here isn't just finding a bug in one library; it’s seeing how these cross-library comparisons expose API level discrepancies that don't show up as CPU versus GPU disagreement signal under fixed inputs.

Lalam: I can confirm that they constructed six hundred seventy-six execution-validated groups involving two thousand five hundred sixty-three matched APIs across all seven libraries, and this process retained seventy two distinct discrepancy cases including four crash cases and sixty eight output inconsistencies.

Jane: Seventy two distinct cases, including crashes and inconsistencies—that’s a solid number to have when you’re trying to ensure reliability in these complex systems. So what does that mean for the people who actually use these libraries every day?

Tom: It means that when we test an operation like matrix multiplication or a specific tensor function, we don't need to just trust the hardware-specific testing; we can trust the cross-framework comparison to flag inconsistencies in the underlying API implementation itself.

Jane: If this holds up, it shifts our focus from just hardware acceleration performance to making sure the logic is sound across different software stacks before it ever hits production. Next time, we need to look at how researchers are actually using these findings in their actual training pipelines.

Bin Duan, Ruican Dong, Naipeng Dong, Dan Dongseong Kim, Guowei Yang

School of Electrical Engineering and Computer Science, The University of Queensland

cs.SE

Submitted: 2026-08-12

Updated: 2026-10-29

Code: https://github.com/beanduan22/SupXamt

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles…

Terminology

Summary

Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles and may miss defects that behave consistently across hardware backends. p. 0

Where the paper says it

au, guowei. au Abstract—Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU–GPU differential oracles and may miss defects that behave consistently across hardware backends.

XAMT constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries. p. 0
Where the paper says it

We present XAMT, a cross-framework differential fuzzing approach for deep learning library APIs. XAMTconstructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries.

XAMT Methodology

XAMT constructs candidate correspondences using explicit API aliases and parameter-role normalization to align library-specific interfaces through shared logical inputs. This process involves several steps:

  1. Collecting callable APIs from the configured namespaces of the seven target libraries and normalizing their operation names and parameter roles. p. 1

    Where the paper says it

    1 presents an overview of XAMT. First, XAMT collects callable APIs from the configured namespaces of the seven target libraries and normalizes their operation names and parameter roles.

  2. Aligning input parameters by mapping framework-specific parameter names into common semantic roles. p. 1

    Where the paper says it

    XAMT therefore normalizes framework-specific parameter names into common semantic roles. The mapping contains 111 parameter-name aliases covering 24 roles.

  3. Validating each supported cross-library API pair through actual execution on canonical-operation-specific ordinary inputs. p. 1

    Where the paper says it

    APIs with the same normalized operation name are placed into candidate groups, while coarse namespacederived categories are used only to exclude obvious semantic mismatches. Second, XAMT aligns their input parameters and validates each supported cross-library API pair through actual execution on canonical-operation-specific ordinary inputs.

  4. Applying variance-guided differential fuzzing over ordinary, boundary, and non-finite inputs. p. 0

    Where the paper says it

    It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs.

API Group Construction and Validation

XAMT constructs candidate groups based on normalized operation names and coarse operation categories to reject obvious semantic mismatches. After pairwise validation, APIs are represented as nodes and PASS pairs as undirected edges, which are then processed using union–find to retain connected components that contain APIs from at least two target libraries. p. 2

Where the paper says it

4) Constructing Executable API Groups: After pairwise validation, XAMT represents each API as a node and each PASS pair as an undirected edge. It applies union–find to the PASS edges and retains connected components that contain APIs from at least two target libraries.

A group is retained only if all members execute successfully and produce mutually consistent outputs. p. 3
Where the paper says it

XAMT therefore reruns all APIs in each component on the same canonical validation input used during pairwise validation. A component is retained as an execution-validated API group only when all members execute successfully and produce mutually consistent outputs; otherwise, the component is excluded from subsequent fuzzing.

Fuzzing Strategy and Oracles

XAMT applies variance-guided differential fuzzing to explore behavioral differences among execution-validated APIs intended to implement the same operation. Input seeds are generated using a unified representation of Tensor, Value Scalar, Index Scalar, Shape, and Boolean Flag inputs. p. 3

Where the paper says it

1) Input Seed Generation: XAMT generates input seeds using a unified parameter representation: • Tensor: Synthetic tensors are generated as multidimensional arrays with valid ranks and shapes and are converted into each library’s native tensor representation. • Value Scalar: Floating-point and integer values are sampled from ranges supported by the corresponding operation.

Two oracles flag abnormal termination (Crash) or inconsistent outputs (Inconsistency) during differential execution. p. 3
Where the paper says it

Oracles During differential execution, XAMT applies two oracles to flag individual executions that exhibit abnormal termination or inconsistent outputs. 1) Crash: The crash oracle detects unexpected crashes during API execution, including aborts, segmentation faults, assertion failures, and memory violations.

Evaluation Results and Effectiveness

XAMT constructs 676 execution-validated groups containing 2,563 matched APIs across seven libraries. The testing campaign identified 72 independently reproduced discrepancy cases, including four Crash cases and 68 output inconsistencies. p. 0

Where the paper says it

Across these seven libraries, XAMT constructs 676 execution-validated crosslibrary groups spanning 74 distinct library combinations and containing 2,563 matched APIs. The testing campaign identifies 72 independently reproduced discrepancy cases, including four Crash cases and 68 output inconsistencies.

Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed. p. 0
Where the paper says it

Among these, XAMT identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies. Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed.

The results show that the same triggering inputs provide no CPU–GPU disagreement signal under the evaluated configurations. p. 0
Where the paper says it

Moreover, among the 50 CPU–GPUapplicable cases in the backend-replay subset, none produces an observable backend disagreement. These results show that the same triggering inputs provide no CPU–GPU disagreement signal under the evaluated configurations.

Complementarity and Sensitivity Analysis

The analysis shows complementarity at three levels: fixed-input backend replay yields no observable backend disagreement, method-level rediscovery finds 2 of 15 applicable defects using existing baselines, and TensorScope covers only 9.7% of the 72 cases fully represented in its scope. p. 7

Where the paper says it

As shown in Table X, only 7 of the 72 cases (9. 7%) are fully represented in TensorScope’s converter-derived correspondence scope.

Execution validation reduces the density of raw discrepancy candidates from 25. p. 9
Where the paper says it

Answer to RQ5. Execution validation reduces the density of raw discrepancy candidates from 25.

4 to 16.1 per 100 tested groups. p. 8
Where the paper says it

As shown in Table XI, the full configuration produces 109 raw discrepancy candidates, corresponding to 16. 1 candidates per 100 tested groups.

The analysis also shows that extended inputs primarily improve the exploration of boundary and non-finite behaviors. p. 8
Where the paper says it

Among the 22 cases not reached, 20 require boundary or non-finite triggering values, while the remaining two are not reached within the testing budget. These results show that extended inputs primarily improve the exploration of boundary and non-finite behaviors.

Conclusion

XAMT constructs candidate API correspondences through explicit API aliases and parameter-role normalization, retains supported correspondences after pairwise and group-level execution validation, and applies variance-guided fuzzing over ordinary, boundary, and non-finite inputs. Across seven libraries, XAMT constructs 676 execution-validated groups containing 2,563 API members. p. 0

Where the paper says it

Crash and inconsistency oracles flag executions exhibiting abnormal termination or inconsistent outputs for subsequent reproduction and analysis. Across the seven libraries, XAMT constructs 676 execution-validated groups containing 2,563 matched APIs.

The evaluation retains 72 distinct discrepancy cases, including four involving abnormal process termination and 68 involving output inconsistencies. p. 6
Where the paper says it

1) Discrepancy Categories: We classify the independently reproduced discrepancy cases by their observed behavior: abnormal process termination or output inconsistency. As shown in Table VII, four cases trigger abnormal process termination and 68 involve output inconsistencies, including differences in output structures, shapes, non-finite masks, infinity signs, or finite values.

Among the 72 submitted reports, 25 have been confirmed, including 23 that have been fixed. p. 7
Where the paper says it

XAMT identifies 72 validated discrepancy cases, including four Crash cases and 68 output inconsistencies. Among the 72 submitted reports, 25 have been confirmed, including 23 that have been fixed; 41 remain pending and six have other dispositions.

The results demonstrate that cross-library comparison exposes API-level discrepancies that provide no CPU–GPU disagreement signal under the evaluated fixed inputs and configurations. p. 9
Where the paper says it

In RQ4, we replay the fixed XAMT triggering inputs under a CPU– GPU differential oracle. None of the 50 applicable output inconsistencies produces an observable backend disagreement, showing that these cross-library discrepancies are not exposed by CPU–GPU comparison under the evaluated configurations.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Cross-framework API comparison allows for detection of defects that behave consistently across hardware backends, providing a signal when an implementation produces the same divergent result on both CPU and GPU.

  2. The system can identify 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies, which are then used to generate developer reports for fixes.

  3. XAMT can construct execution-validated API groups across seven libraries, such as PyTorch, TensorFlow, Keras, JAX, MindSpore, PaddlePaddle, and Chainer to compare equivalent operations.

  4. The system can perform method-level rediscovery by running existing baselines like FreeFuzz and DeepREL on affected APIs to see if they expose the same underlying defects.

  5. The system can identify which API pairs are fully represented in TensorScope's converter-derived correspondence scope, helping researchers understand the limitations of automated correspondence mapping.

Abstract

Deep learning libraries underpin many safety- and reliability-critical applications, yet existing API-level testing techniques often rely on intra-library properties or CPU--GPU differential oracles and may miss defects that behave consistently across hardware backends. We present Xamt, a cross-framework differential fuzzing approach for deep learning library APIs. Xamt constructs and tests execution-validated groups of APIs intended to implement equivalent operations across seven libraries. It uses explicit API aliases and parameter-role normalization to construct candidate correspondences and validates them through pairwise execution and a group-level behavioral check on canonical ordinary inputs. The resulting groups are explored using variance-guided differential fuzzing with ordinary, boundary, and non-finite inputs. Crash and inconsistency oracles flag executions exhibiting abnormal termination or inconsistent outputs for subsequent reproduction and analysis. Across the seven libraries, Xamt constructs 676 execution-validated groups containing 2,563 matched APIs. Among these, Xamt identifies 72 independently reproduced discrepancy cases, including 4 crash cases and 68 output inconsistencies. Among the 72 developer reports, 25 have been confirmed, including 23 that have been fixed.

Sources

Related papers