Grounding Deep Learning Framework Testing in Data Flow

arXiv:2610.15808 · cs.SE, cs.LG · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Grounding Deep Learning Framework Testing in Data Flow".

Jane: The gist The FlowMuT method introduces a mutation-based testing technique grounded in data flow,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to recap, the core thesis of "Grounding Deep Learning Framework Testing in Data Flow" is that we need to look at more than just the model structure when testing deep learning frameworks. They point out that existing methods only capture the operator topology, which misses vital information like runtime tensor values and properties >

Jane: And they propose using a Tensor Flow Graph, or TFG, to record this data flow information. This graph captures the operator topology and also details what is carried on each edge, which includes those tensor values and their properties >

Lu: They then use this TFG to guide the testing process. They build mutation candidates by pairing operators with specific sites in the TFG using a new algorithm, and they filter out illegal mutations based on input, internal, and output constraints >

Meng: I’m wondering how they actually manage building that TFG for a seed model efficiently enough to do this iteratively across many test cases >

Tom: They claim their method combines CG-derived operators with TFG-derived ones into a fixed pool of one hundred forty-seven validated mutation operators after deduplication and validation >

Jane: And they use this pool along with LinUCB selection to guide the process, using coverage and impact features to pick the best mutations >

Lu: They found that this approach helps address challenges like achieving data flow diversity, which is something existing methods struggle with because they only consider structural information about the model >

Tom: The results are pretty strong. FlowMuT detected seventy-nine bugs compared to twenty-eight for the strongest baseline, and it achieved a legal mutant rate of ninety-six point seven percent with those data flow constraints >

Conclusion: Jane: So, wrapping up this discussion on "Grounding Deep Learning Framework Testing in Data Flow," we're looking at how this method changes how we test deep learning frameworks by focusing on the actual data movement during execution rather than just the structure of the model >

Tom: Yeah, it’s about moving from a static view to a dynamic view of computation. The authors are showing that incorporating data flow information into testing significantly improves bug detection and diversity metrics compared to older methods >

Lu: What this means for the field is that we can start building testing methods that are truly grounded in how deep learning frameworks actually run, not just how they look on paper >

Meng: From a deployment standpoint, if you can test based on runtime data flow constraints, it might mean fewer surprises when models move from training to actual inference at scale >

Tom: It’s about making the testing process more robust by using those data flow signatures to select relevant mutations and ensure we’re exercising new behavior in the model >

Jane: So, this paper suggests that for deep learning framework testing, the key is grounding it in data flow information to find bugs that structural methods simply overlook >

CHUNRONG FANG, YANZHOU MU, YINGLONG ZOU, YIBO HU, ZHENYU CHEN

State Key Laboratory for Novel Software Technology, Nanjing University

cs.SE, cs.LG

Submitted: 2026-10-11

Updated: 2026-10-11

Comments: 22 pages, 6 figures, 7 tables

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The FlowMuT method introduces a mutation-based testing technique grounded in data flow, utilizing a Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements

Key concepts

Tensor Flow Graph (TFG): The TFG is a map of a model's execution. Nodes represent operators, and edges show the tensors flowing between them. It records not just the order of operations but also the runtime values and properties of these tensors, which is essential for understanding how data moves through the network during testing.
Data Flow Representation: This concept focuses on tracking how data moves between different parts of a deep learning model. By capturing tensor states and operator inputs, it allows the method to understand the dependencies within the framework execution, making it possible to construct meaningful mutations that test specific data pathways.
Mutation Operator Pool: This pool is a collection of operators used for testing. It combines standard operators with new ones derived directly from the TFG structure. This pool is validated and fixed, ensuring that every mutation candidate has been checked against constraints related to the model's inputs, internal state, and outputs.
LinUCB Selection: LinUCB is an algorithm used to guide which mutation to select next. It uses features like coverage and impact scores derived from the TFG to intelligently choose the most informative mutation candidate. This guided selection leads to higher test case diversity and better framework coverage.

Terminology

Grounding — 1× in this paper · explained in 3 episodes

Summary

The gist The FlowMuT method introduces a mutation-based testing technique grounded in data flow, utilizing a Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements to detect bugs missed by existing methods.

Data Flow Representation

Data flow is crucial for deep learning framework execution aspects like automatic differentiation and operator-fusion optimization because it encompasses the tensors passed between operators and their runtime values and properties; The Tensor Flow Graph (TFG) is introduced to capture the runtime tensor states of model test cases and their operators’ input requirements, where its nodes represent operators and its edges represent tensors flowing between them; The TFG is defined as TFG = (G, F, R), where G records the operator topology of the profiled execution; The method follows a four-stage workflow: Profile & build TFG, Generate candidates, Select mutation, Execute & learn. In each round, the process involves building the TFG for the current seed model, constructing mutation candidates by pairing operators with TFG sites using a novel and efficient algorithm, filtering illegal candidates using input, internal, and output constraints.

Mutation Operator Pool

The operator pool combines CG-derived operators adapted from prior work with TFG-derived operators constructed from TFG objects and mutation primitives. This pool is built by merging CG-derived and TFG-derived operators, resulting in a fixed pool M of 147 validated FlowMuT mutation operators after deduplication and validation. The TFG records operator topology, while the edge annotation F(e) describes the tensor carried by edge e, including tensor values and properties.

Empirical Evaluation

FlowMuT was evaluated on PyTorch and MindSpore using 14 representative models from ten application scenarios. The results show that FlowMuT detects 79 bugs, compared with 28 for the strongest baseline. It achieves a legal mutant rate of 96.7% with data flow constraints, compared with 86.4% without constraints. FlowMuT leads every test case diversity and framework-coverage metric.

Research Questions Addressed

The study investigates four research questions concerning bug detection, diversity and coverage, mutation legality, and mutation selection. The results confirm that FlowMuT detects 79 bugs compared to 28 for the strongest baseline, achieves a legal-mutant rate of 96.7%, and leads in diversity and coverage metrics with a DFSD of 43.4% higher than the strongest baseline. Furthermore, LinUCB selection guides the process effectively, leading to the highest test case diversity and framework coverage.

Threats to Validity

The researchers acknowledge threats to validity including internal validity from implementation differences and stochastic execution, construct validity regarding data flow signatures not representing source coverage, and external validity concerning the scope of frameworks and data flow patterns tested. They mitigate these threats by reporting diversity, coverage, and detected bugs separately, and by recording developer confirmation and fix status for reported bugs.

The FlowMuT method proposes a model-level DL framework testing method grounded in data flow. The TFG records operator topology, runtime tensor values and properties, and operator input requirements. This shared description supports mutation construction, legality checking, and selection with LinUCB using coverage and impact features. Execution feedback guides subsequent selection, and each accepted mutant provides the TFG for the next iteration. FlowMuT submitted 55 bug reports for cases missed by the baselines, improved data flow diversity by 43.4% over the strongest baseline, and achieved a legal-mutant rate of 96.7%. The implementation and all experimental results, including configurations, mutants, triggering test cases, and bug reports are available in the accompanying artifact.

Improvements for AI systems

Bold header: Data Flow Representation via Tensor Flow Graph (TFG)

This representation captures the runtime tensor states of model test cases and their operators’ input requirements by recording runtime values and properties of its tensor, shape, layout, device, memory layout, aliasing information, numerical properties, and gradient state on edges. This allows testing to distinguish between models with similar topologies by capturing the dynamic data flow behavior that existing operator topology representations omit.

Bold header: Mutation Operator Pool Construction

The system constructs a comprehensive pool of 147 mutation operators by combining CG-derived and TFG-derived operators, ensuring coverage of both structural changes and data flow-aware modifications. This pool is validated through positive and negative checks to produce the fixed pool M of 147 validated FlowMuT mutation operators, enhancing the reliability of generated test cases.

Bold header: Contextual Bandit Selection via LinUCB

FlowMuT utilizes LinUCB to select a candidate, where the reward measures the novelty of the resulting framework behavior based on observed events and transitions, and contextual uncertainty is derived from features like site context, tensor state, mutation operator coverage, downstream impact, and gradient influence. This guides selection by using feedback such that each accepted mutant becomes the next seed model, maximizing exploration of novel data flow behaviors.

Bold header: Improved Bug Detection Efficacy

The FlowMuT method is shown to be highly effective at bug detection, reporting 55 bug reports for cases missed by all baseline methods and achieving a legal mutant rate of 96.7%, compared with 86.4% without constraints. This capability allows the AI system to uncover framework bugs related to automatic differentiation and operator-fusion optimization that are missed by traditional topology-based testing.

Bold header: Enhanced Test Case Diversity and Coverage

The integration of data flow guidance leads to superior test case generation, achieving 43.4% higher data flow diversity than the strongest baseline in DFSD metrics. This improvement is quantified by leading metrics like LIC, LPC, and LSC at round 250, ensuring the generated tests exercise a broader range of model states and operator interactions.