TATG: Tracking-Aware Testing Objective for LLM-based Test Generation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "TATG: Tracking-Aware Testing Objective for LLM-based Test Generation".
Jane: Here's the gist of the paper. Complex Java methods remain challenging for automated unit test generation because achieving strong coverage and fault detection often requires satisfying branch-specific testing requirements that are not directly visible from a focal method signature Recent LLM-based frameworks, such as KTester, PANTA, and MUTGEN, leverage project context, static analysis, coverage feedback, or mutation guidance to improve test generation <ref:2607.03194#pg6> However, these approaches do not explicitly represent and track individual testing requirements across iterations <ref:2607.03194#pg6> As a result, generation and repair efforts may repeatedly target already-satisfied requirements while overlooking unresolved branches and weak oracles <ref:2607.03194#pg6> TATG introduces a unified objective representation that captures testing requirements derived from both static program analysis and dynamic feedback <ref:2607.03194#pg6> Pruning redundant reasoning tokens TATG presents a tracking-aware LLM-based unit test generation framework that unifies static program analysis and dynamic evaluation feedback through a unified objective representation This representation captures testing requirements derived from both static program analysis and dynamic feedback Each objective records its evidence, setup, action, oracle, acceptance criterion, status, and de-duplication key <ref:2607.03194#pg6> Building on this representation, TATG employs a two-stage workflow: structural rounds first improve coverage <ref:2607.03194#pg6> while hardening rounds subsequently strengthen assertions using mutation feedback to improve fault-detection capability.
Tom: How the training signal is built TATG's objective state G is a set of objectives where each objective G is represented as a tuple: G = ⟨id, src, cat, E, S, A, Q, α, status, key⟩ <ref:2607.03194#pg6> The field src records where the objective comes from in categories such as static or mutation <ref:2607.03194#pg6> Feedback-driven objectives are generated incrementally throughout iterative generation through compilation/execution diagnostics, coverage reports, and mutation testing feedback The objective state is updated after each round to record which objectives have been satisfied or remain open.
Jane: Where the gains come from TATG's two-stage generation strategy coordinates structural coverage improvement and mutation-based oracle strengthening within a unified generation process The structural stage focuses on producing executable tests and improving reachability using static objectives, while the hardening stage uses mutation feedback to strengthen assertions for code that is already reachable.
Tom: The structural stage prioritizes executable tests and hard-to-reach paths, and the hardening stage prioritizes strengthening oracles for code that is already reachable.
Paper summary: Tom: Looking back at "TATG: Tracking-Aware Testing Objective for LLM-based Test Generation" by Guancheng Wang, Qinghua Xu, and Lionel C. Briand, the core argument is that explicit tracking of testing requirements makes iterative guidance for AI test generation much more effective.
Jane: They developed a unified objective representation that captures both static analysis findings and dynamic execution feedback, which is then used in a two-stage workflow to improve coverage first then fault detection.
Lu: The implication is that instead of relying on vague prompts or coarse project knowledge, you can provide the AI with precise objectives like "exercise the branch where currToken equals Token string," which significantly improves how much code gets covered and how good those tests are at catching errors.
Meng: It sounds like this moves us away from just generating a single test suite and toward building a systematic process that refines the tests round by round based on what failed or succeeded in the last step.
Lalam: The paper shows that TATG consistently achieves higher structural coverage and mutation score than existing baselines, and for fifty methods, it matched industrial tool performance while still outperforming them in line coverage and mutation score.
Tom: It really boils down to using feedback to guide generation in a way that addresses the specific needs of the code path you're currently working on, rather than just hoping the AI guesses what’s important next.
Jane: So, TATG provides a framework where you can systematically track whether each specific requirement—like an exception translation path—has been met or left open across all iterations of test generation.
Conclusion: Tom: So we're wrapping up this look at TATG, which is about this new framework for making AI test generation better by tracking its steps instead of just guessing.
Jane: Right, so you mentioned that the authors are Wang, Xu, and Briand. What’s the main thing to get from that title?
Tom: It means they’re tracking objectives across iterations in this LLM testing setup. They built a way to record where every testing requirement comes from—whether it's static code analysis or dynamic execution feedback.
Lu: Exactly. This unified objective representation, it keeps track of the evidence, the setup, the action taken, the oracle they used for checking success, and even a status marker for each requirement.
Meng: I’m thinking about deployment here. If you're running this on real traffic at scale with that level of tracking in place—how does that affect runtime? Is it adding too much overhead?
Jane: That’s a big question, Meng. But the paper shows they use a two-stage process: first, structural rounds to get basic code coverage up using static data, and then hardening rounds using mutation feedback to fix those weak spots.
Tom: And that’s where the results really show up. They consistently beat baselines like KTester and PANTA by improving line coverage, branch coverage, and mutation score by about twenty points on average.
Lu: It’s a strong performance lift, especially because the structural stage is shown to be critical; without it, you only get about twenty-seven percent line coverage.
Jane: So what does this mean for someone who just wants to build better tests? It suggests that moving from vague hints to explicit tracking of needs makes the whole process much more reliable.
Tom: It means the AI isn't just randomly generating code; it's following a documented plan for hitting every single required test condition.
Meng: I still worry about the fidelity of those dynamic feedback loops—the compilation and execution diagnostics—but if that tracking is accurate, it’s a solid way to ensure you don't miss those edge cases that only show up during runtime.
Lu: The authors explicitly state their evaluation was fair, using the same model version and configuration across all tests to reduce the risk from just one specific setup.
Jane: So it seems TATG is pushing us toward a more structured way of teaching AI how to test complex software, moving beyond simple prompting to something much more goal-oriented.
Guancheng Wang, Qinghua Xu, Lionel C. Briand
Lero Centre for Software, University of Limerick · School of Electrical Engineering and Computer Science, University of Ottawa
cs.SE
Submitted: 2026-07-03
Updated: 2026-11-06
Code: https://github.com/paohaijiao/jquick-curl
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Complex Java methods remain challenging for automated unit test generation because achieving strong coverage and fault detection often requires satisfying branch-specific testing requirements that…
Key concepts
- Unified Objective Representation
- This is the core idea of TATG. It's a structured way to record every testing need derived from both static code analysis (like identifying possible branches) and dynamic feedback (like compilation errors or coverage reports). Each objective tracks its source, setup, action, oracle, and status across generations.
- Structural Rounds
- The first stage of TATG focuses on improving test reachability and overall code coverage. It uses static objectives to prioritize generating executable tests and exploring hard-to-reach paths in the code. This ensures the generated tests cover a broad range of execution paths before focusing on assertion strength.
- Hardening Rounds
- The second stage focuses on strengthening assertions using feedback from mutation testing. After structural coverage is established, this stage targets already reachable code to improve the quality of tests by making them more robust against small changes, thereby enhancing fault detection capabilities.
Terminology
Summary
Complex Java methods remain challenging for automated unit test generation because achieving strong coverage and fault detection often requires satisfying branch-specific testing requirements that are not directly visible from a focal method signature Recent LLM-based frameworks, such as KTester, PANTA, and MUTGEN, leverage project context, static analysis, coverage feedback, or mutation guidance to improve test generation <ref:2607. p. 0 briand@lero. p. 0 TATG: Tracking-Aware Testing Objective for
LLM-based Test Generation
Guancheng Wang∗
, Qinghua Xu∗
, Lionel C. Briand∗†
∗ Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland
† School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
guancheng. Recent LLM-based frameworks, such as
KTester, PANTA, and MUTGEN, leverage project context, static
analysis, coverage feedback, or mutation guidance to improve test
generation. However, these approaches do not explicitly represent
and track individual testing requirements across iterations. However, these approaches do not explicitly represent
and track individual testing requirements across iterations. As
a result, generation and repair efforts may repeatedly target already-satisfied requirements while overlooking unresolved
branches and weak oracles. We present TATG, a tracking-aware LLM-based unit test
generation. TATG introduces a unified objective representation
that captures testing requirements derived from both static
program analysis and dynamic feedback.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
TATG presents a tracking-aware LLM-based unit test generation framework that unifies static program analysis and dynamic evaluation feedback through a unified objective representation This representation captures testing requirements derived from both static program analysis and dynamic feedback Each objective records its evidence, setup, action, oracle, acceptance criterion, status, and de-duplication key <ref:2607.03194#pg6> Building on this representation, TATG employs a two-stage workflow: structural rounds first improve coverage <ref:2607. p. 0 Each objective records
its evidence, setup, action, oracle, acceptance criterion, status,
and de-duplication key, enabling fine-grained tracking of which
requirements have been satisfied and which remain unresolved
throughout generation. Building on this representation, TATG
employs a two-stage workflow: structural rounds first improve
coverage, while hardening rounds subsequently strengthen assertions using mutation feedback to improve fault-detection
capability. Each objective records
its evidence, setup, action, oracle, acceptance criterion, status,
and de-duplication key, enabling fine-grained tracking of which
requirements have been satisfied and which remain unresolved
throughout generation. Building on this representation, TATG
employs a two-stage workflow: structural rounds first improve
coverage, while hardening rounds subsequently strengthen assertions using mutation feedback to improve fault-detection
capability.Where the paper says it
Where the paper says it
How the training signal is built
TATG's objective state G is a set of objectives where each objective G is represented as a tuple: G = ⟨id, src, cat, E, S, A, Q, α, status, key⟩ <ref:2607.03194#pg6> The field src records where the objective comes from in categories such as static or mutation <ref:2607. p. 2 For example, an objective created from a missed
branch may use the source line and branch index, while a
mutation objective may use the mutant identifier reported by
mutation testing. The field src records where the objective comes
from:
src ∈ {static, coverage, validation, oracle, mutation}. Feedback-driven objectives are generated incrementally
throughout iterative generation. Compilation and execution
diagnostics, along with coverage reports, generate ValidationRepair objectives, weak assertions identified in generated
tests generate OracleStrengthening objectives, and mutation
testing generates MutationReachability and MutationAssertion
objectives for uncovered and live mutants, respectively.Where the paper says it
Where the paper says it
Where the gains come from
TATG's two-stage generation strategy coordinates structural coverage improvement and mutation-based oracle strengthening within a unified generation process The structural stage focuses on producing executable tests and improving reachability using static objectives, while the hardening stage uses mutation feedback to strengthen assertions for code that is already reachable. The structural stage prioritizes executable tests and hard-to-reach paths, and the hardening stage prioritizes strengthening oracles for code that is already reachable. p. 2 The structural stage prioritizes executable tests and
hard-to-reach paths. The hardening stage prioritizes strengthening oracles for code that is already reachable.Where the paper says it
Performance comparison
Results show that TATG consistently outperforms state-of-the-art baselines, including KTester and PANTA, improving line coverage, branch coverage, and mutation score by an average of 22.15, 20.14, and 37.66 percentage points <ref:2607. p. 0 14, and 37. 66 percentage points, respectively. 66 percentage points, respectively. On a
randomly selected subset of 50 focal methods, TATG also achieves
performance comparable to that of a proprietary industrial test
generation tool, while achieving higher line coverage and a higher
mutation score. The complete results are reported in Table III. Across
both models, TATG consistently achieves the best overall
performance in terms of line coverage, branch coverage, and
mutation score while generating fewer test cases than the baselines.Where the paper says it
Where the paper says it
Where the paper says it
Ablation study results
The ablation study demonstrates that each stage of TATG contributes to its overall performance by showing that the structural stage plays a critical role in establishing stronger structural coverage, thereby improving the reachability of subsequent mutation-based hardening. Without the structural stage, TATG achieves only 26. p. 8 Since PITest requires at
least one executable test to produce mutation feedback, the first
iteration is performed without mutation feedback to bootstrap
test generation. Without the structural stage, TATG achieves
only 26. We evaluate TATG on 141 complex Java methods, including
the 110 KTester subjects and 31 additional challenging methods. Results show that TATG consistently outperforms state-of-theart baselines, including KTester and PANTA, improving line
coverage, branch coverage, and mutation score by an average
of 22. 48%
and branch coverage by 4. 97% on Qwen, and by 6. The unified objective representation enables more effective iterative guidance by
tracking unresolved testing objectives and providing finegrained feedback. RQ2: Ablation Study
Table III shows that the hardening stage makes a substantial contribution to the overall performance by consistently
improving the mutation score over the structural stage alone.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Threats to validity
To reduce the threat of stochastic LLM outputs, all LLM-based approaches are evaluated with the same model version, configuration, prompting budget, and execution environment <ref:2607.03194#pg6> TATG explicitly targets structural coverage and mutation score when evaluating it using method-level line coverage, branch coverage, and mutation score <ref:2607. p. 9 Construct validity may be affected by
whether the selected metrics accurately reflect the objectives
of the proposed approach. Since TATG explicitly targets
structural coverage and mutation score, we evaluate it using
method-level line coverage, branch coverage, and mutation
score, following prior work.Where the paper says it
Conclusion
These results demonstrate that unified objective tracking enables more effective iterative guidance for LLM-based unit test generation. TATG consistently achieves higher structural coverage and mutation score than existing baselines On a randomly selected subset of 50 methods, TATG also achieves performance comparable to that of an industrial proprietary test generation tool while achieving higher line coverage and a higher mutation score. p. 9 Experimental results show that TATG
consistently achieves higher structural coverage and mutation
score than existing baselines. On a randomly selected subset
of 50 methods, TATG also achieves performance comparable
to that of an industrial proprietary test generation tool while
achieving higher line coverage and a higher mutation score. Experimental results show that TATG
consistently achieves higher structural coverage and mutation
score than existing baselines. On a randomly selected subset
of 50 methods, TATG also achieves performance comparable
to that of an industrial proprietary test generation tool while
achieving higher line coverage and a higher mutation score. CONCLUSION
This paper presents TATG, an LLM-based unit test generation framework based on a unified objective representation. By representing static analysis results, execution feedback,
coverage gaps, and mutation feedback as tracked objectives,
TATG enables iterative guidance across generation rounds,
allowing the LLM to focus on unresolved testing objectives. CONCLUSION
This paper presents TATG, an LLM-based unit test generation framework based on a unified objective representation. By representing static analysis results, execution feedback,
coverage gaps, and mutation feedback as tracked objectives,
TATG enables iterative guidance across generation rounds,
allowing the LLM to focus on unresolved testing objectives.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
[1] Z. Wang, K. Liu, G. Li, and Z. Jin, “Hits: High-coverage llm-based unit test generation via method slicing,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024
[2] A. Li, M. Liu, Z. Chen, Z. Pei, Z. Li, D. Dai, Y. Wang, and Z. p. 0 TATG: Tracking-Aware Testing Objective for
LLM-based Test Generation
Guancheng Wang∗
, Qinghua Xu∗
, Lionel C. Briand∗†
∗ Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland
† School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
guancheng. LLM-based tools can reason about the source and often
generate readable tests, yet a generic branch- or exceptionoriented prompt does not translate P1–P6 into distinct testing
goals with required setup and measurable acceptance criteria. KTester, a recent state-of-the-art approach, combines project
knowledge for test generation, however, the context remains
coarse-grained (e.Where the paper says it
Where the paper says it
[3] S. Gu, N. Nashid, and A. Mesbah, “Llm test generation via iterative hybrid program analysis,” arXiv preprint arXiv:2503. p. 9 By representing static analysis results, execution feedback,
coverage gaps, and mutation feedback as tracked objectives,
TATG enables iterative guidance across generation rounds,
allowing the LLM to focus on unresolved testing objectives. We evaluated TATG on 141 complex Java methods against
representative traditional, hybrid, and LLM-based test generation approaches.Where the paper says it
[4] S. Gu, Q. Zhang, K. Li, C. Fang, F. Tian, L. Zhu, J. Zhou, and Z. Chen, “Testart: Improving llm-based unit testing via co-evolution of automated generation and repair iteration,” arXiv preprint arXiv:2408. p. 0 Building on this objective representation, TATG employs
a two-stage generation strategy. The structural stage focuses
on producing executable tests, improving reachability, and
increasing line/branch coverage using static objectives, along
arXiv:2607.Where the paper says it
[5] J. Altmayer Pizzorno and E. D. Berger, “Coverup: Effective high coverage test generation for python,” Proceedings of the ACM on Software Engineering, vol. p. 0 TATG: Tracking-Aware Testing Objective for
LLM-based Test Generation
Guancheng Wang∗
, Qinghua Xu∗
, Lionel C. Briand∗†
∗ Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland
† School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
guancheng.Where the paper says it
[6] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM computing surveys
[7] A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” Information and Software Technology
[8] M. Harman, J. Ritchey, I. Harper, S. Sengupta, K. Mao, A. Gulati, C. Foster, and H. Robert, “Mutation-guided llm-based test generation at meta,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering
[9] G. Wang, Q. Xu, L. C. Briand, and K. p. 0 TATG: Tracking-Aware Testing Objective for
LLM-based Test Generation
Guancheng Wang∗
, Qinghua Xu∗
, Lionel C. Briand∗†
∗ Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland
† School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
guancheng. On a
randomly selected subset of 50 focal methods, TATG also achieves
performance comparable to that of a proprietary industrial test
generation tool, while achieving higher line coverage and a higher
mutation score. Index Terms—Unit Test Generation, Program Analysis, Large
Language Models
I.Where the paper says it
Where the paper says it
[10] Q. Xu, G. Wang, L. Briand, Z. Guo, and K. Liu, “Llm-based mockless unit test generation for java,” arXiv preprint arXiv:2605. p. 0 Building on this objective representation, TATG employs
a two-stage generation strategy. The structural stage focuses
on producing executable tests, improving reachability, and
increasing line/branch coverage using static objectives, along
arXiv:2607.Where the paper says it
[11] paohaijiao, “jquick-curl: A java wrapper for libcurl,” https://github.com/paohaijiao/jquick-curl, accessed 2024-06-23
[12] Google, “Agent development kit (adk) for java,” https://github.com/google/adk-java, accessed 2024-06-23
[13] agentscope-ai, “Build production-ready ai agents in java,” https://github.com/agentscope-ai/agentscope-java, accessed 2024-06-23
[14] A2A Project, “A2a: An open-source framework for building autonomous agents in java,” https://github.com/a2aproject/a2a-java, accessed 2024-06-23
[15] “Lizard: A code complexity analyzer,” https://github.com/terryyin/lizard, accessed 2026-06-23
[16] M. Stavtsev and Y. Bugayenko, “Evaluating the dependency between cyclomatic complexity and response for class,” arXiv preprint arXiv:2410. p. 6 Consequently, the additional subjects cover a broader
range of access-control constraints and dependency patterns
commonly encountered in real-world Java projects. Table II summarizes the benchmark subjects, including the
number of methods, cyclomatic complexity (CC), and noncomment lines of code (NLOC).Where the paper says it
[17] C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” in Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, 2007
[18] G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, 2011
[19] Y. Lin, Y. S. Ong, J. Sun, G. Fraser, and J. S. Dong, “Graph-based seed object synthesis for search-based unit testing,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
[20] “Jacoco,” Accessed: 2026
[21] “Pitest,” Accessed: 2026
[22] soot-oss, “Sootup: A redesign of the soot static analysis framework,” https://github.com/soot-oss/SootUp, 2023
[23] “vllm: A high-performance llm inference and serving framework,” https://vllm.ai, accessed 2024-06-23
[24] “Fireworks ai,” https://fireworks.ai/models/fireworks/deepseek-v4-flash, accessed 2024-06-23
--- Page 1 ---
TATG: Tracking-Aware Testing Objective for LLM-based Test Generation
Guancheng Wang∗, Qinghua Xu∗, Lionel C. Briand∗† Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland † School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada guancheng. p. 0 TATG: Tracking-Aware Testing Objective for
LLM-based Test Generation
Guancheng Wang∗
, Qinghua Xu∗
, Lionel C. Briand∗†
∗ Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland
† School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
guancheng. Briand∗†
∗ Research Ireland Lero Centre for Software, University of Limerick, Limerick, Ireland
† School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
guancheng. qinghua@gmail. qinghua@gmail. com, lionel. Building on this objective representation, TATG employs
a two-stage generation strategy. The structural stage focuses
on producing executable tests, improving reachability, and
increasing line/branch coverage using static objectives, along
arXiv:2607.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
I I INTRODUCTION
Automated unit test generation has relied on random testing, search-based testing, symbolic execution, and, more recently, large language models (LLMs) <ref:2607.03194#pg6> Despite this progress, generating high-quality tests for complex Java methods remains challenging <ref:2607. p. 0 INTRODUCTION
Automated unit test generation has relied on random testing,
search-based testing, symbolic execution, and, more recently,
large language models (LLMs). Despite this progress, generating high-quality tests for complex Java methods remains
challenging. Despite this progress, generating high-quality tests for complex Java methods remains
challenging. The difficulty often lies not in the size of the
focal method itself, but in satisfying branch-specific testing
requirements that are only partially visible from the method’s
source code. The difficulty often lies not in the size of the
focal method itself, but in satisfying branch-specific testing
requirements that are only partially visible from the method’s
source code. Reaching a target behavior may require establishing a prerequisite program state, exercising a particular switch
branch, triggering an exception-translation path, invoking the
method through a valid entry route, or constructing assertions
strong enough to detect semantic regressions. Some of these
requirements can be inferred statically, while others only
become visible after generated tests are compiled, executed,
and evaluated. Recent LLM-based test generation approaches have moved
beyond focal-method-only prompting by incorporating project
context, static analysis, coverage feedback, and mutation
guidance [1], [2], [3], [4], [5], [6], [7], [8], [9]. For example, KTester injects project structure and usage knowledge [2], PANTA uses coverage feedback to target undertested paths [3], and MUTGEN leverages mutation feedback
to strengthen generated tests [9]. However, these signals are
typically delivered to the LLM as prompts, hints, path descriptions, or post-execution feedback. However, these signals are
typically delivered to the LLM as prompts, hints, path descriptions, or post-execution feedback. Existing approaches do
not maintain an explicit representation of individual testing
requirements and their status across iterations. Existing approaches do
not maintain an explicit representation of individual testing
requirements and their status across iterations. Consequently,
later rounds may revisit already-satisfied behaviors while unresolved branches, or weak oracles, remain unaddressed.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
II. MOTIVATING EXAMPLE
We use a single running example to illustrate the testing problem that TATG addresses <ref:2607.03194#pg6> A useful test suite for Listing 1 must satisfy requirements that are not explicit <ref:2607. p. 1 Running example: stateful entry, finite dispatch, and exception
translation. A useful test suite for Listing 1 must satisfy requirements that are not explicit. A useful test suite for Listing 1 must satisfy requirements that are not explicit. First, most paths are statedependent. A test that simply constructs a fresh reader and calls the target may reach only the
fall-through exception. Second, the number of nested branches
is finite: P1, P2, and P3 correspond to distinct token constants,
and each needs a distinct input or mock behaviour. Second, the number of nested branches
is finite: P1, P2, and P3 correspond to distinct token constants,
and each needs a distinct input or mock behaviour. Third, P5
requires exceptional predecessor behaviour: the collaborator
must throw StreamException at the right call site so that
the method executes the catch-rethrow block. Third, P5
requires exceptional predecessor behaviour: the collaborator
must throw StreamException at the right call site so that
the method executes the catch-rethrow block. Finally, if the
method is package-private or is normally accessed through a
facade, the test must also select an executable entry route. Finally, if the
method is package-private or is normally accessed through a
facade, the test must also select an executable entry route. These requirements also explain why coverage should precede mutation refinement.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 4 ---
TABLE I
PROJECTION FROM SOURCE-GROUNDED EVIDENCE TO OBJECTIVES.
Category cat Evidence E from static analysis or dynamic feedback Required setup S Target action A Oracle shape Q <ref:2607. p. 5 TABLE I
PROJECTION FROM SOURCE-GROUNDED EVIDENCE TO OBJECTIVES. Category cat Evidence E from static analysis or dynamic feedback Required setup S Target action A Oracle shape Q
StateSetup Assignment, update, or predicate over a receiver field, e. TABLE I
PROJECTION FROM SOURCE-GROUNDED EVIDENCE TO OBJECTIVES. Category cat Evidence E from static analysis or dynamic feedback Required setup S Target action A Oracle shape Q
StateSetup Assignment, update, or predicate over a receiver field, e. For each evidence type, the table specifies
how the setup (S), target action (A), and oracle shape (Q) are
constructed. Structural objectives are initialized before the first generation round. Structural objectives are initialized before the first generation round. Most structural objectives are derived through
static analysis, including StateSetup, DispatchArm, ExceptionPath, Construction, BoundaryInput, and EntryRoute. Most structural objectives are derived through
static analysis, including StateSetup, DispatchArm, ExceptionPath, Construction, BoundaryInput, and EntryRoute. However,
some testing requirements cannot be reliably inferred through
static analysis alone. However,
some testing requirements cannot be reliably inferred through
static analysis alone. Therefore, TATG performs additional
LLM-based objective inference, using the objective definitions
and the focal method source code to identify higher-level
testing objectives. Feedback-driven objectives are generated incrementally
throughout iterative generation. Compilation and execution
diagnostics, along with coverage reports, generate ValidationRepair objectives, weak assertions identified in generated
tests generate OracleStrengthening objectives, and mutation
testing generates MutationReachability and MutationAssertion
objectives for uncovered and live mutants, respectively.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
Figure 1. Overview of TATG. p. 2 APPROACH
A. Overview
TATG is a two-stage, objective-guided framework for LLMbased unit test generation.Where the paper says it
TATG addresses this example through two-stage objectiveguided generation <ref:2607. p. 2 As a
result, a passing test suite may still leave deep paths uncovered,
while subsequent repair or regeneration lacks an explicit
objective that identifies the unresolved branch or exception
behavior requiring attention, such as a specific token-dispatch
branch or an exception-translation path. TATG addresses this example through two-stage objectiveguided generation. TATG addresses this example through two-stage objectiveguided generation. Before generation, static analysis translates
branch-relevant facts into explicit objectives, such as “exercise the branch where currToken == Token. advance() to throw StreamException and
assert the resulting IOException”. The structural stage
first uses compilation and execution diagnostics, together with
coverage feedback, to improve target reachability and code
coverage. The structural stage
first uses compilation and execution diagnostics, together with
coverage feedback, to improve target reachability and code
coverage. After each round, TATG updates the objective
state to record which objectives were satisfied, which failed
compilation/execution, and which remain open. After each round, TATG updates the objective
state to record which objectives were satisfied, which failed
compilation/execution, and which remain open. Only after
sufficient structural progress does mutation feedback become
actionable: mutants in unreached code mainly indicate missing
reachability, whereas live mutants in reached code may indicate either insufficient test inputs or weak assertions.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 6 ---
TABLE I
PROJECTION FROM SOURCE-GROUNDED EVIDENCE TO OBJECTIVES.
Category cat Evidence E from static analysis or dynamic feedback Required setup S Target action A Oracle shape Q <ref:2607.03194#pg6> StateSetup Assignment, update, or predicate over a receiver field, e. p. 5 TABLE I
PROJECTION FROM SOURCE-GROUNDED EVIDENCE TO OBJECTIVES. Category cat Evidence E from static analysis or dynamic feedback Required setup S Target action A Oracle shape Q
StateSetup Assignment, update, or predicate over a receiver field, e. Category cat Evidence E from static analysis or dynamic feedback Required setup S Target action A Oracle shape Q
StateSetup Assignment, update, or predicate over a receiver field, e. index++, _token = p. index++, _token = p. nextToken(), if
(this. nextToken(), if
(this. index >= end), or a method call whose receiver resolves
to a receiver field, e. Building on this objective representation, TATG employs
a two-stage generation strategy. The structural stage focuses
on producing executable tests, improving reachability, and
increasing line/branch coverage using static objectives, along
arXiv:2607. index >= end), or a method call whose receiver resolves
to a receiver field, e. inObject()
Establish the required receiver or
collaborator state through visible
constructors, setters, builders, factories,
or public methods
Invoke the focal method after the
required state has been established
Assert return value, state transition,
emitted output, collaborator interaction,
or exception
DispatchArm Branch or switch statement whose predicate/selector depends on a
concrete code element: if (token == VALUE_STRING),
switch (mode), case STRICT:, "json". index >= end), or a method call whose receiver resolves
to a receiver field, e. inObject()
Establish the required receiver or
collaborator state through visible
constructors, setters, builders, factories,
or public methods
Invoke the focal method after the
required state has been established
Assert return value, state transition,
emitted output, collaborator interaction,
or exception
DispatchArm Branch or switch statement whose predicate/selector depends on a
concrete code element: if (token == VALUE_STRING),
switch (mode), case STRICT:, "json". index >= end), or a method call whose receiver resolves
to a receiver field, e. inObject()
Establish the required receiver or
collaborator state through visible
constructors, setters, builders, factories,
or public methods
Invoke the focal method after the
required state has been established
Assert return value, state transition,
emitted output, collaborator interaction,
or exception
DispatchArm Branch or switch statement whose predicate/selector depends on a
concrete code element: if (token == VALUE_STRING),
switch (mode), case STRICT:, "json". inObject()
Establish the required receiver or
collaborator state through visible
constructors, setters, builders, factories,
or public methods
Invoke the focal method after the
required state has been established
Assert return value, state transition,
emitted output, collaborator interaction,
or exception
DispatchArm Branch or switch statement whose predicate/selector depends on a
concrete code element: if (token == VALUE_STRING),
switch (mode), case STRICT:, "json". equals(format),
type == 1, or x instanceof Foo
Choose argument values, receiver state,
or concrete object types that satisfy the
selected predicate, match the selected
label, or satisfy the selected type test.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
json
.equals(format), type == 1, or x instanceof Foo <ref:2607. p. 5Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified objective representation enables explicit tracking of testing requirements
which remain unresolved throughout generation,
allowing subsequent iterations to focus exclusively on unresolved testing goals rather than repeatedly targeting already-satisfied requirements. -
Two-stage objective guided generation coordinates structural coverage improvement and mutation-based oracle strengthening, ensuring that the structural stage focuses on
producing executable tests, improving reachability, and increasing line/branch coverage
before hardening rounds strengthen assertions using mutation feedback to improve fault detection capability. -
The system can perform two distinct types of objective inference: static analysis generates structural objectives for reachability and branch coverage, while LLM-based inference identifies higher-level testing objectives such as
ExceptionPath objective corresponding to the try–catch–rethrow path.
-
Feedback-driven objectives are generated incrementally through iterations, unifying
compilation/execution diagnostics, coverage reports, and mutation results into a single objective rather than handling them independently,
which allows TATG to reason explicitly about which requirements have been satisfied and which remain unresolved. -
The system can generate specific feedback objectives like
OracleStrengthening
orMutationAssertion
based on evidence such asa weak assertion identified in generated tests
or alive mutant reported on a source line that was not executed,
respectively. -
The budget allocation is dynamic, computing the number of requested tests as Br = clip(Bmin, Bmax, ⌈λAr⌉), which allows TATG to allocate resources per active objective rather than using a fixed number of tests for every round.
-
The system can generate highly specific setup instructions from static analysis objectives, such as
configure stream.advance to throw StreamException and assert the resulting IOException,
directly translating abstract requirements into actionable test configurations in the prompt.
Abstract
Complex Java methods challenge automated unit test generation, as achieving high coverage and fault detection requires satisfying intricate testing requirements involving program states, execution paths, and assertions. Existing LLM-based approaches leverage program context, static analysis, and dynamic feedback but lack explicit tracking of individual testing requirements across iterations, potentially leading to redundant generation and unresolved testing objectives. We present TATG, a tracking-aware LLM-based unit test generation framework that unifies structural coverage and mutation-based fault detection. TATG introduces a structured objective representation capturing testing requirements derived from static analysis and dynamic feedback, enabling fine-grained tracking of their satisfaction status. It employs a two-stage workflow that first improves structural coverage and subsequently strengthens assertions through mutation-guided hardening. We evaluate TATG on 141 complex Java methods against state-of-the-art baselines, including KTester and PANTA. Results show average improvements of 22.15, 20.14, and 37.66 percentage points in line coverage, branch coverage, and mutation score, respectively. TATG also outperforms general-purpose coding assistants, including Codex and Claude Code, under the same open-weight model configurations, achieving higher average coverage and mutation scores while generating fewer tests and consuming fewer tokens. These findings highlight the value of explicitly tracking testing objectives for effective LLM-based test generation.
Sources
- Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation
- LLM Test Generation via Iterative Hybrid Program Analysis
- TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration
- Mutation-Guided Unit Test Generation with a Large Language Model
- LLM-based Mockless Unit Test Generation for Java
- Evaluating the Dependency Between Cyclomatic Complexity and Response For Class
- Unit Test Case Generation with Transformers and Focal Context
- Retrieval-Augmented Test Generation: How Far Are We?
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters