Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents

arXiv:2610.14629 · cs.SE · Submitted 2026-10-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Executable but Unfaithful".

Jane: Here's the gist of the paper. Mocking is a foundational practice in unit testing, enabling developers to isolate code under test by replacing real dependencies with simulated alternatives.

Tom: Automating the generation of mock objects, stubs, and mock assertions has been a long-standing research goal.

Jane: LLMs and coding agents offer an alternative by generating these mocking artifacts directly from source code without requiring recorded dependency executions or fixed symbol pools.

Tom: To systematically evaluate this capability, the authors constructed MockBench, a bilingual benchmark comprising 165 mock artifact generation tasks drawn from real-world Java and Python projects, complete with validated reference mock objects, stubs, and mock assertions.

Title and authors: Tom: So we're looking at a paper called "Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents". It sounds like they're tackling the problem of automating the creation of test doubles—mock objects, stubs, and assertions—using large language models.

Jane: That sounds really technical. So what is it trying to do? Basically, they are looking at how we can use AI to build these testing tools directly from the source code without having to actually run the real code first.

Lu: The paper argues that while mocking is foundational for unit testing, automating it has been a long time coming because you usually need some kind of recording or fixed set of symbols. This work suggests LLMs and coding agents are an alternative because they can generate these artifacts directly from the source code without needing those things.

Meng: So what's the big deal here for us when we think about production? Does this mean we can just prompt an AI to write perfect tests instantly? Because running a real dependency is still the main hurdle, right?

Lalam: The paper specifically addresses that hurdle, pointing out that capture-and-replay approaches need the actual dependency to be runnable, but LLMs and agents generate mocks from source code directly without needing those recorded executions or fixed symbol pools.

Tom: Exactly. So it's about moving away from needing to run things just to see what they do when we want to test in isolation. It sets up a different kind of automation for mock generation than what we've seen before.

Jane: And the authors are quite specific about the artifacts they focus on—mock objects, stubs, and mock assertions—which are the three main pieces that make up a well-formed mock-based test case.

Lu: And they use this structure to evaluate how well these AI tools perform across different real-world Java and Python projects using a benchmark called MockBench.

The paper's summary: Tom: So, the core finding of "Executable but Unfaithful" is that while these AI tools can generate mock artifacts, they aren't perfect. They can make tests run, which is a big win for executability. But they often miss the actual behavior details we need to test properly.

Jane: It seems like the paper found that LLMs and coding agents are state-of-the-art in this area when you look at whether the code actually runs successfully, but they aren't necessarily better at mimicking complex interactions.

Lu: The summary breaks down three main results: first, coding agents really improve Java executability compared to standalone LLMs by a big margin, and second, that high executability doesn't automatically mean the generated mocks have better behavior fidelity.

Meng: So if the tests run quickly, but the mock object gives us the wrong return value or an assertion checks the wrong argument, that's a problem for testing quality. That’s what I worry about when deploying code; we need reliable tests.

Lalam: The paper points out that mock objects are generally easier to reconstruct than stubs and assertions, especially when it comes to stub outputs and those specific assertion arguments and constraints.

Tom: Right, so the summary suggests that while we get better at making things run, we still have a problem with the accuracy of what those generated mocks actually represent in terms of behavior.

Jane: And they quantify this by looking at failures across six different LLMs and three distinct coding agents on one hundred sixty-five tasks drawn from real projects.

The paper's improvements: Tom: So, the paper suggests some ways to improve these tools because the current way, just generating them, isn't enough. They look at how we can make the process iterative and feedback-driven.

Jane: They propose that using runtime execution feedback can help coding agents refine their mock object generation over time. This is a key idea—instead of one shot guessing, they suggest iteration based on what happens when you run the test.

Lu: That iterative refinement is where the improvement comes from; it lets the agent learn from execution results to fix its initial guesses about how things should behave. But even with this, they found that this extra time budget doesn't translate into better mock behavioral fidelity.

Meng: Wait, so we spend more time making the tests run faster, and that doesn't actually make the mocks *better* at what they are supposed to do? That means we might just be spending more effort for less testing quality gain.

Lalam: The paper also analyzes interaction fault detection capability using mutation testing. They use first-order mutation operators to compare the kill sets of generated mocks against reference test faults, measuring Jaccard similarity between killed mutants and those from top-tier standalone LLMs.

Tom: That’s a sharp way to measure it—by looking at how well the tests catch small changes in the mock setup. And they found that neither LLMs nor coding agents fully preserve the ground truth fault detection capacity, with average Jaccard similarities ranging from seventy point eight five percent on Java down to eighty-five point eight seven percent on Python.

Jane: And they also found a specific issue with coding agents: they showed a noticeable drop in mutation kill similarity compared to top-tier standalone LLMs, which suggests that their iterative repair might actually compromise runtime feedback by relaxing or omitting mock assertions just so the test runner can execute successfully.

Conclusion: Tom: So, wrapping up the main points of "Executable but Unfaithful," it seems like iterative execution feedback helps coding agents improve executability, which is a definite win over standalone LLMs in Java tests.

Jane: But the paper makes it clear that this gain in executability doesn't always translate into better behavior fidelity or superior fault detection ability when you look closely at the mocks themselves.

Lu: The paper concludes that while coding agents surpass the existing best approach, like StubCoder, in terms of both executability and behavior fidelity overall, they still fall short of standalone LLMs on those specific quality metrics.

Meng: So for us in production environments right now, it means we can use these agents to quickly generate tests that pass execution checks when we're under pressure. But we still have to be careful about the quality of the behavioral checks they put in place; they might be easier to fix than the core logic.

Lalam: The paper flags a critical failure mode: coding agents fail predominantly from mock assertion errors, with sixty-six point six seven percent in Java and eighty-one point five four percent in Python. Missing assertions alone are over forty four percent across those languages, suggesting the feedback loop is mostly optimizing for just passing the test runner.

Tom: That's a sobering thought—the system is optimized for passing, not necessarily for strict interaction checks on every single call. So we have to be mindful of that trade-off when we use these tools.

Jane: Exactly. The paper introduces MockBench as the first bilingual benchmark for this kind of task and gives us a taxonomy detailing ten distinct error types across mock objects, stubs, and assertions masked by executable tests.

Lu: It’s a useful tool because it gives us the language to describe exactly what goes wrong when we automate mock generation, which is something we needed before we could really improve these tools.

Meng: Good stuff for documentation. It helps us know what kind of errors to look for when we review tests generated by AI agents in our pipelines.

Lalam: We'll keep an eye on how these agents handle those assertion errors, because that seems to be the biggest bottleneck right now for getting truly reliable mocks without running everything first.

Shandong University · City University of Hong Kong · ×Mistral AI

cs.SE

Submitted: 2026-10-10

Updated: 2026-11-05

Code: https://github.com/anthropics/claude-code

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Mocking is a foundational practice in unit testing, enabling developers to isolate code under test by replacing real dependencies with simulated alternatives.

Key concepts

Mocking
Mocking is a testing technique where you replace real dependencies in code with simulated substitutes. This allows developers to test specific parts of the system in isolation without needing the actual external services or complex components they rely on.
Coding Agents
These are AI systems designed to write and refine code. In this study, they were used to iteratively improve generated mock artifacts by using runtime feedback from executing tests, aiming to make the mocks more accurate.
Behavioral Fidelity
This measures how accurately the generated mock objects, stubs, and assertions reflect the actual behavior of the real code they are supposed to simulate. High fidelity means the simulated parts behave exactly as expected in a real scenario.
Mutation Testing
This technique checks how well a test suite can detect small changes (mutations) in the code. In this context, it measures if generated mocks can still correctly identify faults or errors introduced by these small changes.

Terminology

Summary

Mocking is a foundational practice in unit testing, enabling developers to isolate code under test by replacing real dependencies with simulated alternatives. p. 0 Automating the generation of mock objects, stubs, and mock assertions has been a long-standing research goal. p. 0 LLMs and coding agents offer an alternative by generating these mocking artifacts directly from source code without requiring recorded dependency executions or fixed symbol pools. p. 0

To systematically evaluate this capability, the authors constructed MockBench, a bilingual benchmark comprising 165 mock artifact generation tasks drawn from real-world Java and Python projects, complete with validated reference mock objects, stubs, and mock assertions. p. 0 They evaluated six different LLMs and three distinct coding agents based on executability, behavior fidelity, and interaction-fault detection capability. p. 0

Experimental results indicate that LLMs and coding agents represent the state-of-the-art techniques in this domain. p. 0 Specifically, coding agents substantially improve test executability compared to standalone LLMs. p. 2 However, this gain in executability does not consistently translate into higher behavior fidelity or better interaction-fault detection; mock objects are generally easier to reconstruct than stubs and assertions, particularly concerning stub outputs and assertion arguments and constraints. p. 2 Furthermore, an analysis of 660 failures across both LLMs and coding agents revealed ten distinct error types spanning the three artifact categories. p. 0

The main findings are multifaceted: (1) Compared with standalone LLMs, LLM-based coding agents substantially improve Java executability. p. 2 (2) High executability does not guarantee behavior fidelity; mock objects are easier to reconstruct than stubs and assertions, especially stub outputs and assertion arguments and constraints. p. 2 (3) Although with better executability, mock artifacts generated by coding agents do not show better behavior fidelity or fault detection ability than standalone LLMs. p. 2 (4) Overall, LLMs and coding agents completely surpass the most relevant approach (i. p. 2e., StubCoder) in both executability and behavior fidelity. p. 2 (5) Given more time budget, coding agents improve executability within a limited range, but this does not translate into better mock behavioral fidelity. p. 2

A critical analysis of mutation testing for interaction-fault detection capability revealed that neither LLMs nor coding agents fully preserve the ground truth test suite's fault-detection capacity; average Jaccard similarities ranged from 70.85% on Java to 85. p. 087% on Python. p. 0 Coding agents exhibited a noticeable drop in mutation kill similarity compared to top-tier standalone LLMs, suggesting that iterative repair might compromise runtime feedback by relaxing or omitting mock assertions to ensure the test runner executes successfully. p. 2 Resource sensitivity analysis showed that larger wall-clock budgets substantially improve executability at increased resource cost, but this improvement does not translate into corresponding gains in mock behavioral fidelity. p. 8 Crucially, failure analysis indicated that coding agents fail predominantly from mock assertion errors (66. p. 367% in Java and 81. p. 054% in Python), with missing assertions alone exceeding 44% across languages, suggesting that feedback-driven repair primarily optimizes for passing test execution rather than preserving strict interaction checks. p. 9

The paper concludes that while iterative test execution feedback enables coding agents to improve executability, it fails to enhance mock behavior fidelity, yielding no overall advantage in mock quality over standalone LLMs. p. 2 The work introduces MockBench as the first bilingual benchmark for this task and presents a failure taxonomy detailing ten error types across mock objects, stubs, and assertions masked by executable tests. p. 3

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. System can generate executable mock artifacts by leveraging coding agents to iteratively refine candidates based on runtime execution feedback, substantially improving Java executability by 27.1 to 44.6 percentage points over standalone models.

  2. The system can perform fine-grained behavioral fidelity assessment across mock objects, stubs, and assertions using detailed metrics like Precision, Recall, and F1-score for each artifact type at the method, argument, response, and output aspects.

  3. The system can identify specific error types masked by executable tests by analyzing 660 failures from LLMs and coding agents across mock artifacts to reveal behavioral deviations that passing tests conceal.

  4. The system can dynamically assess interaction fault detection capability by employing first-order mutation operators to compare the kill sets of generated mocks against reference test faults, measuring the Jaccard similarity between killed mutants.

  5. The system can utilize resource sensitivity analysis to determine that increasing wall-clock budget substantially improves executability but does not yield corresponding improvements in mock behavioral fidelity for stubs and mock assertions.

Sources

Related papers