Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents

summary

The gist

Mocking is a foundational practice in unit testing, enabling developers to isolate code under test by replacing real dependencies with simulated alternatives.

In short

The research tested if coding agents and LLMs could automatically generate mock objects, stubs, and assertions for unit tests. Coding agents significantly improved test executability in Java but failed to enhance behavioral fidelity or fault detection compared to standalone LLMs. The study highlights that while iterative refinement helps execution, it doesn't guarantee better mock quality.

Key concepts

Mocking
Mocking is a testing technique where you replace real dependencies in code with simulated substitutes. This allows developers to test specific parts of the system in isolation without needing the actual external services or complex components they rely on.
Coding Agents
These are AI systems designed to write and refine code. In this study, they were used to iteratively improve generated mock artifacts by using runtime feedback from executing tests, aiming to make the mocks more accurate.
Behavioral Fidelity
This measures how accurately the generated mock objects, stubs, and assertions reflect the actual behavior of the real code they are supposed to simulate. High fidelity means the simulated parts behave exactly as expected in a real scenario.
Mutation Testing
This technique checks how well a test suite can detect small changes (mutations) in the code. In this context, it measures if generated mocks can still correctly identify faults or errors introduced by these small changes.

Terminology used across episodes

This episode discusses

The paper

Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents · Read on arXiv

Shandong University · City University of Hong Kong · ×Mistral AI

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Executable but Unfaithful".

Jane: Here's the gist of the paper. Mocking is a foundational practice in unit testing, enabling developers to isolate code under test by replacing real dependencies with simulated alternatives.

Tom: Automating the generation of mock objects, stubs, and mock assertions has been a long-standing research goal.

Jane: LLMs and coding agents offer an alternative by generating these mocking artifacts directly from source code without requiring recorded dependency executions or fixed symbol pools.

Tom: To systematically evaluate this capability, the authors constructed MockBench, a bilingual benchmark comprising 165 mock artifact generation tasks drawn from real-world Java and Python projects, complete with validated reference mock objects, stubs, and mock assertions.

Title and authors: Tom: So we're looking at a paper called "Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents". It sounds like they're tackling the problem of automating the creation of test doubles—mock objects, stubs, and assertions—using large language models.

Jane: That sounds really technical. So what is it trying to do? Basically, they are looking at how we can use AI to build these testing tools directly from the source code without having to actually run the real code first.

Lu: The paper argues that while mocking is foundational for unit testing, automating it has been a long time coming because you usually need some kind of recording or fixed set of symbols. This work suggests LLMs and coding agents are an alternative because they can generate these artifacts directly from the source code without needing those things.

Meng: So what's the big deal here for us when we think about production? Does this mean we can just prompt an AI to write perfect tests instantly? Because running a real dependency is still the main hurdle, right?

Lalam: The paper specifically addresses that hurdle, pointing out that capture-and-replay approaches need the actual dependency to be runnable, but LLMs and agents generate mocks from source code directly without needing those recorded executions or fixed symbol pools.

Tom: Exactly. So it's about moving away from needing to run things just to see what they do when we want to test in isolation. It sets up a different kind of automation for mock generation than what we've seen before.

Jane: And the authors are quite specific about the artifacts they focus on—mock objects, stubs, and mock assertions—which are the three main pieces that make up a well-formed mock-based test case.

Lu: And they use this structure to evaluate how well these AI tools perform across different real-world Java and Python projects using a benchmark called MockBench.

The paper's summary: Tom: So, the core finding of "Executable but Unfaithful" is that while these AI tools can generate mock artifacts, they aren't perfect. They can make tests run, which is a big win for executability. But they often miss the actual behavior details we need to test properly.

Jane: It seems like the paper found that LLMs and coding agents are state-of-the-art in this area when you look at whether the code actually runs successfully, but they aren't necessarily better at mimicking complex interactions.

Lu: The summary breaks down three main results: first, coding agents really improve Java executability compared to standalone LLMs by a big margin, and second, that high executability doesn't automatically mean the generated mocks have better behavior fidelity.

Meng: So if the tests run quickly, but the mock object gives us the wrong return value or an assertion checks the wrong argument, that's a problem for testing quality. That’s what I worry about when deploying code; we need reliable tests.

Lalam: The paper points out that mock objects are generally easier to reconstruct than stubs and assertions, especially when it comes to stub outputs and those specific assertion arguments and constraints.

Tom: Right, so the summary suggests that while we get better at making things run, we still have a problem with the accuracy of what those generated mocks actually represent in terms of behavior.

Jane: And they quantify this by looking at failures across six different LLMs and three distinct coding agents on one hundred sixty-five tasks drawn from real projects.

The paper's improvements: Tom: So, the paper suggests some ways to improve these tools because the current way, just generating them, isn't enough. They look at how we can make the process iterative and feedback-driven.

Jane: They propose that using runtime execution feedback can help coding agents refine their mock object generation over time. This is a key idea—instead of one shot guessing, they suggest iteration based on what happens when you run the test.

Lu: That iterative refinement is where the improvement comes from; it lets the agent learn from execution results to fix its initial guesses about how things should behave. But even with this, they found that this extra time budget doesn't translate into better mock behavioral fidelity.

Meng: Wait, so we spend more time making the tests run faster, and that doesn't actually make the mocks *better* at what they are supposed to do? That means we might just be spending more effort for less testing quality gain.

Lalam: The paper also analyzes interaction fault detection capability using mutation testing. They use first-order mutation operators to compare the kill sets of generated mocks against reference test faults, measuring Jaccard similarity between killed mutants and those from top-tier standalone LLMs.

Tom: That’s a sharp way to measure it—by looking at how well the tests catch small changes in the mock setup. And they found that neither LLMs nor coding agents fully preserve the ground truth fault detection capacity, with average Jaccard similarities ranging from seventy point eight five percent on Java down to eighty-five point eight seven percent on Python.

Jane: And they also found a specific issue with coding agents: they showed a noticeable drop in mutation kill similarity compared to top-tier standalone LLMs, which suggests that their iterative repair might actually compromise runtime feedback by relaxing or omitting mock assertions just so the test runner can execute successfully.

Conclusion: Tom: So, wrapping up the main points of "Executable but Unfaithful," it seems like iterative execution feedback helps coding agents improve executability, which is a definite win over standalone LLMs in Java tests.

Jane: But the paper makes it clear that this gain in executability doesn't always translate into better behavior fidelity or superior fault detection ability when you look closely at the mocks themselves.

Lu: The paper concludes that while coding agents surpass the existing best approach, like StubCoder, in terms of both executability and behavior fidelity overall, they still fall short of standalone LLMs on those specific quality metrics.

Meng: So for us in production environments right now, it means we can use these agents to quickly generate tests that pass execution checks when we're under pressure. But we still have to be careful about the quality of the behavioral checks they put in place; they might be easier to fix than the core logic.

Lalam: The paper flags a critical failure mode: coding agents fail predominantly from mock assertion errors, with sixty-six point six seven percent in Java and eighty-one point five four percent in Python. Missing assertions alone are over forty four percent across those languages, suggesting the feedback loop is mostly optimizing for just passing the test runner.

Tom: That's a sobering thought—the system is optimized for passing, not necessarily for strict interaction checks on every single call. So we have to be mindful of that trade-off when we use these tools.

Jane: Exactly. The paper introduces MockBench as the first bilingual benchmark for this kind of task and gives us a taxonomy detailing ten distinct error types across mock objects, stubs, and assertions masked by executable tests.

Lu: It’s a useful tool because it gives us the language to describe exactly what goes wrong when we automate mock generation, which is something we needed before we could really improve these tools.

Meng: Good stuff for documentation. It helps us know what kind of errors to look for when we review tests generated by AI agents in our pipelines.

Lalam: We'll keep an eye on how these agents handle those assertion errors, because that seems to be the biggest bottleneck right now for getting truly reliable mocks without running everything first.

More episodes

← Home