Vibe Coding in Software Development: A Multivocal Literature Review
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Vibe Coding in Software Development".
Jane: Here's the gist of the paper. A multivocal literature review (MLR) was conducted to integrate peer-reviewed and grey literature on vibe coding, synthesizing 47 sources (28 peer-reviewed and 19 grey) published between 2022 and October 2025 across eight research questions.
Tom: Vibe coding is defined as an intent-driven software development practice where a developer specifies desired functionality, constraints, and context primarily through natural language prompts, enabling large language models (LLMs) to generate, refine, and iteratively revise code artifacts.
Jane: This is consistently described not as a one-shot activity but as an iterative generation-evaluation-revision loop.
Tom: Key findings indicate that the developer's role fundamentally shifts from direct code authoring toward specification, supervision, and validation.
Title and authors: Tom: The paper starts by introducing this whole concept of vibe coding as a specific software development practice that relies on natural language prompts for the large language models to create code >
Jane: And it points out that the evidence around this practice is really scattered across academic papers and also in things like blogs and industry reports, which is why they did this multivocal literature review to get a fuller picture >
Lu: They looked at forty-seven sources published between two thousand twenty-two and October twenty twenty-five, mixing peer-reviewed studies with grey literature to capture the current state of practice >
Meng: Forty seven sources sounds like a lot of data. Does that mean we’re looking at a really broad range of how people are using these models, or is it mostly just one specific way they are doing things?
Lalam: It’s about capturing the state of practice as well as the state of research, so it tries to bridge that gap between what’s being formally studied and what’s actually happening in the day-to-day coding environment >
Tom: So, fundamentally, they are mapping out how this interaction between a human and an AI is structured when someone uses natural language to direct code generation >
Jane: It really frames it as this loop where the developer states intent, the model generates code, then you evaluate it by running and inspecting that result, and then you refine your next prompt based on what you observe >
The paper's summary: Tom: The main takeaway from this paper is that when vibe coding is done right, it shows short-term gains in productivity and time to prototype for about forty-five percent of the sources they reviewed >
Jane: That’s a concrete number for speed, but then they immediately pivot to the risks because the literature is much weaker on long-term quality, maintainability, and how many defects end up in the final product >
Lu: The paper identifies hallucinations and incorrect generations as the most frequent failure mode reported across all sources, showing that plausible looking code can still be totally wrong >
Meng: Hallucinations are a big concern for production. If an AI generates something that looks right but is fundamentally insecure or doesn't meet the requirements, that’s a deployment killer.
Lalam: They also found that the most common things developers worry about regarding their experience in this process are trust calibration and adaptive reliance, which means how much you decide to trust the AI based on what you see >
Tom: That makes sense. People aren't just blindly trusting the output; they’re constantly adjusting their level of reliance depending on how complex or unfamiliar the task is >
Jane: And it also points out that there are concerns about cognitive load and cognitive debt, where developers might defer understanding a part of the code because they didn't fully grasp it during the initial generation phase >
The paper's improvements: Tom: Now, the authors suggest several ways to make this process more reliable. They are pushing for moving beyond just vague "vibe checks" and demanding structured verification pipelines >
Jane: They strongly recommend that developers use automated tests, linting, and runtime inspection to check the AI-generated code before you even try to integrate it into a working system >
Lu: They also talk about needing explicit context provisioning tactics. This means providing the model with all the relevant context, like existing documentation or specific requirements, so it doesn't guess what you mean >
Meng: Providing all that context sounds like extra friction for development time. How do you balance giving the AI everything versus keeping your workflow fast?
Lalam: They suggest developing robust iteration strategies where error-driven feedback is fed directly back into the prompt refinement process, instead of just making small review commits >
Tom: So, if you find an error during runtime, that error log should immediately become part of your next prompt to fix the output across those turns >
Jane: And they also talk about building in safeguards like human-in-the-loop reviews and sandboxing environments because these are what people reported as the most frequently recommended safety measures in their vibe coding workflows >
Conclusion: Tom: So to wrap up, the main point is that the effectiveness of vibe coding isn't just about how fast you can generate code; it hinges entirely on how rigorous your evaluation loop is >
Jane: It suggests that if you have a systematic approach—using defined acceptance criteria and automated testing—vibe coding becomes a productive feedback-driven process >
Lu: But they also clearly flag the verification gap: the speed of AI generation is often faster than the developer's capacity to verify correctness, security, and where that code actually comes from >
Meng: And for deployment at scale, it sounds like we need more than just good prompts. We need systems with built-in checks that can handle context drift and ensure provenance tracking for the generated code fragments >
Lalam: Exactly. The paper calls for studying vibe coding as an end-to-end sociotechnical workflow, not just isolated prompt-based generation, focusing on causal evidence and long-horizon maintainability >
Tom: That’s the core implication: we need to study this as a whole process rather than just looking at the AI output in isolation >
Jane: It gives us a roadmap for what future research should focus on, specifically needing more causal evidence about session dynamics and testing those specific safeguard mechanisms mentioned in the paper >
Lu: And they point out that since there are still so few long-horizon benchmarks, that’s a huge area where we need to develop standardized ways to assess these workflows accurately >
Meng: So if you're building something with this, the message is clear: don't just trust the first output. Build in the verification and iteration steps from day one >
Lalam: That’s what this multivocal literature review of vibe coding tells us about how to move forward.
Faculty of Information Technology and Communication Sciences, Tampere University
cs.SE
Submitted: 2026-07-22
Updated: 2026-11-05
Comments: Accepted in The Journal of Systems & Software (JSS)
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: A multivocal literature review (MLR) was conducted to integrate peer-reviewed and grey literature on vibe coding…
Key concepts
- Vibe Coding
- An intent-driven software development practice where a developer describes desired functionality using natural language prompts. This allows large language models (LLMs) to generate, refine, and revise code iteratively. It is defined as a loop of generation, evaluation, and revision.
- Validation and Evaluation Pipelines
- The process of checking AI-generated code using structured methods like automated tests or linting before integration. The literature shows this is the most common safeguard (72%) but empirical evidence on its actual effectiveness is still being tested.
- Trust Calibration and Adaptive Reliance
- A developer's ability to adjust how much they rely on the AI based on observed errors and task complexity. This mechanism helps manage trust, ensuring stricter verification for complex tasks while allowing greater reliance for routine ones.
- Verification Gap
- The mismatch between the speed of AI code generation and a developer's capacity to verify its correctness, security, and provenance. Without systematic evaluation, this gap increases the risk of introducing hallucinations and technical debt.
Terminology
Summary
A multivocal literature review (MLR) was conducted to integrate peer-reviewed and grey literature on vibe coding, synthesizing 47 sources (28 peer-reviewed and 19 grey) published between 2022 and October 2025 across eight research questions. p. 0 Vibe coding is defined as an intent-driven software development practice where a developer specifies desired functionality, constraints, and context primarily through natural language prompts, enabling large language models (LLMs) to generate, refine, and iteratively revise code artifacts. p. 0 This is consistently described not as a one-shot activity but as an iterative generation-evaluation-revision loop. p. 0
Key findings indicate that the developer's role fundamentally shifts from direct code authoring toward specification, supervision, and validation. The dominant workflow patterns observed are the use of validation and evaluation pipelines (72%) and chat-based iterative development loops (63%). p. 3 The most reported constructs related to developer experience are trust calibration and adaptive reliance (68%), followed by changes in cognitive load (38%), cognitive debt due to deferred understanding (24%), and flow effects (20%). p. 3
Empirical evidence strongly supports short-term gains in productivity and time-to-prototype, with 45% of sources reporting these benefits. p. 0 However, the literature remains significantly weaker regarding long-term quality, maintainability, defects, and reproducibility. p. 0 The most frequent failure mode identified is hallucinations and incorrect generations (28%). p. 3 While validation and review practices are the most frequently recommended safeguards (88%), empirical evidence on their actual effectiveness is under-tested. p. 0
A conceptual framework positions vibe coding as an iterative control system: a feedback-driven process where intent specification, AI output evaluation, and prompt refinement drive successive iterations. p. 0 The central message derived from the synthesis is that the effectiveness of vibe coding hinges less on generation speed and more on the rigor of the surrounding evaluation loop. p. 0
The literature suggests that when evaluation is systematic—involving defined acceptance criteria, automated testing, and structured prompt refinement—vibe coding functions as a productive feedback-driven process. p. 4 Conversely, superficial evaluation introduces risks such as hallucinations and technical debt. p. 7 The primary risk identified is a verification gap: the mismatch between the speed of AI generation and the developer's capacity to verify correctness, security, and provenance.
The evidence base is strongest for prototyping and early-stage development contexts (RQ6), while production use remains weakly evidenced. p. 0 Tool evidence shows Copilot and chat-based assistants as most visible tools, though visibility does not equate to demonstrated effectiveness due to sparse comparative evaluations (RQ7). p. 5 The overall evidence strength is moderate (34 sources). p. 9
The main research implication is that vibe coding should be studied as an end-to-end sociotechnical workflow rather than isolated prompt-based code generation. p. 4 Future work must prioritize causal evidence, focusing on session-level dynamics, evaluating the effectiveness of specific safeguard mechanisms, addressing domain skew in the evidence base, and studying long-horizon maintainability and trust calibration. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Evaluate generated code through structured verification pipelines rather than relying solely on
vibe
checks. This involves implementing automated tests, linting, and runtime inspection to detect errors introduced by AI-generated code before integration into a working system. -
Implement explicit context provisioning tactics to guide model responses and reduce ambiguity in generated outputs. Developers should provide contextual artifacts such as requirements, constraints, documentation, or existing code snippets to ensure the generated code is relevant and accurate.
-
Develop robust iteration strategies that explicitly incorporate error-driven feedback into the prompt refinement process. This includes a
iterate–prompt–patch loop
where runtime errors or logs are fed back into successive prompts to repair the output across turns, rather than relying on minimal review commits. -
Integrate trust calibration mechanisms that allow developers to adjust their reliance on model outputs based on observed errors and task complexity. This adaptive reliance is crucial for managing
trust calibration and adaptive reliance,
ensuring stricter verification for unfamiliar functionality than for routine tasks. -
Design systems with built-in safeguards such as human-in-the-loop review and sandboxing environments. These mechanisms should be used to confine operations or enforce disciplined review routines, which are reported as the most frequently recommended safeguards in vibe coding workflows.
-
Incorporate provenance tracking and traceability mechanisms into the development workflow to address licensing uncertainty. This ensures that the origin and legal status of generated code fragments can be verified, mitigating risks associated with
licensing and provenance uncertainty.
-
Enhance context management capabilities to prevent context drift during extended development sessions. This involves developing technical solutions for maintaining coherent state across multi-turn interactions, ensuring that earlier information is not lost or misinterpreted as the session progresses.
-
Establish standardized benchmarks capable of assessing realistic long-horizon development workflows. Since
lack of standardized evaluation frameworks and long-horizon benchmarks
is a major open challenge, the AI system should be designed to be evaluated against these missing benchmarks for productivity and quality measures.
Sources
- Vibe coding: programming through conversation with artificial intelligence
- Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
- Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
- Exploring Student-AI Interactions in Vibe Coding
- Academic Vibe Coding: Opportunities for Accelerating Research in an Era of Resource Constraint
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
- Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Coding
- A Survey of Vibe Coding with Large Language Models
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters