Vibe Coding in Software Development: A Multivocal Literature Review
summary
The gist
A multivocal literature review (MLR) was conducted to integrate peer-reviewed and grey literature on vibe coding…
In short
A literature review synthesized 47 sources on 'vibe coding,' an intent-driven practice using natural language prompts to guide AI code generation. Findings show developers shift to specification, with success depending on rigorous evaluation loops rather than just speed. The main risk is a verification gap, where superficial checks fail to prevent hallucinations and technical debt.
Key concepts
- Vibe Coding
- An intent-driven software development practice where a developer describes desired functionality using natural language prompts. This allows large language models (LLMs) to generate, refine, and revise code iteratively. It is defined as a loop of generation, evaluation, and revision.
- Validation and Evaluation Pipelines
- The process of checking AI-generated code using structured methods like automated tests or linting before integration. The literature shows this is the most common safeguard (72%) but empirical evidence on its actual effectiveness is still being tested.
- Trust Calibration and Adaptive Reliance
- A developer's ability to adjust how much they rely on the AI based on observed errors and task complexity. This mechanism helps manage trust, ensuring stricter verification for complex tasks while allowing greater reliance for routine ones.
- Verification Gap
- The mismatch between the speed of AI code generation and a developer's capacity to verify its correctness, security, and provenance. Without systematic evaluation, this gap increases the risk of introducing hallucinations and technical debt.
Terminology used across episodes
This episode discusses
- Vibe Coding in Software Development: A Multivocal Literature Review · Paper Radio
- Vibe coding: programming through conversation with artificial intelligence
- Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
- Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
- Exploring Student-AI Interactions in Vibe Coding
- Academic Vibe Coding: Opportunities for Accelerating Research in an Era of Resource Constraint
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
- Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Coding
- A Survey of Vibe Coding with Large Language Models
The paper
Vibe Coding in Software Development: A Multivocal Literature Review · Read on arXiv
Faculty of Information Technology and Communication Sciences, Tampere University
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Vibe Coding in Software Development".
Jane: Here's the gist of the paper. A multivocal literature review (MLR) was conducted to integrate peer-reviewed and grey literature on vibe coding, synthesizing 47 sources (28 peer-reviewed and 19 grey) published between 2022 and October 2025 across eight research questions.
Tom: Vibe coding is defined as an intent-driven software development practice where a developer specifies desired functionality, constraints, and context primarily through natural language prompts, enabling large language models (LLMs) to generate, refine, and iteratively revise code artifacts.
Jane: This is consistently described not as a one-shot activity but as an iterative generation-evaluation-revision loop.
Tom: Key findings indicate that the developer's role fundamentally shifts from direct code authoring toward specification, supervision, and validation.
Title and authors: Tom: The paper starts by introducing this whole concept of vibe coding as a specific software development practice that relies on natural language prompts for the large language models to create code >
Jane: And it points out that the evidence around this practice is really scattered across academic papers and also in things like blogs and industry reports, which is why they did this multivocal literature review to get a fuller picture >
Lu: They looked at forty-seven sources published between two thousand twenty-two and October twenty twenty-five, mixing peer-reviewed studies with grey literature to capture the current state of practice >
Meng: Forty seven sources sounds like a lot of data. Does that mean we’re looking at a really broad range of how people are using these models, or is it mostly just one specific way they are doing things?
Lalam: It’s about capturing the state of practice as well as the state of research, so it tries to bridge that gap between what’s being formally studied and what’s actually happening in the day-to-day coding environment >
Tom: So, fundamentally, they are mapping out how this interaction between a human and an AI is structured when someone uses natural language to direct code generation >
Jane: It really frames it as this loop where the developer states intent, the model generates code, then you evaluate it by running and inspecting that result, and then you refine your next prompt based on what you observe >
The paper's summary: Tom: The main takeaway from this paper is that when vibe coding is done right, it shows short-term gains in productivity and time to prototype for about forty-five percent of the sources they reviewed >
Jane: That’s a concrete number for speed, but then they immediately pivot to the risks because the literature is much weaker on long-term quality, maintainability, and how many defects end up in the final product >
Lu: The paper identifies hallucinations and incorrect generations as the most frequent failure mode reported across all sources, showing that plausible looking code can still be totally wrong >
Meng: Hallucinations are a big concern for production. If an AI generates something that looks right but is fundamentally insecure or doesn't meet the requirements, that’s a deployment killer.
Lalam: They also found that the most common things developers worry about regarding their experience in this process are trust calibration and adaptive reliance, which means how much you decide to trust the AI based on what you see >
Tom: That makes sense. People aren't just blindly trusting the output; they’re constantly adjusting their level of reliance depending on how complex or unfamiliar the task is >
Jane: And it also points out that there are concerns about cognitive load and cognitive debt, where developers might defer understanding a part of the code because they didn't fully grasp it during the initial generation phase >
The paper's improvements: Tom: Now, the authors suggest several ways to make this process more reliable. They are pushing for moving beyond just vague "vibe checks" and demanding structured verification pipelines >
Jane: They strongly recommend that developers use automated tests, linting, and runtime inspection to check the AI-generated code before you even try to integrate it into a working system >
Lu: They also talk about needing explicit context provisioning tactics. This means providing the model with all the relevant context, like existing documentation or specific requirements, so it doesn't guess what you mean >
Meng: Providing all that context sounds like extra friction for development time. How do you balance giving the AI everything versus keeping your workflow fast?
Lalam: They suggest developing robust iteration strategies where error-driven feedback is fed directly back into the prompt refinement process, instead of just making small review commits >
Tom: So, if you find an error during runtime, that error log should immediately become part of your next prompt to fix the output across those turns >
Jane: And they also talk about building in safeguards like human-in-the-loop reviews and sandboxing environments because these are what people reported as the most frequently recommended safety measures in their vibe coding workflows >
Conclusion: Tom: So to wrap up, the main point is that the effectiveness of vibe coding isn't just about how fast you can generate code; it hinges entirely on how rigorous your evaluation loop is >
Jane: It suggests that if you have a systematic approach—using defined acceptance criteria and automated testing—vibe coding becomes a productive feedback-driven process >
Lu: But they also clearly flag the verification gap: the speed of AI generation is often faster than the developer's capacity to verify correctness, security, and where that code actually comes from >
Meng: And for deployment at scale, it sounds like we need more than just good prompts. We need systems with built-in checks that can handle context drift and ensure provenance tracking for the generated code fragments >
Lalam: Exactly. The paper calls for studying vibe coding as an end-to-end sociotechnical workflow, not just isolated prompt-based generation, focusing on causal evidence and long-horizon maintainability >
Tom: That’s the core implication: we need to study this as a whole process rather than just looking at the AI output in isolation >
Jane: It gives us a roadmap for what future research should focus on, specifically needing more causal evidence about session dynamics and testing those specific safeguard mechanisms mentioned in the paper >
Lu: And they point out that since there are still so few long-horizon benchmarks, that’s a huge area where we need to develop standardized ways to assess these workflows accurately >
Meng: So if you're building something with this, the message is clear: don't just trust the first output. Build in the verification and iteration steps from day one >
Lalam: That’s what this multivocal literature review of vibe coding tells us about how to move forward.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game