K2V: Generative Keyframing For Physically Plausible Video Generation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "K2V: Generative Keyframing For Physically Plausible Video Generation".
Jane: Here's the gist of the paper. Video diffusion models are able to generate samples with high-quality appearance and textural qualities; however, content produced by these methods is known to exhibit significant physical errors.
Tom: The authors propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module.
Jane: This approach allows for explicit reasoning in language and fine-grained control over the appearance of the video at important moments.
Tom: They then condition a video generation model with these keyframes, reducing the role of the diffusion model to interpolating between physically correct intermediate states rather than inferring an entire sequence with diffusion.
Paper summary: Tom: So looking at K2V: Generative Keyframing For Physically Plausible Video Generation, it seems like they've really nailed how to handle physical errors in diffusion-based video generation >
Jane: The main thing is that they shifted the burden of physical reasoning away from the video diffusion model and onto a language model module >
Lu: They achieved this by using a keyframe generation module where a language model plans the physical evolution of the scene, which then conditions an existing video generation model >
Meng: It's interesting how they manage to get that explicit reasoning in language while still getting high-quality visual outputs from the diffusion side >
Tom: The result on Physics-IQ Verified, with that improvement to fifty-five point zero for Wan2 point 2, shows they achieved a significant jump in quality and efficiency compared to prior work > <ref:2611.17324#pg1>
Jane: It means we’re moving toward systems where we can enforce physical constraints at specific moments rather than trying to get the whole sequence right all at once >
Lu: Their limitation is that while they can set moments in a video, they cannot currently determine the intermediate motion between those keyframes, which can still lead to some physical implausibilities in places like when balls are rolling >
Tom: That’s a real caveat. So the paper shows a powerful way to improve physical plausibility by using language models for planning and keyframe conditioning >
Conclusion: Tom: So, K2V is about taking these video diffusion models that look pretty great visually, but they mess up the physics all the time, and they fix that by using a separate language model to plan out keyframes first.
Jane: Right, so instead of the whole video model trying to guess how everything moves at once, this new approach uses the language model to figure out where things *should* be physically correct, and then it just interpolates between those known good moments.
Lu: The core idea is offloading physical reasoning from the diffusion process onto a keyframe generation module, which is driven by a language model. They found that video models are better at connecting two explicit keyframes than guessing the whole path from scratch in diffusion space.
Meng: From an engineering standpoint, it sounds like they’ve built a system where you condition your existing video generation model with these language-planned keyframes, rather than having the diffusion model handle all the planning. That’s a different way to structure the pipeline.
Tom: And the results on PhysicsIQ Verified are pretty clear: they took Wan two point two to fifty point zero, which is a big jump in quality for that specific test track. It also showed that this lighter version of the model, Wan two point two five billion parameters, is performing really well compared to the bigger one.
Jane: What this means for us watching videos right now is that we’re moving toward systems where you can control physical accuracy at important moments in a clip, like making sure a domino stays on track.
Lu: They also showed that the language model isn't just planning; it's doing the reasoning itself. For example, when looking at one clip, the language model was actually thinking about whether a chain of dominoes would cross a gap.
Meng: That level of reasoning is what we want to see in production, but how does this translate to real-time performance? I wonder what the latency is for generating those keyframes themselves.
Tom: Well, they have some limitations they pointed out: it can't currently figure out the motion *between* the keyframes perfectly. If things like a ball are rolling off course between two planned points, the model might still insert physically wrong motion there.
Jane: So it’s not perfect yet; it sets the important points but doesn't always smooth out every tiny physical detail in between those set moments. But that still gives us a much more plausible result overall than before.
Lu: Plus, they noted that the color can drift across the video, but you can fix that easily with a simple adjustment to how colors are mapped onto the frames.
Meng: I’m interested in how robust this is when we try to deploy it on high-traffic scenarios where things are constantly changing. That's where I think we'll see if this holds up under real stress.
Tom: It seems like K2V successfully shifted the heavy lifting of physical planning from the diffusion model onto a language model, and they’ve shown that this is a much more successful way to get physically plausible video generation today.
Jane: This moves us closer to generating videos that don't just look pretty but also make sense in the real world, which is a huge step for things like robotics and simulation.
FELIX O’MAHONY, DANIEL OLMEDA REINO, ROBERTO CIPOLLA, AYUSH TEWARI
University of Cambridge · Toyota Motor Europe
cs.CV
Submitted: 2026-11-12
Updated: 2026-11-12
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 94/100
The gist: Video diffusion models are able to generate samples with high-quality appearance and textural qualities; however, content produced by these methods is known to exhibit significant physical errors.
Key concepts
- Keyframe Generation Strategy
- This strategy separates the complex task of planning physical motion from the video generation process. Instead of a diffusion model trying to guess an entire sequence, a separate module plans explicit keyframes based on language and scene understanding. This simplifies the diffusion model's job to just smoothly connect these pre-planned points.
- Keyframe Module Implementation
- This is an agentic pipeline where a powerful language model acts as a central planner. It takes the video description and generates a sequence of keyframes, each accompanied by instructions on how to transition to the next frame. This allows the model to reason about the scene's physical evolution before generating visuals.
- Conditioning Mechanism
- The generated keyframes are used to guide the video generation model. The authors insert these specific keyframes into a padded image tensor at precise time points within the video sequence, ensuring that intermediate conditioning only works correctly when applied at the end of a fixed temporal block.
Terminology
Summary
Video diffusion models are able to generate samples with high-quality appearance and textural qualities; however, content produced by these methods is known to exhibit significant physical errors. p. 0 Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods
is known to exhibit significant physical errors [Motamed et al. However, content produced by these methods
is known to exhibit significant physical errors [Motamed et al. We
propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which
allows for explicit reasoning in language, and fine-grained control over the
appearance of the video at important moments in the video. However, content produced by these methods
is known to exhibit significant physical errors [Motamed et al. We
propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which
allows for explicit reasoning in language, and fine-grained control over the
appearance of the video at important moments in the video. We
propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which
allows for explicit reasoning in language, and fine-grained control over the
appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of
the diffusion model to the simpler task of interpolating between physicallycorrect intermediate states, rather than inferring an entire sequence with
diffusion. We then condition a video generation model with these keyframes, reducing the role of
the diffusion model to the simpler task of interpolating between physicallycorrect intermediate states, rather than inferring an entire sequence with
diffusion. Results demonstrate that our approach significantly outperforms
existing approaches, achieving state-of-the-art performance on the PhysicsIQ Verified benchmark [Rädsch et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Keyframe Generation Strategy
The authors introduce Keyframes to Video (K2V), a novel approach to video generation which simplifies the task required of a video model to interpolation between keyframes, offloading physical inference to a separate keyframe generation module. Their first insight is that video models are more successful at reconstructing the physical motion connecting two explicit keyframes than at inferring plausible physical motion from a starting state entirely in diffusion space. p. 0 In this paper we introduce Keyframes to Video (K2V), a novel
approach to video generation which simplifies the task required
of a video model to interpolation between keyframes, offloading
physical inference to a separate keyframe generation module. Our
first insight is that video models are more successful at reconstructing the physical motion connecting two explicit keyframes than at
inferring plausible physical motion from a starting state entirely in
diffusion space. Our
first insight is that video models are more successful at reconstructing the physical motion connecting two explicit keyframes than at
inferring plausible physical motion from a starting state entirely in
diffusion space. We identify a simple training-free approach to condition an existing video generation model, Wan2. We identify a simple training-free approach to condition an existing video generation model, Wan2. 2 [Wan et al. 2 [Wan et al. 2025],
to perform the task of keyframe interpolation. 2025],
to perform the task of keyframe interpolation. Our second insight
is that the physical reasoning required to plan those keyframes is
already present in language models. Our second insight
is that the physical reasoning required to plan those keyframes is
already present in language models. We therefore introduce a novel
keyframe generation module, in which a language model plans the
physical evolution of the scene. We therefore introduce a novel
keyframe generation module, in which a language model plans the
physical evolution of the scene. This approach redistributes the load
of physical planning away from a video diffusion model and onto a
language model, which can then finely control an image edit model
to produce the intended physical effect.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Keyframe Module Implementation
The keyframe module is an agentic pipeline where a central planning agent, Gemini 3.8 Flash [DeepMind 2026b], orchestrates the keyframe generation. p. 3 A central planning agent, Gemini 3. 8 Flash [DeepMind 2026b], orchestrates
the keyframe generation. 8 Flash [DeepMind 2026b], orchestrates
the keyframe generation. The planning agent is given the original
caption for the video, a task-agnostic prompt, and, optionally, the
image intended for the video generation model. The agent
generates each keyframe in sequence, enforcing the causal nature
of physically-plausible generation. For each keyframe, the agent writes an editing instruction describing how the scene should change to reach the next keyframe.Where the paper says it
Where the paper says it
Where the paper says it
Conditioning Mechanism
The authors condition Wan2.2 for multiple frames at several points in a given video by constructing a padded image tensor where each keyframe is inserted at specific timestamps. p. 3 However, we observe
that it is trivially possible to condition Wan2. 2 for multiple frames
at several points in a given video. , 𝑁} (3)
Where 𝑓 is the frame rate of the generator in frames per second. The reason that frames are only inserted every 𝐿𝑇 frames is because we note in our experiments that intermediate conditioning
only works if the conditioning is applied at the end of a temporal
block of 𝐿𝑇 frames.Where the paper says it
Where the paper says it
Performance and Scaling
K2V achieves state-of-the-art performance on PhysicsIQ Verified [Rädsch et al. 2026], raising Wan2. p. 1 We also report state-of-the-art results
on PhyGenBench [Meng et al. 2025], raising the state of the art
from 0. 2-14B
from 32. 0 on the Physics-IQ Verified image-to-video track. 0 on the Physics-IQ Verified image-to-video track. The gain also extends to the efficiency of generation: the much
lighter Wan2. The gain also extends to the efficiency of generation: the much
lighter Wan2. 2-5B reaches 50. 2-5B reaches 50. 0 when supported by our method, substantially outperforming the unmodified 14B backbone at roughly
a third of the parameters. 0 when supported by our method, substantially outperforming the unmodified 14B backbone at roughly
a third of the parameters. We also report state-of-the-art results
on PhyGenBench [Meng et al. We also report state-of-the-art results
on PhyGenBench [Meng et al. 2025], raising the state of the art
from 0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Evaluation and Analysis
The approach is highly effective at producing physically plausible generations, reaching state-of-the-art performance by a significant margin on PhysicsIQ Verified [Rädsch et al. 2026]. The keyframe density plot shows that the model does use keyframes at a range of moments within videos. p. 5 The keyframe density plot shown in the lower-left
panel of Figure 7 highlights that the model does use keyframes at a
range of moments within videos. The right panel of Figure 7 shows that the language model also
spends more time on scenes which are particularly complex. The keyframe density plot shown in the lower-left
panel of Figure 7 highlights that the model does use keyframes at a
range of moments within videos. The right panel of Figure 7 shows that the language model also
spends more time on scenes which are particularly complex. must reason about whether the domino chain will cross the gap. The reasoning trace extracted from the language model for this clip
reveals the model considering this problem: “I can see that the gap
between domino 4 and domino 5 is wide enough for the domino 5 to
fall without hitting domino 4 and for it to be a valid demonstration”.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Work
The authors note that while their method can set moments in a video, it cannot currently determine intermediate motion. This can lead to remaining physical implausibilities where the model inserts incorrect motion. p. 9 2-5B, which requires just 4 minutes per
generation, with our method outperforms the 14B model, leading
to an effective acceleration of accurate generation. Although our method does resolve almost all physical plausibility
problems in our test sets, there remain some instances where the
language model fails to reason correctly about the physics of a scene,
leading to an incorrect generation. There is a black pipe that points from
the left side to the right side of the table. The brown tennis ball rolls straight out of
the black pipe and hits the rubber duck. We ablate the benchmark caption, instead
providing the original caption which was provided with the input
frame, which shows that densely prompting the motion in a scene
also supports more accurate video generation. Finally, we ablate color correction, showing that the use of
keyframes does lead to color varying throughout a video, but that
this can be rectified with a simple affine transformation of the frame
colors.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In conclusion, they proposed K2V, a novel approach for video generation which enforces adherence to set keyframes which are generated with a custom language model-backed keyframe module. They proposed their method to redistribute the physical generation task required for accurate video generation from within video diffusion onto a language model. p. 0 We
propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which
allows for explicit reasoning in language, and fine-grained control over the
appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of
the diffusion model to the simpler task of interpolating between physicallycorrect intermediate states, rather than inferring an entire sequence with
diffusion. 4 Results
We select our experiments to show the broad applicability of our
approach in first frame and text conditioned settings. Our results
highlight that, both qualitatively and quantitatively, our approach
outperforms existing methods and produces videos which correctly
follow physical laws. Through comprehensive experiments we show that K2V is highly
effective at producing physically plausible generations, reaching
state-of-the-art performance by a significant margin on PhysicsIQ Verified [Rädsch et al. 2026], a benchmark designed specifically
to evaluate physical plausibility. Results demonstrate that our approach significantly outperforms
existing approaches, achieving state-of-the-art performance on the PhysicsIQ Verified benchmark [Rädsch et al. 2026] and PhyGenBench [Meng et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
AI Disclosure
In this work, they used generative AI tools for implementing their approach in code and for providing feedback on research methodology. They additionally used generative AI tools for suggesting experimental parameters and proposing related work. p. 1 • State-of-the-art results on Physics-IQ Verified and PhyGenBench, together with an analysis showing that planning
effort is spent in proportion to the physical complexity of a
scene. 2 Related Work
2. • State-of-the-art results on Physics-IQ Verified and PhyGenBench, together with an analysis showing that planning
effort is spent in proportion to the physical complexity of a
scene. 2 Related Work
2.Where the paper says it
Where the paper says it
Ethics and Privacy Statement
The rapid proliferation of generative video models has had widespread effects including significant beneficial progress in fields such as computer graphics, robotics, and physical intelligence. However, some effects have been negative such as the distribution of content intended to deceive viewers. p. 0 Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods
is known to exhibit significant physical errors [Motamed et al. Although our method does resolve almost all physical plausibility
problems in our test sets, there remain some instances where the
language model fails to reason correctly about the physics of a scene,
leading to an incorrect generation. For example, in one scene a stack
of blocks was reasoned incorrectly to be stable, whereas in fact
Keyframes Provided
Physics-IQ Score
Generated Keyframes
"Make the inner petals
all white and orange"
Generated Video
t=0.Where the paper says it
Where the paper says it
Acknowledgments
The authors thank Jeffrey Hu for reviewing the manuscript and providing valuable feedback. Research was supported by the NVIDIA Academic Grant Program using NVIDIA RTX PRO 6000 Blackwell GPUs and funded in part by Toyota Motor Europe. p. 0 K2V: Generative Keyframing For Physically Plausible Video Generation
FELIX O’MAHONY✉, University of Cambridge, UK
DANIEL OLMEDA REINO, Toyota Motor Europe, Belgium
ROBERTO CIPOLLA, University of Cambridge, UK
AYUSH TEWARI, University of Cambridge, UK
Physics-IQ Verified Dominos With Space
50
40
30
20
10
0
GT
Ours
Wan2. 2
Siphon
K2V + Wan2.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Keyframe generation module can be implemented as an agentic pipeline where a planning agent orchestrates keyframe generation through tools like render edit, view image, and evaluate render to ensure physical plausibility. This allows the system to
reason physically, from what the frame actually shows
andchoose them sparsely,
concentrating compute on complex scenes. -
Conditioning of video generation models can be extended via a training-free mechanism where keyframes are pinned at arbitrary interior latent frames by replacing the grey frame with the generated keyframe, which is a
trivial extension of this.
This allows conditioning Wan2.2 without further training or fine tuning, leveraging its pre-trained capabilities. -
The system can produce videos that
correct errors made by the image edit model
because the agentic pipeline includes tools toreview and correct errors made by the image edit model, or to refine its image edits.
This allows for iterative refinement of keyframes and their resulting video without incurring the cost of producing a complete sample for every revision. -
The system can effectively allocate planning effort based on scene complexity because
planning effort is spent in proportion to the physical complexity of a scene,
as shown by the language model spending more time on scenes requiring careful planning. This results in state-of-the-art performance by focusing compute where it is most needed.
Abstract
Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.
Sources
- Qwen3-VL Technical Report
- VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
- Authoring for Living Worlds: Tool-Constrained LLM Agents for Executable Multi-Actor Scenarios
- NEWTON: Agentic Planning for Physically Grounded Video Generation
- LTX-Video: Realtime Video Latent Diffusion
- S2PD: Serial-to-Parallel Diffusion for Physically and Logically Consistent Video Generation
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
- PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation
- Exploring the Evolution of Physics Cognition in Video Generation: A Survey
- The Serial Scaling Hypothesis
- Cosmos 3: Omnimodal World Models for Physical AI
- Physics-IQ Verified
- Wan: Open and Advanced Large-Scale Video Generative Models
- A Mechanistic View on Video Generation as World Models: State and Dynamics
- Chain of Event-Centric Causal Thought for Physically Plausible Video Generation
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
- UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs