K2V: Generative Keyframing For Physically Plausible Video Generation

arXiv:2611.17324 · cs.CV · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "K2V: Generative Keyframing For Physically Plausible Video Generation".

Jane: Here's the gist of the paper. Video diffusion models are able to generate samples with high-quality appearance and textural qualities; however, content produced by these methods is known to exhibit significant physical errors.

Tom: The authors propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module.

Jane: This approach allows for explicit reasoning in language and fine-grained control over the appearance of the video at important moments.

Tom: They then condition a video generation model with these keyframes, reducing the role of the diffusion model to interpolating between physically correct intermediate states rather than inferring an entire sequence with diffusion.

Paper summary: Tom: So looking at K2V: Generative Keyframing For Physically Plausible Video Generation, it seems like they've really nailed how to handle physical errors in diffusion-based video generation >

Jane: The main thing is that they shifted the burden of physical reasoning away from the video diffusion model and onto a language model module >

Lu: They achieved this by using a keyframe generation module where a language model plans the physical evolution of the scene, which then conditions an existing video generation model >

Meng: It's interesting how they manage to get that explicit reasoning in language while still getting high-quality visual outputs from the diffusion side >

Tom: The result on Physics-IQ Verified, with that improvement to fifty-five point zero for Wan2 point 2, shows they achieved a significant jump in quality and efficiency compared to prior work > <ref:2611.17324#pg1>

Jane: It means we’re moving toward systems where we can enforce physical constraints at specific moments rather than trying to get the whole sequence right all at once >

Lu: Their limitation is that while they can set moments in a video, they cannot currently determine the intermediate motion between those keyframes, which can still lead to some physical implausibilities in places like when balls are rolling >

Tom: That’s a real caveat. So the paper shows a powerful way to improve physical plausibility by using language models for planning and keyframe conditioning >

Conclusion: Tom: So, K2V is about taking these video diffusion models that look pretty great visually, but they mess up the physics all the time, and they fix that by using a separate language model to plan out keyframes first.

Jane: Right, so instead of the whole video model trying to guess how everything moves at once, this new approach uses the language model to figure out where things *should* be physically correct, and then it just interpolates between those known good moments.

Lu: The core idea is offloading physical reasoning from the diffusion process onto a keyframe generation module, which is driven by a language model. They found that video models are better at connecting two explicit keyframes than guessing the whole path from scratch in diffusion space.

Meng: From an engineering standpoint, it sounds like they’ve built a system where you condition your existing video generation model with these language-planned keyframes, rather than having the diffusion model handle all the planning. That’s a different way to structure the pipeline.

Tom: And the results on PhysicsIQ Verified are pretty clear: they took Wan two point two to fifty point zero, which is a big jump in quality for that specific test track. It also showed that this lighter version of the model, Wan two point two five billion parameters, is performing really well compared to the bigger one.

Jane: What this means for us watching videos right now is that we’re moving toward systems where you can control physical accuracy at important moments in a clip, like making sure a domino stays on track.

Lu: They also showed that the language model isn't just planning; it's doing the reasoning itself. For example, when looking at one clip, the language model was actually thinking about whether a chain of dominoes would cross a gap.

Meng: That level of reasoning is what we want to see in production, but how does this translate to real-time performance? I wonder what the latency is for generating those keyframes themselves.

Tom: Well, they have some limitations they pointed out: it can't currently figure out the motion *between* the keyframes perfectly. If things like a ball are rolling off course between two planned points, the model might still insert physically wrong motion there.

Jane: So it’s not perfect yet; it sets the important points but doesn't always smooth out every tiny physical detail in between those set moments. But that still gives us a much more plausible result overall than before.

Lu: Plus, they noted that the color can drift across the video, but you can fix that easily with a simple adjustment to how colors are mapped onto the frames.

Meng: I’m interested in how robust this is when we try to deploy it on high-traffic scenarios where things are constantly changing. That's where I think we'll see if this holds up under real stress.

Tom: It seems like K2V successfully shifted the heavy lifting of physical planning from the diffusion model onto a language model, and they’ve shown that this is a much more successful way to get physically plausible video generation today.

Jane: This moves us closer to generating videos that don't just look pretty but also make sense in the real world, which is a huge step for things like robotics and simulation.

FELIX O’MAHONY, DANIEL OLMEDA REINO, ROBERTO CIPOLLA, AYUSH TEWARI

University of Cambridge · Toyota Motor Europe

cs.CV

Submitted: 2026-11-12

Updated: 2026-11-12

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 94/100

The gist: Video diffusion models are able to generate samples with high-quality appearance and textural qualities; however, content produced by these methods is known to exhibit significant physical errors.

Key concepts

Keyframe Generation Strategy
This strategy separates the complex task of planning physical motion from the video generation process. Instead of a diffusion model trying to guess an entire sequence, a separate module plans explicit keyframes based on language and scene understanding. This simplifies the diffusion model's job to just smoothly connect these pre-planned points.
Keyframe Module Implementation
This is an agentic pipeline where a powerful language model acts as a central planner. It takes the video description and generates a sequence of keyframes, each accompanied by instructions on how to transition to the next frame. This allows the model to reason about the scene's physical evolution before generating visuals.
Conditioning Mechanism
The generated keyframes are used to guide the video generation model. The authors insert these specific keyframes into a padded image tensor at precise time points within the video sequence, ensuring that intermediate conditioning only works correctly when applied at the end of a fixed temporal block.

Terminology

Summary

Video diffusion models are able to generate samples with high-quality appearance and textural qualities; however, content produced by these methods is known to exhibit significant physical errors. p. 0

Where the paper says it

Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors [Motamed et al.

The authors propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module. p. 0
Where the paper says it

However, content produced by these methods is known to exhibit significant physical errors [Motamed et al. We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video.

This approach allows for explicit reasoning in language and fine-grained control over the appearance of the video at important moments. p. 0
Where the paper says it

However, content produced by these methods is known to exhibit significant physical errors [Motamed et al. We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video.

They then condition a video generation model with these keyframes, reducing the role of the diffusion model to interpolating between physically correct intermediate states rather than inferring an entire sequence with diffusion. p. 0
Where the paper says it

We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physicallycorrect intermediate states, rather than inferring an entire sequence with diffusion.

Results demonstrate that their approach significantly outperforms existing methods, achieving state-of-the-art performance on the PhysicsIQ Verified benchmark. p. 0
Where the paper says it

We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physicallycorrect intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the PhysicsIQ Verified benchmark [Rädsch et al.

Keyframe Generation Strategy

The authors introduce Keyframes to Video (K2V), a novel approach to video generation which simplifies the task required of a video model to interpolation between keyframes, offloading physical inference to a separate keyframe generation module. Their first insight is that video models are more successful at reconstructing the physical motion connecting two explicit keyframes than at inferring plausible physical motion from a starting state entirely in diffusion space. p. 0

Where the paper says it

In this paper we introduce Keyframes to Video (K2V), a novel approach to video generation which simplifies the task required of a video model to interpolation between keyframes, offloading physical inference to a separate keyframe generation module. Our first insight is that video models are more successful at reconstructing the physical motion connecting two explicit keyframes than at inferring plausible physical motion from a starting state entirely in diffusion space.

They identify a simple training-free approach to condition an existing video generation model, Wan2. p. 0
Where the paper says it

Our first insight is that video models are more successful at reconstructing the physical motion connecting two explicit keyframes than at inferring plausible physical motion from a starting state entirely in diffusion space. We identify a simple training-free approach to condition an existing video generation model, Wan2.

2 [Wan et al. p. 0
Where the paper says it

We identify a simple training-free approach to condition an existing video generation model, Wan2. 2 [Wan et al.

2025], to perform the task of keyframe interpolation. p. 0
Where the paper says it

2 [Wan et al. 2025], to perform the task of keyframe interpolation.

Their second insight is that the physical reasoning required to plan those keyframes is already present in language models. p. 0
Where the paper says it

2025], to perform the task of keyframe interpolation. Our second insight is that the physical reasoning required to plan those keyframes is already present in language models.

They introduce a novel keyframe generation module in which a language model plans the physical evolution of the scene. p. 0
Where the paper says it

Our second insight is that the physical reasoning required to plan those keyframes is already present in language models. We therefore introduce a novel keyframe generation module, in which a language model plans the physical evolution of the scene.

This approach redistributes the load of physical planning away from a video diffusion model and onto a language model. p. 0
Where the paper says it

We therefore introduce a novel keyframe generation module, in which a language model plans the physical evolution of the scene. This approach redistributes the load of physical planning away from a video diffusion model and onto a language model, which can then finely control an image edit model to produce the intended physical effect.

Keyframe Module Implementation

The keyframe module is an agentic pipeline where a central planning agent, Gemini 3.8 Flash [DeepMind 2026b], orchestrates the keyframe generation. p. 3

Where the paper says it

A central planning agent, Gemini 3. 8 Flash [DeepMind 2026b], orchestrates the keyframe generation.

This planning agent is given the original caption for the video, a task-agnostic prompt, and optionally, the image intended for the video generation model. p. 3
Where the paper says it

8 Flash [DeepMind 2026b], orchestrates the keyframe generation. The planning agent is given the original caption for the video, a task-agnostic prompt, and, optionally, the image intended for the video generation model.

The agent generates each keyframe in sequence by imagining how the scene will evolve and writing an editing instruction describing how to reach the next keyframe. p. 3
Where the paper says it

The agent generates each keyframe in sequence, enforcing the causal nature of physically-plausible generation. For each keyframe, the agent writes an editing instruction describing how the scene should change to reach the next keyframe.

Conditioning Mechanism

The authors condition Wan2.2 for multiple frames at several points in a given video by constructing a padded image tensor where each keyframe is inserted at specific timestamps. p. 3

Where the paper says it

However, we observe that it is trivially possible to condition Wan2. 2 for multiple frames at several points in a given video.

Specifically, the padded image tensor I'[n] is constructed by setting the frame at each keyframe timestamp to the corresponding generated keyframe K i [Wan et al. 2025]. The reason for inserting frames every L T frames is that intermediate conditioning only works if the conditioning is applied at the end of a temporal block of L T frames. p. 4
Where the paper says it

, 𝑁} (3) Where 𝑓 is the frame rate of the generator in frames per second. The reason that frames are only inserted every 𝐿𝑇 frames is because we note in our experiments that intermediate conditioning only works if the conditioning is applied at the end of a temporal block of 𝐿𝑇 frames.

Performance and Scaling

K2V achieves state-of-the-art performance on PhysicsIQ Verified [Rädsch et al. 2026], raising Wan2. p. 1

Where the paper says it

We also report state-of-the-art results on PhyGenBench [Meng et al. 2025], raising the state of the art from 0.

2 from 32.2 to 55.0 on the Physics-IQ Verified image-to-video track. p. 1
Where the paper says it

2-14B from 32. 0 on the Physics-IQ Verified image-to-video track.

This gain extends to the efficiency of generation, as the much lighter Wan2. p. 1
Where the paper says it

0 on the Physics-IQ Verified image-to-video track. The gain also extends to the efficiency of generation: the much lighter Wan2.

2-5B reaches 50. p. 1
Where the paper says it

The gain also extends to the efficiency of generation: the much lighter Wan2. 2-5B reaches 50.

0 when supported by their method, substantially outperforming the unmodified 14B backbone at roughly a third of the parameters. p. 1
Where the paper says it

2-5B reaches 50. 0 when supported by our method, substantially outperforming the unmodified 14B backbone at roughly a third of the parameters.

They also report state-of-the-art results on PhyGenBench [Meng et al. p. 1
Where the paper says it

0 when supported by our method, substantially outperforming the unmodified 14B backbone at roughly a third of the parameters. We also report state-of-the-art results on PhyGenBench [Meng et al.

2025], raising the state of the art from 0. p. 1
Where the paper says it

We also report state-of-the-art results on PhyGenBench [Meng et al. 2025], raising the state of the art from 0.

56 to 0.63.

Evaluation and Analysis

The approach is highly effective at producing physically plausible generations, reaching state-of-the-art performance by a significant margin on PhysicsIQ Verified [Rädsch et al. 2026]. The keyframe density plot shows that the model does use keyframes at a range of moments within videos. p. 5

Where the paper says it

The keyframe density plot shown in the lower-left panel of Figure 7 highlights that the model does use keyframes at a range of moments within videos. The right panel of Figure 7 shows that the language model also spends more time on scenes which are particularly complex.

The language model also spends more time on scenes which are particularly complex. p. 5
Where the paper says it

The keyframe density plot shown in the lower-left panel of Figure 7 highlights that the model does use keyframes at a range of moments within videos. The right panel of Figure 7 shows that the language model also spends more time on scenes which are particularly complex.

The reasoning trace extracted from the language model for one clip reveals the model considering whether the domino chain will cross a gap. p. 6
Where the paper says it

must reason about whether the domino chain will cross the gap. The reasoning trace extracted from the language model for this clip reveals the model considering this problem: “I can see that the gap between domino 4 and domino 5 is wide enough for the domino 5 to fall without hitting domino 4 and for it to be a valid demonstration”.

Limitations and Future Work

The authors note that while their method can set moments in a video, it cannot currently determine intermediate motion. This can lead to remaining physical implausibilities where the model inserts incorrect motion. p. 9

Where the paper says it

2-5B, which requires just 4 minutes per generation, with our method outperforms the 14B model, leading to an effective acceleration of accurate generation. Although our method does resolve almost all physical plausibility problems in our test sets, there remain some instances where the language model fails to reason correctly about the physics of a scene, leading to an incorrect generation.

They observe this most strongly in scenes with balls rolling, where a ball follows a trajectory which is not straight between keyframes. p. 6
Where the paper says it

There is a black pipe that points from the left side to the right side of the table. The brown tennis ball rolls straight out of the black pipe and hits the rubber duck.

Furthermore, the use of keyframes does lead to color varying throughout a video, but this can be rectified with a simple affine transformation of the frame colors. p. 9
Where the paper says it

We ablate the benchmark caption, instead providing the original caption which was provided with the input frame, which shows that densely prompting the motion in a scene also supports more accurate video generation. Finally, we ablate color correction, showing that the use of keyframes does lead to color varying throughout a video, but that this can be rectified with a simple affine transformation of the frame colors.

Conclusion

In conclusion, they proposed K2V, a novel approach for video generation which enforces adherence to set keyframes which are generated with a custom language model-backed keyframe module. They proposed their method to redistribute the physical generation task required for accurate video generation from within video diffusion onto a language model. p. 0

Where the paper says it

We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physicallycorrect intermediate states, rather than inferring an entire sequence with diffusion.

Through experiments, they show that their approach is significantly more successful at generating videos which are physically plausible than any existing video generators. p. 4
Where the paper says it

4 Results We select our experiments to show the broad applicability of our approach in first frame and text conditioned settings. Our results highlight that, both qualitatively and quantitatively, our approach outperforms existing methods and produces videos which correctly follow physical laws.

On the Physics-IQ Verified benchmark, they demonstrate state-of-the-art results by a significant margin. p. 1
Where the paper says it

Through comprehensive experiments we show that K2V is highly effective at producing physically plausible generations, reaching state-of-the-art performance by a significant margin on PhysicsIQ Verified [Rädsch et al. 2026], a benchmark designed specifically to evaluate physical plausibility.

On PhyGenBench, their method outperforms all comparable approaches in all physical categories tested. p. 0
Where the paper says it

Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the PhysicsIQ Verified benchmark [Rädsch et al. 2026] and PhyGenBench [Meng et al.

AI Disclosure

In this work, they used generative AI tools for implementing their approach in code and for providing feedback on research methodology. They additionally used generative AI tools for suggesting experimental parameters and proposing related work. p. 1

Where the paper says it

• State-of-the-art results on Physics-IQ Verified and PhyGenBench, together with an analysis showing that planning effort is spent in proportion to the physical complexity of a scene. 2 Related Work 2.

They have reviewed all AI-assisted work and take responsibility for the final content of this work. p. 1
Where the paper says it

• State-of-the-art results on Physics-IQ Verified and PhyGenBench, together with an analysis showing that planning effort is spent in proportion to the physical complexity of a scene. 2 Related Work 2.

Ethics and Privacy Statement

The rapid proliferation of generative video models has had widespread effects including significant beneficial progress in fields such as computer graphics, robotics, and physical intelligence. However, some effects have been negative such as the distribution of content intended to deceive viewers. p. 0

Where the paper says it

Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors [Motamed et al.

Research which intends to improve the physical plausibility of generated content could make detection of generated content more difficult. p. 9
Where the paper says it

Although our method does resolve almost all physical plausibility problems in our test sets, there remain some instances where the language model fails to reason correctly about the physics of a scene, leading to an incorrect generation. For example, in one scene a stack of blocks was reasoned incorrectly to be stable, whereas in fact Keyframes Provided Physics-IQ Score Generated Keyframes "Make the inner petals all white and orange" Generated Video t=0.

Acknowledgments

The authors thank Jeffrey Hu for reviewing the manuscript and providing valuable feedback. Research was supported by the NVIDIA Academic Grant Program using NVIDIA RTX PRO 6000 Blackwell GPUs and funded in part by Toyota Motor Europe. p. 0

Where the paper says it

K2V: Generative Keyframing For Physically Plausible Video Generation FELIX O’MAHONY✉, University of Cambridge, UK DANIEL OLMEDA REINO, Toyota Motor Europe, Belgium ROBERTO CIPOLLA, University of Cambridge, UK AYUSH TEWARI, University of Cambridge, UK Physics-IQ Verified Dominos With Space 50 40 30 20 10 0 GT Ours Wan2. 2 Siphon K2V + Wan2.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Keyframe generation module can be implemented as an agentic pipeline where a planning agent orchestrates keyframe generation through tools like render edit, view image, and evaluate render to ensure physical plausibility. This allows the system to reason physically, from what the frame actually shows and choose them sparsely, concentrating compute on complex scenes.

  2. Conditioning of video generation models can be extended via a training-free mechanism where keyframes are pinned at arbitrary interior latent frames by replacing the grey frame with the generated keyframe, which is a trivial extension of this. This allows conditioning Wan2.2 without further training or fine tuning, leveraging its pre-trained capabilities.

  3. The system can produce videos that correct errors made by the image edit model because the agentic pipeline includes tools to review and correct errors made by the image edit model, or to refine its image edits. This allows for iterative refinement of keyframes and their resulting video without incurring the cost of producing a complete sample for every revision.

  4. The system can effectively allocate planning effort based on scene complexity because planning effort is spent in proportion to the physical complexity of a scene, as shown by the language model spending more time on scenes requiring careful planning. This results in state-of-the-art performance by focusing compute where it is most needed.

Abstract

Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.

Sources

Related papers