LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation

arXiv:2610.12442 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation".

Jane: The gist: LEGO proposes a lifting-free approach for exocentric-to-egocentric video generation by replacing explicit scene reconstruction with a learned view synthesizer that renders the egocentric view directly,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper called "LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation," and it tackles a really tricky problem. It’s about taking one video from an external camera and turning it into a video that looks like someone is filming you from their own perspective.

Jane: That's right, Tom. The core challenge they're addressing is how to make that work when the two cameras don't overlap much at all, which means a lot of the target view just isn't visible in the original recording.

Tom: Exactly. And what they’re proposing is ditching the way most current methods do it, which is by reconstructing the scene explicitly first, estimating depth and turning that into a point cloud to feed into a video diffusion model.

Jane: That explicit method has a problem because of those depth errors; they mess up where things are placed in the final video, even if the textures look okay because it preserves them.

Lu: What LEGO claims is they propose something different: a learned view synthesizer that renders the egocentric view directly, skipping all that explicit reconstruction entirely.

Meng: So instead of building a point cloud and replaying it from the exocentric camera, they want this synthesizer to just render what the egocentric camera would see based on the input.

Tom: Right. The paper basically asks what should be given as a condition to a video diffusion model for this task, and their answer is not an explicit scene representation, but this learned view synthesizer.

Jane: They introduce an LVSM-style transformer that the paper calls F, which is fine-tuned to map the exocentric frame and the target camera rays straight into the egocentric image.

Lu: And here's a key mechanism: inside this synthesizer F, each egocentric query attends over exocentric locations with normalized weights w(u→v), which are averaged over heads and layers to give them a correspondence distribution pu(v).

Meng: So instead of one pixel getting one reprojected location like in the old explicit method, this means it’s probabilistic; it averages over candidate source locations according to this learned distribution.

Tom: That averaging makes the errors blur out, which is actually good for a diffusion model because that model is really good at restoring detail when things are noisy.

Jane: So they argue that a condition should prioritize structural alignment first, and sharpness second because the generator can learn the fine details on its own later.

Lu: And to help with that structural alignment, they use something called a confidence map derived from this correspondence distribution pu(v).

Meng: The confidence is defined by taking the probability mass of the k most likely sources, and they set regions below a certain threshold to neutral gray. This gated render C is what gets fed into the diffusion latent space.

Tom: And they use that confidence map for something called Adaptive Rendering Consistency or ARC, which acts like a training-free guidance to steer the early denoising steps toward their render during high-noise times.

Jane: It's interesting because they say this approach consistently beats the state-of-the-art explicit pipelines on both EgoHumans and Nymeria without needing any retraining.

Lu: And another big point is that this synthesizer render is much cheaper to compute than the depth estimation and point cloud rendering that you see in the current state of the art.

Meng: That's good for practical deployment; if you’re trying to build something, having a simpler, faster way to generate structure is a definite plus.

Tom: So we've covered how LEGO proposes replacing explicit reconstruction with this learned synthesizer and confidence map to condition the video diffusion model. But what does this mean for the broader picture of exocentric video generation?

Jane: It means that instead of forcing a hard geometric mapping, we can leverage the diffusion model’s denoising strength by feeding it a structurally aligned but slightly blurred signal.

Tom: The paper introduces LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation, and the authors Suhwan Cho, Yonwoo Choi, Soongjin Kim, and Jicheol Park Taegyu Lim are the ones who put this up.

Lu: The implication here is that we can decouple view structure from detail; the synthesizer supplies the structure of the view, and then the diffusion model handles all that fine detail reconstruction.

Meng: From an engineering standpoint, it means we don't need to spend all our time perfecting a depth estimation pipeline just to get a decent starting point for generation.

Tom: So to wrap up this part: LEGO conditions a video diffusion model on the render of a learned view synthesizer, not on the point cloud reconstruction method.

Jane: The main argument is that alignment must come before sharpness, and their design has three practical advantages: it’s lifting-free with no depth estimation or point cloud needed at any stage, its confidence comes free from the synthesizer's distribution and drives guidance, and the frozen synthesizer weights just carry over to new scenes without retraining.

Lu: The future work they hint at is that this synthesizer can be dropped in as one appears for other tasks.

Meng: It seems like a very solid way to move forward, relying on learned correspondence rather than heavy geometric lifting.

Tom: That’s what we have for LEGO today, moving from the explicit reconstruction pipeline to this direct rendering approach. We'll take a quick break and come back to talk about the implications of this work in conclusion.

Conclusion: Tom: So we’re wrapping up on LEGO, which is this lifting-free approach for making egocentric videos from just one exocentric recording.

Jane: Right, so it replaces all that heavy scene reconstruction stuff—no depth maps or point clouds needed—with this learned view synthesizer that renders the egocentric view directly.

Lu: What’s really interesting is they solve the fundamental question of what to condition a video diffusion model on when there's no overlap between the cameras.

Meng: I mean, instead of giving it a messy point cloud, you give it this probabilistic map and a confidence score that tells the model where to trust what it sees.

Lalam: From a cultural view, this means we can generate more personalized visual content without needing massive datasets for every single camera angle combination.

Tom: So the main idea is structural alignment first, and sharpness second because the diffusion model handles the detail later.

Jane: Exactly, and they show that this method outperforms explicit pipelines on tests like EgoHumans without any retraining required for their synthesizer.

Lu: The fact that it’s lifting-free means those learned weights just transfer over to a brand new scene or recording, which is a big deal for generalizability.

Meng: I guess the trade-off is that if the original exocentric frame doesn't show something, the synthesizer can't invent it.

Tom: That’s their limitation—it can’t recover content that isn't in the input footage—but they argue it’s a worthwhile compromise for speed and structure.

Jane: So LEGO suggests we can decouple view structure from fine detail, letting the diffusion model focus on what it does best.

Lu: And that synthesizer is now a reusable component you can just drop into other projects when you need that kind of view mapping.

Tom: It really shifts the way we think about conditioning generation models—structure over immediate sharpness.

Jane: It makes sense to look at how this learned correspondence distribution works because that’s where the magic happens for guiding the denoising process.

Meng: Next up, we’re going to talk about how this confidence map actually steers those early steps of the diffusion process.

Suhwan Cho, Yonwoo Choi, Soongjin Kim, Jicheol Park Taegyu Lim

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/suhwan-cho/lego

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist: LEGO proposes a lifting-free approach for exocentric-to-egocentric video generation by replacing explicit scene reconstruction with a learned view synthesizer that renders the egocentric

Key concepts

Learned View Synthesizer (F)
Instead of calculating depth and reprojecting points, this transformer learns to directly predict the egocentric frame from the exocentric input and ray maps. It predicts the final view by attending over exocentric locations, effectively learning how to synthesize the target view without needing an explicit 3D model.
Correspondence Distribution (pu)
This is a probabilistic mapping learned by the synthesizer that determines which parts of the source exocentric frame correspond to which parts of the target egocentric frame. Unlike deterministic methods, this distribution averages over several possible source locations, resulting in a blurry but structurally sound render.
Confidence-Gated Conditioning
The synthesizer outputs a confidence map based on how certain it is about each predicted location. Regions with low confidence are masked or set to neutral gray. This confidence signal guides the diffusion model during generation by telling it where to trust the synthesized structure most.

Terminology

Summary

The gist: LEGO proposes a lifting-free approach for exocentric-to-egocentric video generation by replacing explicit scene reconstruction with a learned view synthesizer that renders the egocentric view directly, conditioning a video diffusion model on this render and its confidence map rather than an explicit point cloud or depth estimation.

LEGO Framework and Core Idea

LEGO is introduced as a lifting-free alternative that replaces the reconstruction pipeline with a learned view synthesizer that renders the egocentric view directly The design follows from the paper’s central question: what should a video diffusion model be given as its condition? LEGO proposes an LVSM-style transformer fine-tuned to map the exocentric frame and the target camera rays to the egocentric image, so cross-view correspondence is learned from the rendering objective alone The frozen synthesizer’s render replaces the pointcloud render in the generator’s conditioning input; no depth estimation, point cloud, or reprojection is involved.

Learned View Synthesizer and Correspondence Distribution

The view synthesizer F is trained to predict the egocentric frame directly, taking the exocentric frame and two ray maps as input to predict the egocentric frame. Inside F, each egocentric query attends over exocentric locations with normalized weights w(u→v), which are averaged over heads and layers to give the correspondence distribution pu(v). This probabilistic mapping differs from the explicit mapping where each pixel is assigned a single reprojected location, which preserves texture but translates depth errors into misplaced content. The synthesizer’s render is probabilistic because it averages over candidate source locations according to a learned correspondence distribution, so its errors are blur.

Confidence-Gated Conditioning and Adaptive Rendering Consistency (ARC)

The render of F replaces the point-cloud render in the egocentric half of the conditioning latent. The concentration of pu supplies the missing signal, and locations below a threshold are set to neutral gray. This confidence is defined as the probability mass of its k most likely sources, and C(u) = ˆI ego(u), c(u) ≥ τ, otherwise. This gated render C is encoded into the generator’s latent space and used as the conditioning input. ARC is a training-free guidance that reuses the synthesizer’s confidence to steer the early denoising steps toward its render. It modifies the denoising step by replacing the egocentric part of the clean estimate with a confidence-weighted blend with the render during high-noise steps.

Performance and Key Findings

LEGO consistently outperforms state-of-the-art explicit pipelines on EgoHumans and Nymeria without retraining. The paper argues that a condition should be judged by alignment first and sharpness second, since the generator supplies the second far more readily than the first. The untuned synthesizer render outperforms the point-cloud render as a condition, and ARC at inference alone outperforms depth-derived attention bias with which prior work trains the generator. The synthesizer render is also much cheaper to compute than depth estimation and point-cloud rendering.

Limitations and Generalization

LEGO inherits the limitations of the view synthesizer, as F renders each frame independently from a single exocentric frame, so it cannot recover content that the frame does not show. Its colors also collapse on unfamiliar scenes. The pipeline is lifting-free but still requires the camera poses of both views. However, the frozen synthesizer enters only through its render and confidence map, so the same weights carry over to new scenes and recordings without retraining. The synthesizer supplies the structure of the view, and the diffusion model its detail.

Conclusion

LEGO conditions a video diffusion model on the render of a learned view synthesizer, not a pointcloud reconstruction, on one criterion: a condition should be aligned before it is sharp. Its design has three practical advantages: it is lifting-free, with no depth estimation, point cloud, or reprojection at any stage; its confidence comes free from the synthesizer’s correspondence distribution and drives both the gating and a training-free guidance that replaces a bias the generator must be trained with; and the frozen synthesizer enters only through its render and confidence, so the same weights carry over to new scenes and recordings without retraining, while a stronger synthesizer can be dropped in as one appears. The synthesizer supplies the structure of the view, and the diffusion model its detail.

Improvements for AI systems

  1. Adapt conditioning for better detail restoration: The system can be improved by replacing explicit depth estimation and point cloud rendering with a learned view synthesizer to condition a video diffusion model directly, as this approach is argued to prioritize structural alignment over sharpness: We argue that a condition should be judged by alignment first and sharpness second, since the generator supplies the second far more readily than the first.

  2. Implement confidence-gated conditioning: The system can utilize a confidence map derived from a learned correspondence distribution, where regions below a threshold are masked, as this leverages the probabilistic nature of view synthesis: The concentration of pu supplies the missing signal. We define the confidence of a target location as the probability mass of its k most likely sources.

  3. Incorporate training-free guidance (ARC): The system can be enhanced by applying Adaptive Rendering Consistency (ARC), which guides early denoising steps based on the synthesizer's confidence map, achieving this without additional training: We therefore reuse the same confidence at inference, guiding the early denoising steps toward the render where it is high and leaving the rest unconstrained.

  4. Enable cross-dataset generalization: The improved system can maintain performance on new datasets by utilizing a frozen synthesizer fine-tuned on a source dataset (like Ego-Exo4D) and leveraging stateof-the-art results on EgoHumans and Nymeria without retraining: The same weights also generalize to EgoHumans and Nymeria without retraining.

  5. Improve color fidelity: By using confidence gating, the system can partially recover color quality: Gating the synthesizer render partially recovers the chroma ratio to 0.62, suggesting that removing unreliable conditioning allows the generator to rely more on its own appearance prior.

Sources

Related papers