LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video".
Title and authors: Tom: So, to recap, LiveStre4m is proposing a system that takes video from multiple angles where you don't know exactly where the camera was positioned—that’s unposed video—and it generates a novel viewpoint for you in real time without needing any prior calibration data.
Jane: That’s the big takeaway, and what I find really interesting is how they handle the hard part of knowing *where* the camera was even in the first place. They're skipping that whole slow process of figuring out camera positions from scratch and just predicting them directly from those input images using a vision transformer.
Lu: From a representation learning standpoint, that prediction is key because it sets up this whole pipeline for reconstructing three dee geometry using a spatial module. The idea is they build this three dee Gaussian splatting representation of the scene based on those predicted views.
Meng: But what does that mean for deployment? If we’re talking about shipping this to an app, how much memory are we talking about holding all those intermediate representations while it’s streaming?
Tom: The key is that they keep the processing time per frame really low. They report a reconstruction time of zero point zero seven seconds per frame at one thousand twenty-four by seven hundred sixty-eight resolution, which is incredibly fast compared to optimization based methods.
Jane: That speed is what makes it relevant for live streaming; it’s not just a slow computer rendering a view, it’s actually doing the work fast enough to keep up with video. And they ensure that the temporal consistency—meaning the frames flow smoothly—by using a neural interpolation network that employs diffusion transformers and then feeds that into a super resolution module.
Lu: The interpolation part is smart because it lets them create high-quality intermediate frames between two keyframes, which then get boosted by the super resolution to give you those final high-resolution novel view video streams.
Meng: So, the trade-off they found was interesting; if you want a really smooth frame rate above ten frames per second, they have to cut the resolution down to one thousand one hundred fifty-two by seven hundred sixty-eight. That’s a practical limitation for production systems.
Tom: Yeah, and the ablation studies showed that the spatial module is actually taking up most of their time, at fifty two point one milliseconds of processing time, which tells us where we need to focus our hardware optimization if we want to push this further.
Jane: So they’ve essentially created a self-contained system where pose prediction leads into geometry reconstruction and then temporal smoothing all in one feed forward pipeline. It addresses the lack of reliable calibration data head-on.
Lu: And they're aiming for something even more robust in the future, suggesting online pose refinement and lightweight bundle adjustment to make sure that geometric consistency stays perfect over a long video sequence.
Tom: It seems like a solid step forward for making novel view synthesis practical for live applications, but we still need to see how it handles really complex, dynamic scenes where the camera movement is unpredictable.
The paper's summary: Tom: So, LiveStre4m isn't just finished; they actually suggest some ways to make it better for real use by focusing on improving geometric consistency and visual fidelity in the next version of the model.
Jane: That makes sense because even a fast system can get shaky if the underlying three dee geometry isn't perfectly consistent across different video frames, so adding online pose refinement is a smart move.
Lu: Exactly, they want to incorporate online pose refinement and lightweight bundle adjustment. This means the model can constantly tweak its understanding of where the camera is without needing a slow re-calibration step every time.
Meng: From my production side, if we can get that geometric consistency improved, it means fewer frames will look jumpy or inconsistent when streaming over a long period, which is exactly what we need for reliable outputs.
Tom: And they also mentioned scaling up the training data for multi-view video, which they believe would really boost the generalization and the overall output quality of the system.
Jane: That sounds like a necessary step because training on more diverse video inputs should help it handle more different kinds of scenes better than it currently can.
Lu: The authors also touched on making the system even lighter; they’re looking at methods to keep things efficient while improving accuracy, which is a constant balancing act in this whole field.
Meng: Efficiency is everything for us; if you make it accurate but slow down the runtime, it just won't get deployed on consumer hardware. So, any improvements that keep the runtime low while boosting quality are what we’re really interested in seeing.
Tom: It seems like they’re pushing toward a system that is not only fast and pose free but also highly accurate and ready for more unpredictable real-world video streams.
Jane: And it really highlights that while feed forward models are great for speed, the next big hurdle is making them as geometrically robust as methods that rely on known camera parameters.
Lu: So the path forward involves combining their fast feed forward reconstruction with more traditional optimization techniques to nail down those underlying spatial relationships.
The paper's improvements: Tom: So we’re wrapping up on LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, and basically, they’ve shown us a way to generate novel viewpoints in real time without needing any prior camera calibration data.
Jane: That’s the big picture: it tackles the problem of needing those slow structure from motion pipelines just to get started with novel view synthesis.
Lu: The implication here is that we can move away from needing perfect ground truth camera parameters for every input, which is a huge win for practical applications where you don't have access to that kind of data.
Meng: For deployment, this means we can build systems that are much faster at generating novel views than those heavy optimization based methods, which is pretty crucial when you’re dealing with live video.
Tom: They hit a reconstruction time of zero point zero seven seconds per frame at one thousand twenty-four by seven hundred sixty-eight resolution, which is incredibly fast compared to those older methods they mentioned.
Jane: And they also showed that they can achieve rendering quality on the Neuralthree deeVideo dataset that’s competitive with some state of the art methods, hitting a PSNR of twenty-one point one one at that resolution <ref:2604.06740#pg2>.
Lu: It really validates the combination of their spatial module for scene geometry and their neural interpolation network for temporal consistency, showing it works as a cohesive feed forward system.
Meng: But we have to remember those caveats; they noted that if you want frame rates above ten frames per second, you have to reduce the image resolution down to one thousand one hundred fifty-two by seven hundred sixty-eight.
Tom: That’s the trade off; quality versus streamability, and it shows they are thinking about real deployment constraints right from the start.
Jane: And they pointed out that while it's fast and pose free, you still have to be careful about how far away your novel view is from where you started, because image quality drops as the target viewpoint moves further away.
Lu: The authors also mentioned their future work, focusing on online pose refinement and lightweight bundle adjustment to make that geometric consistency even stronger over time.
Tom: So the future direction is definitely about making it more robust and accurate for longer video sequences by adding those refinement tools they mentioned.
Jane: This paper, LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, gives us a really tangible tool for getting photorealistic novel viewpoints quickly in a streaming environment.
Lu: It’s a strong foundation because it proves that feed forward reconstruction can handle this task efficiently without the heavy computational overhead of traditional methods.
Meng: For me, it means we can start prototyping these features in real-time applications much sooner, rather than waiting for massive optimization pipelines to finish running.
Lalam: I’m confident that this approach is solid; the combination of pose prediction and spatial geometry reconstruction provides a very direct path to novel views when you only have unposed inputs.
Conclusion: Tom: So we’re wrapping up on LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, and essentially, they’ve shown us a way to generate novel viewpoints in real time without needing any prior camera calibration data.
Jane: That’s the big picture: it tackles the problem of needing those slow structure from motion pipelines just to get started with novel view synthesis.
Lu: The implication here is that we can move away from needing perfect ground truth camera parameters for every input, which is a huge win for practical applications where you don't have access to that kind of data.
Meng: For deployment, this means we can build systems that are much faster at generating novel views than those heavy optimization based methods, which is pretty crucial when you’re dealing with live video.
Tom: They hit a reconstruction time of zero point zero seven seconds per frame at one thousand twenty-four by seven hundred sixty-eight resolution, which is incredibly fast compared to those older methods they mentioned.
Jane: And they also showed that they can achieve rendering quality on the Neuralthree deeVideo dataset that’s competitive with some state of the art methods, hitting a PSNR of twenty one point one at that resolution.
Lu: It really validates the combination of their spatial module for scene geometry and their neural interpolation network for temporal consistency, showing it works as a cohesive feed forward system.
Meng: But we have to remember those caveats; they noted that if you want frame rates above ten frames per second, you have to reduce the image resolution down to one thousand one hundred fifty-two by seven hundred sixty-eight.
Tom: That’s the trade off; quality versus streamability, and it shows they are thinking about real deployment constraints right from the start.
Jane: And they pointed out that while it's fast and pose free, you still have to be careful about how far away your novel view is from where you started, because image quality drops as the target viewpoint moves further away.
Lu: The authors also mentioned their future work, focusing on online pose refinement and lightweight bundle adjustment to make that geometric consistency even stronger over time.
Tom: So the future direction is definitely about making it more robust and accurate for longer video sequences by adding those refinement tools they mentioned.
Jane: This paper, LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, gives us a really tangible tool for getting photorealistic novel viewpoints quickly in a streaming environment.
Lu: It’s a strong foundation because it proves that feed forward reconstruction can handle this task efficiently without the heavy computational overhead of traditional methods.
Meng: For me, it means we can start prototyping these features in real-time applications much sooner, rather than waiting for massive optimization pipelines to finish running.
Lalam: I’m confident that this approach is solid; the combination of pose prediction and spatial geometry reconstruction provides a very direct path to novel views when you only have unposed inputs.
Tom: We’ve seen how they managed to do this with just two synchronized input streams at one thousand twenty-four by seven hundred sixty-eight resolution in zero point zero seven seconds per frame on a single GPU.
Jane: It’s a real step toward making novel view synthesis practical for live applications because it cuts out the need for those slow structure from motion pipelines entirely.
Lu: We're moving away from needing perfect ground truth camera parameters for every input, which is a huge win when you don't have that kind of data available.
Meng: From a production standpoint, if we can get this kind of performance on a single GPU for one thousand twenty-four by seven hundred sixty-eight resolution, that drastically lowers the barrier for deploying novel view synthesis features in real applications.
Tom: It’s about making these kinds of complex visual tasks accessible by focusing on feed forward models instead of heavy optimization loops.
Jane: So, to wrap up this discussion on LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, the authors have shown a way to do real time novel view synthesis from sparse unposed multi-view video inputs using just two synchronized input streams at one thousand twenty-four by seven hundred sixty-eight resolution in zero point zero seven seconds per frame.
Lu: They achieved this by combining a multi view vision transformer for keyframe three dimensional scene reconstruction with a diffusion transformer interpolation module that ensures temporal consistency and stable streaming.
Meng: It’s a significant step toward deployable live novel-view synthesis systems because they’ve managed to produce photorealistic novel viewpoints from sparse unposed input views at one thousand twenty-four by seven hundred sixty-eight resolution in only zero point zero seven seconds per frame on a single GPU.
Tom: We’ll keep an eye on those future work plans for the online pose refinement part; that should really nail down geometric consistency over longer sequences.
Jane: Next time, we look at StyleComposer, because it’s training-free style composition and it’s interesting how they route every style attribute through a representation.
Pedro Quesado Erkut Akdag, Yasaman Kashefbahrami, Willem Menu Egor Bondarev
Eindhoven University of Technology
cs.CV
Submitted: 2026-04-08
Updated: 2026-04-08
Journal ref: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2026, pp. 279-288
Code: https://github.com/pedro-quesado/LiveStre4m
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: LiveStre4m proposes a novel viewpoint video live-streaming method that enables real-time Novel View Synthesis from unposed multi-view video inputs by combining a Camera Pose Predictor…
Key concepts
- Camera Pose Predictor
- This component estimates the exact position and orientation of the camera capturing an image directly from unposed RGB video. It uses Vision Transformers (ViTs) to analyze the input images without needing prior calibration data, allowing the system to work in a fully pose-free manner.
- Spatial Module
- This module uses transformer architectures to understand the scene's 3D structure and appearance. It takes camera-centric representations and maps them into a global scene token vector, which is then used to predict the parameters for a 3D Gaussian Splatting representation of the scene.
- Neural Interpolation Network (NIN)
- The NIN ensures temporal consistency in the novel view video. It uses Diffusion Transformers to interpolate between keyframes and includes a high-speed single-image super-resolution module to increase image resolution, resulting in smoother and sharper output frames.
Terminology
Summary
LiveStre4m proposes a novel viewpoint video live-streaming method that enables real-time Novel View Synthesis from unposed multi-view video inputs by combining a Camera Pose Predictor, a Spatial Module, and a Neural Interpolation Network
Core Methodology
LiveStre4m is structured around three key modules: the Camera Pose Predictor, the Spatial Module, and the Neural Interpolation Network (NIN). The Camera Pose Predictor estimates camera parameters directly from unposed RGB images using multi-view ViTs. p. 1 LiveStre4m
combines three key components: a Camera Pose Predictor, a Spatial Module, and a Neural Interpolation Network (NIN). The Camera Pose Predictor, inspired by previous methods on pose regression and scene reconstruction [27, 28, 41], employs multi-view ViTs to estimate camera parameters directly from unposed RGB images. The Camera Pose Predictor, inspired by previous methods on pose regression and scene reconstruction [27, 28, 41], employs multi-view ViTs to estimate camera parameters directly from unposed RGB images. The
Spatial Module leverages a transformer-based architecture
to estimate scene geometry and appearance to generate a 3D
Gaussian Splatting representation. Although this enables
efficient photorealistic NVS, computational costs limit image resolution and video frame rate. The NIN module alleviates this by applying Diffusion Transformers [18] for
temporal frame interpolation alongside a high-speed singleimage super-resolution module, enabling temporal consistency, sufficient frame rate and improved image resolution.Where the paper says it
Where the paper says it
Where the paper says it
Data Flow and Temporal Consistency
The data flow involves predicting camera poses from unposed input RGB views, which are then processed by the Spatial Module to generate a low-resolution novel viewpoint of the scene. After two keyframes are processed, NIN interpolates and increases image resolution to generate an intermediate frame. p. 3 Then, the images and predicted camera information are processed by the Spatial
Module to generate a low-resolution novel viewpoint of the scene. After two keyframes are processed, NIN interpolates and increases
image resolution to generate the high-resolution video snippet. This module is a lightweight pretrained CNN architecture [1] performing fast image super-resolution. The
resulting high-resolution frames are used to compose the
novel view video stream of LiveStre4m.Where the paper says it
Where the paper says it
Key Architectural Components
The Camera Pose Predictor employs a coarse-to-fine strategy to estimate camera poses from tokenized input frames, refining them through attention heads. The Spatial Module uses local view tokens processed by a transformer module Floc(·) to create camera-centric representations, which are then mapped by Fglobal(·) to a single global scene token vector. p. 3 To enhance
robustness against minor inaccuracies in camera pose predictions, explicit geometric projections are avoided [41]. Instead, a transformer module Fglobal(·) learns to map local
point tokens into a single global scene token vector, guided
by the predicted camera poses. By combining appearance features extracted and the predicted geometry, a DPT head
predicts 3D Gaussian parameters centered on the predicted pointmap to achieve photorealistic NVS. form a dense 3D point map.Where the paper says it
Where the paper says it
Performance and Efficiency Gains
LiveStre4m attains an average reconstruction time of 0.07s per-frame at 1024 × 768 resolution, significantly outperforming optimization-based dynamic scene representation methods by orders of magnitude in runtime. p. 0 LiveStre4m attains an
average reconstruction time of 0. 07s per-frame at 1024 ×
768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. The NIN module alleviates this by applying Diffusion Transformers [18] for
temporal frame interpolation alongside a high-speed singleimage super-resolution module, enabling temporal consistency, sufficient frame rate and improved image resolution. LiveStre4m operates end-to-end, taking sparse unposed
input video streams and producing novel viewpoint video at
1024 × 768 resolution in 0.Where the paper says it
Where the paper says it
Experimental Validation
Experiments on the Neural3DVideo dataset show that LiveStre4m achieves competitive rendering quality compared to state-of-the-art methods, attaining a PSNR of 21.11 at 1024 × 768 resolution. p. 0 LiveStre4m attains an
average reconstruction time of 0. 07s per-frame at 1024 ×
768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. 65
compared to 16. In terms of efficiency, LiveStre4m
demonstrates a substantial advantage with a runtime of
0. In terms of efficiency, LiveStre4m
demonstrates a substantial advantage with a runtime of
0. 074s per frame, over 3. 074s per frame, over 3. 3× faster than FLARE (0. We present additional results in
supplementary material, including detailed per-scene analysis (Sec. A), an evaluation on the highly dynamic VRUBasketball [32, 33] dataset in (Sec.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ablation Insights
Ablation studies indicate a trade-off between visual quality and streamability, showing that maintaining frame rates above > 10 fps requires limiting the image resolution to 1152 × 768. The Spatial Module is identified as the component dominating the runtime with 52. p. 7 Runtime of Model Components: Table 6 breaks down
the computational cost of each component, highlighting potential bottlenecks. The Spatial Module dominates the runtime with 52. The Spatial Module dominates the runtime with 52. The Interpolation Net is significantly
lighter at 19. The Interpolation Net is significantly
lighter at 19. 3 ms, validating the choice of interpolating intermediate frames for a more efficient model. The interpolation component of the NIN is a pretrained 12-block DiT with spatial
and temporal attention coupled with a 4-block decoder that
refines image tokens to generate the interpolated frame. The
super-resolution module is a lightweight 3 layer CNN with
32 channels pretrained for 2× image upsampling. Table 5 explores how the number of output
views affects the runtime and NVS quality. Results indicate that increasing the number of output images not only
increases the runtime, but also reduces image quality, as target viewpoints deviate further from the central viewpoint.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Work
LiveStre4m represents a significant step toward livestreaming of novel-viewpoint video in real-time. Potential enhancements include incorporating online pose refinement and lightweight bundle adjustment to improve geometric consistency and visual fidelity. p. 7 Discussion
LiveStre4m prioritizes efficiency, trading some visual
quality for significantly faster inference compared to
optimization-based approaches, but to close this gap, potential enhancements are on exploration. Incorporating online
pose refinement and lightweight bundle adjustment could
improve geometric consistency and visual fidelity. Incorporating online
pose refinement and lightweight bundle adjustment could
improve geometric consistency and visual fidelity. Furthermore, scaling multi-view video training data is expected to
further enhance generalization and output quality. NIN ensures temporal consistency, better image resolution and higher frame rates. Experimental results demonstrate that the proposed
method produces photorealistic novel viewpoints from
sparse unposed input views at 1024×768 resolution in only
0. Experimental results demonstrate that the proposed
method produces photorealistic novel viewpoints from
sparse unposed input views at 1024×768 resolution in only
0. 07s per frame on a single GPU.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Acknowledgments
This work was supported by the ELEVATION Xecs 23 project. The references section lists numerous works supporting various aspects of the proposed method, such as NeRF [16], Gaussian Splatting [10], and various feed-forward NVS approaches like DUSt3R [29] and FLARE [41]. p. 4 Furthermore, this allows LiveStre4m to be applied to
various input frame rates while preserving temporal consistency in the generated novel viewpoint video. Training Strategy
The proposed method is finetuned with visual losses only,
i. To ensure fair comparison with methods that rely on ground-truth
camera parameters, we follow the evaluation protocol established by prior pose-free approaches [5, 30, 38]. Table 1 presents a detailed quantitative comparison
of dynamic scene reconstruction methods evaluated on
the Neural3DVideo [14] and MeetRoom [13] datasets. Moreover, LiveStre4m performs NVS at twice the output
image resolution due to the Super Resolution module. Qualitative results are illustrated in Fig. An
overview of LiveStre4m data flow is first provided, followed by detailed descriptions of the Camera Pose Predictor, Spatial Module, and Neural Interpolation Network
(NIN) in Sections 3. 4, respectively. For instance, DUSt3R [29] and MASt3R [12] leverage pre-trained
Vision Transformers (ViTs) [4] to provide dense geometric
correspondences and scene structure, reducing the reliance
on conventional optimization-based pipelines. Building on
1
arXiv:2604. These results demonstrate that LiveStre4m
makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live
novel-view synthesis systems. Code available at: https:
//github. Code available at: https:
//github. com/pedro-quesado/LiveStre4m
1. For instance, DUSt3R [29] and MASt3R [12] leverage pre-trained
Vision Transformers (ViTs) [4] to provide dense geometric
correspondences and scene structure, reducing the reliance
on conventional optimization-based pipelines. Building on
1
arXiv:2604. Acknowledgments
This work was supported by the ELEVATION Xecs
2023022 project
References
[1] Guillaume Berger, Manik Dhingra, Antoine Mercier,
Yashesh Savani, Sunny Panchal, and Fatih Porikli. Quicksrnet: Plain single-image super-resolution architecture for
faster inference on mobile platforms, 2023. 4, comparing
reference renderings with LiveStre4m outputs across different test scenes, showcasing photorealistic feed-forward
3D scene reconstruction. We present additional results in
supplementary material, including detailed per-scene analysis (Sec. 07s per-frame at 1024 ×
768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m
makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live
novel-view synthesis systems. As shown in Table 2, LiveStre4m achieves competitive rendering quality compared to state-of-the-art methods. On the Neural3DVideo dataset, LiveStre4m attains a
PSNR of 21. Although FLARE is not developed for dynamic scenes, it is capable of zero-shot camera-free NVS
from sparse input views. For comparison, FLARE is evaluated with the same pipeline as LiveStre4m at each each
timestep and its performance is averaged for the entire
video. Although this approach represents a step toward streamable
NVS, the optimization requires 12 seconds per frame,
which is far too slow for real-time applications. Building
upon this idea, IGS [36] is proposed as an even faster approach, reducing reconstruction time to approximately 2. For consistent reporting, runtime is defined as the time required to
generate a novel viewpoint of a complete video, averaged
by the number of frames, excluding pre-processing. To ensure fair comparison with methods that rely on ground-truth
camera parameters, we follow the evaluation protocol established by prior pose-free approaches [5, 30, 38]. Acknowledgments
This work was supported by the ELEVATION Xecs
2023022 project
References
[1] Guillaume Berger, Manik Dhingra, Antoine Mercier,
Yashesh Savani, Sunny Panchal, and Fatih Porikli. Quicksrnet: Plain single-image super-resolution architecture for
faster inference on mobile platforms, 2023.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI system can perform real-time novel view synthesis from sparse unposed multi-view video inputs using only two synchronized input streams at 1024 x 768 resolution in 0.07 seconds per frame, which is orders of magnitude faster than optimization-based methods.
-
The system can achieve zero shot NVS with a transformer based Spatial Module guided by camera parameters predicted directly from RGB images, removing the reliance on known camera calibration information to reconstruct scene geometry and appearance.
-
Temporal consistency in novelview video generation is ensured by a Neural Interpolation Network that applies Diffusion Transformers for temporal frame interpolation alongside a high speed single image super resolution module, resulting in improved image quality and sufficient frame rate.
-
The system can operate in a fully pose free manner for novel view synthesis, which is crucial because
reliable calibration data is unavailable and acquiring these parameters requires either specialized hardware or computationally expensive structure-from-motion pipelines.
-
The system can be adapted for dynamic scenes by using a feed forward method that enables
online per-frame optimization
ideas, although the current focus remains on static camera assumptions to maintain real time streaming.
Abstract
Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations (about 2.67 s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of 0.07 s per-frame at 1024 times 768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m
Sources
- Scalable Diffusion Models with Transformers
- Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- NeRF--: Neural Radiance Fields Without Known Camera Parameters
- LocalDyGS: Multi-view Global Dynamic Scene Modeling via Adaptive Local Implicit Feature Decoupling
- Swift4D:Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs